网管告警:
告警主机:YiDHLWJKFZ-js-app-
主机IP:192.168.***.***
告警项目:system.cpu.util[,iowait]
告警时间:2019.02. ::
告警等级:Warning
告警信息:Disk I/O is overloaded on YiDHLWJKFZ-js-app-
问题详情:CPU iowait time:20.14 %
当前状态:PROBLEM:20.14 %
事件ID:
top查看:(wa值为17.7%)
[root@localhost vmuser]# top top - :: up :, user, load average: 3.56, 3.45, 3.40
Tasks: total, running, sleeping, stopped, zombie
Cpu(s): 1.0%us, 2.6%sy, 0.0%ni, 78.2%id, 17.7%wa, 0.2%hi, 0.3%si, 0.0%st
Mem: 16467984k total, 2823808k used, 13644176k free, 331524k buffers
Swap: 16383992k total, 0k used, 16383992k free, 565436k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
root D 5.3 0.0 :16.37 jbd2/sda3-
postfix 104m 46m S 3.7 0.3 :45.87 qmgr
root 242m S 3.0 0.0 :17.92 rsyslogd
postfix S 1.7 0.0 :00.05 local
root S 1.3 0.0 :44.87 master
postfix S 1.3 0.0 :00.05 cleanup
root D 1.3 0.0 :00.04 local
postfix S 1.0 0.0 :29.32 trivial-rewrite
postfix S 1.0 0.0 :00.06 cleanup
postfix D 0.7 0.0 :39.73 pickup
postfix D 0.7 0.0 :00.03 cleanup
iostat查看:(iowait值一直过高)
[root@localhost vmuser]# iostat -x
Linux 2.6.-.el6.x86_64 (localhost.localdomain) // _x86_64_ ( CPU) avg-cpu: %user %nice %system %iowait %steal %idle
2.96 0.00 2.14 19.78 0.00 75.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 12.58 361.59 28.73 1621.40 616.25 15618.31 9.84 3.72 2.25 0.53 86.86
sdb 307.71 0.01 6.02 0.00 1255.11 0.05 208.45 0.14 22.52 14.07 8.47 avg-cpu: %user %nice %system %iowait %steal %idle
0.84 0.00 2.12 21.38 0.00 75.66 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 465.00 36.00 1641.00 288.00 16508.00 10.02 3.27 1.98 0.58 97.30
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle
1.44 0.00 4.49 15.09 0.00 78.97 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 1167.00 18.00 4016.50 144.00 40720.00 10.13 3.61 0.88 0.22 87.95
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle
0.44 0.00 1.19 24.88 0.00 73.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 351.00 22.50 879.00 180.00 9696.00 10.96 3.25 3.68 1.10 98.90
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle
0.87 0.00 2.31 20.07 0.00 76.75 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 435.50 19.00 2666.50 152.00 24512.00 9.18 3.23 1.21 0.35 93.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [root@localhost vmuser]#
iotop查看:
[root@localhost vmuser]# iotop Total DISK READ: 124.32 K/s | Total DISK WRITE: 6.69 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
be/ root 0.00 B/s 236.99 K/s 0.00 % 92.26 % [jbd2/sda3-]
be/ postfix 124.32 K/s 0.00 B/s 0.00 % 88.09 % pickup -l -t fifo -u
be/ postfix 0.00 B/s 0.00 B/s 0.00 % 6.21 % qmgr -l -t fifo -u
be/ postfix 0.00 B/s 58.28 K/s 0.00 % 6.02 % cleanup -z -t unix -u
be/ root 0.00 B/s 27.20 K/s 0.00 % 3.82 % rsyslogd -c
be/ postfix 0.00 B/s 158.28 K/s 0.00 % 1.79 % cleanup -z -t unix -u
be/ postfix 0.00 B/s 260.31 K/s 0.00 % 1.62 % local -t unix unix -u
be/ postfix 0.00 B/s 19.43 K/s 0.00 % 0.75 % bounce -z -t unix -u
be/ postfix 0.00 B/s 58.28 K/s 0.00 % 0.75 % cleanup -z -t unix -uth :/usr/local/sa/sa-agent/main/~ocal/sa/sa-agent com.transfar.sa.agent.main.Bootstrap
rt/ root 0.00 B/s 54.39 B/s 0.00 % 0.67 % bounce -z -t unix -u
rt/ postfix 0.00 B/s 38.85 B/s 0.00 % 0.60 % bounce -z -t unix -u
be/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/]
rt/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/]
rt/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/]
rt/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/]
rt/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/]]
关闭不必要的postfix进程
[root@localhost vmuser]# service postfix stop
Shutting down postfix: [ OK ]
[root@localhost vmuser]#