进入正题前先说明:OOM killer什么时候出现?

linux下允许程序申请比系统可用内存更多的内存,这个特性叫Overcommit。这样做是出于优化系统考虑,因为不是所有的程序申请了内存就立刻使用的,当你使用的时候说不定系统已经回收了一些资源了。不幸的是,当你用到这个Overcommit给你的内存的时候,系统还没有资源的话,OOM killer就跳出来了

今天外网某个游戏服无法登陆,查到mysqld进程不在了,于是查系统日志(/var/log/message)如下:

----------------------------------------------系统日志----开始---------------------------------------------

Apr 13 04:00:59 localhost kernel: Out of memory: Kill process 6033 (mysqld) score 85 or sacrifice child
Apr 13 04:00:59 localhost kernel: Killed process 6033, UID 498, (mysqld) total-vm:56872260kB, anon-rss:3202560kB, file-rss:40kB
Apr 13 09:28:37 localhost auditd[1979]: Audit daemon rotating log files

----------------------------------------------系统日志----结束---------------------------------------------

我的内存是64G,一般运行游戏服的90个进程是没问题的,而且内存还剩12个G,今天突然被OOM 杀掉,于是查了系统日志,发现 syslog-ng在作怪,日志如下:

--------------------------------syslog-ng--------------系统日志----开始---------------------------------------------

Apr 13 03:32:16 localhost kernel: syslog-ng[178936]: segfault at c06000 ip 00007f77c269a820 sp 00007f77b61f9390 error 4 in libsyslog-ng-3.6.so.0.0.0[7f77c2642000+9b000]
Apr 13 03:32:16 localhost supervise/syslog-ng[105636]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
Apr 13 03:32:19 localhost kernel: syslog-ng[178944]: segfault at c06000 ip 00007f77c269a820 sp 00007f77bca5c390 error 4 in libsyslog-ng-3.6.so.0.0.0[7f77c2642000+9b000]
Apr 13 03:32:19 localhost supervise/syslog-ng[105636]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
Apr 13 03:32:32 localhost kernel: syslog-ng[178960]: segfault at c0cbb0 ip 00007f77c0e4d9df sp 00007f77b6bfa2f8 error 4 in libc-2.12.so[7f77c0d1a000+18b000]
Apr 13 03:32:32 localhost supervise/syslog-ng[105636]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
Apr 13 03:32:44 localhost kernel: syslog-ng[178973]: segfault at c06370 ip 00007f77c0e4d9df sp 00007f77bca5c2f8 error 4 in libc-2.12.so[7f77c0d1a000+18b000]
Apr 13 03:32:45 localhost supervise/syslog-ng[105636]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
Apr 13 03:33:02 localhost kernel: syslog-ng[178997]: segfault at c06050 ip 00007f77c0e4d9df sp 00007f779bffc2f8 error 4 in libc-2.12.so[7f77c0d1a000+18b000]
Apr 13 03:33:02 localhost supervise/syslog-ng[105636]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'

--------------------------------syslog-ng--------------系统日志----结束---------------------------------------------

syslog-ng 一直  restarting。所以猜测是syslog-ng内存占用过大,正好mysql要使用之前申请的系统内存时,而此时系统并没有资源,就被无情KILL了。

total -vm:个人理解是当前占用的虚拟内存。

以上是个人碰到问题收集,如果各路大神觉得有问题,记得给我留言纠正,生命不息,奋斗不止。

05-27 19:45