gyq0618

gyq0618

关注
发信
关注(28)粉丝(399)

zabbix使用python脚本监控java进程

作者：撸大师
链接：https://www.jianshu.com/p/172118ca0262

前言：目前有个需求，需要对线上几台拥有5个tomcat的服务器添加监控，分别监控各个tomcat的gc、连接数、使用内存等，现成的模板是监控一台tomcat，完全不适用目前情况，5个tomcat，如果现在新建模板，那每个tomcat得添加十几个item,也就是要添加60多个 Ttem，而且还要添加 Trigger 和 Graph，也都是成倍的添加，如果这台服务器在增加tomcat，又得再加一次，做工重复且繁琐，于是想到能不能自动发现tomca的端口以及把不同tomcat的监控数据直接发给zabbx-server。

此文章多方百度和多次实验失败后完成的。

参考1：zabbix_sender提交item数据

参考2：zabbix自动发现并监控主机的TCP监听端口

一、发现端口和pid

 

说明：发现端口是为了区分程序；而发现pid则是便于用jstat获取程序的状态数据；
直接上脚本：

点击(此处)折叠或打开

  1. > vim jstat.py
  2. #!/usr/bin/env python
  3. #coding=utf-8
  4. '''
  5. ##
  6. ## 功能: 调用jstat获取JMX的各项指标
  7. ## 说明: 用于zabbix自动发现告警
  8. ## 版本: V1.0 2016-11-02
  9. ## 特性: 1. 线程功能，提高脚本执行速度
  10. ##
  11. '''

  13. import sys
  14. import os
  15. import commands
  16. import subprocess
  17. import json
  18. import argparse
  19. import socket
  20. import threading

  22. jstat_cmd = commands.getoutput("which jstat")
  23. jstack_cmd = commands.getoutput("which jstack")
  24. jvmport_cmd = "netstat -tpnl|grep -oP '(?<=:)\d+.*\d+(?=/java)'|

  26. hostname = socket.gethostname()
  27. zbx_sender='/usr/local/zabbix/bin/zabbix_sender'
  28. zbx_cfg='/usr/local/zabbix/etc/zabbix_agentd.conf'
  29. zbx_tmp_file='/usr/local/zabbix/scripts/.zabbix_jmx_status'

  31. jstat_dict = {
  32.         "S0":"Young.Space0.Percent",
  33.         "S1":"Young.Space1.Percent",
  34.         "E":"Eden.Space.Percent",
  35.         "O":"Old.Space.Percent",
  36.         "P":"Perm.Space.Percent",
  37.         "FGC":"Old.Gc.Count",
  38.         "FGCT":"Old.Gc.Time",
  39.         "YGC":"Young.Gc.Count",
  40.         "YGCT":"Young.Gc.Time",
  41.         "GCT":"Total.Gc.Time",
  42.         "PGCMN":"Perm.Gc.Min",
  43.         "PGCMX":"Perm.Gc.Max",
  44.         "PGC":"Perm.Gc.New",
  45.         "PC":"Perm.Gc.Cur",
  46.         "Tomcat.Thread":"Tomcat.Thread"
  47.     }

  49. jmx_threads = []

  51. def get_status(cmd,opts,pid):
  52.     value = commands.getoutput('sudo %s -%s %s' % (cmd,opts,pid)).strip().split('\n')

  54.     kv = []
  55.     for i in value[0].split(' '):
  56.         if i != '':
  57.             kv.append(i)

  59.     vv = []
  60.     for i in value[1].split(' '):
  61.         if i != '':
  62.             vv.append(i)

  64.     data = dict(zip(kv,vv))
  65.     return data

  67. def get_thread(cmd,pid):
  68.     value = commands.getoutput('sudo %s %s|grep http|wc -l' % (cmd,pid))
  69.     data = {"Tomcat.Thread":value}
  70.     return data

  72. def get_jmx(jport,jprocess):
  73.     '''
  74.       使用jstat获取Java的性能指标
  75.     '''
  76.     
  77.     file_truncate() # 清空zabbix_data_tmp

  79.     gcutil_data = get_status(jstat_cmd,"gcutil",jprocess)
  80.     gccapacity_data = get_status(jstat_cmd,"gccapacity",jprocess)
  81.     thread_data = get_thread(jstack_cmd,jprocess)
  82.     data_dict = dict(gcutil_data.items()+gccapacity_data.items()+thread_data.items())

  84.     for jmxkey in data_dict.keys():
  85.         if jmxkey in jstat_dict.keys():
  86.             cur_key = jstat_dict[jmxkey]
  87.             zbx_data = "%s jstat[%s,%s] %s" %(hostname,jport,cur_key,data_dict[jmxkey])
  88.             with open(zbx_tmp_file,'a') as file_obj: file_obj.write(zbx_data + '\n')

  90. def jvm_port_discovery():
  91.     output = subprocess.Popen(jvmport_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  92.     jvm_port_lists = output.stdout.readlines()
  93.     jvm_port_proce = []
  94.     for jvm_port_tmp in jvm_port_lists:
  95.          jvm_port_proce.append(jvm_port_tmp.split())
  96.     return jvm_port_proce
  97.     

  99. def file_truncate():
  100.     '''
  101.       用于清空zabbix_sender使用的临时文件
  102.     '''
  103.     with open(zbx_tmp_file,'w') as fn: fn.truncate()

  105. def zbx_tmp_file_create():
  106.     '''
  107.       创建zabbix_sender发送的文件内容
  108.     '''
  109.     jvmport_list = jvm_port_discovery()
  110.     for jvm_tmp in jvmport_list:
  111.         jvmport = jvm_tmp[0]
  112.         jvmprocess = jvm_tmp[1]
  113.         th = threading.Thread(target=get_jmx,args=(jvmport,jvmprocess))
  114.         th.start()
  115.         jmx_threads.append(th)

  117. def send_data_zabbix():
  118.     '''
  119.       调用zabbix_sender命令，将收集的key和value发送至zabbix server
  120.     '''
  121.     zbx_tmp_file_create()
  122.     for get_jmxdata in jmx_threads:
  123.         get_jmxdata.join()
  124.     zbx_sender_cmd = "%s -c %s -i %s" %(zbx_sender,zbx_cfg,zbx_tmp_file)
  125.     print zbx_sender_cmd
  126.     zbx_sender_status,zbx_sender_result = commands.getstatusoutput(zbx_sender_cmd)
  127.     #print zbx_sender_status
  128.     print zbx_sender_result

  130. def zbx_discovery():
  131.     '''
  132.       用于zabbix自动发现JVM端口
  133.     '''
  134.     jvm_zabbix = []
  135.     jvmport_list = jvm_port_discovery()
  136.     for jvm_tmp in jvmport_list:
  137.         jvm_zabbix.append({'{#JPORT}' : jvm_tmp[0],
  138.                            '{#JPROCESS}' : jvm_tmp[1],
  139.                          })
  140.     return json.dumps({'data': jvm_zabbix}, sort_keys=True, indent=7,separators=(',', ':'))

  142. def cmd_line_opts(arg=None):
  143.     class ParseHelpFormat(argparse.HelpFormatter):
  144.         def __init__(self, prog, indent_increment=5, max_help_position=50, width=200):
  145.             super(ParseHelpFormat, self).__init__(prog, indent_increment, max_help_position, width)

  147.     parse = argparse.ArgumentParser(description='Jmx监控"',
  148.                                     formatter_class=ParseHelpFormat)
  149.     parse.add_argument('--version', '-v', action='version', version="0.1", help='查看版本')
  150.     parse.add_argument('--jvmport', action='store_true', help='获取JVM端口')
  151.     parse.add_argument('--data', action='store_true', help='发送JMX指标数据至zabbix')

  153.     if arg:
  154.         return parse.parse_args(arg)
  155.     if not sys.argv[1:]:
  156.         return parse.parse_args(['-h'])
  157.     else:
  158.         return parse.parse_args()


  161. if __name__ == '__main__':
  162.     opts = cmd_line_opts()
  163.     if opts.jvmport:
  164.         print zbx_discovery()
  165.     elif opts.data:
  166.         send_data_zabbix()
  167.     else:
  168.         cmd_line_opts(arg=['-h'])

查看下返回数据


点击(此处)折叠或打开

  1. > sudo python jstat.py --jvmport
  2. {
  3.        "data":[
  4.               {
  5.                      "{#JPORT}":"8082",
  6.                      "{#JPROCESS}":"456"
  7.               },
  8.               {
  9.                      "{#JPORT}":"8083",
  10.                      "{#JPROCESS}":"11992"
  11.               },
  12.               {
  13.                      "{#JPORT}":"8084",
  14.                      "{#JPROCESS}":"7713"
  15.               },
  16.               {
  17.                      "{#JPORT}":"7074",
  18.                      "{#JPROCESS}":"11239"
  19.               },
  20.               {
  21.                      "{#JPORT}":"8899",
  22.                      "{#JPROCESS}":"14186"
  23.               }
  24.        ]
  25. }

二、新建 templates

避免麻烦，我直接导出 templates

点击(此处)折叠或打开

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <zabbix_export>
  3.     <version>2.0</version>
  4.     <date>2016-11-02T08:11:35Z</date>
  5.     <groups>
  6.         <group>
  7.             <name>Template java</name>
  8.         </group>
  9.     </groups>
  10.     <templates>
  11.         <template>
  12.             <template>jmx-im-jstat</template>
  13.             <name>jmx-im-jstat</name>
  14.             <groups>
  15.                 <group>
  16.                     <name>Template java</name>
  17.                 </group>
  18.             </groups>
  19.             <applications>
  20.                 <application>
  21.                     <name>JSTAT</name>
  22.                 </application>
  23.             </applications>
  24.             <items>
  25.                 <item>
  26.                     <name>jmxdata</name>
  27.                     <type>0</type>
  28.                     <snmp_community/>
  29.                     <multiplier>1</multiplier>
  30.                     <snmp_oid/>
  31.                     <key>jmxdata</key>
  32.                     <delay>30</delay>
  33.                     <history>90</history>
  34.                     <trends>365</trends>
  35.                     <status>0</status>
  36.                     <value_type>3</value_type>
  37.                     <allowed_hosts/>
  38.                     <units/>
  39.                     <delta>0</delta>
  40.                     <snmpv3_contextname/>
  41.                     <snmpv3_securityname/>
  42.                     <snmpv3_securitylevel>0</snmpv3_securitylevel>
  43.                     <snmpv3_authprotocol>0</snmpv3_authprotocol>
  44.                     <snmpv3_authpassphrase/>
  45.                     <snmpv3_privprotocol>0</snmpv3_privprotocol>
  46.                     <snmpv3_privpassphrase/>
  47.                     <formula>1</formula>
  48.                     <delay_flex/>
  49.                     <params/>
  50.                     <ipmi_sensor/>
  51.                     <data_type>0</data_type>
  52.                     <authtype>0</authtype>
  53.                     <username/>
  54.                     <password/>
  55.                     <publickey/>
  56.                     <privatekey/>
  57.                     <port/>
  58.                     <description/>
  59.                     <inventory_link>0</inventory_link>
  60.                     <applications>
  61.                         <application>
  62.                             <name>JSTAT</name>
  63.                         </application>
  64.                     </applications>
  65.                     <valuemap/>
  66.                     <logtimefmt/>
  67.                 </item>
  68.             </items>
  69.             <discovery_rules>
  70.                 <discovery_rule>
  71.                     <name>jmxport</name>
  72.                     <type>0</type>
  73.                     <snmp_community/>
  74.                     <snmp_oid/>
  75.                     <key>jmxport</key>
  76.                     <delay>30</delay>
  77.                     <status>0</status>
  78.                     <allowed_hosts/>
  79.                     <snmpv3_contextname/>
  80.                     <snmpv3_securityname/>
  81.                     <snmpv3_securitylevel>0</snmpv3_securitylevel>
  82.                     <snmpv3_authprotocol>0</snmpv3_authprotocol>
  83.                     <snmpv3_authpassphrase/>
  84.                     <snmpv3_privprotocol>0</snmpv3_privprotocol>
  85.                     <snmpv3_privpassphrase/>
  86.                     <delay_flex/>
  87.                     <params/>
  88.                     <ipmi_sensor/>
  89.                     <authtype>0</authtype>
  90.                     <username/>
  91.                     <password/>
  92.                     <publickey/>
  93.                     <privatekey/>
  94.                     <port/>
  95.                     <filter>:</filter>
  96.                     <lifetime>1</lifetime>
  97.                     <description/>
  98.                     <item_prototypes>
  99.                         <item_prototype>
  100.                             <name>port:$1 $2</name>
  101.                             <type>2</type>
  102.                             <snmp_community/>
  103.                             <multiplier>1</multiplier>
  104.                             <snmp_oid/>
  105.                             <key>jstat[{#JPORT},Young.Gc.Count]</key>
  106.                             <delay>0</delay>
  107.                             <history>90</history>
  108.                             <trends>365</trends>
  109.                             <status>0</status>
  110.                             <value_type>3</value_type>
  111.                             <allowed_hosts/>
  112.                             <units/>
  113.                             <delta>0</delta>
  114.                             <snmpv3_contextname/>
  115.                             <snmpv3_securityname/>
  116.                             <snmpv3_securitylevel>0</snmpv3_securitylevel>
  117.                             <snmpv3_authprotocol>0</snmpv3_authprotocol>
  118.                             <snmpv3_authpassphrase/>
  119.                             <snmpv3_privprotocol>0</snmpv3_privprotocol>
  120.                             <snmpv3_privpassphrase/>
  121.                             <formula>1</formula>
  122.                             <delay_flex/>
  123.                             <params/>
  124.                             <ipmi_sensor/>
  125.                             <data_type>0</data_type>
  126.                             <authtype>0</authtype>
  127.                             <username/>
  128.                             <password/>
  129.                             <publickey/>
  130.                             <privatekey/>
  131.                             <port/>
  132.                             <description/>
  133.                             <inventory_link>0</inventory_link>
  134.                             <applications>
  135.                                 <application>
  136.                                     <name>JSTAT</name>
  137.                                 </application>
  138.                             </applications>
  139.                             <valuemap/>
  140.                             <logtimefmt/>
  141.                         </item_prototype>
  142.                         <item_prototype>
  143.                             <name>port:$1 $2</name>
  144.                             <type>2</type>
  145.                             <snmp_community/>
  146.                             <multiplier>1</multiplier>
  147.                             <snmp_oid/>
  148.                             <key>jstat[{#JPORT},Old.Gc.Count]</key>
  149.                             <delay>0</delay>
  150.                             <history>90</history>
  151.                             <trends>365</trends>
  152.                             <status>0</status>
  153.                             <value_type>3</value_type>
  154.                             <allowed_hosts/>
  155.                             <units/>
  156.                             <delta>0</delta>
  157.                             <snmpv3_contextname/>
  158.                             <snmpv3_securityname/>
  159.                             <snmpv3_securitylevel>0</snmpv3_securitylevel>
  160.                             <snmpv3_authprotocol>0</snmpv3_authprotocol>
  161.                             <snmpv3_authpassphrase/>
  162.                             <snmpv3_privprotocol>0</snmpv3_privprotocol>
  163.                             <snmpv3_privpassphrase/>
  164.                             <formula>1</formula>
  165.                             <delay_flex/>
  166.                             <params/>
  167.                             <ipmi_sensor/>
  168.                             <data_type>0</data_type>
  169.                             <authtype>0</authtype>
  170.                             <username/>
  171.                             <password/>
  172.                             <publickey/>
  173.                             <privatekey/>
  174.                             <port/>
  175.                             <description/>
  176.                             <inventory_link>0</inventory_link>
  177.                             <applications>
  178.                                 <application>
  179.                                     <name>JSTAT</name>
  180.                                 </application>
  181.                             </applications>
  182.                             <valuemap/>
  183.                             <logtimefmt/>
  184.                         </item_prototype>
  185.                         <item_prototype>
  186.                             <name>port:$1 $2</name>
  187.                             <type>2</type>
  188.                             <snmp_community/>
  189.                             <multiplier>1</multiplier>
  190.                             <snmp_oid/>
  191.                             <key>jstat[{#JPORT},Tomcat.Thread]</key>
  192.                             <delay>0</delay>
  193.                             <history>90</history>
  194.                             <trends>365</trends>
  195.                             <status>0</status>
  196.                             <value_type>3</value_type>
  197.                             <allowed_hosts/>
  198.                             <units/>
  199.                             <delta>0</delta>
  200.                             <snmpv3_contextname/>
  201.                             <snmpv3_securityname/>
  202.                             <snmpv3_securitylevel>0</snmpv3_securitylevel>
  203.                             <snmpv3_authprotocol>0</snmpv3_authprotocol>
  204.                             <snmpv3_authpassphrase/>
  205.                             <snmpv3_privprotocol>0</snmpv3_privprotocol>
  206.                             <snmpv3_privpassphrase/>
  207.                             <formula>1</formula>
  208.                             <delay_flex/>
  209.                             <params/>
  210.                             <ipmi_sensor/>
  211.                             <data_type>0</data_type>
  212.                             <authtype>0</authtype>
  213.                             <username/>
  214.                             <password/>
  215.                             <publickey/>
  216.                             <privatekey/>
  217.                             <port/>
  218.                             <description/>
  219.                             <inventory_link>0</inventory_link>
  220.                             <applications>
  221.                                 <application>
  222.                                     <name>JSTAT</name>
  223.                                 </application>
  224.                             </applications>
  225.                             <valuemap/>
  226.                             <logtimefmt/>
  227.                         </item_prototype>
  228.                     </item_prototypes>
  229.                     <trigger_prototypes>
  230.                         <trigger_prototype>
  231.                             <expression>{jmx-im-jstat:jstat[{#JPORT},Tomcat.Thread].last(0)}>500</expression>
  232.                             <name>Tomcat [#JPORT] Thread is too high</name>
  233.                             <url/>
  234.                             <status>0</status>
  235.                             <priority>0</priority>
  236.                             <description/>
  237.                             <type>0</type>
  238.                         </trigger_prototype>
  239.                     </trigger_prototypes>
  240.                     <graph_prototypes>
  241.                         <graph_prototype>
  242.                             <name>port:{#JPORT} Tomcat.Thread</name>
  243.                             <width>900</width>
  244.                             <height>200</height>
  245.                             <yaxismin>0.0000</yaxismin>
  246.                             <yaxismax>100.0000</yaxismax>
  247.                             <show_work_period>1</show_work_period>
  248.                             <show_triggers>1</show_triggers>
  249.                             <type>0</type>
  250.                             <show_legend>1</show_legend>
  251.                             <show_3d>0</show_3d>
  252.                             <percent_left>0.0000</percent_left>
  253.                             <percent_right>0.0000</percent_right>
  254.                             <ymin_type_1>0</ymin_type_1>
  255.                             <ymax_type_1>0</ymax_type_1>
  256.                             <ymin_item_1>0</ymin_item_1>
  257.                             <ymax_item_1>0</ymax_item_1>
  258.                             <graph_items>
  259.                                 <graph_item>
  260.                                     <sortorder>0</sortorder>
  261.                                     <drawtype>0</drawtype>
  262.                                     <color>C80000</color>
  263.                                     <yaxisside>0</yaxisside>
  264.                                     <calc_fnc>2</calc_fnc>
  265.                                     <type>0</type>
  266.                                     <item>
  267.                                         <host>jmx-im-jstat</host>
  268.                                         <key>jstat[{#JPORT},Tomcat.Thread]</key>
  269.                                     </item>
  270.                                 </graph_item>
  271.                             </graph_items>
  272.                         </graph_prototype>
  273.                     </graph_prototypes>
  274.                     <host_prototypes/>
  275.                 </discovery_rule>
  276.             </discovery_rules>
  277.             <macros/>
  278.             <templates/>
  279.             <screens/>
  280.         </template>
  281.     </templates>
  282. </zabbix_export>

注：模板太长，我只取2个item 放置其中；

下面附上几张配置截图


zabbix使用python脚本监控java进程-LMLPHP



zabbix使用python脚本监控java进程-LMLPHP


zabbix使用python脚本监控java进程-LMLPHP



目前只配置了item，还没有添加报警和图形


三、创建数据，并用zabbix_sender发送到zabbix-server

zabbix_agentd配置文件

点击(此处)折叠或打开

  1. > cat /usr/local/zabbix/etc/zabbix_agentd.conf.d/jstat.conf
  2. UserParameter=jmxport,sudo /usr/bin/python /usr/local/zabbix/scripts/jstat.py --jvmport
  3. UserParameter=jmxdata,sudo /usr/bin/python /usr/local/zabbix/scripts/jstat.py --data

执行脚本

点击(此处)折叠或打开

  1. > sudo python jstat.py --data
  2. /usr/local/zabbix/bin/zabbix_sender -c /usr/local/zabbix/etc/zabbix_agentd.conf -i /usr/local/zabbix/scripts/.zabbix_jmx_status
  3. info from server: "processed: 45; failed: 0; total: 45; seconds spent: 0.000531"
  4. sent: 45; skipped: 0; total: 45

看上面结果，成功发送45个数据，数据会先临时存到 /usr/local/zabbix/scripts/.zabbix_jmx_status，我们查看下数据内容


点击(此处)折叠或打开

  1. > cat /usr/local/zabbix/scripts/.zabbix_jmx_status
  2. test-01 jstat[8083,Young.Gc.Time] 36.572
  3. test-01 jstat[8083,Old.Gc.Time] 3.971
  4. test-01 jstat[8083,Perm.Gc.New] 80384.0
  5. test-01 jstat[8083,Total.Gc.Time] 40.543
  6. ...
  7. test-01 jstat[7074,Old.Gc.Time] 1.734
  8. test-01 jstat[7074,Perm.Gc.New] 55296.0
  9. test-01 jstat[7074,Total.Gc.Time] 26.635

第一列表示主机名，这个与zabbix_agentd.conf的配置保持一致，而且 zabbix-web添加的主机名也要一样；
第二列表示 key，这个和 templates里定义的一致；
第三列是 数据；

四、查看数据


zabbix使用python脚本监控java进程-LMLPHP



可以看到数据已经在zabbix展示了，之后我们可以添加相应的 报警和图形，使这个 template更加完善。

#############################################
zabbix_sender发送信息两种方式
1.zabbix_get主动去获取item时会提示timeout，可以修改server，agentd的conf文件里的timeout参数为300，然后重启服务
2.通过crontd定时执行脚本









12-20 20:57
Powered by LMLPHP ©2024 gyq0618 0.069356