使用ganglia监控mongodb集群

    前几天提交了一篇ganglia监控storm集群的博文，本文将介绍使用ganglia监控mongdb集群。因为我们需要使用ganglia一统天下。
1. ganglia扩展机制
    首先要使用ganglia监控mongodb集群必须先明白ganglia的扩展机制。通过ganglia插件可以给我们提供两种扩展ganglia监控功能的方法：

    1）、通过添加内嵌（in-band）插件，主要是通过gmetric命令来实现。

    这是通常使用的一种方法，主要是通过cronjob方法并调用ganglia的gmetric命令来向gmond输入数据，进而实现统一监控，这种方法简单，对于少量的监控可以采用，但是对于大规模自定义监控时，监控数据难以统一管理。

    2）、通过添加一些额外的脚本来实现对系统的监控，主要是通过C或者python接口来实现。

    在ganglia3.1.x版本以后，增加了C或者Python接口，通过这个接口可以自定义数据收集模块，并且这些模块可以被直接插入到gmond中以监控用户自定义的应用。
2. python脚本监控mongdb
    我们使用python脚本来监控mongodb集群，毕竟通过python脚本扩展比较方便，需要增加监控信息时在相应的py脚本中添加监控数据就可以了，十分方便，扩展性强，移植也比较简单。
2.1 环境配置
    要使用python脚本来实现ganglia监控扩展，首先需要明确modpython.so文件是否存在，这个文件是ganglia调用python的动态链接库，要通过python接口开发ganglia插件，必须要编译安装此模块。modpython.so文件存放在ganglia安装目录下的lib(or lib64)/ganglia/目录中。如果存在则可以进行下面的脚本编写；如果不存在，那么需要你重新编译安装gmond哦，编译安装时带上参数“--with-python”。
2.2 编写监控脚本
     我们打开ganglia安装目录下的/etc/gmond.conf文件，可以发现在客户端监控中可以看到include ("/usr/local/ganglia/etc/conf.d/*.conf")，说明gmond服务直接扫描目录下的监控配置文件，所以我们需要将监控配置脚本放在/etc/conf.d/目录下并命名为XX.conf，所以我们将要监控mongdb的配置脚本命名为mongdb.conf
    1)、查看modpython.conf文件
    modpython.conf位于/etc/conf.d/目录下。文件内容如下：

点击(此处)折叠或打开

1. modules {
2. module {
3. name = "python_module"   #主模块文成
   
4. path = "modpython.so"    #ganglia扩展python脚本需要的动态链接库
   
5. params = "/usr/local/ganglia/lib64/ganglia/python_modules"     #python脚本存放的位置
   
6. }
7. }
8. 
   
9. include ("/usr/local/ganglia/etc/conf.d/*.pyconf")     #ganglia扩展存放配置脚本的路径  
   

    所以我们使用python来扩展ganglia监控mongodb需要将配置脚本和py脚本放在相应的目录下，再重启ganglia服务就可以完成mongdb监控，下面将介绍如何编写脚本。
    2)、创建mongodb.pyconf脚本
    注意这里需要使用root权限来创建编辑脚本，将此脚本存放在conf.d目录下。具体要收集mongdb那些参数可以参考https://github.com/ganglia/gmond_python_modules/tree/master/mongodb，根据自己的需求酌量增删。

点击(此处)折叠或打开

1. modules {
   
2.   module {
3.     name = "mongodb"   #模块名，该模块名必须与开发的存放于"/usr/lib64/ganglia/python_modules"指定的路径下的python脚本名称一致
4.     language = "python"   #声明使用python语言
5.     #参数列表，所有的参数作为一个dict(即map)传给python脚本的metric_init(params)函数。
   
6.     param server_status{
   
7.         value = "mongo路径 --host host --port 27017 --quiet --eval 'printjson(db.serverStatus())'"
   
8.     }
   
9.     param rs_status{
   
10.         value = "mongo路径 --host host --port 2701 --quiet --eval 'printjson(rs.status())'"
    
11.     }
    
12.   }
    
13. }
    
14. 
    
15. #需要收集的metric列表，一个模块中可以扩展任意个metric
    
16. collection_group {
    
17.   collect_every = 30
    
18.   time_threshold = 90 #最大发送间隔
    
19.   metric {
20.     name = "mongodb_opcounters_insert" #metric在模块中的名字
    
21.     title = "Inserts" #图形界面上显示的标题
    
22.   }
    
23.   metric {
24.     name = "mongodb_opcounters_query"
25.     title = "Queries"
26.   }
27.   metric {
28.     name = "mongodb_opcounters_update"
29.     title = "Updates"
30.   }
31.   metric {
32.     name = "mongodb_opcounters_delete"
33.     title = "Deletes"
34.   }
35.   metric {
36.     name = "mongodb_opcounters_getmore"
37.     title = "Getmores"
38.   }
39.   metric {
40.     name = "mongodb_opcounters_command"
41.     title = "Commands"
42.   }
43.   metric {
44.     name = "mongodb_backgroundFlushing_flushes"
45.     title = "Flushes"
46.   }
47.   metric {
48.     name = "mongodb_mem_mapped"
49.     title = "Memory-mapped Data"
50.   }
51.   metric {
52.     name = "mongodb_mem_virtual"
53.     title = "Process Virtual Size"
54.   }
55.   metric {
56.     name = "mongodb_mem_resident"
57.     title = "Process Resident Size"
58.   }
59.   metric {
60.     name = "mongodb_extra_info_page_faults"
61.     title = "Page Faults"
62.   }
63.   metric {
64.     name = "mongodb_globalLock_ratio"
65.     title = "Global Write Lock Ratio"
66.   }
67.   metric {
68.     name = "mongodb_indexCounters_btree_miss_ratio"
69.     title = "BTree Page Miss Ratio"
70.   }
71.   metric {
72.     name = "mongodb_globalLock_currentQueue_total"
73.     title = "Total Operations Waiting for Lock"
74.   }
75.   metric {
76.     name = "mongodb_globalLock_currentQueue_readers"
77.     title = "Readers Waiting for Lock"
78.   }
79.   metric {
80.     name = "mongodb_globalLock_currentQueue_writers"
81.     title = "Writers Waiting for Lock"
82.   }
83.   metric {
84.     name = "mongodb_globalLock_activeClients_total"
85.     title = "Total Active Clients"
86.   }
87.   metric {
88.     name = "mongodb_globalLock_activeClients_readers"
89.     title = "Active Readers"
90.   }
91.   metric {
92.     name = "mongodb_globalLock_activeClients_writers"
93.     title = "Active Writers"
94.   }
95.   metric {
96.     name = "mongodb_connections_current"
97.     title = "Open Connections"
98.   }
99.   metric {
100.     name = "mongodb_connections_current_ratio"
101.     title = "Open Connections"
102.   }
103.   metric {
104.     name = "mongodb_slave_delay"
105.     title = "Replica Set Slave Delay"
106.   }
107.   metric {
108.     name = "mongodb_asserts_total"
109.     title = "Asserts per Second"
110.   }
111. }

    从上面你可以发现这个配置文件的写法跟gmond.conf的语法一致，所以有什么不明白的可以参考gmond.conf的写法。
3)、创建mongodb.py脚本
    将mongodb.py文件存放在lib64/ganglia/python_modules目录下，在这个目录中可以看到已经有很多python脚本存在，比如：监控磁盘、内存、网络、mysql、redis等的脚本。我们可以参考这些python脚本完成mongodb.py的编写。我们打开其中部分脚本可以看到在每个脚本中都有一个函数metric_init(params)，前面也说过mongodb.pyconf传来的参数传递给metric_init函数。
    

点击(此处)折叠或打开

1. #!/usr/bin/env python
   
2. import json
3. import os
4. import re
5. import socket
6. import string
7. import time
8. import copy
9. 
   
10. NAME_PREFIX = 'mongodb_'
11. PARAMS = {
12.     'server_status' : '/bin/mongo路径 --host host --port 27017 --quiet --eval "printjson(db.serverStatus())"',
13.     'rs_status' : '/bin/mongo路径 --host host --port 27017 --quiet --eval "printjson(rs.status())"'
14. }
15. METRICS = {
16.     'time' : 0,
17.     'data' : {}
18. }
19. LAST_METRICS = copy.deepcopy(METRICS)
20. METRICS_CACHE_TTL = 3
21. def flatten(d, pre = '', sep = '_'):
22.     """Flatten a dict (i.e. dict['a']['b']['c'] => dict['a_b_c'])"""
23.     new_d = {}
24.     for k,v in d.items():
25.         if type(v) == dict:
26.             new_d.update(flatten(d[k], '%s%s%s' % (pre, k, sep)))
27.         else:
28.             new_d['%s%s' % (pre, k)] = v
29.     return new_d
30. 
    
31. def get_metrics():
32.     """Return all metrics"""
33.     global METRICS, LAST_METRICS
34.     if (time.time() - METRICS['time']) > METRICS_CACHE_TTL:
35.         metrics = {}
36.         for status_type in PARAMS.keys():
37.             # get raw metric data
38.             o = os.popen(PARAMS[status_type])
39.             # clean up
40.             metrics_str = ''.join(o.readlines()).strip() # convert to string
41.             metrics_str = re.sub('\w+\((.*)\)', r"\1", metrics_str) # remove functions
42.             # convert to flattened dict
43.             try:
44.                 if status_type == 'server_status':
45.                     metrics.update(flatten(json.loads(metrics_str)))
46.                 else:
47.                     metrics.update(flatten(json.loads(metrics_str), pre='%s_' % status_type))
48.             except ValueError:
49.                 metrics = {}
50. 
    
51.         # update cache
52.         LAST_METRICS = copy.deepcopy(METRICS)
53.         METRICS = {
54.             'time': time.time(),
55.             'data': metrics
56.         }
57.     return [METRICS, LAST_METRICS]
58. 
    
59. def get_value(name):
60.     """Return a value for the requested metric"""
61.     # get metrics
62.     metrics = get_metrics()[0]
63.     # get value
64.     name = name[len(NAME_PREFIX):] # remove prefix from name
65.     try:
66.         result = metrics['data'][name]
67.     except StandardError:
68.         result = 0
69.     return result
70. 
    
71. def get_rate(name):
72.     """Return change over time for the requested metric"""
73.     # get metrics
74.     [curr_metrics, last_metrics] = get_metrics()
75.     # get rate
76.     name = name[len(NAME_PREFIX):] # remove prefix from name
77.     try:
78.         rate = float(curr_metrics['data'][name] - last_metrics['data'][name]) / \
79.         float(curr_metrics['time'] - last_metrics['time'])
80.         if rate < 0:
81.             rate = float(0)
82.     except StandardError:
83.         rate = float(0)
84.     return rate
85. 
    
86. def get_opcounter_rate(name):
87.     """Return change over time for an opcounter metric"""
88.     master_rate = get_rate(name)
89.     repl_rate = get_rate(name.replace('opcounters_', 'opcountersRepl_'))
90.     return master_rate + repl_rate
91. 
    
92. def get_globalLock_ratio(name):
93.     """Return the global lock ratio"""
94.     try:
95.         result = get_rate(NAME_PREFIX + 'globalLock_lockTime') / \
96.         get_rate(NAME_PREFIX + 'globalLock_totalTime') * 100
97.     except ZeroDivisionError:
98.         result = 0
99.     return result
100. 
     
101. def get_indexCounters_btree_miss_ratio(name):
102.     """Return the btree miss ratio"""
103.     try:
104.         result = get_rate(NAME_PREFIX + 'indexCounters_btree_misses') / \
105.         get_rate(NAME_PREFIX + 'indexCounters_btree_accesses') * 100
106.     except ZeroDivisionError:
107.         result = 0
108.     return result
109. 
     
110. def get_connections_current_ratio(name):
111.     """Return the percentage of connections used"""
112.     try:
113.         result = float(get_value(NAME_PREFIX + 'connections_current')) / \
114.         float(get_value(NAME_PREFIX + 'connections_available')) * 100
115.     except ZeroDivisionError:
116.         result = 0
117.     return result
118. 
     
119. def get_slave_delay(name):
120.     """Return the replica set slave delay"""
121.     # get metrics
122.     metrics = get_metrics()[0]
123.     # no point checking my optime if i'm not replicating
124.     if 'rs_status_myState' not in metrics['data'] or metrics['data']['rs_status_myState'] != 2:
125.         result = 0
126.     # compare my optime with the master's
127.     else:
128.         master = {}
129.         slave = {}
130.     try:
131.         for member in metrics['data']['rs_status_members']:
132.             if member['state'] == 1:
133.                 master = member
134.             if member['name'].split(':')[0] == socket.getfqdn():
135.                 slave = member
136.         result = max(0, master['optime']['t'] - slave['optime']['t']) / 1000
137.     except KeyError:
138.         result = 0
139.     return result
140. 
     
141. def get_asserts_total_rate(name):
142.     """Return the total number of asserts per second"""
143.     return float(reduce(lambda memo,obj: memo + get_rate('%sasserts_%s' % (NAME_PREFIX, obj)),['regular', 'warning', 'msg', 'user', 'rollovers'], 0))
144. 
     
145. def metric_init(lparams):
146.     """Initialize metric descriptors"""
147.     global PARAMS
148.     # set parameters
149.     for key in lparams:
150.         PARAMS[key] = lparams[key]
151.     # define descriptors
152.     time_max = 60
153.     groups = 'mongodb'
154.     descriptors = [
155.         {
156.             'name': NAME_PREFIX + 'opcounters_insert',
157.             'call_back': get_opcounter_rate,
158.             'time_max': time_max,
159.             'value_type': 'float',
160.             'units': 'Inserts/Sec',
161.             'slope': 'both', 
162.             'format': '%f',
163.             'description': 'Inserts',
164.             'groups': groups
165.         },
166.         {
167.             'name': NAME_PREFIX + 'opcounters_query',
168.             'call_back': get_opcounter_rate,
169.             'time_max': time_max,
170.             'value_type': 'float',
171.             'units': 'Queries/Sec',
172.             'slope': 'both',
173.             'format': '%f',
174.             'description': 'Queries',
175.             'groups': groups
176.         },
177.         {
178.             'name': NAME_PREFIX + 'opcounters_update',
179.             'call_back': get_opcounter_rate,
180.             'time_max': time_max,
181.             'value_type': 'float',
182.             'units': 'Updates/Sec',
183.             'slope': 'both',
184.             'format': '%f',
185.             'description': 'Updates',
186.             'groups': groups
187.         },
188.         {
189.             'name': NAME_PREFIX + 'opcounters_delete',
190.             'call_back': get_opcounter_rate,
191.             'time_max': time_max,
192.             'value_type': 'float',
193.             'units': 'Deletes/Sec',
194.             'slope': 'both',
195.             'format': '%f',
196.             'description': 'Deletes',
197.             'groups': groups
198.         },
199.         {
200.             'name': NAME_PREFIX + 'opcounters_getmore',
201.             'call_back': get_opcounter_rate,
202.             'time_max': time_max,
203.             'value_type': 'float',
204.             'units': 'Getmores/Sec',
205.             'slope': 'both',
206.             'format': '%f',
207.             'description': 'Getmores',
208.             'groups': groups
209.         },
210.         {
211.             'name': NAME_PREFIX + 'opcounters_command',
212.             'call_back': get_opcounter_rate,
213.             'time_max': time_max,
214.             'value_type': 'float',
215.             'units': 'Commands/Sec',
216.             'slope': 'both',
217.             'format': '%f',
218.             'description': 'Commands',
219.             'groups': groups
220.         },
221.         {
222.             'name': NAME_PREFIX + 'backgroundFlushing_flushes',
223.             'call_back': get_rate,
224.             'time_max': time_max,
225.             'value_type': 'float',
226.             'units': 'Flushes/Sec',
227.             'slope': 'both',
228.             'format': '%f',
229.             'description': 'Flushes',
230.             'groups': groups
231.         },
232.         {
233.             'name': NAME_PREFIX + 'mem_mapped',
234.             'call_back': get_value,
235.             'time_max': time_max,
236.             'value_type': 'uint',
237.             'units': 'MB',
238.             'slope': 'both',
239.             'format': '%u',
240.             'description': 'Memory-mapped Data',
241.             'groups': groups
242.         },
243.         {
244.             'name': NAME_PREFIX + 'mem_virtual',
245.             'call_back': get_value,
246.             'time_max': time_max,
247.             'value_type': 'uint',
248.             'units': 'MB',
249.             'slope': 'both',
250.             'format': '%u',
251.             'description': 'Process Virtual Size',
252.             'groups': groups
253.         },
254.         {
255.             'name': NAME_PREFIX + 'mem_resident',
256.             'call_back': get_value,
257.             'time_max': time_max,
258.             'value_type': 'uint',
259.             'units': 'MB',
260.             'slope': 'both',
261.             'format': '%u',
262.             'description': 'Process Resident Size',
263.             'groups': groups
264.         },
265.         {
266.             'name': NAME_PREFIX + 'extra_info_page_faults',
267.             'call_back': get_rate,
268.             'time_max': time_max,
269.             'value_type': 'float',
270.             'units': 'Faults/Sec',
271.             'slope': 'both',
272.             'format': '%f',
273.             'description': 'Page Faults',
274.             'groups': groups
275.         },
276.         {
277.             'name': NAME_PREFIX + 'globalLock_ratio',
278.             'call_back': get_globalLock_ratio,
279.             'time_max': time_max,
280.             'value_type': 'float',
281.             'units': '%',
282.             'slope': 'both',
283.             'format': '%f',
284.             'description': 'Global Write Lock Ratio',
285.             'groups': groups
286.         },
287.         {
288.             'name': NAME_PREFIX + 'indexCounters_btree_miss_ratio',
289.             'call_back': get_indexCounters_btree_miss_ratio,
290.             'time_max': time_max,
291.             'value_type': 'float',
292.             'units': '%',
293.             'slope': 'both',
294.             'format': '%f',
295.             'description': 'BTree Page Miss Ratio',
296.             'groups': groups
297.         },
298.         {
299.             'name': NAME_PREFIX + 'globalLock_currentQueue_total',
300.             'call_back': get_value,
301.             'time_max': time_max,
302.             'value_type': 'uint',
303.             'units': 'Operations',
304.             'slope': 'both',
305.             'format': '%u',
306.             'description': 'Total Operations Waiting for Lock',
307.             'groups': groups
308.         },
309.         {
310.             'name': NAME_PREFIX + 'globalLock_currentQueue_readers',
311.             'call_back': get_value,
312.             'time_max': time_max,
313.             'value_type': 'uint',
314.             'units': 'Operations',
315.             'slope': 'both',
316.             'format': '%u',
317.             'description': 'Readers Waiting for Lock',
318.             'groups': groups
319.         },
320.         {
321.             'name': NAME_PREFIX + 'globalLock_currentQueue_writers',
322.             'call_back': get_value,
323.             'time_max': time_max,
324.             'value_type': 'uint',
325.             'units': 'Operations',
326.             'slope': 'both',
327.             'format': '%u',
328.             'description': 'Writers Waiting for Lock',
329.             'groups': groups
330.         },
331.         {
332.             'name': NAME_PREFIX + 'globalLock_activeClients_total',
333.             'call_back': get_value,
334.             'time_max': time_max,
335.             'value_type': 'uint',
336.             'units': 'Clients',
337.             'slope': 'both',
338.             'format': '%u',
339.             'description': 'Total Active Clients',
340.             'groups': groups
341.         },
342.         {
343.             'name': NAME_PREFIX + 'globalLock_activeClients_readers',
344.             'call_back': get_value,
345.             'time_max': time_max,
346.             'value_type': 'uint',
347.             'units': 'Clients',
348.             'slope': 'both',
349.             'format': '%u',
350.             'description': 'Active Readers',
351.             'groups': groups
352.         },
353.         {
354.             'name': NAME_PREFIX + 'globalLock_activeClients_writers',
355.             'call_back': get_value,
356.             'time_max': time_max,
357.             'value_type': 'uint',
358.             'units': 'Clients',
359.             'slope': 'both',
360.             'format': '%u',
361.             'description': 'Active Writers',
362.             'groups': groups
363.         },
364.         {
365.             'name': NAME_PREFIX + 'connections_current',
366.             'call_back': get_value,
367.             'time_max': time_max,
368.             'value_type': 'uint',
369.             'units': 'Connections',
370.             'slope': 'both',
371.             'format': '%u',
372.             'description': 'Open Connections',
373.             'groups': groups
374.         },
375.         {
376.             'name': NAME_PREFIX + 'connections_current_ratio',
377.             'call_back': get_connections_current_ratio,
378.             'time_max': time_max,
379.             'value_type': 'float',
380.             'units': '%',
381.             'slope': 'both',
382.             'format': '%f',
383.             'description': 'Percentage of Connections Used',
384.             'groups': groups
385.         },
386.         {
387.             'name': NAME_PREFIX + 'slave_delay',
388.             'call_back': get_slave_delay,
389.             'time_max': time_max,
390.             'value_type': 'uint',
391.             'units': 'Seconds',
392.             'slope': 'both',
393.             'format': '%u',
394.             'description': 'Replica Set Slave Delay',
395.             'groups': groups
396.         },
397.         {
398.             'name': NAME_PREFIX + 'asserts_total',
399.             'call_back': get_asserts_total_rate,
400.             'time_max': time_max,
401.             'value_type': 'float',
402.             'units': 'Asserts/Sec',
403.             'slope': 'both',
404.             'format': '%f',
405.             'description': 'Asserts',
406.             'groups': groups
407.         }
408.     ]
409.     return descriptors
410. 
     
411. def metric_cleanup():
412.     """Cleanup"""
413.     pass
414. 
     
415. # the following code is for debugging and testing
416. if __name__ == '__main__':
417.     descriptors = metric_init(PARAMS)
418.     while True:
419.         for d in descriptors:
420.             print (('%s = %s') % (d['name'], d['format'])) % (d['call_back'](d['name']))
421.         print ''
422.         time.sleep(METRICS_CACHE_TTL)
     

    python扩展脚本中必须要重写的函数有：metric_init(params)，metric_cleanup()
    metric_init()函数在模块初始化的时候调用，必须要返回一个metric描述字典或者字典列表，mongdb.py就返回了字典列表。

    Metric字典定义如下：

    d = {‘name’ : ‘’,   #这个name必须跟pyconf文件中的名字保持一致

            'call_back’ : ,

            'time_max’ : int(),

            'value_type’ : ‘’,

            'units’ : ’’,

            'slope’ : ‘’,

            'format’ : ‘’,

            'description’ : ‘’
        }
    metric_cleanup()函数在模块结束时调用，无数据返回
4)、在web端查看监控统计
    完成脚本编写后，重启gmond服务。