处理Zenoss HA DRBD脑裂

今天做HA切换测试时，发现Zenoss数据只记录到很早以前的情况。判断是Drbd脑裂。主上显示：

点击(此处)折叠或打开

1. # service drbd status
2. drbd driver loaded OK; device status:
3. version: 8.2.6 (api:88/proto:86-88)
4. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
5. m:res cs st ds p mounted fstype
6. 1:mysql StandAlone Primary/Unknown UpToDate/DUnknown - /var/lib/mysql ext3
7. 2:zenhome StandAlone Primary/Unknown UpToDate/DUnknown - /opt/zenoss ext3
8. 3:zenperf StandAlone Primary/Unknown UpToDate/DUnknown - /opt/zenoss/perf ext3

从上显示：

点击(此处)折叠或打开

1. # service drbd status
2. drbd driver loaded OK; device status:
3. version: 8.2.6 (api:88/proto:86-88)
4. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
5. m:res cs st ds p mounted fstype
6. 1:mysql StandAlone Secondary/Unknown UpToDate/DUnknown -
7. 2:zenhome StandAlone Secondary/Unknown UpToDate/DUnknown -
8. 3:zenperf StandAlone Secondary/Unknown UpToDate/DUnknown -

对于在从上入手，修复脑裂

点击(此处)折叠或打开

1. # drbdadm disconnect mysql
2. # drbdadm disconnect zenhome
3. # drbdadm disconnect zenperf
4. # drbdadm secondary mysql
5. # drbdadm secondary zenhome
6. # drbdadm secondary zenperf
7. # drbdadm -- --discard-my-data connect mysql
8. # drbdadm -- --discard-my-data connect zenhome
9. # drbdadm -- --discard-my-data connect zenperf

查看从的Drbd状态

点击(此处)折叠或打开

1. more /proc/drbd
2. version: 8.2.6 (api:88/proto:86-88)
3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
4. ild, 2008-10-03 11:30:17
6. 1: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
7. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:7584
8. 2: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
9. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:47888
10. 3: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
11. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:1936

显示WFConnection状态。看来，还得重新在主上进行连接。

点击(此处)折叠或打开

1. # drbdadm connect mysql
2. # drbdadm connect zenhome
3. # drbdadm connect zenperf

再次查看主从状态，DRBD脑裂恢复。

点击(此处)折叠或打开

1. more /proc/drbd
2. version: 8.2.6 (api:88/proto:86-88)
3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
4. ild, 2008-10-03 11:30:17
6. 1: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
7. ns:89456 nr:0 dw:45201920 dr:162898 al:97 bm:98 lo:1 pe:9 ua:253 ap:1 oos:16
8. 6712
9. [======>.............] sync'ed: 36.6% (166712/255948)K
10. finish: 0:00:36 speed: 4,464 (6,864) K/sec
11. 2: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
12. ns:47360 nr:0 dw:4351072 dr:532070 al:229 bm:394 lo:1 pe:9 ua:1325 ap:2 oos:
13. 339072
14. [=>..................] sync'ed: 13.7% (339072/386240)K
15. finish: 0:01:04 speed: 5,240 (5,240) K/sec
16. 3: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
17. ns:15372 nr:0 dw:8790796 dr:41474 al:66 bm:89 lo:2 pe:6 ua:344 ap:1 oos:1214
18. 48
19. [=>..................] sync'ed: 14.8% (121448/136668)K
20. finish: 0:00:31 speed: 3,804 (3,804) K/sec

点击(此处)折叠或打开

1. # more /proc/drbd
2. version: 8.2.6 (api:88/proto:86-88)
3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
5. 1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
6. ns:0 nr:192920 dw:192920 dr:0 al:0 bm:14 lo:1 pe:1980 ua:0 ap:0 oos:63216
7. [==============>.....] sync'ed: 76.2% (63216/255948)K
8. finish: 0:00:26 speed: 2,368 (4,816) K/sec
9. 2: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
10. ns:0 nr:150944 dw:150944 dr:0 al:0 bm:18 lo:2 pe:7320 ua:0 ap:0 oos:235328
11. [=======>............] sync'ed: 40.0% (235328/386240)K
12. finish: 0:00:53 speed: 4,332 (4,192) K/sec
13. 3: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
14. ns:0 nr:119000 dw:119000 dr:0 al:0 bm:7 lo:1 pe:688 ua:0 ap:0 oos:17684
15. [================>...] sync'ed: 88.3% (17684/136668)K
16. finish: 0:00:03 speed: 4,600 (3,836) K/sec

看来，以后需要多注意DRBD状态，否则，Zenoss很有可能会切换到其它以住时期的运行状态。