Ceph集群rbd-mirror A、B区域备份实施方案
备注：首先准备两个集群， 并确认其状态，集群的准备过程在这就不做陈述
1.查看集群状态
A区域
[root@ceph2111 ceph]# ceph -s
cluster 05440e8c-bbd9-4dca-b4e7-c0c88b3fa270
health HEALTH_OK
monmap e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0}
election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113
osdmap e63: 4 osds: 3 up, 3 in
flags sortbitwise
pgmap v4121: 64 pgs, 1 pools, 557 bytes data, 14 objects
111 MB used, 45935 MB / 46046 MB avail
64 active+clean
[root@ceph2111 ceph]# B区域 [root@ceph111 ceph]# ceph -s
cluster e091a437-85a5-4561-b445-ef67ea98cfb2
health HEALTH_OK
monmap e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0}
election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115
osdmap e40: 6 osds: 6 up, 6 in
flags sortbitwise
pgmap v293: 64 pgs, 1 pools, 555 bytes data, 14 objects
205 MB used, 30448 MB / 30653 MB avail
64 active+clean
client io 0 B/s rd, 0 op/s rd, 0 op/s wr 2.在两个集群里都安装tbd-mirror rpm包 yum install rbd-mirror 3.在A和B两端都启动rbd-mirror的进程 [root@ceph2111 ceph]# rbd-mirror -m 10.10.2.111 -d
2016-10-27 12:17:28.140639 7f788b294c40 0 ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process rbd-mirror, pid 1867 这里是用nohup 在后台运行
A区域
[root@ceph2111 ceph]# nohup rbd-mirror -m 10.10.2.111 -d > /dev/null 2>&1 &
[1] 1953 B区域
[root@ceph111 ~]# nohup rbd-mirror -m 10.10.1.111 -d > /dev/null 2>&1 &
[1] 15408 确认是否在运行 [root@ceph2111 ceph]# ps -ef|grep rbd
root 1953 30723 0 12:19 pts/0 00:00:00 rbd-mirror -m 10.10.2.111 -d
root 1984 30723 0 12:19 pts/0 00:00:00 grep --color=auto rbd 4.在ceph.conf 里添加 rbd_default_features = 125 A区域
[root@ceph2111 ceph]# vi ceph.conf
[global]
fsid = 05440e8c-bbd9-4dca-b4e7-c0c88b3fa270
mon_initial_members = ceph2111, ceph2112, ceph2113
mon_host = 10.10.2.111,10.10.2.112,10.10.2.113
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 2
public network = 10.10.2.0/24
rbd_default_features = 125
~
"ceph.conf" 10L, 323C written B区域
[root@ceph111 my_cluster]# vi ceph.conf
[global]
fsid = e091a437-85a5-4561-b445-ef67ea98cfb2
mon_initial_members = ceph111, ceph112, ceph115
mon_host = 10.10.1.111,10.10.1.112,10.10.1.115
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 2
public network = 10.10.1.0/24
rbd_default_features = 125 5.对A进行设置 [root@ceph2111 ceph]# cp ceph.conf local.conf
[root@ceph2111 ceph]# cp ceph.client.admin.keyring local.client.admin.keyring 把A区域 cp到B区域环境中 [root@ceph2111 ceph]# scp local.conf local.client.admin.keyring [email protected]:/etc/ceph/
local.conf 100% 323 0.3KB/s 00:00
local.client.admin.keyring 在A 集群里进行 同步配置 [root@ceph2111 ceph]# ceph-deploy --overwrite-conf admin ceph{2111..2114} 检验上面设置是否完成 A区域
[root@ceph2111 ceph]# ceph --cluster local mon stat
e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0}, election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113
[root@ceph2111 ceph]# ceph --cluster remote mon stat
e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0}, election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115 B区域 root@ceph111 ceph]# cp ceph.conf remote.conf
[root@ceph111 ceph]# cp ceph.client.admin.keyring remote.client.admin.keyring
[root@ceph111 ceph]# ls
ceph.client.admin.keyring ceph.conf local.client.admin.keyring local.conf rbdmap remote.client.admin.keyring remote.conf tmpr_jrwC tmptBCcx2
[root@ceph111 ceph]# scp remote.c
remote.client.admin.keyring remote.conf
[root@ceph111 ceph]# scp remote.c* [email protected]:/etc/ceph/
remote.client.admin.keyring 100% 129 0.1KB/s 00:00
remote.conf [root@ceph111 ceph]# ceph --cluster local mon stat
e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0}, election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113
[root@ceph111 ceph]# ceph --cluster remote mon stat
e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0}, election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115 100% 320 0.3KB/s 00:00 6.准备工作完成了，开始做相关配置
A区域
[root@ceph2111 ceph]# rbd --cluster local mirror pool enable rbd pool
[root@ceph2111 ceph]# rbd --cluster remote mirror pool enable rbd pool 如果需要关闭那么执行 1. rbd --cluster local mirror pool disable rbd
2. rbd --cluster remote mirror pool disable rbd 增加 CLUSTER PEER
格式
rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}
使用admin这个账户就可以了 执行
A区域 [root@ceph2111 ceph]# rbd --cluster local mirror pool peer add rbd client.admin@remote
d79c69c5-5fac-4bcb-8e93-28a18994d637
[root@ceph2111 ceph]# rbd --cluster remote mirror pool peer add rbd client.admin@local
9dd66d08-89e4-4c91-a389-649b9c6940de 7.查询是否连接好
A区域 [root@ceph2111 ceph]# rbd --cluster local mirror pool info
Mode: pool
Peers:
UUID NAME CLIENT
d79c69c5-5fac-4bcb-8e93-28a18994d637 remote client.admin
[root@ceph2111 ceph]# rbd --cluster remote mirror pool info
Mode: pool
Peers:
UUID NAME CLIENT
9dd66d08-89e4-4c91-a389-649b9c6940de local client.admin 如果需要删除
rbd mirror pool peer remove {pool-name} {peer-uuid} 执行
1. rbd --cluster local mirror pool peer remove image-pool uuid
2. rbd --cluster remote mirror pool peer remove image-pool uuid 8.验证是否成功 A区域
[root@ceph2111 ceph]# rbd create test2111 --size 400 开启tbd-mirror pool enable 功能 [root@ceph2111 ceph]# rbd-mirror pool enable
[root@ceph2111 ceph]# rbd ls
test111
test2111
[root@ceph2111 ceph]# B区域
[root@ceph111 ceph]# rbd create test111 --size 400
[root@ceph111 ceph]# rbd ls [root@ceph111 ceph]# rbd-mirror pool enable
[root@ceph111 ceph]# rbd ls
test111
test2111 9.备注镜像的升级与降级 1. ceph]# rbd --cluster local mirror image demote rbd/test111
2. ceph]# rbd --cluster local mirror image promote rbd/test111
3. ceph]# rbd --cluster local mirror image promote rbd/test111
4. rbd: error promoting image to primary
5. 2016-03-30 23:35:13.477096 7ffa50a3dc00 -1 librbd: image is already primary