iSCSI网关集成了Ceph存储和iSCSI标准，以提供一个高可用性(HA) iSCSI目标，该目标将RADOS块设备(RBD)映像导出为SCSI磁盘。iSCSI协议允许客户机 (initiator) 通过TCP/IP网络向SCSI存储设备( targets )发送SCSI命令。这允许异构客户机访问Ceph存储集群。
每个iSCSI网关运行Linux IO目标内核子系统(LIO)，以提供iSCSI协议支持。LIO利用用户空间通过( TCMU ) 与Ceph的librbd库交互，并向iSCSI客户机暴露RBD镜像。使用Ceph的iSCSI网关，可以有效地运行一个完全集成的块存储基础设施，它具有传统存储区域网络(SAN)的所有特性和优点。
1、创建池和镜像
（1）创建池
# ceph osd pool create iscsi-images 128 128 replicated # ceph osd pool application enable iscsi-images rbd
（2）创建images
# rbd --pool iscsi-images create --size=2048 'iscsi-gateway-image001' # rbd --pool iscsi-images create --size=4096 'iscsi-gateway-image002' # rbd --pool iscsi-images create --size=2048 'iscsi-gateway-image003' # rbd --pool iscsi-images create --size=4096 'iscsi-gateway-image004'
（3）显示images
# rbd ls -p iscsi-images iscsi-gateway-image001 iscsi-gateway-image002 iscsi-gateway-image003 iscsi-gateway-image004
2、deepsea 方式安装iSCSI网关
（1）node001 和 node002节点上安装，编辑policy.cfg 文件
vim /srv/pillar/ceph/proposals/policy.cfg ...... # IGW role-igw/cluster/node00[1-2]*.sls ......
（2）运行 stage 2 和 stage 4
# salt-run state.orch ceph.stage.2 # salt 'node001*' pillar.items public_network: 192.168.2.0/24 roles: - mon - mgr - storage - igw time_server: admin.example.com # salt-run state.orch ceph.stage.4
3、手动方式安装iSCSI网关
（1）node003 节点安装 iscsi 软件包
# zypper -n in -t pattern ceph_iscsi # zypper -n in tcmu-runner tcmu-runner-handler-rbd \ ceph-iscsi patterns-ses-ceph_iscsi python3-Flask python3-click python3-configshell-fb \ python3-itsdangerous python3-netifaces python3-rtslib-fb \ python3-targetcli-fb python3-urwid targetcli-fb-common
（2）admin节点创建key,并复制到 node003
# ceph auth add client.igw.node003 mon 'allow *' osd 'allow *' mgr 'allow r' # ceph auth get client.igw.node003 client.igw.node003 key: AQC0eotdAAAAABAASZrZH9KEo0V0WtFTCW9AHQ== caps: [mgr] allow r caps: [mon] allow * caps: [osd] allow *
# ceph auth get client.igw.node003 >> /etc/ceph/ceph.client.igw.node003.keyring # scp /etc/ceph/ceph.client.igw.node003.keyring node003:/etc/ceph
（3）node003 节点启动服务
# systemctl start tcmu-runner.service
# systemctl enable tcmu-runner.service （4）node003 节点创建配置文件
# vim /etc/ceph/iscsi-gateway.cfg [config] cluster_client_name = client.igw.node003 pool = iscsi-images trusted_ip_list = 192.168.2.42,192.168.2.40,192.168.2.41 minimum_gateways = 1 fqdn_enabled=true # Additional API configuration options are as follows, defaults shown. api_port = 5000 api_user = admin api_password = admin api_secure = false # Log level logger_level = WARNING
（5）启动 RBD target 服务
# systemctl start rbd-target-api.service
# systemctl enable rbd-target-api.service（6）显示配置信息
# gwcli info HTTP mode : http Rest API port : 5000 Local endpoint : http://localhost:5000/api Local Ceph Cluster : ceph 2ndary API IP's : 192.168.2.42,192.168.2.40,192.168.2.41
# gwcli ls o- / ...................................................................... [...] o- cluster ...................................................... [Clusters: 1] | o- ceph ......................................................... [HEALTH_OK] | o- pools ....................................................... [Pools: 1] | | o- iscsi-images ........ [(x3), Commit: 0.00Y/15718656K (0%), Used: 192K] | o- topology ............................................. [OSDs: 6,MONs: 3] o- disks .................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................ [DiscoveryAuth: None, Targets: 0]
4、Dashboard 添加 iscsi 网关
（1）Admin节点上，查看 dashboard iSCSI 网关
admin:~ # ceph dashboard iscsi-gateway-list
{"gateways": {"node002.example.com": {"service_url": "http://admin:[email protected]:5000"},
"node001.example.com": {"service_url": "http://admin:[email protected]:5000"}}}
（2）添加 iSCSI 网关
# ceph dashboard iscsi-gateway-add http://admin:[email protected]:5000 # ceph dashboard iscsi-gateway-list {"gateways": {"node002.example.com": {"service_url": "http://admin:[email protected]:5000"}, "node001.example.com": {"service_url": "http://admin:[email protected]:5000"}, "node003.example.com": {"service_url": "http://admin:[email protected]:5000"}}}
（3）登陆 Dashboard 查看 iSCSI 网关
5、Export RBD Images via iSCSI
（1）创建 iSCSI target name
# gwcli
gwcli > /> cd /iscsi-targets
gwcli > /iscsi-targets> create iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01（2）添加 iSCSI 网关
gwcli > /iscsi-targets> cd iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/gateways
/iscsi-target...tvol/gateways> create node001.example.com 172.200.50.40
/iscsi-target...tvol/gateways> create node002.example.com 172.200.50.41
/iscsi-target...tvol/gateways> create node003.example.com 172.200.50.42
/iscsi-target...ay01/gateways> ls
o- gateways ......................................................... [Up: 3/3, Portals: 3]
o- node001.example.com ............................................. [172.200.50.40 (UP)]
o- node002.example.com ............................................. [172.200.50.41 (UP)]
o- node003.example.com ............................................. [172.200.50.42 (UP)]注意：安装主机名来定义
/iscsi-target...tvol/gateways> create node002 172.200.50.41
The first gateway defined must be the local machine（3）添加 RBD 镜像
/iscsi-target...tvol/gateways> cd /disks
/disks> attach iscsi-images/iscsi-gateway-image001
/disks> attach iscsi-images/iscsi-gateway-image002（4）target 和 RBD 镜像建立映射关系
/disks> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/disks
/iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image001
/iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image002（5）设置不验证
gwcli > /> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/hosts
/iscsi-target...teway01/hosts> auth disable_acl
/iscsi-target...teway01/hosts> exit（6）查看配置信息
node001:~ # gwcli ls
o- / ............................................................................... [...]
o- cluster ............................................................... [Clusters: 1]
| o- ceph .................................................................. [HEALTH_OK]
| o- pools ................................................................ [Pools: 1]
| | o- iscsi-images .................. [(x3), Commit: 6G/15717248K (40%), Used: 1152K]
| o- topology ...................................................... [OSDs: 6,MONs: 3]
o- disks ................................................................ [6G, Disks: 2]
| o- iscsi-images .................................................. [iscsi-images (6G)]
| o- iscsi-gateway-image001 ............... [iscsi-images/iscsi-gateway-image001 (2G)]
| o- iscsi-gateway-image002 ............... [iscsi-images/iscsi-gateway-image002 (4G)]
o- iscsi-targets ..................................... [DiscoveryAuth: None, Targets: 1]
o- iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 .............. [Gateways: 3]
o- disks ................................................................ [Disks: 2]
| o- iscsi-images/iscsi-gateway-image001 .............. [Owner: node001.example.com]
| o- iscsi-images/iscsi-gateway-image002 .............. [Owner: node002.example.com]
o- gateways .................................................. [Up: 3/3, Portals: 3]
| o- node001.example.com ...................................... [172.200.50.40 (UP)]
| o- node002.example.com ...................................... [172.200.50.41 (UP)]
| o- node003.example.com ...................................... [172.200.50.42 (UP)]
o- host-groups ........................................................ [Groups : 0]
o- hosts .................................................... [Hosts: 0: Auth: None]
RBD 作为 VMware ESXI datastore 是否支持？
（1）目前来说，RBD是不支持datastore形式。
（2）iSCSI 是支持 datastore 这种方式，可以作为VMware Esxi 虚拟机提供存储功能，性价比非常不错的选择。
# gwcli ls
o- / ............................................................................. [...]
o- cluster ............................................................. [Clusters: 1]
| o- ceph ................................................................ [HEALTH_OK]
| o- pools .............................................................. [Pools: 1]
| | o- iscsi-images ............... [(x3), Commit: 0.00Y/15719936K (0%), Used: 576K]
| o- topology .................................................... [OSDs: 6,MONs: 3]
o- disks ........................................................... [0.00Y, Disks: 0]
o- iscsi-targets ................................... [DiscoveryAuth: None, Targets: 0]
# vim /etc/multipath.conf
defaults {
user_friendly_names yes
}
devices {
device {
vendor "(LIO-ORG|SUSE)"
product "RBD"
path_grouping_policy "multibus" # 所有有效路径在一个优先组群中
path_checker "tur" # 在设备中执行 TEST UNIT READY 命令。
features "0"
hardware_handler "1 alua" # 在切换路径组群或者处理I/O错误时用来执行硬件具体动作的模块。
prio "alua"
failback "immediate"
rr_weight "uniform" # 所有路径都有相同的加权
no_path_retry 12 # 路径故障后，重试12次，每次5秒
rr_min_io 100 # 指定切换到当前路径组的下一个路径前路由到该路径的I/O请求数。
}
}
> esxcli iscsi adapter list
> Adapter Driver State UID Description
> ------- --------- ------ ------------- ----------------------
> vmhba64 iscsi_vmk online iscsi.vmhba64 iSCSI Software Adapter