一、什么是iscsi
iSCSI技术是一种由IBM公司研究开发的，是一个供硬件设备使用的可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。
二、需求分析
最近在做虚拟化，需要通过一个主机挂载来自两个iscsi服务器的targets，如图：
三、实现思路
问题：刚开始觉得这个应该问题不大，挂载第一台iscsi服务器的时候没有碰到问题，但是在挂载第二台iscsi时，发现怎样都无法挂载，挂载报错。
最终找到了正确的解决方法，在centos7/RHEL7上挂载时，两台iscsi服务器端的target命名需要一致，下面进行搭建演示：
系统：centos 7.2
iscsi客户端：192.168.1.156
iscsi服务端：192.168.1.157
iscsi服务端：192.168.1.158
首先要确保3台服务器的selinux和firewalld关闭
[root@iscsi-client ~]# setenforce ; systemctl stop firewalld ; systemctl disable firewalld
iscsi服务端配置如下：
[root@iscsi-server1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
sr0 : 4G rom
使用sdb作为target。为sdb进行分区并使用lvm进行发布出去，使用lvm的好处是后期便于扩展，在rhel7和centos7中，因为采用的是xfs文件系统，
lvm只能扩展，不能缩小，这点要牢记。
[root@iscsi-server1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.). Changes will remain in memory only, until you decide to write them.
Be careful before using the write command. Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x966a38a7. Command (m for help): n
Partition type:
p primary ( primary, extended, free)
e extended
Select (default p):
Using default response p
Partition number (-, default ):
First sector (-, default ):
Using default value
Last sector, +sectors or +size{K,M,G} (-, default ):
Using default value
Partition of type Linux and of size GiB is set Command (m for help): w
The partition table has been altered! Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
└─sdb1 : 10G part
sr0 : 4G rom
以上就为sdb做了分区，接下来将sdb1创建为独立的lvm
# 创建卷组
[root@iscsi-server1 ~]# vgcreate vg_iscsi_1_156 /dev/sdb1
Physical volume "/dev/sdb1" successfully created
Volume group "vg_iscsi_1_156" successfully created # 创建逻辑分区 [root@iscsi-server1 ~]# lvcreate -l + -n lv_iscsi_1_156 vg_iscsi_1_156
Logical volume "lv_iscsi_1_156" created.
[root@iscsi-server1 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root centos -wi-ao---- .47g
swap centos -wi-ao---- .00g
lv_iscsi_1_156 vg_iscsi_1_156 -wi-a----- .00g
lvm已经创建完毕，接下来创建target
# 安装target工具和服务
[root@iscsi-server1 ~]# yum install scsi-target-utils -y # 开始创建target
[root@iscsi-server1 ~]# targetcli
targetcli shell version 2.1.fb41
Copyright - by Datera, Inc and others.
For help on commands, type 'help'. /> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: ]
| o- fileio ................................................................................................. [Storage Objects: ]
| o- pscsi .................................................................................................. [Storage Objects: ]
| o- ramdisk ................................................................................................ [Storage Objects: ]
o- iscsi ............................................................................................................ [Targets: ]
o- loopback ......................................................................................................... [Targets: ]
创建块设备，这里要注意，因为我们使用的lvm，因此不能将sdb添加到块设备，而是将逻辑卷添加进去
/> backstores/block create node1.disk /dev/sdb1
Cannot configure StorageObject because device /dev/sdb1 is already in use
/> backstores/block create node1.disk /dev/vg_iscsi_1_156/lv_iscsi_1_156
Created block storage object node1.disk using /dev/vg_iscsi_1_156/lv_iscsi_1_156.
在创建wwn的时候，又踩了一个坑。
wwn命名必须 iqn.2017-01.com.xxx:server 注意这个月份 2017-01 这里必须要01 如果写成1就会报错。
# 错误写法
/> iscsi/ create iqn.-.com.node1:server
WWN not valid as: iqn, naa, eui # 正确写法
/> iscsi/ create iqn.-.com.node1:server
Created target iqn.-.com.node1:server.
Created TPG .
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port .
# 创建acl，在创建acl这里一定要注意，如果准备在同一客户端挂载两个iscsi这里的acl命名一定要一致
/> iscsi/iqn.-.com.node1:server/tpg1/acls create iqn.-.com.node1:client
Created Node ACL for iqn.-.com.node1:client/> iscsi/iqn.-.com.node1:server/tpg1/luns create /backstores/block/node1.disk
Created LUN .
Created LUN -> mapping in node ACL iqn.-.com.node1:client # 删除自动生成的监听
/> iscsi/iqn.-.com.node1:server/tpg1/portals/ delete 0.0.0.0
Deleted network portal 0.0.0.0: # 创建绑定固定ip地址的监听
/> iscsi/iqn.-.com.node1:server/tpg1/portals/ create 192.168.1.157
Using default IP port
Created network portal 192.168.1.157:.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: ]
| | o- node1.disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (.0GiB) write-thru activated]
| o- fileio ................................................................................................. [Storage Objects: ]
| o- pscsi .................................................................................................. [Storage Objects: ]
| o- ramdisk ................................................................................................ [Storage Objects: ]
o- iscsi ............................................................................................................ [Targets: ]
| o- iqn.-.com.node1:server ...................................................................................... [TPGs: ]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: ]
| | o- iqn.-.com.node1:client ......................................................................... [Mapped LUNs: ]
| | o- mapped_lun0 ............................................................................ [lun0 block/node1.disk (rw)]
| o- luns .......................................................................................................... [LUNs: ]
| | o- lun0 .......................................................... [block/node1.disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
| o- portals .................................................................................................... [Portals: ]
| o- 192.168.1.157: ............................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: ]
/> saveconfig
Last configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
最后一定要执行 saveconfig 再退出
另一台iscsi服务端配置一样，targetcli如下：
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: ]
| | o- node1:disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (.0GiB) write-thru activated]
| o- fileio ................................................................................................. [Storage Objects: ]
| o- pscsi .................................................................................................. [Storage Objects: ]
| o- ramdisk ................................................................................................ [Storage Objects: ]
o- iscsi ............................................................................................................ [Targets: ]
| o- iqn.-.com.node1:server ...................................................................................... [TPGs: ]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: ]
| | o- iqn.-.com.node1:client ......................................................................... [Mapped LUNs: ]
| | o- mapped_lun0 ............................................................................ [lun0 block/node1:disk (rw)]
| o- luns .......................................................................................................... [LUNs: ]
| | o- lun0 .......................................................... [block/node1:disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
| o- portals .................................................................................................... [Portals: ]
| o- 192.168.1.158: ............................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: ]
其中acl命名一定要和server1保持一致
这里要注意，target服务建议要配置开机启动。在搭建的时候，有出现过没有开机启动配置丢失的情况。
[root@iscsi-server2 ~]# systemctl start target ; systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
iscsi客户端配置：
[root@iscsi-client yum.repos.d]# yum install iscsi* -y
1. 修改/etc/iscsi/initiatorname.iscsi文件
[root@iscsi-client ~]# vim /etc/iscsi/initiatorname.iscsi # 这里InitiatorName 后面就是iscsi服务端通过targetcli创建的iscsi acl命名，因为要挂载两个iscsi所以名称必须一致
InitiatorName=iqn.-.com.node1:client
2. 启动服务
# 这里客户端依然建议开机启动
[root@iscsi-client ~]# systemctl start iscsid ; systemctl enable iscsid
3. 挂载iscsi
# 挂载192.168.1.，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.157
192.168.1.157:, iqn.-.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.-.com.node1:server -p 192.168.1.157 -l
Logging in to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.157,] (multiple)
Login to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.157,] successful. # 挂载192.168.1.，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.158
192.168.1.158:, iqn.-.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.-.com.node1:server -p 192.168.1.158 -l
Logging in to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.158,] (multiple)
Login to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.158,] successful.
查看是否生效：
[root@iscsi-client ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
sdc : 10G disk
sr0 : 4G rom /mnt/iso
sdb 和 sdc都已经生成，说明挂载成功
在配置完成iscsi客户端的时候，建议进行重启服务器进行验证
# 重启成功，查看没问题。
[root@iscsi-client ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
sdc : 10G disk
sr0 : 4G rom
四、对iscsi进行扩容的操作
对192.168.1.157这台主机添加一块硬盘，并扩容客户端iscsi的容量
192.168.1.157 iscsi服务端操作如下：
# 已经为157添加了一块新的硬盘 sdc，首先将sdc加入到lvm中
[root@iscsi-server1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
└─sdb1 : 10G part
└─vg_iscsi_1_156-lv_iscsi_1_156 : 10G lvm
sdc : 10G disk
sr0 : 4G rom # 对sdc进行分区
[root@iscsi-server1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.). Changes will remain in memory only, until you decide to write them.
Be careful before using the write command. Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x68a23069. Command (m for help): n
Partition type:
p primary ( primary, extended, free)
e extended
Select (default p):
Using default response p
Partition number (-, default ):
First sector (-, default ):
Using default value
Last sector, +sectors or +size{K,M,G} (-, default ):
Using default value
Partition of type Linux and of size GiB is set Command (m for help):
Command (m for help): w
The partition table has been altered! Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]#
[root@iscsi-server1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
└─sdb1 : 10G part
└─vg_iscsi_1_156-lv_iscsi_1_156 : 10G lvm
sdc : 10G disk
└─sdc1 : 10G part
sr0 : 4G rom # 将sdc1添加到之前lvm卷组vg_iscsi_1_156里
[root@iscsi-server1 ~]# vgextend vg_iscsi_1_156 /dev/sdc1
Physical volume "/dev/sdc1" successfully created
Volume group "vg_iscsi_1_156" successfully extended # 扩展逻辑卷
[root@iscsi-server1 ~]# lvresize -l + -n /dev/vg_iscsi_1_156/lv_iscsi_1_156
Size of logical volume vg_iscsi_1_156/lv_iscsi_1_156 changed from 10.00 GiB ( extents) to 19.99 GiB ( extents).
Logical volume lv_iscsi_1_156 successfully resized. # 重读下逻辑卷大小
[root@iscsi-server1 ~]# resize2fs /dev/vg_iscsi_1_156/lv_iscsi_1_156
resize2fs 1.42. (-Dec-)
resize2fs: Device or resource busy while trying to open /dev/vg_iscsi_1_156/lv_iscsi_1_156
Couldn't find valid filesystem superblock.
到此，iscsi服务端扩容操作完毕。
192.168.1.156 iscsi客户端操作如下：
重新加载sdb和sdc容量，可以看到，sdc已经变成了20G
[root@iscsi-client ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 : 4K disk
sda : 20G disk
├─sda1 : 500M part /boot
└─sda2 : .5G part
├─centos-root : .5G lvm /
└─centos-swap : 2G lvm [SWAP]
sdb : 10G disk
sdc : 20G disk
sr0 : 4G rom
这里验证一个问题，因为挂载了两台iscsi，尝试多次重启，看看sdb和sdc会不会出现互换设备名的情况
经过3次重启的验证，并没有出现过互换设备名的情况，但是在挂载的时候，这里还是建议使用UUID
我在虚拟机上分区后无法查看到UUID，物理设备做iscsi服务端是可以查看的。
[root@db1 ~]# blkid /dev/vda1
/dev/vda1: UUID="e1b26164-3ec5-435a-bf0c-b9f4c0989941" TYPE="xfs"
至此，已经完成了一台服务器上挂载两个iscsi服务端的问题。
五、故障问题
在 192.168.1.156上查看到 sdb 所对应的是server 192.168.1.158
测试1：当sdb分区并挂载至某个目录
当158 down掉，客户端会有怎样的提示，如下验证：
dmesg 报错信息如下：
ls 挂载目录错误如下：
很明显的I/O错误。
解决方案：再次启动server 192.168.1.158, target开机自动启动，客户端需要重新挂载到目录才会恢复正常。
测试2：当sdb分区并挂载到某个目录做为web程序存放点，故障时，看web会返回什么状态码
通过apache测试，当iscsi服务器down掉，web服务器并不会马上故障。大概5分钟后报错
查看错误日志：
所以，当iscsi客户端出现IO错误时，先查看磁盘空间是否不足，再次检查下iscsi服务端网络通信是否正常。