之前写了《Linux bridge开启hairpin模拟测试macvlan vepa模式》，记录了Linux bridge开启hairpin，模拟测试macvlan vepa模式下，同一父接口下两个子接口的网络通讯情况，文中缺少了子接口与外部网络通讯的部分，本文补上，详见下：
参考
1.Linux bridge开启hairpin模拟测试macvlan vepa模式
2.Linux虚拟网络设备—之使用Veth pair连接linux网桥bridge
3.Linux 网络设备 - Bridge 详解
4.brctl快速入门与基础
环境
与《Linux bridge开启hairpin模拟测试macvlan vepa模式》环境相同。
测试
1. 测试流程
《Linux bridge开启hairpin模拟测试macvlan vepa模式》只验证了macvlan vepa模式下，同一父网卡下的两个子接口（子网卡）通过开启hairpin的外部交换转发，实现网络通讯。
本文在上述测试基础上，补充Linux bridge开启hairpin模拟测试macvlan vepa时，子接口（子网卡）与外部网络（宿主机以外网络）的通讯情况，测试步骤如下：
- 详见《Linux bridge开启hairpin模拟测试macvlan vepa模式》中 1.测试流程
- 补充测试流程：
- 将宿主机网卡enp0s5加入Linux bridge br0
- 删掉网卡enp0s5绑定的IP（10.211.55.18），并将该IP绑定到br0 上
- 修改宿主机默认路由，默认路由设备由enp0s5改为br0
- 增加ns101下默认路由，默认路由设备使用veth0_1.101
- 增加ns102下默认路由，默认路由设备使用veth0_1.102
- 测试macvlan vepa网络子接口与宿主机以外网络的通讯情况
详见下图：
- 网络连接图
- 物理网卡enp0s5连接外部网络
- br0连接物理网卡enp0s5和虚拟网卡veth0
- 虚拟网卡veth0和veth0_1是一对veth pair
- veth0_1.101和veth0_1.102是veth0的子网卡
- 网络通讯图
- veth0_1.101和veth0_1.102，通过br0（veth0接口开启hairpin）转发通讯。
- veth0_1.101和veth0_1.102，通过br0的接口（enp0s5）与宿主机以外网络通讯，例如：网关10.211.55.1
2. 将宿主机网卡enp0s5加入Linux bridge br0
请务必确保有其他方式可以登录宿主机，网卡enp0s5加入br0后，当前网络连接会断开。
- enp0s5 加入br0
// enp0s5 加入 br0
[root@centos7-18 ~]# brctl addif br0 enp0s5
// 查看bridge
[root@centos7-18 ~]# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.001c426087b2 no enp0s5
veth0
virbr0 8000.5254009f1377 yes virbr0-nic
[root@centos7-18 ~]#
3. 删掉网卡enp0s5 IP（10.211.55.18），将该IP绑定到br0 上
- 删掉宿主机网卡enp0s5绑定的IP 10.211.55.18
// 删掉宿主机网卡enp0s5的IP
[root@centos7-18 ~]# ip addr del 10.211.55.18/24 dev enp0s5
- 增加br0 绑定IP 10.211.55.18
// br0绑定IP 10.211.55.18
[root@centos7-18 ~]# ip addr add 10.211.55.18/24 dev br0
- 查看当前IP
- enp0s5已经没有IP了
- br0的IP为10.211.55.18
// 查看当前IP
[root@centos7-18 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic
valid_lft 2581396sec preferred_lft 594196sec
inet6 fe80::21c:42ff:fe60:87b2/64 scope link
valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
inet 10.211.55.18/24 scope global br0
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic
valid_lft 2591815sec preferred_lft 604615sec
inet6 fe80::8413:97ff:fe70:a2e2/64 scope link
valid_lft forever preferred_lft forever
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
inet6 fdb2:2c26:f4e4:0:7887:efff:fec6:779b/64 scope global mngtmpaddr dynamic
valid_lft 2591815sec preferred_lft 604615sec
inet6 fe80::7887:efff:fec6:779b/64 scope link
valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
inet6 fe80::8408:8eff:fe91:9fe/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
4. 修改宿主机默认路由，设备由enp0s5改为br0
- 删掉宿主机enp0s5的默认路由
// 删掉宿主机网卡enp0s5的IP
[root@centos7-18 ~]# ip route del default via 10.211.55.1 dev enp0s5
- 增加宿主机br0的默认路由
// 增加宿主机**br0**的默认路由
[root@centos7-18 ~]# ip route add default via 10.211.55.1 dev br0
- 查看当前路由
// 查看当前路由
[root@centos7-18 ~]# ip route
default via 10.211.55.1 dev br0
10.211.55.0/24 dev br0 proto kernel scope link src 10.211.55.18
10.211.55.0/24 dev enp0s6 proto kernel scope link src 10.211.55.21 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
5. 增加namespace ns101和ns102的默认路由
- 增加ns101下默认路由，设备使用veth0_1.101
// 增加ns101下默认路由，默认路由设备使用veth0_1.101
[root@centos7-18 ~]# ip netns exec ns101 ip route add default via 10.211.55.1 dev veth0_1.101
// 查看ns101路由表
[root@centos7-18 ~]# ip netns exec ns101 ip route
default via 10.211.55.1 dev veth0_1.101
10.211.55.0/24 dev veth0_1.101 proto kernel scope link src 10.211.55.101
[root@centos7-18 ~]#
- 增加ns102下默认路由，设备使用veth0_1.102
// 增加ns102下默认路由，默认路由设备使用veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip route add default via 10.211.55.1 dev veth0_1.102
// 查看ns102路由表
[root@centos7-18 ~]# ip netns exec ns102 ip route
default via 10.211.55.1 dev veth0_1.102
10.211.55.0/24 dev veth0_1.102 proto kernel scope link src 10.211.55.102
[root@centos7-18 ~]#
6. 测试macvlan vepa网络子接口与宿主机以外网络通讯
- 检查测试环境-enp0s5、br0、veth0和veth0_1已启用
// 显示已启用的设备：enp0s5、br0、veth0、veth0_1
[root@centos7-18 ~]# ip address show up
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic
valid_lft 2569303sec preferred_lft 582103sec
inet6 fe80::21c:42ff:fe60:87b2/64 scope link
valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
inet 10.211.55.18/24 scope global br0
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic
valid_lft 2591897sec preferred_lft 604697sec
inet6 fe80::8413:97ff:fe70:a2e2/64 scope link
valid_lft forever preferred_lft forever
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
inet6 fdb2:2c26:f4e4:0:7887:efff:fec6:779b/64 scope global mngtmpaddr dynamic
valid_lft 2591897sec preferred_lft 604697sec
inet6 fe80::7887:efff:fec6:779b/64 scope link
valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
inet6 fe80::8408:8eff:fe91:9fe/64 scope link
valid_lft forever preferred_lft forever
- 检查测试环境-br0的veth0接口已开启hairpin
// 开启hairpin
[root@centos7-18 ~]# brctl hairpin br0 veth0 on
// 查看br0的veth0开启hairpin结果
[root@centos7-18 ~]# bridge -d link | grep -A5 veth0
8: veth0 state UP @veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2
hairpin on guard off root_block off fastleave off learning on flood on mcast_flood on
[root@centos7-18 ~]#
- 检查测试环境-veth0_1.101和veth0_1.102已启用
// 显示ns101已启用的设备：veth0_1.101
[root@centos7-18 ~]# ip netns exec ns101 ip a show up
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global veth0_1.101
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic
valid_lft 2591507sec preferred_lft 604307sec
inet6 fe80::b03e:6eff:feae:7457/64 scope link
valid_lft forever preferred_lft forever
// 显示ns102已启用的设备：veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip a show up
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
10: veth0_1.102@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.102/24 scope global veth0_1.102
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:20f8:d5ff:fe8b:c163/64 scope global mngtmpaddr dynamic
valid_lft 2591491sec preferred_lft 604291sec
inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link
valid_lft forever preferred_lft forever
- 测试namespace ns101 访问宿主机以外网络
- Ping 网关IP 10.211.55.1，通
- Ping 网络内其它主机IP 10.211.55.10，通
// 查看ns101的IP
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A5 veth
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global veth0_1.101
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic
valid_lft 2591841sec preferred_lft 604641sec
inet6 fe80::b03e:6eff:feae:7457/64 scope link
valid_lft forever preferred_lft forever
// Ping 网关IP 10.211.55.1，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.164 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.323 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.164/0.243/0.323/0.081 ms
[root@centos7-18 ~]#
// Ping 网络内其它主机IP 10.211.55.10，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.10
PING 10.211.55.10 (10.211.55.10) 56(84) bytes of data.
64 bytes from 10.211.55.10: icmp_seq=1 ttl=64 time=0.288 ms
64 bytes from 10.211.55.10: icmp_seq=2 ttl=64 time=0.526 ms
--- 10.211.55.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.288/0.407/0.526/0.119 ms
[root@centos7-18 ~]#
- 测试namespace ns101和ns102 可以互相访问
// ns101 ping ns102的IP 10.211.55.102，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.085 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.083 ms
--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.083/0.084/0.085/0.001 ms
// ns102 ping ns101的IP 10.211.55.101，通
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
64 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.057 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.087 ms
--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.057/0.072/0.087/0.015 ms
[root@centos7-18 ~]#
结果
bridge工作在二层，打通了namespace与宿主机以外的网络，查看br0的mac table可以看到mac结果
1. 查看br0的mac table，包含网关（外部）的mac
- 先通过宿主机Ping网关IP 10.211.55.1，获得网关对应的mac：00:1c:42:00:00:18
// 宿主机Ping网关IP 10.211.55.1后，查看网关mac
[root@centos7-18 ~]# ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.170 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.264 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.170/0.217/0.264/0.047 ms
// 网关mac为 00:1c:42:00:00:18
[root@centos7-18 ~]# arp
Address HWtype HWaddress Flags Mask Iface
10.211.55.2 ether 00:1c:42:00:00:08 C br0
10.211.55.1 ether 00:1c:42:00:00:18 C br0
10.211.55.10 (incomplete) br0
10.211.55.1 ether 00:1c:42:00:00:18 C enp0s6
10.211.55.102 ether 22:f8:d5:8b:c1:63 C br0
10.211.55.101 ether b2:3e:6e:ae:74:57 C br0
- 查看br0的mac table，已包含网关的mac(00:1c:42:00:00:18)
// 查看br0 mac table
[root@centos7-18 ~]# brctl showmacs br0
port no mac addr is local? ageing timer
2 00:1c:42:00:00:08 no 0.00
2 00:1c:42:00:00:18 no 10.43
2 00:1c:42:60:87:b2 yes 0.00
2 00:1c:42:60:87:b2 yes 0.00
2 00:1c:42:d1:70:62 no 193.31
1 86:08:8e:91:09:fe yes 0.00
1 86:08:8e:91:09:fe yes 0.00
2. 查看br0的mac table，包含macvlan vepa子网卡（内部）的mac
- 查看ns101 网卡veth0_1.101的mac为b2:3e:6e:ae:74:57
// ns101网卡veth0_1.101的mac为b2:3e:6e:ae:74:57
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A2 veth
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global veth0_1.101
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic
- 在ns101中 ping 网关IP 10.211.55.1
// ns101 ping 网关IP 10.211.55.1
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.169 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.225 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.169/0.197/0.225/0.028 ms
- 查看br0的mac table，新增加了veth0_1.101的mac（b2:3e:6e:ae:74:57）
// 查看br0 mac table，增加了veth0_1.101的mac b2:3e:6e:ae:74:57
[root@centos7-18 ~]# brctl showmacs br0
port no mac addr is local? ageing timer
2 00:1c:42:00:00:08 no 0.00
2 00:1c:42:00:00:18 no 0.93
2 00:1c:42:60:87:b2 yes 0.00
2 00:1c:42:60:87:b2 yes 0.00
2 00:1c:42:d1:70:62 no 177.70
1 86:08:8e:91:09:fe yes 0.00
1 86:08:8e:91:09:fe yes 0.00
1 b2:3e:6e:ae:74:57 no 0.94
总结
通过Linux bridge 开启接口hairpin的方式，模拟macvlan vepa在外部交换支持802.1q的情况下，同一父网卡下的多个子网卡之间是可以通讯的。(详见 Linux bridge开启hairpin模拟测试macvlan vepa模式 )
通过将宿主机物理网卡加入到Linux bridge中，使用bridge桥接内外网络，可以实现内部macvlan vepa子网卡访问外部网络。