说明:iptables调试的最好方式应该是输出日志了。并且iptables有个raw的表,优先级别最好,且调试时针对icmp协议(ping)进行,那么日志输出就是整条链路串起来输出的,非常的清晰。
前提:
必须配置了日志输出,参考:http://www.cnblogs.com/EasonJim/p/8413715.html
背景:
在配置folsom版openstack的quantum时出现vm无法ping通外网的问题,经过抓包分析确定问题是iptables中的snat规则不生效,需要调试iptables定位下有问题的iptables规则。 iptables有5个链: PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING,4个表: filter, nat, mangle, raw,其工作流在下图表示的很清楚。
从图中可见raw拥有最高的优先级,raw表使用PREROUTING和OUTPUT两个链,因此raw可以覆盖所有包。在raw表中支持一个特殊的目标:TRACE,使内核记录下每条匹配该包的对应iptables规则信息。使用raw表内的TRACE target即可实现对iptables规则的跟踪调试。
配置:
假设需要对ipv4的ICMP包进行跟踪调试,抓取所有流经本机的ICMP包
iptables -t raw -A OUTPUT -p icmp -j TRACE
iptables -t raw -A PREROUTING -p icmp -j TRACE
加载对应内核模组
modprobe ipt_LOG
调试信息记录在/var/log/kern.log文件(也可以查看/var/log/messages)。
示例调试:
在vm内对外部作ping操作,vm的ip为10.0.0.4
[root@--- ~]# ping -c 192.168.0.19
PING 192.168.0.19 (192.168.0.19)() bytes of data. --- 192.168.0.19 ping statistics --- packets transmitted, received, % packet loss, time 0ms
在/var/log/kern.log中的对应调试信息如下
Apr :: openstack-network kernel: [1038991.870882] TRACE: raw:PREROUTING:policy:2IN=tap5c42978b-ac OUT= MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870902] TRACE: nat:PREROUTING:rule:1IN=tap5c42978b-ac OUT= MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870909] TRACE: nat:quantum-l3-agent-PREROUTING:return:4IN=tap5c42978b-ac OUT= MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870915] TRACE: nat:PREROUTING:policy:2IN=tap5c42978b-ac OUT= MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870938] TRACE: filter:FORWARD:rule:1IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870944] TRACE: filter:quantum-filter-top:rule:1IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870950] TRACE: filter:quantum-l3-agent-local:return:1IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870957] TRACE: filter:quantum-filter-top:return:2IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870962] TRACE: filter:FORWARD:rule:2IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870969] TRACE: filter:quantum-l3-agent-FORWARD:return:1IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870974] TRACE: filter:FORWARD:policy:3IN=tap5c42978b-ac OUT=br-ex MAC=fa::3e:a7:0c:f3:fa::3e:a4:::: SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870979] TRACE: nat:POSTROUTING:rule:1IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
Apr :: openstack-network kernel: [1038991.870985] TRACE: nat:quantum-l3-agent-POSTROUTING:rule:1IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID= DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=
可见数据包流在nat表的quantum-l3-agent-POSTROUTING的第一条规则处被截断了,查看iptables中的nat表的规则如下
*nat
:PREROUTING ACCEPT [:]
:INPUT ACCEPT [:]
:OUTPUT ACCEPT [:]
:POSTROUTING ACCEPT [:]
:quantum-l3-agent-OUTPUT - [:]
:quantum-l3-agent-POSTROUTING - [:]
:quantum-l3-agent-PREROUTING - [:]
:quantum-l3-agent-float-snat - [:]
:quantum-l3-agent-snat - [:]
:quantum-postrouting-bottom - [:]-A PREROUTING -j quantum-l3-agent-PREROUTING
-A OUTPUT -j quantum-l3-agent-OUTPUT
-A POSTROUTING -j quantum-l3-agent-POSTROUTING
-A POSTROUTING -j quantum-postrouting-bottom
-A quantum-l3-agent-OUTPUT -d 192.168.0.16/-j DNAT --to-destination 10.0.0.4
-A quantum-l3-agent-OUTPUT -d 192.168.0.17/-j DNAT --to-destination 10.0.0.3
-A quantum-l3-agent-POSTROUTING !-i qg-91757ded-c4 !-o qg-91757ded-c4 -m conntrack !--ctstate DNAT -j ACCEPT
-A quantum-l3-agent-POSTROUTING -s 10.0.0.0/-d 192.168.1.1/-j ACCEPT
-A quantum-l3-agent-PREROUTING -d 169.254.169.254/-p tcp -m tcp --dport80-j DNAT --to-destination 192.168.1.1:-A quantum-l3-agent-PREROUTING -d 192.168.0.16/-j DNAT --to-destination 10.0.0.4
-A quantum-l3-agent-PREROUTING -d 192.168.0.17/-j DNAT --to-destination 10.0.0.3
-A quantum-l3-agent-float-snat -s 10.0.0.4/-j SNAT --to-source 192.168.0.16
-A quantum-l3-agent-snat -j quantum-l3-agent-float-snat
-A quantum-l3-agent-snat -s 10.0.0.0/-j SNAT --to-source 192.168.0.15
-A quantum-postrouting-bottom -j quantum-l3-agent-snat
COMMIT
确定有问题的规则为
-A quantum-l3-agent-POSTROUTING !-i qg-91757ded-c4 !-o qg-91757ded-c4 -m conntrack !--ctstate DNAT -j ACCEPT
把这条规则删掉后重启iptables,vm能顺利连接外网,问题解决。
参考:
http://blog.51cto.com/flymanhi/1276331(以上内容转自此篇文章)