问题:
官网方式yum安装完saltstack之后,master与minion都成功启动后,但是却发现怎么都收不到minion的key。(zmq版本不一致所致,困扰了我好几天,希望能帮到大家。)
分析:
master上的状态:
# 查看master 上的配置文件,只修改了interface:
[root@master doubles]# cat /etc/salt/master interface: 192.168.188.128 [root@master doubles]# systemctl status salt-master
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2019-01-08 09:55:59 HKT; 7min ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltstack.com/en/latest/contents.html
Main PID: 9331 (salt-master)
CGroup: /system.slice/salt-master.service
├─9331 /usr/bin/python /usr/bin/salt-master
├─9340 /usr/bin/python /usr/bin/salt-master
├─9345 /usr/bin/python /usr/bin/salt-master
├─9348 /usr/bin/python /usr/bin/salt-master
├─9349 /usr/bin/python /usr/bin/salt-master
├─9350 /usr/bin/python /usr/bin/salt-master
├─9351 /usr/bin/python /usr/bin/salt-master
├─9352 /usr/bin/python /usr/bin/salt-master
├─9353 /usr/bin/python /usr/bin/salt-master
├─9354 /usr/bin/python /usr/bin/salt-master
├─9361 /usr/bin/python /usr/bin/salt-master
├─9362 /usr/bin/python /usr/bin/salt-master
└─9364 /usr/bin/python /usr/bin/salt-master Jan 08 09:55:54 master.doubles.cn systemd[1]: Starting The Salt Master Server...
Jan 08 09:55:59 master.doubles.cn systemd[1]: Started The Salt Master Server. [root@master doubles]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:
# 始终看不到minion的key。
[root@master doubles]# salt-key -a 192.168.188.129
The key glob '192.168.188.129' does not match any unaccepted keys.
[root@master doubles]# salt '*' test.ping
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
看下minion上的salt状态:
# 查看minion上的配置文件,发现只修改了master与id如下:
[root@slave doubles]# vim /etc/salt/minion master: 192.168.188.128 id: 192.168.188.129 # salt-minion开启调试模式
[root@slave doubles]# salt-minion -l debug
...
[DEBUG ] SaltReqTimeoutError, retrying. (1/7)
[DEBUG ] SaltReqTimeoutError, retrying. (2/7)
[DEBUG ] SaltReqTimeoutError, retrying. (3/7)
[DEBUG ] SaltReqTimeoutError, retrying. (4/7)
[DEBUG ] SaltReqTimeoutError, retrying. (5/7)
[DEBUG ] SaltReqTimeoutError, retrying. (6/7)
[DEBUG ] SaltReqTimeoutError, retrying. (7/7)
[DEBUG ] Re-init ZMQ socket: Message timed out
[DEBUG ] Trying to connect to: tcp://192.168.188.128:4506
[ERROR ] Error while bringing up minion for multi-master. Is master at 192.168.188.128 responding?
[DEBUG ] Connecting to master. Attempt 1 of 1
[DEBUG ] Master URI: tcp://192.168.188.128:4506
[DEBUG ] Re-using AsyncAuth for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506')
[DEBUG ] Generated random reconnect delay between '1000ms' and '11000ms' (7456)
[DEBUG ] Setting zmq_reconnect_ivl to '7456ms'
[DEBUG ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG ] Re-using AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506', 'clear')
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] SaltReqTimeoutError, retrying. (1/7)
[DEBUG ] SaltReqTimeoutError, retrying. (2/7)
[DEBUG ] SaltReqTimeoutError, retrying. (3/7)
[DEBUG ] SaltReqTimeoutError, retrying. (4/7)
[DEBUG ] SaltReqTimeoutError, retrying. (5/7)
[DEBUG ] SaltReqTimeoutError, retrying. (6/7)
[DEBUG ] SaltReqTimeoutError, retrying. (7/7)
[DEBUG ] Re-init ZMQ socket: Message timed out
[DEBUG ] Trying to connect to: tcp://192.168.188.128:4506
[ERROR ] Error while bringing up minion for multi-master. Is master at 192.168.188.128 responding?
...
在这里,本机上的master也是无法收到本机上的minion的key。找了好久,有的说是配置里的master与id没有配对,有的说是增大线程数,有的说是防火墙,有的说是删掉pki下的id,试了都没有用。最后,认真思考一下,其实是minion无法与master 通信,偶然对比了一下minion与master的版本:
master上:
[root@master doubles]# salt-master --versions-report
Salt Version:
Salt: 2018.3.3 Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.3.1
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 3.2.2 System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
minion上:
[root@slave doubles]# salt-minion --versions-report
Salt Version:
Salt: 2018.3.3 Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4 System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
发现master与minion上的ZMQ版本不一样,这应该就是问题所在了。
解决:
尝试更新master上的zmq版本。
master上:
[root@master doubles]# yum install *zmq* –y
# 再对比版本
[root@master doubles]# salt-master --versions-report
Salt Version:
Salt: 2018.3.3 Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4 System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
此时发现zmq版本已经和minion一致了。重启master再试一下:
master:
[root@master doubles]# systemctl restart salt-master
[root@master doubles]# systemctl status salt-master
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2019-01-08 11:01:43 HKT; 17s ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltstack.com/en/latest/contents.html
Main PID: 15436 (salt-master)
CGroup: /system.slice/salt-master.service
├─15436 /usr/bin/python /usr/bin/salt-master
├─15445 /usr/bin/python /usr/bin/salt-master
├─15450 /usr/bin/python /usr/bin/salt-master
├─15453 /usr/bin/python /usr/bin/salt-master
├─15454 /usr/bin/python /usr/bin/salt-master
├─15455 /usr/bin/python /usr/bin/salt-master
├─15456 /usr/bin/python /usr/bin/salt-master
├─15457 /usr/bin/python /usr/bin/salt-master
├─15458 /usr/bin/python /usr/bin/salt-master
├─15459 /usr/bin/python /usr/bin/salt-master
├─15460 /usr/bin/python /usr/bin/salt-master
├─15461 /usr/bin/python /usr/bin/salt-master
└─15462 /usr/bin/python /usr/bin/salt-master Jan 08 11:01:42 master.doubles.cn systemd[1]: Stopped The Salt Master Server.
Jan 08 11:01:42 master.doubles.cn systemd[1]: Starting The Salt Master Server...
Jan 08 11:01:43 master.doubles.cn systemd[1]: Started The Salt Master Server.
minion上,重启:
[root@slave doubles]# salt-minion -l debug
...
[DEBUG ] Trying to connect to: tcp://192.168.188.128:4506
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.188.128:4506
[DEBUG ] Trying to connect to: tcp://192.168.188.128:4506
[INFO ] Minion is ready to receive requests!
[DEBUG ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_add_complete'
[DEBUG ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
...
# 跟之前不一样了
看下master上能否收到key了。
master上:
[root@master doubles]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
192.168.188.129
Rejected Keys:
[root@master doubles]# salt-key -a 192.168.188.129
The following keys are going to be accepted:
Unaccepted Keys:
192.168.188.129
Proceed? [n/Y] y
Key for minion 192.168.188.129 accepted.
[root@master doubles]# salt-key -L
Accepted Keys:
192.168.188.129
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[root@master doubles]# salt "*" test.ping
192.168.188.129:
True
发现已经OK了。