问题描述
我知道这个问题被问过很多次，但是关于docker的一切，这次是crio.
I know this question is asked many times, but all about docker, this time is crio.
CentOS Linux release 7.6
CRI-O Version: 1.16.1
Kubernetes: v1.16.3
KubeAdm: v1.16.3
CoreDNS Pod处于Error/CrashLoopBackOff状态，并且audit.log显示selinux阻止CoreDNS从/var/lib/kubelet/container_id/volumes/
CoreDNS pods are in Error/CrashLoopBackOff state, and audit.log shows selinux prevents CoreDNS to read from /var/lib/kubelet/container_id/volumes/
type=AVC msg=audit(1576203392.727:1431): avc: denied { read } for pid=15866 comm="coredns" name="Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1576203392.727:1431): avc: denied { open } for pid=15866 comm="coredns" path="/etc/coredns/..2019_12_13_02_13_30.965446608/Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1576203393.049:1432): avc: denied { open } for pid=15866 comm="coredns" path="/var/run/secrets/kubernetes.io/serviceaccount/..2019_12_13_02_13_30.605147375/token" dev="tmpfs" ino=124481 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1
如果我使用的Docker版本高于1.7，则可以正常工作，我认为这可能与使用z/Z选项安装卷的补丁有关.
if I use docker newer than 1.7, it works fine, I assume this may related with the patch of mounting volume with z/Z option.
我可以在下面添加策略，但这会损害安全性.
I can add policy like underneath, but it will compromise security.
module coredns 0.1;
require {
type tmpfs_t;
type container_t;
type var_lib_t;
class file { open read };
}
allow container_t tmpfs_t:file open;
allow container_t var_lib_t:file { open read };
还有更好的解决方案吗?就像docker，只需付出一点努力，就不会损害安全性.
any better solution exists? just like docker, with a little efforts and don't compromise security.
推荐答案
我已经调查过，看来问题出在 kubelet版本.让我详细说明一下:
I've looked into it and it seems that the problem lays in kubelet version. Let me elaborate on that:
在1.16中未重新标记的SELinux卷-此链接提供了有关问题.
SELinux Volumes not relabeled in 1.16 - this link is providing more details about the issue.
我试图在不同版本的Kubernetes上重现此coredns问题.
I tried to reproduce this coredns issue on different versions of Kubernetes.
问题在1.16版及更高版本上显示.在1.15.6版
Issue shows on version 1.16 and newer. It seems to work properly with SELinux enabled on the version 1.15.6
要使其正常工作，您将需要有效的CentOS和CRI-O环境.
For this to work you will need working CentOS and CRI-O environment.
CRI-O版本:
Version: 0.1.0
RuntimeName: cri-o
RuntimeVersion: 1.16.2
RuntimeApiVersion: v1alpha1
要部署此基础结构，我大部分时间关注此站点: KubeVirt
To deploy this insfrastructure I followed this site for the most part: KubeVirt
复制步骤:
- 禁用SELinux并重新启动计算机:
-
$ setenforce 0 -
$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config -
$ reboot
- Disable SELinux and restart machine:
$ setenforce 0$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config$ reboot
- kubelet-1.15.7-0.x86_64
- kubeadm-1.15.7-0.x86_64
- kubectl-1.15.7-0.x86_64
使用以下命令检查coredns Pod是否正常运行:
$ kubectl get pods -ACheck if coredns pods are running correctly with command:
$ kubectl get pods -A它应该提供类似的输出:
It should give similar output to that:
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5c98db65d4-2c7lt 1/1 Running 2 7m59s kube-system coredns-5c98db65d4-5dp9s 1/1 Running 2 7m59s kube-system etcd-centos-kube-master 1/1 Running 2 7m20s kube-system kube-apiserver-centos-kube-master 1/1 Running 2 7m4s kube-system kube-controller-manager-centos-kube-master 1/1 Running 2 6m55s kube-system kube-flannel-ds-amd64-mzh27 1/1 Running 2 7m14s kube-system kube-proxy-bqll8 1/1 Running 2 7m58s kube-system kube-scheduler-centos-kube-master 1/1 Running 2 6m58s禁用 SELinux 的kubernetes集群中的
Coredns pods正常运行.
Coredns pods in kubernetes cluster with SELinux disabled are working properly.
启用SELinux :
从root帐户调用命令以启用SELinux并重新启动计算机:
From root account invoke commands to enable SELinux and restart the machine:
-
$ setenforce 1 -
$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config -
$ reboot
$ setenforce 1$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config$ reboot
检查coredns Pod是否正常运行.在运行时，它们应该不会出现crashloopbackoff错误:
kubectl get pods -ACheck if coredns pods are running correctly. They should not get crashloopbackoff error when running:
kubectl get pods -A复制步骤:
- 如果来自其他版本，请运行
$ kubeadm reset - 使用
$ yum remove OLD_PACKAGES删除旧的Kubernetes软件包 - 禁用SELinux并重新启动计算机:
-
$ setenforce 0 -
$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config -
$ reboot
- Run
$ kubeadm resetif coming from another another version - Remove old Kubernetes packages with
$ yum remove OLD_PACKAGES - Disable SELinux and restart machine:
$ setenforce 0$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config$ reboot
- kubelet-1.16.4-0.x86_64
- kubeadm-1.16.4-0.x86_64
- kubectl-1.16.4-0.x86_64
使用以下命令检查coredns Pod是否正常运行:
$ kubectl get pods -ACheck if coredns pods are running correctly with command:
$ kubectl get pods -A它应该提供类似的输出:
It should give similar output to that:
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5644d7b6d9-fgbkl 1/1 Running 1 13m kube-system coredns-5644d7b6d9-x6h4l 1/1 Running 1 13m kube-system etcd-centos-kube-master 1/1 Running 1 12m kube-system kube-apiserver-centos-kube-master 1/1 Running 1 12m kube-system kube-controller-manager-centos-kube-master 1/1 Running 1 12m kube-system kube-proxy-v52ls 1/1 Running 1 13m kube-system kube-scheduler-centos-kube-master 1/1 Running 1 12m启用SELinux :
从root帐户调用命令以启用SELinux并重新启动计算机:
From root account invoke commands to enable SELinux and restart the machine:
-
$ setenforce 1 -
$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config -
$ reboot
$ setenforce 1$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config$ reboot
重新启动coredns pods后，应进入crashloopbackoff状态，如下所示:
After reboot coredns pods should enter crashloopbackoff state as shown below:
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5644d7b6d9-fgbkl 0/1 CrashLoopBackOff 25 113m kube-system coredns-5644d7b6d9-x6h4l 0/1 CrashLoopBackOff 25 113m kube-system etcd-centos-kube-master 1/1 Running 1 112m kube-system kube-apiserver-centos-kube-master 1/1 Running 1 112m kube-system kube-controller-manager-centos-kube-master 1/1 Running 1 112m kube-system kube-proxy-v52ls 1/1 Running 1 113m kube-system kube-scheduler-centos-kube-master 1/1 Running 1 112m豆荚
coredns-5644d7b6d9-fgbkl中的日志显示:plugin/kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied这篇关于当使用crio并在selinux上打开时，coredns无法启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！
-
-