背景
系统环境：k8s+docker+cri-dockerd
因为我不想把镜像通过Docker hub公开，以及将来在不联网的生产环境部署，自己运行一个docker存储库，在k8s部署工作负载时从中拉取镜像。
相关命令形如：
docker run -d -p 5000:5000 --restart=always --name registry registry:2
docker push localhost:5000/user/user-image
问题
没有修改环境配置，进行了一些k8s和docker相关操作后，再推送镜像时突然发生错误。
Get "http://localhost:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused
解决
一开始我按一般排查故障的方法，检查 registry 容器日志，docker 服务日志，重启docker服务，重新部署 registry 容器等等，均未解决问题，百思不得其解。
后续我进行k8s操作，部署时发现问题，大概是在k8s部署的容器可以分配一个Node端口，同一个Node的同一个端口只能分配一次，导致只有一个Node时不能部署第二份。表现如下：
root@vmi1640551:~# kubectl -n test-cinema-2 get po
NAME READY STATUS RESTARTS AGE
a-bookings-1-756694bb6b-sdqbg 0/1 Pending 0 8m4s
a-movies-1-66785d95ff-6jp27 0/1 Pending 0 8m4s
a-showtimes-1-fcb9d8bc6-9txh5 0/1 Pending 0 8m4s
a-users-1-59bb6845cf-zb7xw 0/1 Pending 0 8m4s
proxy 1/1 Running 0 8m14s
root@vmi1640551:~# kubectl -n test-cinema-2 describe po a-bookings-1-756694bb6b-sdqbg
Name: a-bookings-1-756694bb6b-sdqbg
Namespace: test-cinema-2
Priority: 0
Service Account: default
Node: <none>
Labels: app=a-bookings-1
pod-template-hash=756694bb6b
Annotations: kompose.cmd: kompose --file docker-compose.yml convert
kompose.version: 1.32.0 (HEAD)
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/a-bookings-1-756694bb6b
Containers:
bookings:
Image: localhost:5050/cinema-2/bookings
Port: 5003/TCP
Host Port: 5003/TCP
Limits:
cpu: 100m
Requests:
cpu: 100m
Readiness: http-get http://:5003/health-check delay=0s timeout=1s period=3s #success=1 #failure=2
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pmx4b (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-pmx4b:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m9s (x2 over 8m20s) default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
root@vmi1640551:~#
我猜测可能是有k8s中容器占用了5000端口。修改 registry 绑定的本地端口后（比如改为5050），推送成功了。
检查确实如此，有一个工作负载的配置如下：
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose --file docker-compose.yml convert
kompose.version: 1.32.0 (HEAD)
labels:
io.kompose.service: users
name: users
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: users
template:
metadata:
annotations:
kompose.cmd: kompose --file docker-compose.yml convert
kompose.version: 1.32.0 (HEAD)
labels:
io.kompose.network/cinema-2-default: "true"
io.kompose.service: users
spec:
containers:
- image: localhost:5000/cinema-2/users
name: users
ports:
- containerPort: 5000
hostPort: 5000 # ! 注意这里 !
protocol: TCP
readinessProbe:
httpGet:
path: /health-check
port: 5000
periodSeconds: 3 # 默认 10
failureThreshold: 2 # 默认 3
successThreshold: 1
timeoutSeconds: 1
restartPolicy: Always
删除实际未使用的 hostPort 后恢复正常。
后续疑问
有些疑问还没来得及解决：
- k8s、docker的网络原理是怎样的？
- 特别的，由k8s pull镜像、从主机
docker push镜像和curl localhost:5000的请求会被如何路由？是否有区别？ - registry 和 k8s中部署的工作负载应该只有一个能监听唯一Node的5000端口，为什么看起来似乎都部署成功了，看不到错误？