我的Mac上的Minikube中有一个本地Kubernetes集群。我将Minio独立服务器部署为具有指定资源限制的单个容器。当我上载大于容器内存限制的文件时,容器因OOMKilled
原因终止。在Ubuntu上,上传的安装文件相同,没有错误。
Minikube在VirtualBox中运行,并配置为使用4GB内存。我还使用Heapster和Metric Server来检查一段时间内的内存使用情况。
$ minikube config set memory 4096
$ minikube addons enable heapster
$ minikube addons enable metrics-server
$ minikube start
我使用了Kubernetes confiuration for Minio standalone setup provided in Minio documentation的稍微修改的版本。我创建了PV和PVC,用于Minio服务器的存储,部署和服务。容器配置:
资源限制设置为具有保证的QoS。容器限制为256 MB内存和0.5 CPU。
resources:
requests:
cpu: '500m'
memory: '256Mi'
limits:
cpu: '500m'
memory: '256Mi'
完整的
videos-storage.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: videos-storage-pv
labels:
software: minio
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /data/storage-videos/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: videos-storage-pv-claim
spec:
storageClassName: ''
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
selector:
matchLabels:
software: minio
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: videos-storage-deployment
spec:
selector:
matchLabels:
app: videos-storage
template:
metadata:
labels:
app: videos-storage
spec:
initContainers:
- name: init-minio-buckets
image: minio/minio
volumeMounts:
- name: data
mountPath: /data/storage-videos
command: ['mkdir', '-p', '/data/storage-videos/videos']
containers:
- name: minio
image: minio/minio
volumeMounts:
- name: data
mountPath: /data/storage-videos
args:
- server
- /data/storage-videos
env:
- name: MINIO_ACCESS_KEY
value: 'minio'
- name: MINIO_SECRET_KEY
value: 'minio123'
ports:
- containerPort: 9000
resources:
requests:
cpu: '500m'
memory: '256Mi'
limits:
cpu: '500m'
memory: '256Mi'
readinessProbe:
httpGet:
path: /minio/health/ready
port: 9000
initialDelaySeconds: 5
periodSeconds: 20
livenessProbe:
httpGet:
path: /minio/health/live
port: 9000
initialDelaySeconds: 5
periodSeconds: 20
volumes:
- name: data
persistentVolumeClaim:
claimName: videos-storage-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: videos-storage-service
spec:
type: LoadBalancer
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
app: videos-storage
我将配置部署到集群:
$ kubectl apply -f videos-storage.yaml
并使用Minikube访问Minio仪表板,以下命令在浏览器中将其打开:
$ minikube service videos-storage-service
然后,我选择一个
videos
存储桶并上传一个1 GB的文件。上载约250 MB的内存后,我在Minio仪表板中收到错误消息。玩限制并分析Heapster图,我可以看到文件大小和内存使用之间的相关性。容器使用与文件大小完全相同的内存量,这对我来说很奇怪。我的印象是文件上传期间不会将文件直接存储在内存中。描述 pods
Name: videos-storage-deployment-6cd94b697-p4v8n
Namespace: default
Priority: 0
Node: minikube/10.0.2.15
Start Time: Mon, 22 Jul 2019 11:05:53 +0300
Labels: app=videos-storage
pod-template-hash=6cd94b697
Annotations: <none>
Status: Running
IP: 172.17.0.8
Controlled By: ReplicaSet/videos-storage-deployment-6cd94b697
Init Containers:
init-minio-buckets:
Container ID: docker://09d75629a39ad1dc0dbdd6fc0a6a6b7970285d0a349bccee2b0ed2851738a9c1
Image: minio/minio
Image ID: docker-pullable://minio/minio@sha256:456074355bc2148c0a95d9c18e1840bb86f57fa6eac83cc37fce0212a7dae080
Port: <none>
Host Port: <none>
Command:
mkdir
-p
/data/storage-videos/videos
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 22 Jul 2019 11:08:45 +0300
Finished: Mon, 22 Jul 2019 11:08:45 +0300
Ready: True
Restart Count: 1
Environment: <none>
Mounts:
/data/storage-videos from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zgs9q (ro)
Containers:
minio:
Container ID: docker://1706139f0cc7852119d245e3cfe31d967eb9e9537096a803e020ffcd3becdb14
Image: minio/minio
Image ID: docker-pullable://minio/minio@sha256:456074355bc2148c0a95d9c18e1840bb86f57fa6eac83cc37fce0212a7dae080
Port: 9000/TCP
Host Port: 0/TCP
Args:
server
/data/storage-videos
State: Running
Started: Mon, 22 Jul 2019 11:08:48 +0300
Last State: Terminated
Reason: OOMKilled
Exit Code: 0
Started: Mon, 22 Jul 2019 11:06:06 +0300
Finished: Mon, 22 Jul 2019 11:08:42 +0300
Ready: True
Restart Count: 1
Limits:
cpu: 500m
memory: 256Mi
Requests:
cpu: 500m
memory: 256Mi
Liveness: http-get http://:9000/minio/health/live delay=5s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:9000/minio/health/ready delay=5s timeout=1s period=20s #success=1 #failure=3
Environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
Mounts:
/data/storage-videos from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zgs9q (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: videos-storage-pv-claim
ReadOnly: false
default-token-zgs9q:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-zgs9q
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Minikube登录
dmesg
:[ +3.529889] minio invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[ +0.000006] CPU: 1 PID: 8026 Comm: minio Tainted: G O 4.15.0 #1
[ +0.000001] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ +0.000000] Call Trace:
[ +0.000055] dump_stack+0x5c/0x82
[ +0.000010] dump_header+0x66/0x281
[ +0.000006] oom_kill_process+0x223/0x430
[ +0.000002] out_of_memory+0x28d/0x490
[ +0.000003] mem_cgroup_out_of_memory+0x36/0x50
[ +0.000004] mem_cgroup_oom_synchronize+0x2d3/0x310
[ +0.000002] ? get_mem_cgroup_from_mm+0x90/0x90
[ +0.000002] pagefault_out_of_memory+0x1f/0x4f
[ +0.000002] __do_page_fault+0x4a3/0x4b0
[ +0.000003] ? page_fault+0x36/0x60
[ +0.000002] page_fault+0x4c/0x60
[ +0.000002] RIP: 0033:0x427649
[ +0.000001] RSP: 002b:000000c0002eaae8 EFLAGS: 00010246
[ +0.000154] Memory cgroup out of memory: Kill process 7734 (pause) score 0 or sacrifice child
[ +0.000013] Killed process 7734 (pause) total-vm:1024kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB
最初,当我没有任何资源限制时便出现了问题。当我尝试上传一个大文件时,带有Minio的容器将使用节点中所有可用的内存,并且由于没有剩余的内存,Kubernetes服务变得无响应,并开始杀死其他容器,例如
apiserver
;并且文件也不会上传。之后,我向Minio容器添加了资源限制,并且群集本身变得稳定,但是Minio容器仍然死亡。我希望Minio在规定的范围内运行,并且不消耗等于文件大小的内存。我不确定问题出在哪方面,无论是Minio还是Minikube还是VirtualBox还是Docker还是Kubernetes。我也不熟悉此设置中内存的工作方式。就像我说的那样,相同的设置在Ubuntu 18.04上运行良好。
版本:
最佳答案
我也posted this issue to Minio repository,得到的响应是256mb
太低。将内存增加到512mb
后,它可以正常工作。