问题描述
我使用社区图kube-prometheus-stack在我的Kubernetes集群上使用Helm安装了Prometheus--我在捆绑的Grafana实例中得到了一些漂亮的仪表板。我现在希望Vertical Pod Autoscaler的推荐者使用普罗米修斯作为历史指标的数据源as described here。这意味着,我必须更改cAdvisor的普罗米修斯刮板设置,this answer为我指明了正确的方向,因为在进行更改后,我现在可以从cAdvisor中看到正确的指标job
标记。不幸的是,Grafana仪表板中的一些图表现在已经损坏。它看起来不再拾取CPU指标,而是只为与CPU相关的图表显示&无数据&。所以,我假设我必须调整图表才能再次正确地提取指标,但我在Grafana中看不到任何明显的地方可以这样做?
不确定它是否与问题相关,但我正在Azure Kubernetes服务(AKS)上运行我的Kubernetes群集。
这是我在安装普罗米修斯时向Helm图表提供的全部values.yaml
:
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
kubeEtcd:
enabled: false
kubeProxy:
enabled: false
kubelet:
serviceMonitor:
# Diables the normal cAdvisor scraping, as we add it with the job name "kubernetes-cadvisor" under additionalScrapeConfigs
# The reason for doing this is to enable the VPA to use the metrics for the recommender
# https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/FAQ.md#how-can-i-use-prometheus-as-a-history-provider-for-the-vpa-recommender
cAdvisor: false
prometheus:
prometheusSpec:
retention: 15d
storageSpec:
volumeClaimTemplate:
spec:
# the azurefile storage class is created automatically on AKS
storageClassName: azurefile
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 50Gi
additionalScrapeConfigs:
- job_name: 'kubernetes-cadvisor'
scheme: https
metrics_path: /metrics/cadvisor
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
Kubernetes版本:1.21.2
Kube-Prometheus-STACK版本:18.1.1
Helm版本:version.BuildInfo{版本:v3.6.3;,GitCommit:";d506314abfb5d21419df8c7e7e68012379db2354";,GitTreeState:;脏;,GoVersion:;go1.16.5;}
推荐答案
不幸的是,我没有访问Azure AKS的权限,因此我在我的GKE集群上重现了此问题。下面我将提供一些可能有助于解决您的问题的解释。
首先,您可以尝试执行此node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
规则以查看它是否返回任何结果:
如果它没有返回任何记录,请阅读以下段落。
为cAdvisor创建擦除配置
与其为cAdvisor创建全新的擦除配置,我建议使用默认情况下在kubelet.serviceMonitor.cAdvisor: true
时生成的配置,但做了一些修改,如将标签更改为job=kubernetes-cadvisor
。
在我的示例中,‘kubernetes-cvisor’擦除配置如下:
注意:我在values.yaml
文件中的additionalScrapeConfigs
下添加了此配置(values.yaml
文件的其余部分可能与您的相似)。
- job_name: 'kubernetes-cadvisor'
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics/cadvisor
scheme: https
authorization:
type: Bearer
credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
follow_redirects: true
relabel_configs:
- source_labels: [job]
separator: ;
regex: (.*)
target_label: __tmp_prometheus_job_name
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
separator: ;
regex: kubelet
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: kubelet
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: https-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: https-metrics
action: replace
- source_labels: [__metrics_path__]
separator: ;
regex: (.*)
target_label: metrics_path
replacement: $1
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
namespaces:
names:
- kube-system
修改普罗米修斯规则
默认情况下,从cAdvisor获取数据的Prometheus规则在其PromQL表达式中使用job="kubelet"
:
job=kubelet
更改为job=kubernetes-cadvisor
后,我们还需要在普罗米修斯规则中修改此标签:注意:我们只需要修改metrics_path="/metrics/cadvisor
的规则(这些是从cAdvisor检索数据的规则)。$ kubectl get prometheusrules prom-1-kube-prometheus-sta-k8s.rules -o yaml
...
- name: k8s.rules
rules:
- expr: |-
sum by (cluster, namespace, pod, container) (
irate(container_cpu_usage_seconds_total{job="kubernetes-cadvisor", metrics_path="/metrics/cadvisor", image!=""}[5m])
) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""})
)
record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
...
here we have a few more rules to modify...
修改普罗米修斯规则并等待一段时间后,我们可以看到它是否像预期的那样工作。我们可以尝试像开始时一样执行node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
。
这篇关于更改cAdvisor刮板中的普罗米修斯作业标签会破坏Grafana控制面板的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!