Prometheus 指标在删除后不会消失

问题描述

我需要删除一些源自一台主机的 Prometheus 指标.为简单起见，我们假设需要删除来自主机的所有指标.

I need to delete some Prometheus metrics originating from one host. For simplicity let's assume all metrics from the host need to be deleted.

对两个 VM 上的 delete_series 端点执行 POST.得到了预期的 204.指标没有消失(即使在通过 CTRL+R、CTRL+F5 强制刷新缓存后，仍然可以在 Web/浏览器界面中看到).

Executed POST to a delete_series endpoint on both VMs. Got 204 as expected. The metrics did not disappear (still visible in web / browser interface, even after forcing cache refresh via CTRL+R, CTRL+F5).

也试过强制clean_tombstones，得到204，但没有改变.

Also tried forcing clean_tombstones, got 204, but not change.

重新启动了两个实例，然后又等了一天，希望某物"会重新加载自己.

Restarted both instances, then also waited a day in a hope that "something" will reload itself.

使用的命令:# curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="my-hostname"}' -v

# curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones' -v

输出:

*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9090 (#0)
> POST /api/v1/admin/tsdb/delete_series?match[]={instance="my-hostname"} HTTP/1.1
> Host: localhost:9090
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Date: Tue, 23 Jul 2019 07:54:30 GMT
<
* Connection #0 to host localhost left intact

我的 Prometheus 集群由两个运行 Prometheus 2.6.1 的虚拟机组成.

My Prometheus cluster consists of two VMs running Prometheus 2.6.1.

admin-api 已启用:

           └─28557 /opt/prometheus/prometheus-server/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus/databases --storage.tsdb.retention=15d --web.enable-admin-api --web.external-url=http://redacted --log.level=debug

推荐答案

进一步检查发现源代码目录中的 /var/spool/prometheus 目录中有一个意外的 node-exporter 文件主机导致节点导出器一遍又一遍地重新刮取古老的指标.删除文件并重新运行删除命令后，指标消失了.

Upon further inspection it turned out that there was an unexpected node-exporter file in /var/spool/prometheus directory on the source host causing node-exporter to re-scrape ancient metrics over and over again. After deleting the file and re-running the delete command the metrics are gone.

这篇关于Prometheus 指标在删除后不会消失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！