Filebeat日期字段映射为type关键字

问题描述

Filebeat正在从文件中读取日志，其中日志的格式如下:

Filebeat is reading logs from a file, where logs are in the following format:

{"logTimestamp":"2019-11-29T16:39:43.027Z","@version":"1","message":"Hello world","logger_name":"se.lolotron.App","thread_name":"thread-1","level":"INFO","level_value":40000,"application":"my-app"}

因此，存在一个以 ISO 8601 时间格式记录的字段 logTimestamp .问题在于该字段在Elasticsearch filebeat索引中被映射为关键字

So there is a field logTimestamp logged in ISO 8601 time format.The problem is that this field is mapped as a keyword In Elasticsearch filebeat index

"logTimestamp": {
    "type": "keyword",
    "ignore_above": 1024
},

另一方面，如果我在相同的Elasticsearch实例中为相似的文档建立了索引，但索引不同，例如

On the other hand if I index a similar document in the same Elasticsearch instance but different index, e.g.

POST /new_index/_doc/
{
    "message": "hello world",
    "logTimestamp":"2019-11-29T16:39:43.027Z"
}

映射是

"logTimestamp": {
     "type": "date"
},

根据文档 和此处默认情况下，如果使用 strict_date_optional_time 格式设置，Elastic应该检测到日期.而strict_date_optional_time被描述为

According to docs here and here by default Elastic should detect a date if formatted with strict_date_optional_time. And strict_date_optional_time is described as

我假定是ISO 8601，并认为我在上面的示例中将新文档索引到new_index证明了这一点.

Which I presume is ISO 8601 and think I proved that with indexing a new doc to new_index in the example above.

在Filebeat的情况下，为什么logTimestamp保存为关键字?有什么想法吗?

Why is logTimestamp saved as keyword in the case of Filebeat? Any ideas?

我正在使用Filbeat 7.2.1，Elasticsearch 7.2.1.还使用默认的 fields.yml .

I'm using Filbeat 7.2.1, Elasticsearch 7.2.1.Also the default fields.yml is used.

推荐答案

我刚刚发现默认情况下，filebeat索引已禁用 date_detection (Filebeat版本7.2.1).可以在此处

I just found out that date_detection is disabled for filebeat indices by default (Filebeat version 7.2.1).This can be seen here

var (
    // Defaults used in the template
    defaultDateDetection         = false
    ...

看起来好像不能被覆盖.

Does not look like it can be overridden.

此问题的解决方法是使用实​​验性功能 append_fields (至少在撰写本文时为实验性功能.请参见此处了解更多信息.)并将以下内容添加到filebeat.yml配置

The workaround for this is to use experimental feature append_fields (experimental at least at the time of writing this post. See here for more.) and add the following to the filebeat.yml config

setup.template.overwrite: true
setup.template.append_fields:
- name: logTimestamp
  type: date

这将确保logTimestamp的映射是日期.

This will make sure that the mapping for logTimestamp is date.

这篇关于Filebeat日期字段映射为type关键字的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！