问题描述
好吧，环顾四周后，我找不到解决我的问题的方法，因为它应该"起作用，但显然没有.我在 Ubuntu 14.04 LTS 机器 Logstash 1.4.2-1-2-2c0f5a1 上使用，我收到如下消息:
2014-08-05 10:21:13,618 [17] INFO Class.Type - 这是来自类的日志消息:BTW，我也是多线的在输入配置中，我有一个 multiline 编解码器并且事件被正确解析.我还将事件文本分成几个部分，以便于阅读.
最后，如 Kibana 中所见，我获得了如下内容(JSON 视图):
{"_index": "logstash-2014.08.06","_type": "自定义类型","_id": "PRtj-EiUTZK3HWAm5RiMwA",_score":空，_来源": {"@timestamp": "2014-08-06T08:51:21.160Z","@version": "1",标签":[多线"],"type": "utg-su","host": "ubuntu-14","path": "/mnt/folder/thisIsTheLogFile.log","logTimestamp": "2014-08-05;10:21:13.618","logThreadId": "17","logLevel": "信息","logMessage": "Class.Type - 这是来自类的日志消息:
顺便说一句，我也是多行的
"},种类": ["21",1407315081160]}你可能已经注意到我放了一个;"在时间戳中.原因是我希望能够使用时间戳字符串对日志进行排序，而显然 logstash 并不擅长(例如:http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/multi-fields.html)..>
我尝试以多种方式使用 date 过滤器失败，但它显然不起作用.
日期{语言环境 =>恩"匹配 =>["logTimestamp", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601"]时区 =>欧洲/维也纳"目标 =>@时间戳"add_field =>{调试"=>时间戳匹配"}}因为我读到 Joda 库可能有问题，如果字符串不严格符合 ISO 8601 标准(非常挑剔并期望一个 T，请参阅 https://logstash.jira.com/browse/LOGSTASH-180)，我也尝试使用 mutate将字符串转换为类似 2014-08-05T10:21:13.618 的内容，然后使用 "YYYY-MM-dd'T'HH:mm:ss.SSS".这也不起作用.
我不想在时间上手动设置 +02:00，因为这会给夏令时带来问题.
在任何这些情况下，事件都会转到 elasticsearch，但 date 显然什么也没做，因为 @timestamp 和 logTimestamp 是不同的，并且没有添加 debug 字段.
知道如何使 logTime 字符串正确排序吗?我专注于将它们转换为正确的时间戳，但也欢迎任何其他解决方案.
如下图所示:
在对 @timestamp 进行排序时，elasticsearch 可以正确执行，但是由于这不是真正的"日志时间戳，而是读取 logstash 事件时，我需要(显然)也可以对 logTimestamp 进行排序.这就是输出.显然没那么有用:
欢迎任何帮助！如果我忘记了一些可能有用的信息，请告诉我.
更新:
这是最终起作用的过滤器配置文件:
# 过滤消息如下:# 2014-08-05 10:21:13,618 [17] INFO Class.Type - 这是来自类的日志消息:# BTW，我也是多线的# 只获取类型事件(类型组件A，类型组件B，等等)筛选 {# 你不能在过滤器之外写一个if"！如果 [type] { 中的type-"神通{# 解析时间戳数据.我们需要 "(?m)" 以便 grok(内部 Oniguruma)正确解析多行事件patterns_dir =>./模式"匹配 =>[ "message", "(?m)%{TIMESTAMP_ISO8601:logTimestampString}[ ;][%{DATA:logThreadId}][ ;]%{LOGLEVEL:logLevel}[ ;]*%{GREEDYDATA:logMessage}" ]}# 时间戳可能有逗号而不是点.转换以便以相同的方式存储所有内容变异{gsub =>[# 用点替换所有逗号"logTimestampString", ",", "."]}变异{gsub =>[# 使 logTimestamp 可排序.有了空间，就不行了！这最终效果不佳# 但不知何故显然使日期过滤器的事情变得更容易"logTimestampString", " ", ";"]}日期 {语言环境 =>恩"匹配 =>["logTimestampString", "YYYY-MM-dd;HH:mm:ss.SSS"]时区 =>欧洲/维也纳"目标 =>日志时间戳"}}}筛选 {如果 [type] { 中的type-"# 删除已经解析的数据变异{remove_field =>[ 信息" ]}}}我已经测试了您的 date 过滤器.它对我有用！
这是我的配置
输入{标准输入{}}筛选 {日期 {语言环境 =>恩"匹配 =>["消息", "YYYY-MM-dd;HH:mm:ss.SSS"]时区 =>欧洲/维也纳"目标 =>@时间戳"add_field =>{调试"=>时间戳匹配"}}}输出 {标准输出{编解码器 =>红宝石调试"}}我使用这个输入:
2014-08-01;11:00:22.123输出为:
{消息"=>"2014-08-01;11:00:22.123",@version" =>"1",@timestamp" =>"2014-08-01T09:00:22.123Z",主机"=>"ABCDE",调试" =>时间戳匹配"}因此，请确保您的 logTimestamp 具有正确的值.这可能是其他问题.或者您能否提供您的日志事件和日志存储配置以进行更多讨论.谢谢.
Well, after looking around quite a lot, I could not find a solution to my problem, as it "should" work, but obviously doesn't.I'm using on a Ubuntu 14.04 LTS machine Logstash 1.4.2-1-2-2c0f5a1, and I am receiving messages such as the following one:
2014-08-05 10:21:13,618 [17] INFO Class.Type - This is a log message from the class:
BTW, I am also multiline
In the input configuration, I do have a multiline codec and the event is parsed correctly. I also separate the event text in several parts so that it is easier to read.
In the end, I obtain, as seen in Kibana, something like the following (JSON view):
{
"_index": "logstash-2014.08.06",
"_type": "customType",
"_id": "PRtj-EiUTZK3HWAm5RiMwA",
"_score": null,
"_source": {
"@timestamp": "2014-08-06T08:51:21.160Z",
"@version": "1",
"tags": [
"multiline"
],
"type": "utg-su",
"host": "ubuntu-14",
"path": "/mnt/folder/thisIsTheLogFile.log",
"logTimestamp": "2014-08-05;10:21:13.618",
"logThreadId": "17",
"logLevel": "INFO",
"logMessage": "Class.Type - This is a log message from the class:
BTW, I am also multiline
"
},
"sort": [
"21",
1407315081160
]
}
You may have noticed that I put a ";" in the timestamp. The reason is that I want to be able to sort the logs using the timestamp string, and apparently logstash is not that good at that (e.g.: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/multi-fields.html).
I have unsuccessfull tried to use the date filter in multiple ways, and it apparently did not work.
date {
locale => "en"
match => ["logTimestamp", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601"]
timezone => "Europe/Vienna"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
Since I read that the Joda library may have problems if the string is not strictly ISO 8601-compliant (very picky and expects a T, see https://logstash.jira.com/browse/LOGSTASH-180), I also tried to use mutate to convert the string to something like 2014-08-05T10:21:13.618 and then use "YYYY-MM-dd'T'HH:mm:ss.SSS". That also did not work.
I do not want to have to manually put a +02:00 on the time because that would give problems with daylight saving.
In any of these cases, the event goes to elasticsearch, but date does apparently nothing, as @timestamp and logTimestamp are different and no debug field is added.
Any idea how I could make the logTime strings properly sortable? I focused on converting them to a proper timestamp, but any other solution would also be welcome.
As you can see below:
When sorting over @timestamp, elasticsearch can do it properly, but since this is not the "real" log timestamp, but rather when the logstash event was read, I need (obviously) to be able to sort also over logTimestamp. This is what then is output. Obviously not that useful:
Any help is welcome! Just let me know if I forgot some information that may be useful.
Update:
Here is the filter config file that finally worked:
# Filters messages like this:
# 2014-08-05 10:21:13,618 [17] INFO Class.Type - This is a log message from the class:
# BTW, I am also multiline
# Take only type- events (type-componentA, type-componentB, etc)
filter {
# You cannot write an "if" outside of the filter!
if "type-" in [type] {
grok {
# Parse timestamp data. We need the "(?m)" so that grok (Oniguruma internally) correctly parses multi-line events
patterns_dir => "./patterns"
match => [ "message", "(?m)%{TIMESTAMP_ISO8601:logTimestampString}[ ;][%{DATA:logThreadId}][ ;]%{LOGLEVEL:logLevel}[ ;]*%{GREEDYDATA:logMessage}" ]
}
# The timestamp may have commas instead of dots. Convert so as to store everything in the same way
mutate {
gsub => [
# replace all commas with dots
"logTimestampString", ",", "."
]
}
mutate {
gsub => [
# make the logTimestamp sortable. With a space, it is not! This does not work that well, in the end
# but somehow apparently makes things easier for the date filter
"logTimestampString", " ", ";"
]
}
date {
locale => "en"
match => ["logTimestampString", "YYYY-MM-dd;HH:mm:ss.SSS"]
timezone => "Europe/Vienna"
target => "logTimestamp"
}
}
}
filter {
if "type-" in [type] {
# Remove already-parsed data
mutate {
remove_field => [ "message" ]
}
}
}
I have tested your date filter. it works on me!
Here is my configuration
input {
stdin{}
}
filter {
date {
locale => "en"
match => ["message", "YYYY-MM-dd;HH:mm:ss.SSS"]
timezone => "Europe/Vienna"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
}
output {
stdout {
codec => "rubydebug"
}
}
And I use this input:
2014-08-01;11:00:22.123
The output is:
{
"message" => "2014-08-01;11:00:22.123",
"@version" => "1",
"@timestamp" => "2014-08-01T09:00:22.123Z",
"host" => "ABCDE",
"debug" => "timestampMatched"
}
So, please make sure that your logTimestamp has the correct value.It is probably other problem. Or can you provide your log event and logstash configuration for more discussion. Thank you.
这篇关于使用日期过滤器将 Logstash 日期解析为时间戳的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！