CloudWatch日志表现异常

CloudWatch日志表现异常

本文介绍了CloudWatch日志表现异常的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有两个带有多行日志语句的日志文件。在每个日志语句的开头,它们都具有相同的日期时间格式。配置如下所示:

  state_file = / var / lib / awslogs / agent-state 

[/opt/logdir/log1.0]
datetime_format =%Y-%m-%d%H:%M:%S
文件= /opt/logdir/log1.0
log_stream_name = /opt/logdir/logs/log1.0
initial_position = start_of_file
multi_line_start_pattern = {datetime_format}
log_group_name = my.log.group


[/opt/logdir/log2-console.log]
datetime_format =%Y-%m-%d%H:%M:%S
文件= / opt / logdir / log2-console。日志
log_stream_name = /opt/logdir/log2-console.log
初始位置= start_of_file
multi_line_start_pattern = {datetime_format}
log_group_name = my.log.group

cloudwatch日志代理正在将log1.0日志正确发送到我在cloudwatch上的日志组,但是,它没有发送log2的日志文件-console.log。



awslogs.log说:

  2016 -11-15 08:11:41,308-cwlogs.push.batch-警告-3593-线程4-跳过事件:{时间戳:1479196444000,开始位置:42330916L,结束位置:42331504L},原因:将来的时间戳超过2小时。 
2016-11-15 08:11:41,308-cwlogs.push.batch-警告-3593-线程4-跳过事件:{'时间戳':1479196451000,'开始位置':42331504L,'结束位置':42332092L },原因:时间戳记在未来超过2小时。

尽管服务器时间正确。同样奇怪的是,在开始推送的实际日志文件中,start_position和end_position中提到的行号不存在。



还有其他人遇到此问题吗?

解决方案

我能够解决此问题。



awslogs的状态已损坏。状态存储在/ var / awslogs / state / agent-state的sqlite数据库中。您可以通过

  sudo sqlite3 / var / awslogs / state / agent-state 

须具备sudo才能具有写访问权限。



列出所有具有

  select * from stream_state; 

查找日志流,并注意 source_id v列中的json数据结构。

  select * from push_state其中k = 7675f84405fcb8fe5b6bb14eaa0c4bfd; 

生成的记录在v列中具有json数据结构,其中包含batch_timestamp。而且这个batch_timestamp接错了。是过去的记录,不再处理任何新的日志记录(超过2小时)。



解决方案是更新此记录。复制v列,用当前时间戳替换batch_timestamp,并用类似的东西进行更新

  update push_state set v ='...在此处插入新值...',其中k ='7675f84405fcb8fe5b6bb14eaa0c4bfd'; 

使用以下命令重新启动服务

  sudo /etc/init.d/awslogs重新启动

我希望它能工作为你服务!


I have two log files with multi-line log statements. Both of them have same datetime format at the begining of each log statement. The configuration looks like this:

state_file = /var/lib/awslogs/agent-state

[/opt/logdir/log1.0]
datetime_format = %Y-%m-%d %H:%M:%S
file = /opt/logdir/log1.0
log_stream_name = /opt/logdir/logs/log1.0
initial_position = start_of_file
multi_line_start_pattern = {datetime_format}
log_group_name = my.log.group


[/opt/logdir/log2-console.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /opt/logdir/log2-console.log
log_stream_name = /opt/logdir/log2-console.log
initial_position = start_of_file
multi_line_start_pattern = {datetime_format}
log_group_name = my.log.group

The cloudwatch logs agent is sending log1.0 logs correctly to my log group on cloudwatch, however, its not sending log files for log2-console.log.

awslogs.log says:

2016-11-15 08:11:41,308 - cwlogs.push.batch - WARNING - 3593 - Thread-4 - Skip event: {'timestamp': 1479196444000, 'start_position': 42330916L, 'end_position': 42331504L}, reason: timestamp is more than 2 hours in future.
2016-11-15 08:11:41,308 - cwlogs.push.batch - WARNING - 3593 - Thread-4 - Skip event: {'timestamp': 1479196451000, 'start_position': 42331504L, 'end_position': 42332092L}, reason: timestamp is more than 2 hours in future.

Though server time is correct. Also weird thing is Line numbers mentioned in start_position and end_position does not exist in actual log file being pushed.

Anyone else experiencing this issue?

解决方案

I was able to fix this.

The state of awslogs was broken. The state is stored in a sqlite database in /var/awslogs/state/agent-state. You can access it via

sudo sqlite3 /var/awslogs/state/agent-state

sudo is needed to have write access.

List all streams with

select * from stream_state;

Look up your log stream and note the source_id which is part of a json data structure in the v column.

Then, list all records with this source_id (in my case it was 7675f84405fcb8fe5b6bb14eaa0c4bfd) in the push_state table

select * from push_state where k="7675f84405fcb8fe5b6bb14eaa0c4bfd";

The resulting record has a json data structure in the v column which contains a batch_timestamp. And this batch_timestamp seams to be wrong. It was in the past and any newer (more than 2 hours) log entries were not processed anymore.

The solution is to update this record. Copy the v column, replace the batch_timestamp with the current timestamp and update with something like

update push_state set v='... insert new value here ...' where k='7675f84405fcb8fe5b6bb14eaa0c4bfd';

Restart the service with

sudo /etc/init.d/awslogs restart

I hope it works for you!

这篇关于CloudWatch日志表现异常的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-11 18:15