本文介绍了如何在Flume中处理多行日志条目的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我刚开始玩Flume.我有一个关于如何将多行日志条目作为一个事件处理的问题.类似于错误情况下的堆栈跟踪.例如,将以下内容视为一个事件,而不是每行一个事件

I have just started playing with Flume. I have a question on how to handle log entries that are multiline, as a single event. Like stack traces during error conditions. For example, treat the below as a single event rather than one event for each line

2013-04-05 05:00:41,280错误(ClientRequestPool-PooledExecutionEngine-Id#4)[com.ms.fw.rexs.gwy.api.service.AbstractAutosysJob]作业失败,导致228794java.lang.NullPointerException 在com.ms.fw.rexs.core.impl.service.job.ReviewNotificationJobService.createReviewNotificationMessageParameters(ReviewNotificationJobService.java:138)处....

2013-04-05 05:00:41,280 ERROR (ClientRequestPool-PooledExecutionEngine-Id#4 ) [com.ms.fw.rexs.gwy.api.service.AbstractAutosysJob] job failed for 228794java.lang.NullPointerException at com.ms.fw.rexs.core.impl.service.job.ReviewNotificationJobService.createReviewNotificationMessageParameters(ReviewNotificationJobService.java:138)....

我已将源配置为spooldir类型.

I have configured the source to a spooldir type.

谢谢苏曼

推荐答案

您需要研究扩展假脱机源使用的行反序列化器,一种简单(但可能有缺陷)的方法是在换行符上划界,但合并行在前一行前加上一定数量的空格作为前缀.

You'll want to look into extending the line deserializer used by spool source, one simple (but potentially flawed) approach would be delimit on newlines, but combine lines that are prefixed with a set number of spaces to the previous line.

事实上,已经有一个带有补丁的Jira问题:

In fact there is already a Jira issue for this with a patch:

这篇关于如何在Flume中处理多行日志条目的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-24 14:06