问题描述
有人可以发布一个简单的FileBasedSource子类示例吗?我是Google Dataflow的新手,对Java经验不足.我的目标是在读取文件时将行号作为键,或者根据行号跳过行.
Can someone post a simple example of subclassing FileBasedSource? I'm new to Google Dataflow and very inexperienced with Java. My goal is to read files while including line numbers as a key, or to skip lines based on the line number.
推荐答案
XMLSource 是了解FileBasedSource的工作原理的一个很好的起点.对于您的阅读器,您可能需要这样的东西(其中readNextLine()读到一行的末尾并更新偏移量):
The implementation of XMLSource is a good starting point for understanding how FileBasedSource works. You'll likely want something like this for your reader (where readNextLine() reads to the end of a line and updates the offset):
protected void startReading(ReadableByteChannel channel) throws IOException {
if (getCurrentSource().getMode() == FileBasedSource.Mode.SINGLE_FILE_OR_SUBRANGE) {
// If we are not at the beginning of a line, we should ignore the current line.
if (getCurrentSource().getStartOffset() > 0) {
SeekableByteChannel seekChannel = (SeekableByteChannel) channel;
// Start from one character back and read till we find a new line.
seekChannel.position(seekChannel.position() - 1);
nextOffset = seekChannel.position() + readNextLine(new ByteArrayOutputStream());
}
}
}
我用完整的 LineIO 示例创建了要点,它可能比XMLSource更简单
I've created a gist with the complete LineIO example, which may be simpler than XMLSource.
这篇关于Google Cloud Dataflow中FileBasedSource用法的示例的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!