哪些文件作为输入映射器忽略

哪些文件作为输入映射器忽略

本文介绍了哪些文件作为输入映射器忽略?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在链接多个MapReduce作业,并希望传递/存储一些元信息(例如原始输入的配置或名称)和结果。至少文件_SUCCESS以及目录_logs中的任何接缝都被忽略。

是否有任何文件名模式默认被InputReader忽略?或者这只是一个固定的限制列表?

解决方案

FileInputFormat 以下

 public boolean accept(Path p){
String name = p.getName();
return!name.startsWith(_)&& !name.startsWith( );
}
};所以如果你使用任何 FileInputFormat (比如如 TextInputFormat , KeyValueTextInputFormat , SequenceFileInputFormat ), (文件名以_或。开头)将被忽略。



您可以使用设置你的自定义 PathFilter 。请记住 hiddenFileFilter 始终处于活动状态。


I'm chaining multiple MapReduce jobs and want to pass along/store some meta information (e.g. configuration or name of original input) with the results. At least the file "_SUCCESS" and also anything in the directory "_logs" seams to be ignored.

Are there any filename patterns which are by default ignored by the InputReader? Or is this just a fixed limited list?

解决方案

The FileInputFormat uses the following hiddenFileFilter by default:

  private static final PathFilter hiddenFileFilter = new PathFilter(){
      public boolean accept(Path p){
        String name = p.getName();
        return !name.startsWith("_") && !name.startsWith(".");
      }
    };

So if you uses any FileInputFormat (such as TextInputFormat, KeyValueTextInputFormat, SequenceFileInputFormat), the hidden files (the file name starts with "_" or ".") will be ignored.

You can use FileInputFormat.setInputPathFilter to set your custom PathFilter. Remember that the hiddenFileFilter is always active.

这篇关于哪些文件作为输入映射器忽略?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-20 13:45