映射器将忽略哪些文件作为输入?

问题描述

我正在链接多个MapReduce作业，并希望随结果传递/存储一些元信息(例如配置或原始输入的名称).至少文件"_SUCCESS"以及目录"_logs"中的所有内容都将被忽略.

I'm chaining multiple MapReduce jobs and want to pass along/store some meta information (e.g. configuration or name of original input) with the results. At least the file "_SUCCESS" and also anything in the directory "_logs" seams to be ignored.

在默认情况下，InputReader是否会忽略任何文件名模式?还是这只是一个固定的有限列表?

Are there any filename patterns which are by default ignored by the InputReader? Or is this just a fixed limited list?

推荐答案

FileInputFormat使用以下 hiddenFileFilter 默认为:

  private static final PathFilter hiddenFileFilter = new PathFilter(){
      public boolean accept(Path p){
        String name = p.getName();
        return !name.startsWith("_") && !name.startsWith(".");
      }
    };

因此，如果您使用任何FileInputFormat(例如TextInputFormat，KeyValueTextInputFormat，SequenceFileInputFormat)，则隐藏文件(文件名以"_"或."开头)将被忽略.

So if you uses any FileInputFormat (such as TextInputFormat, KeyValueTextInputFormat, SequenceFileInputFormat), the hidden files (the file name starts with "_" or ".") will be ignored.

您可以使用 FileInputFormat.setInputPathFilter 来设置您的自定义PathFilter.请记住，hiddenFileFilter始终处于活动状态.

You can use FileInputFormat.setInputPathFilter to set your custom PathFilter. Remember that the hiddenFileFilter is always active.

这篇关于映射器将忽略哪些文件作为输入?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！