java - 在MapReduce中处理用户输入字符串

我开始使用MapReduce的Hadoop变体，因此对来龙去脉的了解零。我了解它在概念上应该如何工作。

我的问题是在我提供的一堆文件中找到特定的搜索字符串。我对文件不感兴趣-已排序。但是，您将如何寻求输入？您会在该计划的JobConf部分中询问吗？如果是这样，我如何将字符串传递给工作？

如果它在map()函数中，您将如何实现它？难道每次调用map()函数时都只是要求搜索字符串吗？

这是应该给您一个想法的主要方法和JobConf()部分:

public static void main(String[] args) throws IOException {

    // This produces an output file in which each line contains a separate word followed by
    // the total number of occurrences of that word in all the input files.

    JobConf job = new JobConf();

    FileInputFormat.setInputPaths(job, new Path("input"));
    FileOutputFormat.setOutputPath(job, new Path("output"));

    // Output from reducer maps words to counts.
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(LongWritable.class);

    // The output of the mapper is a map from words (including duplicates) to the value 1.
    job.setMapperClass(InputMapper.class);

    // The output of the reducer is a map from unique words to their total counts.
    job.setReducerClass(CountWordsReducer.class);

    JobClient.runJob(job);
}

和map()函数:

public void map(LongWritable key, Text value, OutputCollector<Text, LongWritable> output, Reporter reporter) throws IOException {

    // The key is the character offset within the file of the start of the line, ignored.
    // The value is a line from the file.

    //This is me trying to hard-code it. I would prefer an explanation on how to get interactive input!
    String inputString = "data";
    String line = value.toString();
    Scanner scanner = new Scanner(line);

    while (scanner.hasNext()) {
        if (line.contains(inputString)) {
            String line1 = scanner.next();
            output.collect(new Text(line1), new LongWritable(1));
        }
    }
    scanner.close();
}

我被认为不需要解决这个问题。任何建议/解释深表感谢!

最佳答案

JobConf 类是 Configuration 类的扩展，因此，您可以设置自定义属性:

JobConf job = new JobConf();
job.set("inputString", "data");
...

然后，如Mapper的文档所述:映射器实现可以通过JobConfigurable.configure(JobConf)访问作业的JobConf并进行初始化。这意味着您必须在Mapper中重新实现这种方法才能获得所需的参数:

private static String inputString;

public void configure(JobConf job)
    inputString = job.get("inputString");
}

无论如何，这是使用旧的API。使用新的配置，由于上下文(以及配置)作为参数传递给map方法，因此更易于访问配置。

关于java - 在MapReduce中处理用户输入字符串，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/29751188/