使用Hadoop mapreduce,我正在编写代码以获取不同长度的子字符串。
给定示例字符串“ZYXCBA”和长度3。我的代码必须返回所有可能的字符串,这些字符串的长度为3(“ZYX”,“YXC”,“XCB”,“CBA”),长度为4(“ZYXC”,“YXCB”, “XCBA”)最后的长度为5(“ZYXCB”,“YXCBA”)。
在 map 阶段,我做了以下工作:
键=我想要的子串的长度
值=“ZYXCBA”。
所以映射器输出是
3,"ZYXCBA"
4,"ZYXCBA"
5,"ZYXCBA"
在reduce中,我采用字符串(“ZYXCBA”)和键3来获取长度为3的所有子字符串。对于4,5来说,情况相同。结果收集在ArrayList中。
我正在使用以下命令运行代码:
hduser@Ganesh:~/Documents$ hadoop jar Saishingles.jar hadoopshingles.Saishingles Behara/Shingles/input Behara/Shingles/output
我的代码如下所示:
package hadoopshingles;
import java.io.IOException;
import java.util.ArrayList;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class Saishingles{
public static class shinglesmapper extends Mapper<Object, Text, IntWritable, Text>{
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
String str = new String(value.toString());
String[] list = str.split(" ");
int index = Integer.parseInt(list[0]);
String val = list[1];
int length = val.length();
for(int i = index; i <= length; i++)
{
context.write(new IntWritable(index),new Text(val));
}
}
}
public static class shinglesreducer extends Reducer<IntWritable,Text,IntWritable,ArrayList<String>> {
private ArrayList<String> result = new ArrayList<String>();
public void reduce(IntWritable key, Text value, Context context
) throws IOException, InterruptedException {
String str = new String(value.toString());
int newkey = key.get();
int Tz = str.length() - newkey + 1;
int position = 0;
while (position <= Tz)
{
result.add(str.substring(position,position + newkey -1));
position = position + 1;
}
context.write(new IntWritable(newkey),result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Saishingles");
job.setJarByClass(hadoopshingles.Saishingles.class);
job.setMapperClass(shinglesmapper.class);
job.setCombinerClass(shinglesreducer.class);
job.setReducerClass(shinglesreducer.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(ArrayList.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
它给出了以下错误:
Exception in thread "main" java.lang.ClassNotFoundException: hadoopshingles.Saishingles
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
请帮助我,并提前谢谢:)
最佳答案
我相信您不应在类名中包含“.class”。
代替
它应该是