问题描述
试图发出2个双维数组从映射器的价值。
在Hadoop中,我们有 TwoDArrayWritable
这需要1 - 二维数组
作为输入。
为了实现我的用例,我试图修改 TwoDArrayWritable
取输入 2
- 二维数组
Trying to emit 2 double dimensional array as value from mapper.In hadoop we have TwoDArrayWritable
which takes 1 - 2D array
as input.In order to achieve my usecase, I tried to edit TwoDArrayWritable
to take input of 2
- 2D array
/**
* A Writable for 2D arrays containing a matrix of instances of a class.
*/
public class MyTwoDArrayWritable implements Writable {
private Class valueClass;
private Writable[][] values;
private Class valueClass1;
private Writable[][] values1;
public MyTwoDArrayWritable(Class valueClass,Class valueClass1) {
this.valueClass = valueClass;
this.valueClass1 = valueClass1;
}
public MyTwoDArrayWritable(Class valueClass, DoubleWritable[][] values,Class valueClass1, DoubleWritable[][] values1) {
this(valueClass, valueClass1);
this.values = values;
this.values1 = values1;
}
public Object toArray() {
int dimensions[] = {values.length, 0};
Object result = Array.newInstance(valueClass, dimensions);
for (int i = 0; i < values.length; i++) {
Object resultRow = Array.newInstance(valueClass, values[i].length);
Array.set(result, i, resultRow);
for (int j = 0; j < values[i].length; j++) {
Array.set(resultRow, j, values[i][j]);
}
}
return result;
}
/**
* @return the valueClass
*/
public Class getValueClass() {
return valueClass;
}
/**
* @param valueClass the valueClass to set
*/
public void setValueClass(Class valueClass) {
this.valueClass = valueClass;
}
/**
* @return the values
*/
public Writable[][] getValues() {
return values;
}
/**
* @param values the values to set
*/
public void setValues(DoubleWritable[][] values,DoubleWritable[][] values1) {
this.values = values;
this.values = values1;
}
/**
* @return the valueClass1
*/
public Class getValueClass1() {
return valueClass1;
}
/**
* @param valueClass1 the valueClass1 to set
*/
public void setValueClass1(Class valueClass1) {
this.valueClass1 = valueClass1;
}
/**
* @return the values1
*/
public Writable[][] getValues1() {
return values1;
}
public void readFields(DataInput in) throws IOException {
// construct matrix
values = new Writable[in.readInt()][];
for (int i = 0; i < values.length; i++) {
values[i] = new Writable[in.readInt()];
}
// construct values
for (int i = 0; i < values.length; i++) {
for (int j = 0; j < values[i].length; j++) {
Writable value; // construct value
try {
value = (Writable) valueClass.newInstance();
} catch (InstantiationException e) {
throw new RuntimeException(e.toString());
} catch (IllegalAccessException e) {
throw new RuntimeException(e.toString());
}
value.readFields(in); // read a value
values[i][j] = value; // store it in values
}
}
}
public void write(DataOutput out) throws IOException {
out.writeInt(values.length); // write values
for (int i = 0; i < values.length; i++) {
out.writeInt(values[i].length);
}
for (int i = 0; i < values.length; i++) {
for (int j = 0; j < values[i].length; j++) {
values[i][j].write(out);
}
}
}
}
和来自映射emited 2 2D双阵列。
And emited 2 2D double array from mapper.
MyTwoDArrayWritable array = new MyTwoDArrayWritable (DoubleWritable.class,DoubleWritable.class);
DoubleWritable[][] myInnerArray = new DoubleWritable[EtransEkey.length][EtransEkey[0].length];
DoubleWritable[][] myInnerArray1 = new DoubleWritable[EtransDevalue.length][EtransDevalue[0].length];
// set values in myInnerArray
for (int k1 = 0; k1 < EtransEkey.length; k1++) {
for(int j1=0;j1< EtransEkey[0].length;j1++){
myInnerArray[k1][j1] = new DoubleWritable(EtransEkey[k1][j1]);
}
}
for (int k1 = 0; k1 < EtransDevalue.length; k1++) {
for(int j1=0;j1< EtransDevalue[0].length;j1++){
myInnerArray1[k1][j1] = new DoubleWritable(EtransDevalue[k1][j1]);
}
}
array.set(myInnerArray,myInnerArray1);
显示误差 array.set(myInnerArray,myInnerArray1);
/*
* The method set(DoubleWritable[][], DoubleWritable[][]) is undefined for the type MyTwoDArrayWritableritable
*/
编辑:如何通过减速这些值重复,以获得myInnerArray矩阵,矩阵myInnerArray1
How to iterate through these values in Reducer to get myInnerArray matrix and myInnerArray1 matrix?
到目前为止,我所做的就是
So far what I did is
for (MyTwoDArrayWritable c : values) {
System.out.println(c.getValues());
DoubleWritable[][] myInnerArray = new DoubleWritable[KdimRow][KdimCol];
for (int k1 = 0; k1 < KdimRow; k1++) {
for(int j1=0;j1< KdimCol;j1++){
myInnerArray[k1][j1] = new DoubleWritable();
}
}
但是,如何将它们存储回双阵列?
推荐答案
您还没有定义的设置
方法 MyTwoDArrayWritable
,这就是为什么表明出错。而不是调用 array.set
,你应该用你已经定义了不正是你需要的方法: setValues方法
,所以替换
You have not defined the set
method in MyTwoDArrayWritable
, that is why that error is shown. Instead of calling array.set
, you should use the method you have already defined which does exactly what you need: setValues
, so replace
array.set(myInnerArray,myInnerArray1);
与
array.setValues(myInnerArray,myInnerArray1);
这篇关于在Hadoop的定制TwoDArrayWritable并不能在迭代减速相同的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!