本文介绍了将Scala代码转换为Java for Spark Partitioner的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
因此,我尝试使用 Spark
和 Java
来实现自定义分区程序,我找到了一个很好的在线示例,但它使用的是 Scala
,我一生都无法弄清楚它如何正确转换为Java,因此我可以尝试实现它.有人可以帮忙吗?这是我在 Scala
中找到的示例代码:
So I am trying to implement a custom partitioner using Spark
with Java
, and I found a great example of how to do this online, but it is using Scala
, and I cannot for the life of me figure out how it translates properly into Java so I can try to implement it. Can anyone help? Here is the example code I found for it in Scala
:
class DomainNamePartitioner(numParts: Int) extends Partitioner {
override def numPartitions: Int = numParts
override def getPartition(key: Any): Int = {
val domain = new Java.net.URL(key.toString).getHost()
val code = (domain.hashCode % numPartitions)
if (code < 0) {
code + numPartitions // Make it non-negative
} else {
code
}
}
// Java equals method to let Spark compare our Partitioner objects
override def equals(other: Any): Boolean = other match {
case dnp: DomainNamePartitioner =>
dnp.numPartitions == numPartitions
case _ =>
false
}
}
推荐答案
首先, Scala
是编写 Spark
的首选.
这是相应的 Java
代码(不是唯一版本):
Here is the corresponding Java
code (it is not the unique version):
查看更多信息: https://spark.apache.org/docs/latest/api/java/index.html
class DomainNamePartitioner extends Partitioner{
private int numParts;
public Partitioner()
{
}
public Partitioner(int numParts)
{
this.numParts = numParts;
}
@Override
public int numPartitions()
{
return numParts;
}
@Override
public int getPartition(Object key){
String domain = new Java.net.URL(key.toString).getHost();
int code = domain.hashCode % numPartitions;
if (code < 0){
return code + this.numPartitions();
}else{
return code;
}
}
@Override
public boolean equals(Object obj){
if (obj instanceof DomainNamePartitioner){
DomainNamePartitioner pObj = (DomainNamePartitioner)obj;
return pObj.numPartitions() == this.numPartitions;
}else{
return false;
}
}
}
这篇关于将Scala代码转换为Java for Spark Partitioner的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!