本文介绍了将Scala代码转换为Java for Spark Partitioner的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

因此,我尝试使用 Spark Java 来实现自定义分区程序,我找到了一个很好的在线示例,但它使用的是 Scala ,我一生都无法弄清楚它如何正确转换为Java,因此我可以尝试实现它.有人可以帮忙吗?这是我在 Scala 中找到的示例代码:

So I am trying to implement a custom partitioner using Spark with Java, and I found a great example of how to do this online, but it is using Scala, and I cannot for the life of me figure out how it translates properly into Java so I can try to implement it. Can anyone help? Here is the example code I found for it in Scala:

class DomainNamePartitioner(numParts: Int) extends Partitioner {
  override def numPartitions: Int = numParts
  override def getPartition(key: Any): Int = {
    val domain = new Java.net.URL(key.toString).getHost()
    val code = (domain.hashCode % numPartitions)
    if (code < 0) {
      code + numPartitions  // Make it non-negative
    } else {
      code
    }
  }
  // Java equals method to let Spark compare our Partitioner objects
  override def equals(other: Any): Boolean = other match {
    case dnp: DomainNamePartitioner =>
      dnp.numPartitions == numPartitions
    case _ =>
      false
  }
}

推荐答案

首先, Scala 是编写 Spark 的首选.

这是相应的 Java 代码(不是唯一版本):

Here is the corresponding Java code (it is not the unique version):

查看更多信息: https://spark.apache.org/docs/latest/api/java/index.html

class DomainNamePartitioner extends Partitioner{

private int numParts;

public Partitioner()
{

}
public Partitioner(int numParts)
{
  this.numParts = numParts;
}

@Override
public int numPartitions()
{
    return numParts;
}
@Override
public int getPartition(Object key){

   String domain = new Java.net.URL(key.toString).getHost();
   int code = domain.hashCode % numPartitions;
   if (code < 0){
    return code + this.numPartitions();
   }else{
    return code;
   }
}

@Override
public boolean equals(Object obj){

    if (obj instanceof DomainNamePartitioner){
        DomainNamePartitioner pObj = (DomainNamePartitioner)obj;
        return pObj.numPartitions() == this.numPartitions;
    }else{
        return false;
    }
}
}

这篇关于将Scala代码转换为Java for Spark Partitioner的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-03 05:47
查看更多