问题描述
如果我想在 Spark DataSet 列，最佳编码策略是什么?
If I want to store an Algebraic Data Type (ADT) (ie a Scala sealed trait hierarchy) within a Spark DataSet column, what is the best encoding strategy?
例如，如果我有一个 ADT，其中叶类型存储不同类型的数据:
For example, if I have an ADT where the leaf types store different kinds of data:
sealed trait Occupation
case object SoftwareEngineer extends Occupation
case class Wizard(level: Int) extends Occupation
case class Other(description: String) extends Occupation
构建一个的最佳方法是什么:
Whats the best way to construct a:
org.apache.spark.sql.DataSet[Occupation]
推荐答案
TL;DR 目前没有好的解决方案，鉴于 Spark SQL/Dataset 的实现，在可预见的未来，不太可能有.
TL;DR There is no good solution right now, and given Spark SQL / Dataset implementation, it is unlikely there will be one in the foreseeable future.
您可以使用通用的kryo 或java 编码器
You can use generic kryo or java encoder
val occupation: Seq[Occupation] = Seq(SoftwareEngineer, Wizard(1), Other("foo"))
spark.createDataset(occupation)(org.apache.spark.sql.Encoders.kryo[Occupation])
但在实践中几乎没有用.
but is hardly useful in practice.
UDT API 目前提供了另一种可能的方法(Spark 1.6、2.0、2.1-SNAPSHOT)，它是私有的，需要相当多的大量样板代码(您可以检查 oasml.linalg.VectorUDT 以查看示例实现).
UDT API provides another possible approach as for now (Spark 1.6, 2.0, 2.1-SNAPSHOT) it is private and requires quite a lot boilerplate code (you can check o.a.s.ml.linalg.VectorUDT to see example implementation).
这篇关于将 ADT/密封特征层次结构编码到 Spark DataSet 列中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！