我正在浏览 JohnSnowLabs SpellChecker here 。
我在那里找到了 Norvig
的算法实现,示例部分只有以下两行:
import com.johnsnowlabs.nlp.annotator.NorvigSweetingModel
NorvigSweetingModel.pretrained()
如何在下面的数据框 (
df
) 上应用这个预训练模型来纠正“names
”列的拼写?+----------------+---+------------+
| names|age| color|
+----------------+---+------------+
| [abc, cde]| 19| red, abc|
|[eefg, efa, efb]|192|efg, efz efz|
+----------------+---+------------+
我试过这样做:
val schk = NorvigSweetingModel.pretrained().setInputCols("names").setOutputCol("Corrected")
val cdf = schk.transform(df)
但是上面的代码给了我以下错误:
java.lang.IllegalArgumentException: requirement failed: Wrong or missing inputCols annotators in SPELL_a1f11bacb851. Received inputCols: names. Make sure such columns have following annotator types: token
at scala.Predef$.require(Predef.scala:224)
at com.johnsnowlabs.nlp.AnnotatorModel.transform(AnnotatorModel.scala:51)
... 49 elided
最佳答案
spark-nlp
旨在用于其自己的特定管道,不同转换器的输入列必须包含特殊元数据。
异常已经告诉你 NorvigSweetingModel
的输入应该被标记化:
如果我没记错的话,至少您将在这里组装文档并标记化。
import com.johnsnowlabs.nlp.DocumentAssembler
import com.johnsnowlabs.nlp.annotator.NorvigSweetingModel
import com.johnsnowlabs.nlp.annotators.Tokenizer
import org.apache.spark.ml.Pipeline
val df = Seq(Seq("abc", "cde"), Seq("eefg", "efa", "efb")).toDF("names")
val nlpPipeline = new Pipeline().setStages(Array(
new DocumentAssembler().setInputCol("names").setOutputCol("document"),
new Tokenizer().setInputCols("document").setOutputCol("tokens"),
NorvigSweetingModel.pretrained().setInputCols("tokens").setOutputCol("corrected")
))
像这样的
Pipeline
可以通过小的调整应用于您的数据 - 输入数据必须是 string
而不是 array<string>
*:val result = df
.transform(_.withColumn("names", concat_ws(" ", $"names")))
.transform(df => nlpPipeline.fit(df).transform(df))
result.show()
+------------+--------------------+--------------------+--------------------+
| names| document| tokens| corrected|
+------------+--------------------+--------------------+--------------------+
| abc cde|[[document, 0, 6,...|[[token, 0, 2, ab...|[[token, 0, 2, ab...|
|eefg efa efb|[[document, 0, 11...|[[token, 0, 3, ee...|[[token, 0, 3, ee...|
+------------+--------------------+--------------------+--------------------+
如果你想要一个可以导出的输出,你应该用
Pipeline
扩展你的 Finisher
。import com.johnsnowlabs.nlp.Finisher
new Finisher().setInputCols("corrected").transform(result).show
+------------+------------------+
| names|finished_corrected|
+------------+------------------+
| abc cde| [abc, cde]|
|eefg efa efb| [eefg, efa, efb]|
+------------+------------------+
* 根据 the docs
DocumentAssembler
但它看起来不像在 1.7.3 中的实际工作:
df.transform(df => nlpPipeline.fit(df).transform(df)).show()
org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(names)' due to data type mismatch: argument 1 requires string type, however, '`names`' is of array<string> type.;;
'Project [names#62, UDF(names#62) AS document#343]
+- AnalysisBarrier
+- Project [value#60 AS names#62]
+- LocalRelation [value#60]
关于scala - 如何使用 JohnSnowLabs NLP 拼写校正模块 NorvigSweetingModel?,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/53418267/