如何删除DataFrame中特定列的NULL和空?

问题描述

我想从demo_name为NULL和demo_name为空的数据框中删除记录.

I would like to remove records from a dataframe having demo_name as NULL and demo_name as empty.

demo_name是该dataFrame中具有String数据类型的列

demo_name is a column in that dataFrame with String datatype

我正在尝试以下代码.我要进行修整，因为demo_name的记录中有多个空格.

I am trying the below code . I want to apply trim as there are records for demo_name with multiple spaces.

   val filterDF = demoDF.filter($"demo_name".isNotNull && $"demo_name".trim != "" )

但是出现错误，因为无法解决符号修剪

But I get error as cannot resolve symbol trim

有人可以帮助我解决此问题吗?

Could someone help me to fix this issue ?

推荐答案

您正在调用trim就像在对String进行操作一样，但是$函数使用implicit转换来转换implicit的名称. Column实例本身的列.问题是Column没有trim函数.

You are calling trim as if you are acting on a String, but the $ function uses implicit conversion to convert the name of the column to the Column instance itself. The problem is that Column doesn't have a trim function.

您需要导入库函数并将其应用于您的列:

You need to import the library functions and apply them to your column:

import org.apache.spark.sql.functions._

demoDF.filter($"demo_name".isNotNull && length(trim($"demo_name")) > 0)

在这里，我使用库函数trim和length-trim来去除空格，然后使用length来验证结果中是否包含任何内容.

Here I use the library functions trim and length--trim to strip the spaces of course and then length to verify that the result has anything in it.

这篇关于如何删除DataFrame中特定列的NULL和空?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！