我正在尝试使用Pig中的DIFF()方法找出两个表(源表和目标表)之间的差异,以实现以下目标:

sourcenew = LOAD 'hdfs://HADOOPMASTER:54310/DVTTest/Source.txt' USING PigStorage(',') as (ID:chararray,Name:chararray,FirstName:chararray ,LastName:chararray,Vertical_Name:chararray ,Vertical_ID:chararray,Gender:chararray,DOB:chararray,Degree_Percentage:chararray ,Salary:chararray,StateName:chararray);


destnew = LOAD 'hdfs://HADOOPMASTER:54310/DVTTest/Destination.txt' USING PigStorage(',') as (ID:chararray,Name:chararray,FirstName:chararray ,LastName:chararray,Vertical_Name:chararray ,Vertical_ID:chararray,Gender:chararray,DOB:chararray,Degree_Percentage:chararray ,Salary:chararray,StateName:chararray);

cogroupnew= COGROUP sourcenew by ID inner, destnew by ID inner;

diffnew = FOREACH cogroupnew GENERATE DIFF(sourcenew,destnew);

DUMP diffnew;

给出两个表之间的差异,或者如果元组匹配,则返回空包{},在此之前它可以正常工作,下一步是在源文件中找到目标中没有的额外记录,为此
cogroupextrainsource= COGROUP sourcenew by ID inner, destnew by ID;
filterextrainsource= FILTER cogroupextrainsource BY ID NOT (cogroupnew)

预期的 throw 错误。
需要帮助以找到更多资源。
帮助将不胜感激。

谢谢!

最佳答案

您不需要列名ID旁边的$符号。仅当您不想按名称访问列时,才使用$。

cogroupextrainsource = COGROUP sourcenew by ID inner, destnew by ID;

10-05 21:04
查看更多