本文介绍了从AWS Glue中的动态框架覆盖实木复合地板文件的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我使用动态框架在S3中写入镶木地板文件,但是如果文件已经存在,我的程序将附加一个新文件而不是替换它.我使用的句子是这样的:
I use dynamic frames to write a parquet file in S3 but if a file already exists my program append a new file instead of replace it. The sentence that I use is this:
glueContext.write_dynamic_frame.from_options(frame = table,
connection_type = "s3",
connection_options = {"path": output_dir,
"partitionKeys": ["var1","var2"]},
format = "parquet")
是否有诸如"mode":"overwrite"
之类的东西可以替换我的实木复合地板文件?
Is there anything like "mode":"overwrite"
that replace my parquet files?
推荐答案
当前,AWS Glue不支持覆盖"模式,但他们正在使用此功能.
Currently AWS Glue doesn't support 'overwrite' mode but they are working on this feature.
作为一种解决方法,您可以将DynamicFrame对象转换为spark的DataFrame并使用spark而不是Glue编写它:
As a workaround you can convert DynamicFrame object to spark's DataFrame and write it using spark instead of Glue:
table.toDF()
.write
.mode("overwrite")
.format("parquet")
.partitionBy("var_1", "var_2")
.save(output_dir)
这篇关于从AWS Glue中的动态框架覆盖实木复合地板文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!