我定义了一个基本脚本来创建DF,其中的数据来自redshift中我的一张表。我运行了该过程,但是一段时间以来,我一直在努力传达我无法解释的消息。
日志中的错误输出为:"/mnt/yarn/usercache/root/appcache/application_1525803778049_0004/container_1525803778049_0004_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o60.getDynamicFrame. : java.lang.UnsupportedOperationException: empty.reduceLeft at scala.collection.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame, DynamicFrameReader, DynamicFrameWriter, DynamicFrameCollection
from pyspark.sql.functions import lit
from awsglue.job import Job
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
table = glueContext.create_dynamic_frame.from_options(connection_type="redshift", connection_options =
{"url": "jdbc:redshift://xxxxx.yyyyy.us-east-1.redshift.amazonaws.com:5439/db",
"user": "yyyy",
"password": "yyyyy",
"dbtable": "schema.table_name",
"redshiftTmpDir": "s3://aws-glue-temporary-accountnumber-us-east-1/"},
format="orc",
transformation_ctx="table" )
table.show()
dfred = table.toDF().createOrReplaceTempView("table_df")
job.commit()
请感谢您能为我提供的任何帮助。非常感谢
最佳答案
好吧,在继续为此苦苦挣扎之后,我经过了DynamicFrame的官方代码类,所以,我在代码中添加了一个apply格式转换,以映射来自redshift中读取表的结果以及提取该表的方法跳过错误o60中失败的参数transformation_ctx
我的最终版本代码是:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame, DynamicFrameReader, DynamicFrameWriter, DynamicFrameCollection
from pyspark.sql.functions import lit
from awsglue.job import Job
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
table = glueContext.create_dynamic_frame.from_options(connection_type="redshift", connection_options =
{"url": "jdbc:redshift://xxxxx.yyyyy.us-east-1.redshift.amazonaws.com:5439/db",
"user": "yyyy",
"password": "yyyyy",
"dbtable": "schema.table_name",
"redshiftTmpDir": "s3://aws-glue-temporary-accountnumber-us-east-1/"}
)
applyformat = ApplyMapping.apply(frame =table, mappings =
[("field1","string","field1","string"),
("field2","string","field2","string") ], transformation_ctx = "applyformat")
dfred = table.toDF().createOrReplaceTempView("table_df")
sqlDF = spark.sql(
"SELECT COUNT(*) FROM table_df"
)
print sqlDF.show()
job.commit()
关于pyspark - 胶水AWS:调用o60.getDynamicFrame时发生错误,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/50240834/