如何获取使用Dataproc工作流模板提交的jobId

本文介绍了如何获取使用Dataproc工作流模板提交的jobId的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 我已经在Airflow操作员（用Python编写的DataprocWorkflowTemplateInstantiateInlineOperator ）。提交作业后，会将一些名称指定为jobId（例如： job0-abc2def65gh12 ）。I have submitted a Hive job using Dataproc Workflow Template with the help of Airflow operator (DataprocWorkflowTemplateInstantiateInlineOperator) written in Python. Once the job is submitted some name will be assigned as jobId (example: job0-abc2def65gh12).无法获取jobId，我试图通过不起作用的REST API传递jobId作为参数。Since I was not able to get jobId I tried to pass jobId as a parameter from REST API which isn't working.我可以获取jobId还是，如果无法获取，可以我将jobId作为参数传递吗？Can I fetch jobId or, if it's not possible, can I pass jobId as a parameter?推荐答案 JobId将作为元数据 Operation 对象中的c $ c>字段。有关如何使用元数据，请参见此[1]文章。The JobId will be available as part of metadata field in Operation object that is returned from Instantiate operation. See this [1] article for how to work with metadata. Airflow操作员仅对Operation轮询[2]，但不返回最终的Operation对象。您可以尝试在 execute 中添加返回值。The Airflow operator only polls [2] on the Operation but does not return the final Operation object. You could try to add a return to execute.另一种选择是使用dataproc rest API [3 ]在工作流程完成之后。分配给工作流本身的所有标签都将传播到群集和作业，因此您可以进行列表作业调用。例如，filter参数可能类似于： filter = labels.my-label = 12345 Another option would to be to use dataproc rest API [3] after workflow finishes. Any labels assigned to the workflow itself will be propagated to clusters and jobs so you can do a list jobs call. For example the filter parameter could look like: filter = labels.my-label=12345 [1] https://cloud.google.com/dataproc/docs/concepts/workflows / debugging＃using_workflowmetadata [2] https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1376 [3] https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.jobs/list 这篇关于如何获取使用Dataproc工作流模板提交的jobId的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！