GoogleCloudStorageDownloadOperator“任务已退出，返回代码为-6”

问题描述

我是新手，我正在尝试使用GoogleCloudStorageDownloadOperator做一些简单的事情：

I am new to airflow and I am trying something simple with GoogleCloudStorageDownloadOperator:

default_args = {
    'start_date': airflow.utils.dates.days_ago(0),
    'schedule_interval': None,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'params': {
        'work_dir': '/tmp'
    }
}

dag = DAG(
    'foo',
    default_args=default_args,
    description='This is foobar',
    schedule_interval=timedelta(weeks=1),
    dagrun_timeout=timedelta(minutes=60))

mock_download = GoogleCloudStorageDownloadOperator(
    task_id='download-foo-from-gcp',
    bucket='foo-data',
    object='{% if (task_instance.pid % 2 == 0) %}foo{% else %}bar{% endif %}/data.tar.gz',
    filename='{{ params.work_dir }}/data.tar.gz',
    google_cloud_storage_conn_id='google_cloud_default',
    dag=dag
)

虽然我可以跑步例如，在PyCharm中执行此任务（使用 airflow test ），从Web界面触发（预定）后，它始终会失败。至少可以说，日志中的错误消息完全没有用：

While I can run this task in PyCharm for example (using airflow test), it fails all the time when triggered from the web interface (scheduled). The error message in the log is completely useless, to say the least:

...
[2020-01-09 17:04:18,871] {gcs_download_operator.py:86} INFO - Executing download: crunchbase-mock-data, foo/data.tar.gz, /tmp/data.tar.gz
[2020-01-09 17:04:28,751] {logging_mixin.py:112} INFO - [2020-01-09 17:04:28,751] {local_task_job.py:103} INFO - Task exited with return code -6

有人能对此有所了解吗？ -6应该是什么意思？有没有办法查看那里发生的事情的更多细节？

Can anyone shed any light on this? What the heck is -6 supposed to mean? Is there a way to see a little more details about what happened there?

推荐答案

我在。

Mac OS High Sierra（或更高版本）？如果是这样，请参考。它解决了我的问题。

Are you on Mac OS High Sierra (or above)? If so, refer https://stackoverflow.com/a/52230415/4434664. It solved my issue.

基本上，气流测试 / PyCharm 仅在进程内运行任务，但是调度程序将启动一个工作进程，该工作进程将调用 fork（），显然，High Sierra引入了一些新的安全更改，这些更改已被打破 fork（）在python中的用法。

Basically, airflow test/PyCharm merely runs the task in-process, but the scheduler would start a worker process which would call fork(), and apparently, High Sierra introduced some new security changes that's breaking fork() usages in python.

这也引起了ansible问题。请参考

This also caused problems in ansible. Refer https://github.com/ansible/ansible/issues/32499#issuecomment-341578864

这篇关于GoogleCloudStorageDownloadOperator“任务已退出，返回代码为-6”的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！