向Dask工作人员推送纯Python模块

问题描述

Dask中有一种简单的方法可以将纯Python模块推向工作人员吗？

Is there an easy way in Dask to push a pure-python module to the workers?

我在集群中有很多工作人员，我想分发我在客户端上拥有的本地模块。我知道，对于像NumPy或Python这样的大型软件包，我应该以更健壮的方式分发内容，但是我有一个经常更改的小模块，应该花很多时间去移动。

I have many workers in a cluster and I want to distribute a local module that I have on my client. I understand that for large packages like NumPy or Python I should distribute things in a more robust fashion, but I have a small module that changes frequently that shouldn't be too much work to move around.

推荐答案

或者，如果您希望在工作人员启动后将其部署到工作人员，则可以使用和

Alternative if you wish to deploy a package to the workers after they have started you can do something similar to this using Client.run and Client.restart

def deploy_env(packages):
    conda_prefix = pathlib.Path(sys.executable).parent.parent
    res = subprocess.check_output(['conda', 'install', '-p', conda_prefix] + packages)
    return res

# Run the deploy command on all the workers
result = client.run(deploy_env, packages)

# Restart all the worker processes
client.restart()

之后

在向调度程序中添加其他工作器时，此方法将不起作用。

This approach will not work when adding additional workers to the scheduler.

这篇关于向Dask工作人员推送纯Python模块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！