问题描述
我的代码一次可以准备1个，我想将其升级为精美的多任务处理程序.我正在寻求有关我可以用来实现目标的帮助.
I have my codes ready for 1 at a time performance, I wanna upgrade it to something fancy, multi-tasking. I am seeking helps about what I can use to achieve my goal.
我的代码按以下顺序执行:解析多页，解析多帖子，解析多图像.我试图用pool.map()进行多页处理，这是Daemonic的KeyError不能有子进程的结果.我对这种多任务处理程序的理解是，解析页面速度很快，解析帖子和图像的时间可能非常长.
my codes performs in this order: parsing multi-pages, parsing multi-posts, parsing multi-images. I tried to do multi-pages with multi-processing with pool.map(), it came out with KeyError of Daemonic can't have children processes. My understanding of this multi-tasking procedure is that parsing pages are fast, parsing posts and images can be really long.
如果我在一页上同时解析帖子和图片，该怎么办?
What if I do parsing posts and parsing images together on single page, can it be allowed?
我应该使用哪个模块?线?多处理?异步吗?最近，我经历了很多事情，我在为应该使用的东西而苦苦挣扎.
Which modules should i use to do so? thread? multiprocessing? asyncio? I went through a lot lately, I am struggling with what I should use.
推荐答案
因此，从我的头顶上可以看到两件事.
So off the top of my head you can look at 2 things.
1)Asyncio(请注意，此示例使用线程化，并且不是线程安全的，特别是asyncio.gather函数)
1) Asyncio (be careful this example uses threading and is not thread safe specifically the function asyncio.gather)
import asyncio
for work in [1,2,3,4,5]:
tasks.append(method_to_be_called(work))
results = await asyncio.gather(*tasks)
2)Asyncio +多处理 https://github.com/jreese/aiomultiprocess
2) Asyncio + multiprocessinghttps://github.com/jreese/aiomultiprocess
这篇关于具有多处理，线程或Asyncio的多任务，具体取决于方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！