问题描述
我正在使用 concurrent.futures 模块进行多处理和多线程.我在配备 16GB RAM、intel i7 第 8 代处理器的 8 核机器上运行它.我在 Python 3.7.2 甚至 Python 3.8.2 上试过这个
I am using concurrent.futures module to do multiprocessing and multithreading. I am running it on a 8 core machine with 16GB RAM, intel i7 8th Gen processor. I tried this on Python 3.7.2 and even on Python 3.8.2
import concurrent.futures
import time
获取列表并将每个元素乘以 2
takes list and multiply each elem by 2
def double_value(x):
y = []
for elem in x:
y.append(2 *elem)
return y
将元素乘以 2
def double_single_value(x):
return 2* x
定义一个
import numpy as np
a = np.arange(100000000).reshape(100, 1000000)
运行多个线程并将每个元素乘以 2 的函数
function to run multiple thread and multiple each elem by 2
def get_double_value(x):
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(double_single_value, x)
return list(results)
下面显示的代码运行了 115 秒.这仅使用多处理.这段代码的 CPU 利用率为 100%
t = time.time()
with concurrent.futures.ProcessPoolExecutor() as executor:
my_results = executor.map(double_value, a)
print(time.time()-t)
下面的函数花费了超过 9 分钟并消耗了系统的所有 RAM,然后系统杀死了所有进程.这段代码中的 CPU 利用率也没有达到 100% (~85%)
t = time.time()
with concurrent.futures.ProcessPoolExecutor() as executor:
my_results = executor.map(get_double_value, a)
print(time.time()-t)
我真的很想明白:
(我已经阅读了许多描述多处理和多线程的帖子,我得到的症结之一是多线程用于 I/O 进程和多处理用于 CPU 进程?)
(I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes ? )
推荐答案
正如你所说:我已经阅读了许多描述多处理和多线程的帖子,我得到的症结之一是多线程是为了我/O 进程和 CPU 进程的多处理".
As you say: "I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes".
你需要弄清楚,你的程序是IO-bound还是CPU-bound,然后应用正确的方法来解决你的问题.随机或同时使用各种方法通常只会让事情变得更糟.
You need to figure out, if your program is IO-bound or CPU-bound, then apply the correct method to solve your problem. Applying various methods at random or all together at the same time usually makes things only worse.
这篇关于Python 中的 Multiprocessing 中的多线程的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!