本文介绍了Python 中的 Multiprocessing 中的多线程的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在使用 concurrent.futures 模块进行多处理和多线程.我在配备 16GB RAM、intel i7 第 8 代处理器的 8 核机器上运行它.我在 Python 3.7.2 甚至 Python 3.8.2 上试过这个

I am using concurrent.futures module to do multiprocessing and multithreading. I am running it on a 8 core machine with 16GB RAM, intel i7 8th Gen processor. I tried this on Python 3.7.2 and even on Python 3.8.2

import concurrent.futures
import time

获取列表并将每个元素乘以 2

takes list and multiply each elem by 2

def double_value(x):
  y = []
  for elem in x:
    y.append(2 *elem)
  return y

将元素乘以 2

def double_single_value(x):
  return 2* x

定义一个

import numpy as np
a = np.arange(100000000).reshape(100, 1000000)

运行多个线程并将每个元素乘以 2 的函数

function to run multiple thread and multiple each elem by 2

 def get_double_value(x):
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(double_single_value, x)
  return list(results)

下面显示的代码运行了 115 秒.这仅使用多处理.这段代码的 CPU 利用率为 100%

t = time.time()

with concurrent.futures.ProcessPoolExecutor() as executor:
  my_results = executor.map(double_value, a)
print(time.time()-t)

下面的函数花费了超过 9 分钟并消耗了系统的所有 RAM,然后系统杀死了所有进程.这段代码中的 CPU 利用率也没有达到 100% (~85%)

t = time.time()
with concurrent.futures.ProcessPoolExecutor() as executor:
  my_results = executor.map(get_double_value, a)

print(time.time()-t)

我真的很想明白:

(我已经阅读了许多描述多处理和多线程的帖子,我得到的症结之一是多线程用于 I/O 进程和多处理用于 CPU 进程?)

(I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes ? )

推荐答案

正如你所说:我已经阅读了许多描述多处理和多线程的帖子,我得到的症结之一是多线程是为了我/O 进程和 CPU 进程的多处理".

As you say: "I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes".

你需要弄清楚,你的程序是IO-bound还是CPU-bound,然后应用正确的方法来解决你的问题.随机或同时使用各种方法通常只会让事情变得更糟.

You need to figure out, if your program is IO-bound or CPU-bound, then apply the correct method to solve your problem. Applying various methods at random or all together at the same time usually makes things only worse.

这篇关于Python 中的 Multiprocessing 中的多线程的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-20 06:34