多处理:仅使用物理核心?

问题描述

我有一个函数foo，该函数消耗大量内存，并且我想并行运行多个实例.

I have a function foo which consumes a lot of memory and which I would like to run several instances of in parallel.

假设我有一个具有4个物理核心的CPU，每个核心都有两个逻辑核心.

Suppose I have a CPU with 4 physical cores, each with two logical cores.

我的系统有足够的内存来并行容纳4个foo实例，但不能容纳8个实例.此外，由于这8个内核中的4个仍然是逻辑内核，因此我也不希望使用所有8个内核在上述情况下都能获得很大的收益.除了只使用4个物理的.

My system has enough memory to accommodate 4 instances of foo in parallel but not 8. Moreover, since 4 of these 8 cores are logical ones anyway, I also do not expect using all 8 cores will provide much gains above and beyond using the 4 physical ones only.

所以我只想在 的4个物理内核上运行foo.换句话说，我想确保执行multiprocessing.Pool(4)(由于内存限制，我可以在此计算机上容纳的函数的并发运行的最大次数为4)将作业分派到四个物理核心(而不是为例如两个物理核心及其两个逻辑后代的组合).

So I want to run foo on the 4 physical cores only. In other words, I would like to ensure that doing multiprocessing.Pool(4) (4 being the maximum number of concurrent run of the function I can accommodate on this machine due to memory limitations) dispatches the job to the four physical cores (and not, for example, to a combo of two physical cores and their two logical offsprings).

如何在python中做到这一点?

How to do that in python?

我先前使用了multiprocessing中的代码示例，但是我与图书馆无关，因此为了避免混淆，我删除了该示例.

I earlier used a code example from multiprocessing but I am library agnostic ,so to avoid confusion, I removed that.

推荐答案

我找到了不涉及更改python模块源代码的解决方案.它使用此处建议的方法.一个人只能检查通过执行以下操作运行该脚本后，物理核心将处于活动状态:

I found a solution that doesn't involve changing the source code of a python module. It uses the approach suggested here. One can check that only the physical cores are active after running that script by doing:

lscpu

bash中的

返回:

in the bash returns:

CPU(s):                8
On-line CPU(s) list:   0,2,4,6
Off-line CPU(s) list:  1,3,5,7
Thread(s) per core:    1

[一个人可以从 python 中运行上面链接的脚本].无论如何，运行上面的脚本后，在python中键入以下命令:

[One can run the script linked above from within python]. In any case, after running the script above, typing these commands in python:

import multiprocessing
multiprocessing.cpu_count()

返回4.

这篇关于多处理:仅使用物理核心?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！