Python多线程多解释器C API

问题描述

我正在使用Python的C API，但是很难理解一些特殊情况.我可以对其进行测试，但它似乎容易出错且很耗时.所以我来这里看看是否有人已经这样做了.

I'm playing around with the C API for Python, but it is quite difficult to understand some corner cases. I could test it, but it seems a bug-prone and time consuming. So I come here to see if somebody has already done this.

问题是，这是使用子解释器管理多线程，线程和子解释器之间没有直接关系的正确方法吗?

The question is, which is the correct way to manage a multi-thread with sub-interpreters, with no direct relation between threads and sub-interpreters?

Py_Initialize();
PyEval_InitThreads(); /* <-- needed? */
_main = PyEval_SaveThread(); /* <-- acquire lock? does it matter? */
/* maybe do I not need it? */
i1 = Py_NewInterpreter();
i2 = Py_NewInterpreter();

我是否使用互斥锁?需要使用锁吗?线程化函数应类似于以下内容:(线程是非python，可能是POSIX线程)

Do I use a mutex? Is required to use locks? The threaded function should be something like the following: (The threads are non-python, probably POSIX threads)

线程1

_save = PyThreadState_Swap(i1);
  // python work
PyThreadState_Restore(_save);

线程2 (几乎完全相同)

_save = PyThreadState_Swap(i1);
  // python work
PyThreadState_Restore(_save);

线程3 (几乎完全相同，但带有子解释器i2)

Thread3 (almost identical, but with the sub-interpreter i2)

_save = PyThreadState_Swap(i2);
  // python work
PyThreadState_Restore(_save);

这是正确的吗?这是我要实现的一般情况吗?有比赛条件吗?

Is this correct? Is this the general case for which I want to achieve? Are there race conditions?

谢谢！

推荐答案

Python中的

子解释器没有充分的文档记录，甚至没有得到很好的支持.以下是我最大的理解.在实践中似乎效果很好.

Sub interpreters in Python are not well documented or even well supported. The following is to the best of my undestanding. It seems to work well in practice.

在Python中处理线程和子解释器时，有两个重要的概念需要理解.首先，Python解释器不是真正的多线程.它具有全局解释器锁(GIL)，几乎所有Python操作都需要获取全局解释器锁(该规则有一些罕见的例外).

Threre are two important concepts to understand when dealing with threads and sub interpreters in Python. First, the Python interpreter is not really multi threaded. It has a Global Interpreter Lock (GIL) that needs to be acquired to perform almost any Python operation (there are a few rare exceptions to this rule).

第二，线程和子解释器的每个组合都必须具有自己的线程状态.解释器会为其创建的每个线程创建一个线程状态，但是如果您想从该解释器未创建的线程中使用Python，则需要创建一个新的线程状态.

Second, every combination of thread and sub interpreter has to have its own thread state. The interpreter creates a thread state for every thread managed by it, but if you want to use Python from a thread not created by that interpreter, you need to create a new thread state.

首先，您需要创建子解释器:

First you need to create the sub interpreters:

初始化Python

Py_Initialize();

初始化Python线程支持

如果您打算从多个线程调用Python，则为必需.此调用还将获取GIL.

Required if you plan to call Python from multiple threads). This call also acquires the GIL.

PyEval_InitThreads();

保存当前线程状态

我本可以使用PyEval_SaveThread()，但是它的副作用之一是释放了GIL，然后需要重新获取.

I could have used PyEval_SaveThread(), but one of its side effects is releasing the GIL, which then needs to be reacquired.

PyThreadState* _main = PyThreadState_Get();

创建子解释器

PyThreadState* ts1 = Py_NewInterpreter();
PyThreadState* ts2 = Py_NewInterpreter();

还原主解释器线程状态

PyThreadState_Swap(_main);

我们现在有两个用于子解释器的线程状态.这些线程状态仅在创建它们的线程中有效.每个要使用子解释器之一的线程都需要为该线程和解释器的组合创建一个线程状态.

We now have two thread states for the sub interpreters. These thread states are only valid in the thread where they were created. Every thread that wants to use one of the sub interpreters needs to create a thread state for that combination of thread and interpreter.

使用新线程中的子解释器

这是在不是由子解释器创建的新线程中使用子解释器的示例代码.新线程必须获取GIL，为该线程和解释器组合创建新的线程状态，并使其成为当前线程状态.最后，必须进行相反的操作以进行清理.

Here is an example code for using a sub interpreter in a new thread that is not created by the sub interpreter. The new thread must acquire the GIL, create a new thread state for the thread and interpretere combination and make it the current thread state. At the end the reverse must be done to clean up.

void do_stuff_in_thread(PyInterpreterState* interp)
{
    // acquire the GIL
    PyEval_AcquireLock();

    // create a new thread state for the the sub interpreter interp
    PyThreadState* ts = PyThreadState_New(interp);

    // make ts the current thread state
    PyThreadState_Swap(ts);

    // at this point:
    // 1. You have the GIL
    // 2. You have the right thread state - a new thread state (this thread was not created by python) in the context of interp

    // PYTHON WORK HERE

    // release ts
    PyThreadState_Swap(NULL);

    // clear and delete ts
    PyThreadState_Clear(ts);
    PyThreadState_Delete(ts);

    // release the GIL
    PyEval_ReleaseLock();
}

在新线程中使用子解释器(Python 3.3之后)

先前的do_stuff_in_thread()仍适用于所有当前的Python版本.但是，Python 3.3弃用了PyEval_AcquireLock()/PyEval_ReleaseLock()，这带来了一些难题.

The previous do_stuff_in_thread() still works with all current Python versions. However, Python 3.3 deprecated PyEval_AcquireLock()/PyEval_ReleaseLock(), which resulted in a bit of a conundrum.

记录的释放GIL的唯一方法是调用PyEval_ReleaseThread()或PyEval_SaveThread()，这两个都需要线程状态，而清理和删除当前线程状态则需要保留GIL.这意味着一个人可以释放GIL或清除线程状态，但不能同时清除两者.

The only documented way to release the GIL is by calling PyEval_ReleaseThread() or PyEval_SaveThread(), both of which require a thread state, while cleaning and deleting the current thread state requires the GIL to be held. That means that one can either release the GIL or clean up the thread state, but not both.

幸运的是，有一个解决方案- PyThreadState_DeleteCurrent() 删除当前线程状态，然后释放GIL. [此API自3.9起才被记录，但至少从Python 2.7起就存在]

Fortunately, there is a solution - PyThreadState_DeleteCurrent() deletes the current thread state and then releases the GIL. [This API has only been documented since 3.9, but it existed since Python 2.7 at least]

此修改后的do_stuff_in_thread()还可用于所有当前的Python版本.

This modified do_stuff_in_thread() also works with all current Python versions.

void do_stuff_in_thread(PyInterpreterState* interp)
{
    // create a new thread state for the the sub interpreter interp
    PyThreadState* ts = PyThreadState_New(interp);

    // make it the current thread state and acquire the GIL
    PyEval_RestoreThread(ts);

    // at this point:
    // 1. You have the GIL
    // 2. You have the right thread state - a new thread state (this thread was not created by python) in the context of interp

    // PYTHON WORK HERE

    // clear ts
    PyThreadState_Clear(ts);

    // delete the current thread state and release the GIL
    PyThreadState_DeleteCurrent();
}

现在每个线程都可以执行以下操作:

Now each thread can do the following:

线程1

do_stuff_in_thread(ts1->interp);

线程2

do_stuff_in_thread(ts1->interp);

线程3

do_stuff_in_thread(ts2->interp);

调用Py_Finalize()会销毁所有子解释器.或者，可以手动销毁它们.这需要使用创建子解释器时创建的线程状态在主线程中完成.最后，使主解释器线程状态为当前状态.

Calling Py_Finalize() destroys all sub interpreters. Alternatively they can be destroyed manually. This needs to be done in the main thread, using the thread states created when creating the sub interpreters. At the end make the main interpreter thread state the current state.

// make ts1 the current thread state
PyThreadState_Swap(ts1);
// destroy the interpreter
Py_EndInterpreter(ts1);

// make ts2 the current thread state
PyThreadState_Swap(ts2);
// destroy the interpreter
Py_EndInterpreter(ts2);

// restore the main interpreter thread state
PyThreadState_Swap(_main);

我希望这可以使事情变得更清楚.

I hope this make things a bit clearer.

我在 github 和另一个也在GitHub上(Python 3.3变体).

I have a small complete example written in C++ on github, and another also on github (post Python 3.3 variant).

这篇关于Python多线程多解释器C API的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！