Python ctypes cdll.LoadLibrary，实例化一个对象，执行其方法，截断私有变量地址

问题描述

我用c写了一个dll库，用vs2017 64位编译，尝试用python3.6 64位加载.然而，对象的成员变量的地址被截断为 32 位.

这是我编译为 sim.dll 的 sim.c 文件:

类检测器{民众:探测器();void process(int* pin, int* pout, int n);私人的:int member_var；};探测器::探测器(){memset(&member_var, 0, sizeof(member_var));myfile.open("addr_debug.txt");我的文件<

这是我的python脚本:

from ctypes import cdlllib = cdll.LoadLibrary(r'sim.dll')类检测器(对象):def __init__(self):self.obj = lib.Detector_new()定义过程(self，pin，pout，n):lib.Detector_process(self.obj,pin, pout, n)检测器 = 检测器()n = 1024a = np.arange(n, dtype=np.uint32)b = np.zeros(n, dtype=np.int32)aptr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_int))bptr = b.ctypes.data_as(ctypes.POINTER(ctypes.c_int))检测器进程(aptr，bptr，n)

这是addr_debug.txt中member_var的地址:

member_var 初始化地址:0000025259E123C4member_var 进程地址:0000000059E123C4

所以访问它会触发内存访问错误:

OSError:异常:访问冲突读取 0000000059E123C4

我试图理解这个问题的一些尝试:

• 将 member_var 定义为 public 而不是 private，不是帮助，地址仍然被截断.
• 定义member_var为全局变量，那么地址就ok了.所以我猜在将对象返回给 python 或将对象传递回 dll 时会发生 member_var 地址截断.

解决方案

始终(正确)为 C 中定义的函数指定 argtypes 和 restype，否则(C89 风格)它们将默认为 int(通常是 32bit)，生成！！！未定义的行为！！！.在64 位 上，地址(大于2 GiB)将被截断(这正是您遇到的情况).检查 [SO]:C 函数调用从 Python 通过 ctypes 返回不正确的值(@CristiFati 的回答) 了解更多详情.

此外，在遇到问题时，不要忘记 [Python 3.Docs]: ctypes - Python 的外部函数库.

下面是您的代码的改编版本.

detector.cpp:

#include #include #include <fstream>#define C_TAG来自C"#define PRINT_MSG_2SP(ARG0, ARG1) printf("%s - [%s] (%d) - [%s]: %s: 0x%0p
", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1)使用 std::endl;std::ofstream 输出文件；类检测器{民众:探测器();void process(int *pIn, int *pOut, int n);私人的:int m_var;};探测器::探测器(): m_var(0) {outFile.open(addr_debug.txt");输出文件

code.py:

导入系统从 ctypes 导入 CDLL、指针、c_int, c_void_p将 numpy 导入为 npsim_dll = CDLL(./sim.dll")detector_new_func = sim_dll.DetectorNewdetector_new_func.restype = c_void_pdetector_process_func = sim_dll.DetectorProcessdetector_process_func.argtypes = [c_void_p, POINTER(c_int), POINTER(c_int), c_int]Detector_delete_func = sim_dll.DetectorDeletedetector_delete_func.argtypes = [c_void_p]类检测器():def __init__(self):self.obj =detector_new_func()定义过程(自我，引脚，噘嘴，n):detector_process_func(self.obj, pin, pout, n)def __del__(self):检测器_delete_func(self.obj)定义主():检测器 = 检测器()n = 1024a = np.arange(n, dtype=np.uint32)b = np.zeros(n, dtype=np.int32)aptr = a.ctypes.data_as(POINTER(c_int))bptr = b.ctypes.data_as(POINTER(c_int))检测器进程(aptr，bptr，n)如果 __name__ == __main__":打印(Python {:s} on {:s}
".format(sys.version, sys.platform))主要的()

注意事项:

• 正如我在开头所说的，问题是没有指定 argtypes 和 restype(例如对于 DetectorNew:comment detector_new_func.restype = c_void_p，你又会遇到这个问题)
• 问题中的代码缺少部分(#includes、imports、...)，还有一些语法错误，因此无法编译，因此不遵循[SO]: How to create a Minimal, Complete, and Verifiable example (mcve) 指南.请在询问时确保 mcve
• 你分配的对象(new Detector())，也必须被释放(否则会产生内存泄漏)，所以我加了一个函数(DetectorDelete - 这样做)，它是从 (Python) Detector 的析构函数调用的
• 其他(非关键)更改(标识符重命名、一些重构、打印到 stdout、...)

输出:

(py35x64_tes1) e:WorkDevStackOverflowq052268294>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>(py35x64_tes1)&vc&vcvarsall.bat"x64(py35x64_test) e:WorkDevStackOverflowq052268294>dir/b代码.py检测器.cpp(py35x64_test) e:WorkDevStackOverflowq052268294>cl/nologo/DDLL/EHsc detection.cpp/link/DLL/OUT:sim.dll检测器.cpp创建库 sim.lib 和对象 sim.exp(py35x64_test) e:WorkDevStackOverflowq052268294>dir/b代码.py检测器.cpp检测器.objsim.dllsim.expsim.lib(py35x64_test) e:WorkDevStackOverflowq052268294>e:WorkDevVEnvspy35x64_testScriptspython.exe"./code.pyWin32 上的 Python 3.5.4(v3.5.4:3f56838，2017 年 8 月 8 日，02:17:05)[MSC v.1900 64 位 (AMD64)]来自 C - [detector.cpp] (28) - [Detector::Detector]: &m_var: 0x0000020CE366E270来自 C - [detector.cpp] (34) - [Detector::process]: &m_var: 0x0000020CE366E270

I wrote a dll library in c, compile with vs2017 64-bit, and try to load it with python3.6 64-bit. However the object's member variable's address got truncated to 32-bit.

Here's my sim.c file, which is compiled to sim.dll:

class Detector {
public:
    Detector();
    void process(int* pin, int* pout, int n);

private:
    int member_var;
};

Detector::Detector()
{
    memset(&member_var, 0, sizeof(member_var));
    myfile.open("addr_debug.txt");
    myfile << "member_var init address: " << &member_var << endl;
}
void Detector::process(int* pin, int* pout, int n);
{
    myfile << "member_var process address: " << &member_var << endl;
    myfile.close();
}

#define DllExport   __declspec( dllexport )

extern "C" {
    DllExport Detector* Detector_new() { return new Detector(); }
    DllExport void Detector_process(Detector* det, int* pin, int* pout, int n)
    {
        det->process(pin, pout, n);
    }
}

Here's my python script:

from ctypes import cdll
lib = cdll.LoadLibrary(r'sim.dll')

class Detector(object):
    def __init__(self):
        self.obj = lib.Detector_new()

    def process(self,pin, pout, n):
        lib.Detector_process(self.obj,pin, pout, n)

detector = Detector()

n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)

aptr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
bptr = b.ctypes.data_as(ctypes.POINTER(ctypes.c_int))

detector.process(aptr, bptr, n)

Here's the address of the member_var in addr_debug.txt:

member_var init address:    0000025259E123C4
member_var process address: 0000000059E123C4

So accessing it trigger memory access error:

OSError: exception: access violation reading 0000000059E123C4

---

Some attempts I tried to understand the issue:

• Define member_var as public instead of private, not help, address still truncated.
• Define member_var as global variable, then the address is ok. So I guess the member_var address truncation happens either when returning the object to python, or passing the object back to dll.

解决方案

Always (CORRECTLY) specify argtypes and restype for functions defined in C, otherwise (C89 style) they will default to int (generally 32bit), generating !!! Undefined Behavior !!!. On 64bit, addresses (larger than 2 GiB) will be truncated (which is exactly what you're experiencing). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for more details.

Also, when running into issues, don't forget about [Python 3.Docs]: ctypes - A foreign function library for Python.

Below it's an adapted version of your code.

detector.cpp:

#include <stdio.h>
#include <memory.h>
#include <fstream>

#define C_TAG "From C"
#define PRINT_MSG_2SP(ARG0, ARG1) printf("%s - [%s] (%d) - [%s]:  %s: 0x%0p
", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1)


using std::endl;

std::ofstream outFile;


class Detector {
    public:
        Detector();
        void process(int *pIn, int *pOut, int n);

    private:
        int m_var;
};


Detector::Detector()
: m_var(0) {
    outFile.open("addr_debug.txt");
    outFile << "m_var init address: " << &m_var << endl;
    PRINT_MSG_2SP("&m_var", &m_var);
}

void Detector::process(int *pIn, int *pOut, int n) {
    outFile << "m_var process address: " << &m_var << endl;
    outFile.close();
    PRINT_MSG_2SP("&m_var", &m_var);
}


#define SIM_EXPORT __declspec(dllexport)

#if defined(__cplusplus)
extern "C" {
#endif

    SIM_EXPORT Detector *DetectorNew() { return new Detector(); }
    SIM_EXPORT void DetectorProcess(Detector *pDet, int *pIn, int *pOut, int n) {
        pDet->process(pIn, pOut, n);
    }
    SIM_EXPORT void DetectorDelete(Detector *pDet) { delete pDet; }

#if defined(__cplusplus)
}
#endif

code.py:

import sys
from ctypes import CDLL, POINTER,
    c_int, c_void_p
import numpy as np


sim_dll = CDLL("./sim.dll")

detector_new_func = sim_dll.DetectorNew
detector_new_func.restype = c_void_p

detector_process_func = sim_dll.DetectorProcess
detector_process_func.argtypes = [c_void_p, POINTER(c_int), POINTER(c_int), c_int]

detector_delete_func = sim_dll.DetectorDelete
detector_delete_func.argtypes = [c_void_p]


class Detector():
    def __init__(self):
        self.obj = detector_new_func()

    def process(self, pin, pout, n):
        detector_process_func(self.obj, pin, pout, n)

    def __del__(self):
        detector_delete_func(self.obj)


def main():
    detector = Detector()

    n = 1024
    a = np.arange(n, dtype=np.uint32)
    b = np.zeros(n, dtype=np.int32)

    aptr = a.ctypes.data_as(POINTER(c_int))
    bptr = b.ctypes.data_as(POINTER(c_int))

    detector.process(aptr, bptr, n)


if __name__ == "__main__":
    print("Python {:s} on {:s}
".format(sys.version, sys.platform))
    main()

Notes:

• As I stated at the beginning, the problem was argtypes and restype not being specified (e.g. for DetectorNew: comment detector_new_func.restype = c_void_p, and you'll run into the problem again)
• Code in the question is missing parts (#includes, imports, ...), also there are some syntax errors, so it doesn't compile, and therefore doesn't follow [SO]: How to create a Minimal, Complete, and Verifiable example (mcve) guidelines. Please when make sure to have mcve when asking
• The object that you allocate (new Detector()), must also be deallocated (otherwise, it will generate a memory leak), so I added a function (DetectorDelete - to do that), which is called from (Python) Detector's destructor
• Other (non critical) changes (identifiers renaming, a bit of refactoring, printing to stdout, ...)

Output:

这篇关于Python ctypes cdll.LoadLibrary，实例化一个对象，执行其方法，截断私有变量地址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！