Python callback invocation from C++ fails on native site

1

I have a shared library (DLL) in C++ with some C-style API which I use from Python. There is a function which takes a C callback as an argument. As far as I understood, Python extension module (a separate DLL) is required to do that. This module shall pass a native "proxy" callback to API, then call Python object from that callback. I try do use it as follows (it fails):

from ctypes import CDLL
from test_extension import set_python_callback

def callback():
    pass

if __name__ == '__main__':
    library = CDLL('test_dll.shared-library')
    set_python_callback(library, callback)
    library.trigger_callback()

I'm using Windows 8 x64 environment with Python 2.7.6 (32 bit) and MinGW 32-bit compiler (mingw32). Python extension is built by distutils. API library can be changed if necessary, however, Python-specific code has to be kept in a separate library.

Here is my code, reduced. All error-checking was removed, however, when performed it showed no errors either from Python API or from Windows API.

API library:

/* typedef void (*callback_t)(); */
static callback_t s_callback = nullptr;

void DLL_PUBLIC set_callback(callback_t callback) {
    s_callback = callback;
}

void DLL_PUBLIC trigger_callback() {
    s_callback(); // <--------------------------(1)
}

Python extension module function set_python_callback(library, callback) takes a ctypes.CDLL object and Python callable. It then extracts native DLL handle from the former:

PyObject* handle_object = PyObject_GetAttrString(library, "_handle");
const HMODULE handle = static_cast<const HMODULE>(
    PyLong_AsVoidPtr(handle_object));
const native_set_callback_t native_api_routine =
    reinterpret_cast<const native_set_callback_t>(
        GetProcAddress(handle, "set_callback"));
native_api_routine(common_callback);

C function common_callback() is implemented as follows:

extern "C" void DLL_PUBLIC common_callback() {
    // <--------------------------------------- (2)
    PyObject *arguments = Py_BuildValue("()");
    PyObject *callback_result = PyEval_CallObject(s_callback, arguments);
    Py_DECREF(arguments);
    Py_XDECREF(callback_result);
}

The error message says:

Invoking callback from DLL... Traceback (most recent call last):
  File "script.py", line 14, in <module>
    library.trigger_callback()
WindowsError: exception: access violation reading 0x00000028

Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute. The strange thing is, changing the way common_callback() call Python code (e. g. passing Py_None instead of empty tuple) changes the address in the error message.

Maybe this is somehow related to the calling convention, but I have no idea where it's wrong exactly or how to fix it.

python
c++
callback
shared-libraries
ctypes
asked on Stack Overflow Mar 9, 2014 by PlushBeaver • edited Jul 24, 2016 by J.J. Hakala

1 Answer

1

Solved the issue by registering the thread for GIL as described in documentation.

Please note that original API DLL, Python extension DLL, and the script were executed in single-threaded mode and no additional thread had been created, either Python-managed, or not. However, there had been a synchronization issue for sure, because using Py_AddPendingCall() also worked. (Using is is discouraged; I found it analyzing signals module sources, it lead to the solution above.)

This statement of mine was incorrect:

Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute.

I was mislead by the fact that some of Python API functions still worked (e. g. Py_BuildValue or Py_Initialize), but most didn't (e. g. PySys_WriteStdout, PyObject_Call, etc).

answered on Stack Overflow Mar 9, 2014 by PlushBeaver

User contributions licensed under CC BY-SA 3.0