Python3 CTypes error loading dylib on Mac 10.14 (Mojave)

Background

I have read countless GitHub project threads and everything I can find on StackOverflow about this problem, though so far no luck. I have a Mac 10.14 box running with the stock CommandLineTools and/or Xcode. I'm trying to "port" a Python wrapper library I have written around an older C and C++ library using CTypes in Python3. It works well already on Ubuntu Linux. However, there is no end to the problems I have been coming across since moving to a Mac platform. There just doesn't seem to be an easy answer to fixing what I'm trying to do on the broken Mac OS platform right now -- at least for the uninitiated Linux person like myself.

I have one question right off the bat before I describe how I'm compiling the dylib I'm trying to load up with CTypes: Do I now need to sign my dylib somehow before I can use it on Mac 10.14? Otherwise, I guess my question boils down to how the $%^@ (and that is truncated speak for what I do mean right now) can I deal with shared / dynamic libraries on Mac with a Python C extension interface?

My preference is to not even touch Xcode and just use the stock Mac tools that come with the system out of the box. My solution must work on the command line without defering to some auto configuration magic that Xcode will give you in GUI form. Really, this is all fairly painless for what it is under Linux.

Compilation and linking

The scenario is actually more complicated than I can describe. I will just sketch what seems to me to be the relevant parts of the solution, and then let you all experts who have gotten this working before tell me the obvious missteps in between.

I'm compiling the older C/C++ source code library as a static archive first using the following gcc (read clang) options on Mac (some of them get ignored):

-O0 -march=native -force_cpusubtype_ALL -fPIC -I../include -fPIC -m64 \
-fvisibility=default -Wno-error -lc++ -lc++abi

Then I'm compiling and linking with a combination of

-Wl,-all_load $(LIBGOMPSTATIC).a $(LIBGMPSTATIC) -Wl,-noall_load \
-ldl -lpthread -lc++ -lc++abi

and

 -dynamiclib -install_name $(MODULENAME) \
 -current_version 1.0.0 -compatibility_version 1.0

to generate the dylib output.

For comparison, on Linux, the analogs to these flags that work are approximately

-Wl,-export-dynamic -Wl,--no-undefined -shared -fPIC \
                        -fvisibility=default -Wl,-Bsymbolic
-Wl,-Bstatic -Wl,--whole-archive $(LIBGOMPSTATIC).a $(LIBGMPSTATIC) -Wl,--no-whole-archive \
                       -Wl,-Bdynamic -Wl,--no-as-needed -ldl -lpthread

and

-Wl,-soname,$(MODULENAME)

The dylib output

The above procedure gives me a dylib file that I can scan with nm to see the symbols I am trying to import with CTypes. This is a good start. When I try to run the test python script to test my CTypes wrapper library, I get a SEGFAULT immediately. Since gdb is apparently useless on Mac these days (sorry), I used the stock llvm to load up a brew-installed python3 with extra debugging symbols:

lldb /usr/local/Cellar/python-dbg\@3.7/3.7.6_13/bin/python3
(lldb) run myscript.py
Process 75435 launched: '/usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3' (x86_64)
Process 75435 stopped
* thread #2, stop reason = exec
    frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100005000 <+0>: popq   %rdi
    0x100005001 <+1>: pushq  $0x0
    0x100005003 <+3>: movq   %rsp, %rbp
    0x100005006 <+6>: andq   $-0x10, %rsp
Target 0: (Python) stopped.
(lldb) bt
* thread #2, stop reason = exec
  * frame #0: 0x0000000100005000 dyld`_dyld_start
(lldb) c

... redacted path information ...

File "/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 442, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(GTFoldPython.dylib, 6): image not found
Process 75435 exited with status = 1 (0x00000001)

I do have each of the environment variables PYTHONAPPSDIR=/usr/local/Cellar/python-dbg@3.7/3.7.6_13, PYTHONPATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH set to correct paths.

So the question is what do I try next to get this working? Many, many hours of thanks in advance!

In my case, the issue turned out to be two things.

The first is that I was running my python script with a different version of python than the C extensions were linked with. For example, the following is the output of my python3-config --ldflags command:

-L/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7dm-darwin -lpython3.7dm -ldl -framework CoreFoundation

So running it with /usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3 caused errors for me. This can be resolved by running the script with /usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3.7dm. Not an obvious fix given that brew installs each with an only slightly modified tap formula.

Second, in my C code, I am frequently writing to an extern'ed character buffer that lives on the stack. When this happens, the default clang stack protection mechanisms throw a SIGABRT at the script. To avoid this, you can recompile by passing the following flags to both the compiler and linker (might be more disabling than is actually needed):

-fno-stack-check -fno-stack-protector -no-pie -D_FORTIFY_SOURCE=0

With these two fixes in place my script runs. And still crashes for other reasons related to multithreading with Python in C. However, this is to be expected, but still has yet to show up in my testing on Linux.

Thanks to @l'L'l for helping me to work through this.