Boost Python, Mac Osx Yosemite, Maya's Python compatibilities?

[daniele.niero]

Hello,

I have a simple math library written in c++ (https://bitbucket.org/daniele-niero/gmath/src), with a porting to python through boost-python.

On windows I have no problem at all as long as I compile everything (boost included) with msvc 10.0, and linking with the internal Maya's python library.

On Mac however, the internal python has been compiled with gcc 4.2.1

$ /Applications/Autodesk/maya2016/Maya.app/Contents/bin/mayapy

Python 2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:52:43)

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

So if I want to build my math library wrappers to work with Maya I must compile boost-python and my library with gcc 4.2.1

Now this is where things get tricky.
Since Maverick, gcc on Mac is not gcc anymore!

$ which gcc

/usr/bin/gcc

$ gcc --version

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer//usr --with-gxx-include-dir=/usr/include/c++/4.2.1

Apple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM 3.6.0svn)

Target: x86_64-apple-darwin14.3.0

Thread model: posix

gcc on the latest mac osx is actually clang, which makes impossible to compile a boost-python wrapper compatible with Maya's internal python.

In fact the latest python shipped with Yosemite has been compiled with clang.

$ ptyhon

Python 2.7.6 (default, Sep  9 2014, 15:04:36)

[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin

Am I missing something or am I interpreting all this data wrongly?
Do you have any suggestion?

Kind Regards,

    Dan

[warnold1]

Maya 2016, version of Python is 2.7.6  

/Applications/Autodesk/maya2016/Maya.app/Contents/bin/mayapy
Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

its actully from:  https://www.python.org/download/releases/2.7.6/

what errors are you actually getting.

[daniele.niero]

Hi there,

Yes, maybe I didn't explain myself clear enough. The python versions are indeed the same but this is not the issue, when you want to wrap c++ to python the compiler is even more important.

Maya's python has been compiled with GCC 4.2.1
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

This means that if I want to wrap my library to work with this specific python, I must compile boost and my library with the same version of gcc.

Unfortunately on the latest Mac OSXs and XCodes, GCC has been replaced with CLANG, even if GCC still being reconised as a valid command, it actually invoking Clang.

As prove of that, I posted what the Mac's Python (mac ships with multiple version of python already installed) says when launched:
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin

GCC 4.2.1 COMPATIBLE but actually built with clang-600.0.39 (Basically XCode 6).

So I haven't a GCC on my Mac to use for compiling my library and link it to Maya's python without causign a segmentation fault (works totally fine with Mac's python)

From Maya documentation is stated that:
To compile plug-ins and standalone applications on Mac OS X, you must have the following installed: Mountain Lion 10.8.5, Xcode 5.0.2 with SDK 10.8 (Mountain Lion), clang with libstdc++

Mountain Lion is quite old now (there is Maverick and Yosemite now) but that shouldn't be a problem becasue they state that you should use XCode 5 (which is just one verison behind). But XCode 5 is the Xcode that, as far as I know, dropped support for GCC and in fact they state you should use clang with libstdc++.
This for building Maya's plugins though, and it is totally reasonable, but the internal python (i need to say this again) IS COMPILED WITH AN OLD TOOLCHAIN. A toolchain that makes impossible, without Gcc 4.2.1 to wrap c/c++ to python's modules usable from Maya's python.

Now the problem is that Installing Gcc 4.2.1 manually on mac has been proven quite a challenge. When Apple drop support, they drop it for real...

If I'm interpreting all this data wrong, please advise.
If instead I have understood correctly the problem, I would suggest Autodesk updates Maya's internal python to be more compatible with the latest Mac toolchains. XCode 5 as they suggested for typical plugin development sounds fine, but that means having a python compiled with CLANG.

[warnold1]

that bit about version of Xcode used to build Maya 2016 is a bit out of date.  was Xcode 6.1 on Mavericks 10.9.5 with 10.9.sdk and libstdc++

thought all of that had been updated

Maya is a mix of components that come from many places.  some C, some C++, some Objective-C with different compilers

gcc and clang both generate object code that can be linked together.   

for c++  one sticking point is the -stdlib that is used.  either the old gcc libstdc++ or the newer clang libc++     even here they can both be used in the same application, with some caution and not mixing incompatible objects.

you haven't said what the actual build or runtime errors that you are getting.   without that info its kind of hard to know what to suggest

you say a segmentation fault, but what is the stack trace,  what is actually crashing

[daniele.niero]

It's hard to say, because if I simply run this two line of python code:

import gmath
print gmath

where gmath is actually the gmath.so library that I compiled, when in Maya, Maya just close itself, if in terminal (running the internal Maya's python) it just says "segmentation fault". I should compile all in debug, I guess, to have more information, but than I would need the python compiled in debug too, isn't it?

Now there is another thing to say. If I compile my library and link it to mac's python and boost compiled against the same python, everythign works for that specific python (but not with mayapy - maya's python)

If I compile my library linking it to maya's python and boost compiled against this python, both mac's python and maya's python fail and crash with the two lines of code above.

This is why I'm assuming is a compiler incompatibility. But I'm just guessing here.

[warnold1]

Compiling with -g is usually a good idea to have symbols available

In your ~/Library/Logs/Diagnostics directory are there related crash reports ?

Running Maya under lldb should trap at the point of error. Should at least give module level info

lldb /Applications/Autodesk/maya2016/Contents/MacOS/Maya

lldb command r to run

r

When crashes lldb command bt

bt



Sent from my iPhone

[daniele.niero]

After a very long time I had the change to go back to this issue.

I did what you asked and this is the outcome.

Fatal Python error: PyThreadState_Get: no current thread
Process 3094 stopped
* thread #1: tid = 0x19cff, 0x00007fff8e05f286 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
frame #0: 0x00007fff8e05f286 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
-> 0x7fff8e05f286 <+10>: jae 0x7fff8e05f290 ; <+20>
0x7fff8e05f288 <+12>: movq %rax, %rdi
0x7fff8e05f28b <+15>: jmp 0x7fff8e05ac53 ; cerror_nocancel
0x7fff8e05f290 <+20>: retq
(lldb) bt
* thread #1: tid = 0x19cff, 0x00007fff8e05f286 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
* frame #0: 0x00007fff8e05f286 libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff8bf7642f libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x00007fff930a3b53 libsystem_c.dylib`abort + 129
frame #3: 0x0000000152a7aa57 Python`Py_FatalError + 49
frame #4: 0x0000000152a79f39 Python`PyThreadState_Get + 28
frame #5: 0x0000000152a76d96 Python`Py_InitModule4_64 + 62
frame #6: 0x0000000152940b89 _gmath.so`init_gmath + 73
frame #7: 0x0000000109910471 Python`_PyImport_LoadDynamicModule + 177
frame #8: 0x000000010990fd88 Python`imp_load_module + 184
frame #9: 0x00000001098f4e7d Python`PyEval_EvalFrameEx + 25213
frame #10: 0x00000001098f586d Python`PyEval_EvalFrameEx + 27757
frame #11: 0x00000001098f7149 Python`PyEval_EvalCodeEx + 2137
frame #12: 0x00000001098f7266 Python`PyEval_EvalCode + 54
frame #13: 0x000000010990b970 Python`PyImport_ExecCodeModuleEx + 208
frame #14: 0x000000010990c7f2 Python`load_source_module + 514
frame #15: 0x000000010990e73f Python`import_submodule + 383
frame #16: 0x000000010990e9fa Python`load_next + 234
frame #17: 0x000000010990ef5b Python`PyImport_ImportModuleLevel + 363
frame #18: 0x00000001098ec7d3 Python`builtin___import__ + 131
frame #19: 0x000000010983f362 Python`PyObject_Call + 98
frame #20: 0x00000001098ed947 Python`PyEval_CallObjectWithKeywords + 87
frame #21: 0x00000001098f2df5 Python`PyEval_EvalFrameEx + 16885
frame #22: 0x00000001098f7149 Python`PyEval_EvalCodeEx + 2137
frame #23: 0x00000001098f7266 Python`PyEval_EvalCode + 54
frame #24: 0x0000000100bd1ee7 libCommandEngine.dylib`TpythonInterpreter::executeCommand(Tstring const&, bool, bool, TinterpreterResult*) + 903
frame #25: 0x00000001013f7daa libExtensionLayer.dylib`virtual thunk to TcommandFieldExecuterTypePython::executeCommand(Tstring const&) + 90
frame #26: 0x00000001013f7635 libExtensionLayer.dylib`QmayaCommandScrollFieldExecuter::doRun() + 341
frame #27: 0x00000001013f7e85 libExtensionLayer.dylib`QmayaCommandScrollFieldExecuter::keyPressEvent(QKeyEvent*) + 117
frame #28: 0x0000000107f29804 QtGui`QWidget::event(QEvent*) + 3300
frame #29: 0x00000001082801b7 QtGui`QFrame::event(QEvent*) + 183
frame #30: 0x0000000108302227 QtGui`QAbstractScrollArea::event(QEvent*) + 791
frame #31: 0x00000001082eb0f1 QtGui`QTextEdit::event(QEvent*) + 401
frame #32: 0x00000001013f7e05 libExtensionLayer.dylib`QmayaCommandScrollFieldExecuter::event(QEvent*) + 53
frame #33: 0x0000000107eddd7f QtGui`QApplicationPrivate::notify_helper(QObject*, QEvent*) + 271
frame #34: 0x0000000107edfc0d QtGui`QApplication::notify(QObject*, QEvent*) + 2861
frame #35: 0x00000001013b14e0 libExtensionLayer.dylib`QmayaApplication::notify(QObject*, QEvent*) + 320
frame #36: 0x0000000107c2e095 QtCore`QCoreApplication::notifyInternal(QObject*, QEvent*) + 117
frame #37: 0x0000000107ee144c QtGui`qt_sendSpontaneousEvent(QObject*, QEvent*) + 44
frame #38: 0x0000000107f4cbec QtGui`QKeyMapper::sendKeyEvent(QWidget*, bool, QEvent::Type, int, QFlags<Qt::KeyboardModifier>, QString const&, bool, int, unsigned int, unsigned int, unsigned int, bool*) + 156
frame #39: 0x0000000107f4c7c0 QtGui`QKeyMapperPrivate::translateKeyEvent(QWidget*, OpaqueEventHandlerCallRef*, OpaqueEventRef*, void*, bool) + 1136
frame #40: 0x0000000107e8b0bc QtGui`qt_dispatchKeyEvent(void*, QWidget*) + 252
frame #41: 0x0000000107e8050a QtGui`-[QCocoaView keyDown:] + 266
frame #42: 0x00007fff95a690fb AppKit`-[NSWindow _reallySendEvent:isDelayedEvent:] + 5452
frame #43: 0x00007fff953fac86 AppKit`-[NSWindow sendEvent:] + 470
frame #44: 0x0000000107e8375e QtGui`-[QCocoaWindow sendEvent:] + 110
frame #45: 0x00007fff953f78b1 AppKit`-[NSApplication sendEvent:] + 4199
frame #46: 0x0000000107e885fe QtGui`-[QNSApplication sendEvent:] + 78
frame #47: 0x00007fff95320b68 AppKit`-[NSApplication run] + 711
frame #48: 0x0000000107e91880 QtGui`QEventDispatcherMac::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 528
frame #49: 0x0000000107c2b377 QtCore`QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 487
frame #50: 0x0000000107c2e627 QtCore`QCoreApplication::exec() + 199
frame #51: 0x00000001013af15b libExtensionLayer.dylib`Tapplication::start() + 171
frame #52: 0x000000010001190d Maya`appmain() + 1869
frame #53: 0x000000010002300d Maya`main + 109
frame #54: 0x00007fff8c5515c9 libdyld.dylib`start + 1
frame #55: 0x00007fff8c5515c9 libdyld.dylib`start + 1

By the way, I was re-reading our previous discussion, and I noticed that meanwhile you where talking about Maya 2016 I was talking about Maya 2015.
Also I answered incorrectly to one post. Maya 2015 uses Python 2.7.3 no python 2.7.6, so the system python and Maya python indeed have different version.

[daniele.niero]

I switched from boost::python to swig and somehow I have been able to build a wrapper that works.
At this point I'm assuming that I was doing something wrong with boost, maybe I didn't compile it against the right python.
Whatever the cause was, I'm happier with swig as wrapper solution.

[jrb]

Hi,

 You would get this exact crash (I just ran into this) if you compiled your python module with -stdlib=libc++ (the default on Xcode6 and up). If you change it to use the old runtime with -stdlib=libstdc++ it would likely work. Not much can have happened at the time you've called Py_InitModule. I got the exact stack and error message when I accidentally compiled the new way. Nothing to do with boost at all.

 Note that the stdlib++ (still shipping with Xcode8) is frozen in time to way back when apple switched to clang so likely much older than any version you'd find on linux or any other gcc-based environment.

- James