Issues debugging 3dsmax 2016 due to object memory layout

[Anonymous]

Hello All,

I was running some tests on my plugins with VS2013 and compiled by code with debug flags and started encountering an error that I could not reproduce on a release version of my libraries. At one point in my code I called IParamBlock2 -> getValue(PBBitmap) and would try to load it and do various operations on the bitmap. In my release it worked fine, the bitmap would load, i could use it as expected. In debug I would crash with some improperly instantiated bitmap, this issue did not occur in previous iterations of our plugins (max08/max11).

Soon it was realized that when getting the values in a debug configuration the bm member of the PBBitmap would not come instantiated as NULL instead would be a pointer with a no mans land address (0xabababab). After various amounts of poking around we decided to check on the sizeof the classes in both debug and release and got the following:

debug  | PBBitmap:704 BitmapInfo:696 Bitmap:96
release | PBBitmap:696 BitmapInfo:688 Bitmap:96

So we dug a bit further into BitmapInfo and found this in the MaxSDK::Util::Path which is a member of the class

class PathImpl;
const AutoPtr<PathImpl> mImpl;
#if defined(_DEBUG)
//! Useful for inspecting the path in the debugger
const MCHAR* mdata;
#endif

So it seems that when we compile under debug, VS assumes that we have this mdata and starts pulling all sorts of strange quirks with the code while our included libraries are only release compiled.

Assuming we can't get the debug libraries for max, is there some sort of workaround for this issue?

I can delete the #if code from my local header files but i wonder why this code was left here anyways

[denis_grigor · Accepted solution]

Hi, 

In general, when developing C++ plugins for applications like 3ds Max, a developer should not forget about things like ABI.
You must keep in mind that a plugin developer is compiling his plugin against an application compiled in release build (in our case 3ds Max), and since the allocation/dealocation procedure is different in case of release and debug procedure - it is understandable that if you compile your plugin in debug mode, it will result in crashes when the memory allocated in "release way" is dealocated in "debug way" or vice-verso.

To solve this problem, there is a special Hybrid configuration, which in essence is a mix of release and debug configuration and whose main idea is to ensure the same memory allocation/dealocation procedure is used by your plugin and 3ds Max.

Thus, the recommended way of debugging your plugins is to use the Hybrid configuration along with debug symbols as described here.

Regarding the existance of _DEBUG within the SDK, this is because the very same SDK is used internally by 3ds Max itself, which could be seen as "dogfooding".

[Anonymous]

Thank you for this reply,

I was not aware of the Hyrid configuration as I am returning to a project that was originally designed for Max2011 and at the time the original developer had already been working in a debug configuration where the sdk would allocate and deallocate in its way but since the header didnt change VS was able to work around it. Since here we had the header change while, of course, the compiled application uses release libraries, a conflict in reading occured by exactly the size of that one additional pointer. 

my last question in the original comment was whether this was left intentionally because it seemed that this was the only _debug check in the sdk and going back to the older project we had, I saw 0 occurences of _debug in the included headers and just made the assumption that there were specifically compiled headers that did not have these checks. Now i see that this was as designed because of the existance of the hybrid option.

[michaelsonbritt]

I request that the use of defined(_DEBUG) in class Path be treated as a bug and logged.  This clause appears in only two places in the SDK, thus it is not very "standard" and creates a tricky pitfall.  The solution could be a different define, like defined(MAX_DEBUG).  Long term, a potentially better solution is to always define Path::mdata even in release builds, but only populate this value in debug builds (would require an SDK break through).

Otherwise, every plug-in developer must avoid compiling in debug configuration, instead using "release-without-optimization" (hybrid).  It's standard practice for plugin developers to compile in debug, regardless of whether 3ds Max is a release configuration app.  To understand that, remember that developers are working with many different apps (including non-Autodesk apps like Blender), not just 3ds Max, and in a complex build environment.  Imagine if all 3dsmax projects needed to be changed because of one pitfall in one third-party dependency, it would be costly.

Another solution for third-party developers is to change the header to remove the defined(_DEBUG) clause.  But that would need to be done every time setting up a new machine or reinstalling the SDK, not really a better solution.

Also, developers need to see the value of Path objects while debugging.  Right now that's very tedious because mImpl is private.  Is there a standard way to see the Path value in a debug watch window, other than changing the source code and adding to call GetCStr()?

Thank you for the help,

Michaelson Britt

[michaelsonbritt]

Update, there is a way to inspect MaxSDK::Util::Path objects in the Visual Studio watch window.  Use the following, replacing myPath with the path object,

(wchar_t*)(8+(uint64_t)(myPath.mImpl.mPtr))

More detail is here.

Cheers,

Michaelson Britt

[Anonymous]

I can understand if they decide to not remove the _DEBUG define in their headers, as plugin developers we are expected to work within their API specifications.

I think the debugging of Path objects should be written in the headers as a comment instead of requiring lost time to researching. 

as for your comment "Another solution for third-party developers is to change the header to remove the defined(_DEBUG) clause.  But that would need to be done every time setting up a new machine or reinstalling the SDK, not really a better solution." Well this is what I did for my team at my company, and created a quick batch script that does this and is explicitly mentioned in documentation that this should be run with sdk installation. it was the easiest approach for our existing debug configuration. The hybrid configuration actually took much more work to get running than expected because of the simultaneous need to include my engine header/sources and the 3dsmax headers.

sidenote: Also I am the same user that originally posted this issue, just using a different account due to company license