MRichSelection causes a heap corruption when deleted.

[jmreinhart]

I am working on a custom context in C++ and I was getting crashes when using undo, specifically from a heap corruption according to the debugger.

When I run my custom toolCommand, undo it, and then do something else, the crash occurs.

I traced the issue to the process that deletes the entries from the undo stack when we do something after undoing.

The issue occurs if commandC (the instance of my custom command) has an MRichSelection as a variable. When Maya tries to delete all the commandC data, deleting the MRichSelection is causing the corruption to occur.

If the MRichSelection is the default MRichSelection() then the issue does not occur.

This issue only seems to occur with MRichSelection, not MSelectionList (those are the two classes I tested).

Specs:

Compiling using VS2017

OS: Windows 8

Graphics: tested with Intel(R) HD Graphics 4600 and NVIDIA GeForce GTX 850M (most up to date drivers on both)

Maya Version: tested on 2020 and 2022

I have attached the minimal code needed to reproduce the issue.

To trigger the crash follow the below :

import maya.mel as mel
cmds.loadPlugin('...')

# You need to run this one line at a time to actually be in component mode
cmds.polySphere()
cmds.selectMode(component=True )
cmds.select('pSphere1.vtx[0]')

# Make sure soft select is on!

mel.eval('setToolTo {}'.format('weightSculptContext1'))

# Click twice
# undo twice
# Click
# pan camera

I have also attached a video of the crash occurring.

Let me know if any other information would be helpful.

[jmreinhart]

Hi, I've now attached the crash log and a txt containing the output windows contents after I added the some print statements to check on the memory addresses.

I tried clearing the MRichSelection during the destructor but that did not work.

I also tried in DG and Parallel mode and it crashed for both.

I also ran Windows Memory Diagnostic in case it was an issue with my RAM, but it detected no issues.

[jmreinhart]

I've done some additional testing:

I tried the same code compiled for Maya 2023 and the result was the same.

Hiding the geometry that has the softSelection before using the context leads to the same results.

I tried having the MPxToolCommand instances removed from memory by setting the undo queue to be short (10) and then running the command 12 times resulted in a crash. 

I did find something that may be useful:

DO, DO, DO, DO, UNDO, DO = No crash

DO, DO, DO, DO, UNDO, DO, DO, DO, UNDO, DO = Crash

So it seems like the first undo messes up the memory in that area, and then because the MRichSelections for each command instance are stored close together in memory, the second undo accesses that messed up memory region and causes a crash.

I tried doing more DO's between the two undos, so that the two MRichSelections that get deleted are further apart in memory, (00000061023DFBD8 and 00000061023D6A58) but that did not work. 

[jmreinhart]

So I did another test, where the MPxToolCommand gets the MRichSelections using MGlobal::getRichSelection() in the redoIt method, and that did not result in a crash. So now I'm wondering if something about how I am passing the MRichSelection to the MPxToolCommand is the source of the issue.

[jmreinhart]

After some additional testing, it seems like the copy constructor of the MRichSelection is the source of the crash. 

The workaround I used was to set the richSelection in the scene, then get it using MGlobal::richSelecton and then set it back if needed.

I'll leave this post unresolved until this is fixed in the API.

[brentmc · Accepted solution]

Thanks for the detailed repro steps and investigation.

I have opened a defect for this:
MAYA-125774- MRichSelection copy constructor causes memory corruption

[jmreinhart]

This issue has now been fixed:

https://help.autodesk.com/view/MAYAUL/2024/ENU/?guid=Maya_ReleaseNotes_2024_release_notes_fixed_issu...