.Net Plugin Causes High Memory Consumption / Leak (not in Managed Heap)

.Net Plugin Causes High Memory Consumption / Leak (not in Managed Heap)

Anonymous
Not applicable
12,101 Views
50 Replies
Message 1 of 51

.Net Plugin Causes High Memory Consumption / Leak (not in Managed Heap)

Anonymous
Not applicable

I have a very large and complex .net Plugin for AutoCAD. I'm trying to avoid explaining the workings of this plugin in detail since my question is a general one. I apologize if I have not used the terms AppDomain and Managed Heap correctly. I lack detailed knowledge of the memory allocation system in .net and how the .net plugin system in autocad is implemented, so I have done my best.

The plugin utilizes a set of complex dynamic blocks. The properties of these dynamic blocks are manipulated programatically. The plugin subscribes to Document.ImpliedSelectionChanged in order to detect when one or more dynamic blocks created by the plugin are selected. I also subscribe to each BlockReference's ObjectClosed event to detect when a block is modified. When the ImpliedSelectionChanged event is fired for relevant blocks, the plugin reads the values of the selected block's properties. The properties may also be set. Essentially I expose a custom block property editor so that complex engineering rules can be enforced on which block properties are available given other factors. Also, these rules may dictate that if a block property is set to a certain value, other property values are also automatically changed. Thus most block properties are hidden from the user and can only be edited via my property editor control.

The issue I have is that large amounts of memory are allocated by AutoCAD when a number of these blocks are selected repeatedly. This memory is never released, resulting in GBs of memory being consumed by acad.exe. This occurs even if one selects/deselects the same set of blocks over and over.

I've analysed the managed heap for the AppDomain my plugin runs in using a memory profiler (JustTrace). The memory leak is not present in this AppDomain. Acad.exe could be using 4GB of memory, and the managed heap is only using a few hundred MBs. Comparing Snapshots of the heap before and after the problematic actions are performed does not reveal the source of this memory consumption.

I have painstakingly wrapped all ACAD API class instances which implement IDisposable in Using blocks or otherwise ensured that Dispose is explicitly called on the main UI thread to eliminate access violations caused by the GC calling dispose on a different thread, so I do not believe this is the source of the problem. And the GC should call Dispose when I force collection, which I have tried.

I'm seeking general insight about finding the cause of this memory consumption, which seems to be occurring in AutoCAD's native code. Saving the drawing and closing/opening it causes the memory to be deallocated. It seems I have caused a memory leak (well many memory leaks), but I am at a loss as to how to track down the source. Any advice would be much appreciated. 

 

Note that I have tried disabling UNDO but there was no decernible change in memory consumption.

Thanks,
Kevin

12,102 Views
50 Replies
Replies (50)
Message 2 of 51

Norman_Yuan
Mentor
Mentor

I have no idea on how the native code wrapped by .NET API handle memory, but one thing surely results in more memory user: manipulating BlockReference's DynamicBlockReferenceProperties, especially when the dynamic block is quite complicated, because whenever a dynamic propery is set to a different value, AutoCAD creats a new anonymous block definition so that the instance of the blockreference with that specific dynamic property value can have a block definition to reference to. After using your app to manipulate the complicated dynamic blocks for a while, if you go to "Purge" dialog box, you would see tons of purge-able anonymous blocks, which surely take up momeries. It is know side effect that using many dynamic blocks with many dynamic properties slows AutoCAD down noticeably.

 

Norman Yuan

Drive CAD With Code

EESignature

0 Likes
Message 3 of 51

Norman_Yuan
Mentor
Mentor

I have no idea on how the native code wrapped by .NET API handle memory, but one thing surely results in more memory user: manipulating BlockReference's DynamicBlockReferenceProperties, especially when the dynamic block is quite complicated, because whenever a dynamic propery is set to a different value, AutoCAD creats a new anonymous block definition so that the instance of the blockreference with that specific dynamic property value can have a block definition to reference to. After using your app to manipulate the complicated dynamic blocks for a while, if you go to "Purge" dialog box, you would see tons of purge-able anonymous blocks, which surely take up momeries. It is known side effect that using many dynamic blocks with many dynamic properties slows AutoCAD down noticeably.

 

Norman Yuan

Drive CAD With Code

EESignature

0 Likes
Message 4 of 51

Anonymous
Not applicable

Assuming this is the cause of the high memory consumption, is there a way to cause AutoCAD to release this memory? My application purges the BlockTable before each save because of the anonymous blocks you describe and how they affect drawing file size. 

 

I have tried doing a full purge and waiting several minutes to see if this causes AutoCAD to release any memory. Unfortunately there is only a small change, if any, to the amount of memory used by acad.exe. Note that I have also tried combining having UNDO completely disabled before the drawing is even created, with a full purge. Again with no discernible change to the amount of memory consumed by acad.exe. 

 

I created an identical post on theswamp.org (http://www.theswamp.org/index.php?topic=49452.new#new). Someone there suggested ensuring Commit is called for every transaction, even when data is only read and not written. I'm working on this change now.

 

As always, I thank you for taking the time to share your insights. I will report back when I have determined if calling Commit when objects or only opened for read has any effect.

0 Likes
Message 5 of 51

Anonymous
Not applicable

Calling Commit for all transactions, even when no writes occur, had no effect on memory consumption or performance. At least none that I could discern.

 

I am considering creating a small plugin which includes some of the dynamic blocks from my application and a small subset of my codebase which can be used to reproduce the memory leak. I will then post the code for the plugin here. Also, I'll ask ADN for their take on the problem.

0 Likes
Message 6 of 51

Anonymous
Not applicable

I've prepared a runnable plugin which can be used to reproduce the memory leak I've described. The attached solution was writen in VS 2010 and tested with AutoCAD 2014 and 2016 (I only debugged in 2014). It should run in any version between 2013 and 2016. I included a working dll in bin\Debug.

 

I've included a small subset of the dynamic blocks utilized by my real plugin in order to reproduce the problem. There is a single command defined by the test plugin (RunAutomatedMemoryLeakTest). The test plugin creates 10 instances of each dynamic block, then sets every available property value 3 times, where the property type code is 5. The amount of memory is directly proportional to the number of blocks/times each property is set. One can easily adjust the amount of memory consumed by altering the for loops.

 

Neither inserting the blocks nor reading the block properties seems to cause memory consumption. The memory consumption is caused by calls to DynamicBlockHelper.SetParameter. 

 

Here is the code for the command (sorry, my employer requires vb.net):

 

    <CommandMethod("RunAutomatedMemoryLeakTest")> _
    Public Shared Sub AutoMemoryLeakTestCommand()
        If Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument IsNot Nothing Then
            Dim blockObjectIDs As New List(Of ObjectId)
            Dim directoryPath As String = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location)
            For x As Integer = 1 To 10
                For Each filePath In Directory.GetFiles(directoryPath)
                    If Path.GetExtension(filePath).Equals(".dwg", StringComparison.OrdinalIgnoreCase) Then
                        DynamicBlockHelper.AddDynBlockToDrawingFromFile(filePath, Path.GetFileNameWithoutExtension(filePath))
                        blockObjectIDs.Add(DynamicBlockHelper.InsertBlock(HostApplicationServices.WorkingDatabase, BlockTableRecord.ModelSpace, New Point3d(0, 0, 0), Path.GetFileNameWithoutExtension(filePath), 1, 1, 1, "NONE"))
                    End If
                Next
            Next
            For x As Integer = 1 To 3
                For Each objID In blockObjectIDs
                    Dim blockProperties = DynamicBlockHelper.GetBlockProperties(objID)
                    For Each blockProp In blockProperties
                        Dim propertyAttribs() As String = blockProp.Key.Split("~")
                        Dim propertyName As String = propertyAttribs(0)
                        Dim propertyType As String = propertyAttribs(2)
                        If propertyType = 5 AndAlso blockProp.Value.Count > 1 Then
                            For Each propValue As String In blockProp.Value
                                DynamicBlockHelper.SetParameter(objID, propertyName, propValue)
                            Next
                        End If
                    Next
                Next
            Next
        End If
    End Sub

And here is the code for the SetParameter method:

 

    Public Shared Function SetParameter(ByVal BlockID As ObjectId, ByVal ParameterName As String, ByVal Value As String) As Boolean
        Dim doc As Document = ApplicationServices.Application.DocumentManager.GetDocument(BlockID.Database)
        Using lock As DocumentLock = doc.LockDocument()
            Using myTrans As Transaction = BlockID.Database.TransactionManager.StartOpenCloseTransaction()
                Try
                    Using myBRef As BlockReference = myTrans.GetObject(BlockID, OpenMode.ForWrite)
                        For Each myDynamProp As DynamicBlockReferenceProperty In _
                           myBRef.DynamicBlockReferencePropertyCollection
                            If myDynamProp.PropertyName.Equals( _
                               ParameterName, StringComparison.OrdinalIgnoreCase) = True Then
                                myDynamProp.Value = Value
                                myTrans.Commit()
                                Return True
                            End If
                        Next
                    End Using
                Catch ex As System.Exception
                    myTrans.Abort()
                    Try
                        EventLog.WriteEntry("MemLeakTest", ex.GetType().Name + ": " + ex.Message + Environment.NewLine + ex.StackTrace, EventLogEntryType.Error)
                    Catch eex As System.Exception

                    End Try
                End Try
                Return False
            End Using
        End Using
    End Function

Thanks,

Kevin

 

0 Likes
Message 7 of 51

Anonymous
Not applicable

I received a response from ADN saying they could reproduce the problem (the message follows). This is my first "Change Request". Does anyone have advice on how to get the fix expedited?

 

Yes, I am able to reproduce the issue. I feel issue is related to not releasing the memory of objects created during the dynamic block update operations as same memory is getting released when drawing is closed. 

 

We have logged Change Request number 70788 with our development team as this issue requires a modification to our software. Please make a note of this number for future reference. You are welcome to request an update on the status of this issue, or to provide us with additional information, at any time by submitting a new Case through DevHelp Online quoting the Change Request number or this Case number.

 

This issue is important to me. What can I do to help?
This issue needs to be assessed by our engineering team, and prioritized against all of the other change requests that are outstanding. As a result any information that you can provide to influence this assessment will help. Please provide the following where possible:

  • Impact on your application and/or your development.
  • The number of users affected.
  • The potential revenue impact to you.
  • The potential revenue impact to Autodesk.
  • Realistic timescale over which a fix would help you.
  • In the case of a request for a new feature or a feature enhancement, please also provide detailed Use Cases for the workflows that this change would address.

This information is extremely important. Our engineering team have limited resources, and so must focus their efforts on the highest impact Change Requests. We do understand that this will cause you delays and affect your development planning, and we appreciate your cooperation and patience.

 

Best Regards,

Virupaksha Aithal

 

Developer Technical Services
http://adn.autodesk.com

0 Likes
Message 8 of 51

Anonymous
Not applicable

got a chance to play a little bit on this, see the below code in c#, since i do not do anything in vb.net, it works here, did a lot of refactoring and also in some code spots you are passing a double instead of a string on your locations like this: myDynamProp.GetAllowedValues(); before adding the values into your results dictionary.

 

    public class DynamicBlockHelper
    {
        #region "Block Insertion Methods"

        public static void AddDynBlockToDrawingFromFile(string blockFilePath, string blockName, bool forceInsert = false)
        {
            var doc = Application.DocumentManager.MdiActiveDocument;
            var destDb = doc.Database;

            try
            {
                var blockDefFound = true;

                if (!forceInsert)
                {
                    using (var transaction = destDb.TransactionManager.StartTransaction())
                    {
                        var myBlockTable = (BlockTable)transaction.GetObject(destDb.BlockTableId, OpenMode.ForRead);
                        
                        if (!myBlockTable.Has(blockName))
                        {
                            blockDefFound = false;
                        }
                        
                        transaction.Commit();
                    }
                }

                if (blockDefFound) return;

                using (var sourceDb = new Database(false, true))
                {
                    sourceDb.ReadDwgFile(blockFilePath, FileShare.Read, true, "");
                    //sourceDb.CloseInput(true);
                    destDb.Insert(blockName, sourceDb, false);
                }
            }
            catch (Exception ex)
            {
             
            }
        }

        public static ObjectId InsertBlock(Database databaseIn, string btrToAddTo, Point3d insPt, string blockName,
            double xScale, double yScale, double zScale, string layerName, double rotation = 0)
        {
            if (string.IsNullOrEmpty(layerName)) return ObjectId.Null;

            var id = ObjectId.Null;

            using (var transaction = databaseIn.TransactionManager.StartTransaction())
            {
                try
                {
                    var blockTbl = (BlockTable) transaction.GetObject(databaseIn.BlockTableId, OpenMode.ForRead);

                    if (blockTbl.Has(blockName) && blockTbl.Has(btrToAddTo))
                    {
                        var blockDef =
                            (BlockTableRecord) transaction.GetObject(blockTbl[blockName], OpenMode.ForRead);

                        var blockTblRecord =
                            (BlockTableRecord) transaction.GetObject(blockTbl[btrToAddTo], OpenMode.ForWrite);

                        var blockRef = new BlockReference(insPt, blockDef.Id)
                        {
                            ScaleFactors = new Scale3d(xScale, yScale, zScale),
                            Rotation = rotation
                        };

                        
                        blockTblRecord.AppendEntity(blockRef);
                       
                        transaction.AddNewlyCreatedDBObject(blockRef, true);

                        var blockEnt = (Entity) transaction.GetObject(blockRef.Id, OpenMode.ForWrite);

                        try
                        {
                            blockEnt.Layer = layerName;
                        }
                        catch (Exception ex)
                        {
                        }

                        var attribs = blockRef.AttributeCollection;
                        
                        foreach (var entObjectId in blockDef)
                        {
                            var entity =
                                (Entity) transaction.GetObject(entObjectId, OpenMode.ForWrite);

                            if (!(entity is AttributeDefinition)) continue;

                            var attribDef = entity as AttributeDefinition;

                            using (var attributeReference = new AttributeReference())
                            {
                                attributeReference.SetAttributeFromBlock(attribDef, blockRef.BlockTransform);
                                attribs.AppendAttribute(attributeReference);
                                transaction.AddNewlyCreatedDBObject(attributeReference, true);
                            }
                        }

                        transaction.Commit();

                        id = blockRef.Id;
                    }
                }
                catch (Exception ex)
                {
                    transaction.Abort();
                }
            }

            return id;
        }

        #endregion

        #region "Get / Set Block Properties"
        public static Dictionary<string, List<string>> GetBlockProperties(ObjectId blockId)
        {
            var results = new Dictionary<string, List<string>>();

            using (var transaction = blockId.Database.TransactionManager.StartTransaction())
            {
                var dbObject = transaction.GetObject(blockId, OpenMode.ForRead);

                if (dbObject is BlockTableRecord)
                {
                    var blockTableRecord = dbObject as BlockTableRecord;

                    var blockReference =
                        (BlockReference)
                            blockTableRecord.GetBlockReferenceIds(false, false)[0].GetObject(OpenMode.ForRead);

                    foreach (DynamicBlockReferenceProperty myDynamProp in
                            blockReference.DynamicBlockReferencePropertyCollection)
                    {
                        var propKey = string.Format("{0}~{1}~{2}~{3}", myDynamProp.PropertyName, myDynamProp.Description,
                            myDynamProp.PropertyTypeCode, myDynamProp.UnitsType);

                        if (results.ContainsKey(propKey)) continue;

                        AddPropertyKeyAndValues(myDynamProp, results, propKey);
                    }
                }
                else
                {
                    var tempBlockRef = dbObject as BlockReference;
                    if (tempBlockRef != null)
                    {
                        foreach (DynamicBlockReferenceProperty myDynamProp in
                            tempBlockRef.DynamicBlockReferencePropertyCollection)
                        {
                            var propKey = string.Format("{0}~{1}~{2}~{3}", myDynamProp.PropertyName,
                                myDynamProp.Description, myDynamProp.PropertyTypeCode, myDynamProp.UnitsType);

                            if (results.ContainsKey(propKey)) continue;

                            AddPropertyKeyAndValues(myDynamProp, results, propKey);
                        }
                    }
                }

                transaction.Commit();
            }

            return results;
        }

        private static void AddPropertyKeyAndValues(DynamicBlockReferenceProperty myDynamProp, Dictionary<string, List<string>> results,
            string propKey)
        {
            var vals = new List<string>();

            var getAllowedValues = myDynamProp.GetAllowedValues();

            foreach (var v in getAllowedValues)
            {
                if (!(v is string)) continue;

                var s = v as string;
                if (!vals.Contains(s))
                {
                    vals.Add(s);
                }
            }

            results.Add(propKey, vals);
        }

        public static bool SetParameter(ObjectId blockId, string parameterName, string value)
        {
            using (var transaction = blockId.Database.TransactionManager.StartTransaction())
            {
                try
                {
                    var blockReference = (BlockReference)transaction.GetObject(blockId, OpenMode.ForWrite);
                    
                    foreach (DynamicBlockReferenceProperty property in blockReference.DynamicBlockReferencePropertyCollection)
                    {
                        if (property.ReadOnly || !property.PropertyName.Equals(parameterName, StringComparison.CurrentCultureIgnoreCase))
                            continue;

                        switch (property.PropertyTypeCode)
                        {
                            case 5:
                                property.Value = value;
                                break;
                        }

                        transaction.Commit();

                        return true;
                    }
                }
                catch (Exception ex)
                {
                    transaction.Abort();
                }

                return false;
            }
        }
        #endregion
    }
        [CommandMethod("RunAutomatedMemoryLeakTest")]
        public static void AutoMemoryLeakTestCommand()
        {
            var blockObjectIDs = new List<ObjectId>();

            var directoryPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
            if (string.IsNullOrWhiteSpace(directoryPath)) return;

            foreach (var filePathLoopVariable in Directory.GetFiles(directoryPath))
            {
                var filePath = filePathLoopVariable;
                if (string.IsNullOrWhiteSpace(filePath)) continue;

                if (!Path.GetExtension(filePath).Equals(".dwg", StringComparison.CurrentCultureIgnoreCase)) continue;

                DynamicBlockHelper.AddDynBlockToDrawingFromFile(filePath,
                    Path.GetFileNameWithoutExtension(filePath));

                blockObjectIDs.Add(DynamicBlockHelper.InsertBlock(HostApplicationServices.WorkingDatabase,
                    BlockTableRecord.ModelSpace, new Point3d(0, 0, 0),
                    Path.GetFileNameWithoutExtension(filePath), 1,
                    1, 1, "NONE"));
            }

            foreach (var objIdLoopVariable in blockObjectIDs)
            {
                var objId = objIdLoopVariable;
                //dynamic blockProperties = DynamicBlockHelper.GetBlockProperties(objId);
                var blockProperties = DynamicBlockHelper.GetBlockProperties(objId);

                foreach (var blockPropLoopVariable in blockProperties)
                {
                    var blockProp = blockPropLoopVariable;
                    var propertyAttribs = blockProp.Key.Split(Convert.ToChar("~"));
                    var propertyName = propertyAttribs[0];
                    var propertyType = propertyAttribs[2];

                    if (propertyType == null || (propertyType != "5" || blockProp.Value.Count <= 1)) continue;

                    foreach (var propValue in blockProp.Value)
                    {
                        DynamicBlockHelper.SetParameter(objId, propertyName, propValue);
                    }
                }
            }
        }

HTH.-

 

0 Likes
Message 9 of 51

Anonymous
Not applicable

LE3,

 

I'm confused. Are you saying you were able to eliminate the memory leak with the changes you made? Looking at the code in AutoMemoryLeakTestCommand, you have removed the outer for loops which will result in much less memory being leaked overall, which may lead you to believe that memory is not being leaked. I suggest you add the two for loops you removed (the first inserted 10 instances of each block, and the second set all available block properties for each block instance 3 times). If you did eliminate the memory leak, I am very excited to implement the needed changes in my code base. 

 

Thanks,

Kevin

0 Likes
Message 10 of 51

Anonymous
Not applicable

i will add those, as soon i get some time, and comment any findings.

0 Likes
Message 11 of 51

Anonymous
Not applicable

Thank you for the effort in any case. I'll look over your suggested changes. 

0 Likes
Message 12 of 51

Anonymous
Not applicable

ok, i wrapped both foreach into the for's blocks, indeed takes a long time, but works without an error (see image attached with the output), i do not know if by adding those for's end up being a good code practice, maybe breaking it into another thread or something, can't offer any other solution now.

 

good luck.

 

HTH.-

0 Likes
Message 13 of 51

Anonymous
Not applicable

Did AutoCAD leak a bunch of memory? That is what this thread is all about. A memory leak that occurs when editing complex dynamic block properties programatically. The plugin is just a demonstration of how to reproduce the memory leak. It has no real purpose other than that.

0 Likes
Message 14 of 51

Anonymous
Not applicable

i see, on my test it tooks less than 38seconds to complete the task, using a2014.

well, at least now there is a c# version of your test code, so others may be willing to give it a take too.

 

good luck.

0 Likes
Message 15 of 51

Virupaksha_aithal
Autodesk Support
Autodesk Support

Hi

 

I feel the issue is about not releasing the memory provided during the dynamic block edit. As soon as drawing is closed, all (most) the memory allocated to the dynamic block edit operation is released. Secondly, dynamic block edit is a complex operation as AutoCAD need to create a new anonymous block if the new value provided to a dynamic block parameter is unique.



Virupaksha Aithal KM
Developer Technical Services
Autodesk Developer Network

0 Likes
Message 16 of 51

Anonymous
Not applicable

What is causing the memory usage has been established. So are you stating you are canceling the change request (70788)? AutoCAD never releases the memory allocated during these operations (not until the drawing is closed). Are you actually suggesting this is by design and therefore no change needs to be made to AutoCAD? 

 

Or do you have a change I can make to my source code which will eliminate the memory leak? Because this is a memory leak. And for a plugin like mine, which makes heavy use of dynamic blocks and manipulates the properties of those blocks via your .net API, this is a huge problem. AutoCAD will literally consume all available physical memory because of this leak. If you don't believe me, adjust the for loops in my demonstration plugin so that the properties of the same blocks are changed 1000s of times. When AutoCAD has used all available physical memory, it will crash.

Message 17 of 51

Anonymous
Not applicable

Also, you can purge the anonymous blocks from the drawing, and the memory will not be released (maybe 1% will be released). 

0 Likes
Message 18 of 51

Anonymous
Not applicable

Just so I'm sure you understand my previous message as intended (I feel I may not have been given enough detail), I can run the plugin I provided you so that AutoCAD uses say 10GB of physical memory. Then I can delete all blocks from the drawing and do a purge all. And no memory is released by AutoCAD. 

0 Likes
Message 19 of 51

Norman_Yuan
Mentor
Mentor

As I said in my earlier reply, I suspect the increased memory consumed in AutoCAD is due to the manipulation of large number of dynamic blocks.

 

The issue you have might be better called "memory fragmentation", rather than "memory leak". I could be wrong here: memory leak refers to memory lost in a application, even quiting the applicationo would not recover the lost memory; only restart the OS can recover it. On the other hand, memory fragmentation is memory being fragmented due to objects of different sizes in memory being created/destroyed, so that some memory become so fragmented that can hardly be reused within the application. However, shutting down the application, the fragmented memory is returned to OS.

 

There is a post on AutoCAD's memory fragment here:

 

http://adndevblog.typepad.com/autocad/2012/07/problem-with-autocads-memory-usage.html

 

With all said, I do not see there is a way to cure your problem, if it is indeed due to your unique business process that requires heavy handling dynamic blocks.

 

I have learned a bit from my experience that using large number of comlicated dynamic blocks not only eats a lot memory, but also significantly slows down AutoCAD (well, depending on what one does with the drawing/AutoCAD).

 

Norman Yuan

Drive CAD With Code

EESignature

0 Likes
Message 20 of 51

Anonymous
Not applicable

My response was directed only at Virupaksha Aithal from AutoDesk Support. I will read the link you provided. But suggestion that 95% of a system's physical memory would be unrecoverable due to fragmentation, and therefore nothing can be done by AutoDesk to correct this problem, doesn't sit well with me. I will run this by a friend of mine who works with native code at a low level, but I expect he will tell me AutoDesk has some work to do.

0 Likes