Out of memory error while working with ReferenceKey

[amitabhVA4SD]

Hello All,

I am experiencing an out-of-memory error while working with Topology Tagging APIs on Inventor 2020.3.4. 

The Assembly is loaded in "Load Full" mode since the "Express Mode" does not give access to the Topology/B-Rep information.

I am getting this issue while trying to collect the B-rep topology using ReferenceKey and ReferenceKeyContext APIs for each Face and Edge in the Assembly using the C# .Net external application. I need this topology persistent id information to bind back this data in a disconnected Inventor Add-In application.

The system has 16 GB RAM and we did check the GDI limit as well.

We found the memory to exhaust at ~33,000 keys.

We are working with one ReferenceKeyManager and Multiple Key Context Tables

Please see the image below from the Task Manager showing the GDI count and Memory consumed while running the application. It is at this exact moment that the out of memory error is being thrown and Inventor crashes

A piece of the code implementation is also attached

[TestMethod]
        public void ReferenceKeyApi()
        {
            // Get Inventor App
            InventorApp = (Inventor.Application)System.Runtime.InteropServices.Marshal.GetActiveObject("Inventor.Application");
            VctApplication.InventorApp = InventorApp;
            Document document = VctApplication.InventorSession.ActiveDoc;
            SelectSet selectSet = document.SelectSet;
            dynamic selectedObject = null;
            if (selectSet.Count > 0)
                 selectedObject = selectSet[1];
       
            // Initialize the VctDocument class
            VctDocument vctDocument = VctApplication.InventorSession.VctDocument;

            // Example for an assembly document
            switch (selectedObject)
            {
                case FaceProxy faceProxy:
                    faceProxy = (FaceProxy) selectedObject;

                    // To get reference key and key context
                    string fKey = vctDocument.GetReferenceKey(document, faceProxy, out string fKeyContext);
                    selectSet.Clear();

                    // later to bind back the reference key to the entity using the key context
                    FaceProxy fRetrievedObject = vctDocument.GetEntityFromReferenceKey(document, fKey, fKeyContext);
                    selectSet.Select(fRetrievedObject);
                    break;

                case EdgeProxy edgeProxy:
                    edgeProxy = (EdgeProxy) selectedObject;

                    // To get reference key and key context
                    string eKey = vctDocument.GetReferenceKey(document, edgeProxy, out string eKeyContext);
                    selectSet.Clear();

                    // later to bind back the reference key to the entity using the key context
                    EdgeProxy eRetrievedObject = vctDocument.GetEntityFromReferenceKey(document, eKey, eKeyContext);
                    selectSet.Select(eRetrievedObject);
                    break;
            }
        }

Thanks,

Amitabh Mukherjee

[amitabhVA4SD]

We are running this plugin with the Inventor application in Hidden mode.

[CattabianiI]

Hi @amitabhVA4SD,
I reproduced your behaviour with the iLogic rule below.
It doesn't crash but it's very unstable and many commands don't work.
The limit of reference keys I reached is 32764 which is 2^15-4 so it sounds like a system limitation.
Maybe @adam.nagy or @BrianEkins can tell us if the limitation is real and how to workaround it for your workflow.

Sub main

	TraverseAssemblyRule.InvApp = ThisApplication
	TraverseAssemblyRule.Run()

End Sub

Public Class TraverseAssemblyRule
	
	
	Public Shared InvApp As Inventor.Application
	Public Shared ThisDoc As Document
	Public Shared count As Integer = 0
	
	Public Shared Sub Run() 
		
		ThisDoc = InvApp.ActiveDocument 
			
		TraverseAssembly(ThisDoc.ComponentDefinition.Occurrences) 
				
	End Sub 
	
	

			
	Private Shared Sub TraverseAssembly(Occurrences As ComponentOccurrences) 
		Dim oOcc As ComponentOccurrence 
		
		For Each oOcc In Occurrences 

			For Each sb As SurfaceBody In oOcc.SurfaceBodies 

				For Each ff As Object In sb.Faces
					ff = TryCast(ff, Face)
					If ff Is Nothing Then
						ff = ff.Parent
					End If 
					For Each ej As Object In ff.Edges
						ej = TryCast(ej, Edge)
						If ej Is Nothing Then
							ej = ej.Parent
						End If 
						ManageRefKey(ej)
					Next
				Next
			
				If oOcc.DefinitionDocumentType = kAssemblyDocumentObject Then 
					TraverseAssembly(oOcc.SubOccurrences) 
				End If 		
					
			Next
		Next 
	End Sub
	
	Private Shared Sub ManageRefKey(element As Object)
		Try

		    ' Set a reference to the ReferenceKeyManager object.
	    Dim refKeyMgr As ReferenceKeyManager
	    refKeyMgr = ThisDoc.ReferenceKeyManager()
		
	    ' Create a key context.
	    Dim keyContext As Long
	    keyContext = refKeyMgr.CreateKeyContext()
		
	    ' Get a reference key from the selected entity. This will
	    ' fail if the selected object doesn't support reference keys.
	    Dim refKey() As Byte = New Byte() {}
		
	  	element.GetReferenceKey(refKey, keyContext)	
		
            ' Get the key context as an array of bytes and
	    ' convert it to a string.
	    Dim contextArray() As Byte = New Byte() {}
	    Call refKeyMgr.SaveContextToArray(keyContext, contextArray)
	    Dim strContext As String
	    strContext = refKeyMgr.KeyToString(contextArray)
	   
	    ' Convert the reference keys to strings to make saving it easier.
	    Dim strElemKey As String
	    strElemKey = ThisDoc.ReferenceKeyManager.KeyToString(refKey)
		count = count + 1

	Catch ex As OutOfMemoryException
		MsgBox(count)
		Throw
	Catch ex As Exception

	End Try
	End Sub

End Class

[amitabhVA4SD]

Exactly @CattabianiI . Thank you for doing a quick analysis.

That is our finding as well. Since it is an external application at our end, Inventor is crashing with an out-of-memory error.

@Anonymous @adam.nagy @BrianEkins - Please help!

[MatthiasMinich]

Have you tried to erase the byte-variable at the end of the function.

Something like this: 

    ReDim refKey(0)
    ReDim contextArray(0)

[YuhanZhang]

Hi @amitabhVA4SD , in your code when call the GetEntityFromReferenceKey methid will it always generate a new key context for each BRep entity?  I can see @CattabianiI 's code will.  Please be aware of that you only need to create one key context for the same document, and this key context can be used for all the BRep entities(faces, edges and their proxies), so when you call GetReferenceKey method from multiple BRep entities in the same document you can input the same key context. A key context actually indicates a reference keys table object but not only a Long value,  so each time you call ReferenceKeyManager.CreateKeyContext it will create a new reference key table object(since Inventor 2020 we expose a new method ReferenceKeyManager.ReleaseKeyContext to allow users to destroy unused reference key tables),  but usually you just have to keep one reference key table for all your BRep entities in the same document. So the suggestion here is:

1. Open a document and create a reference key context using ReferenceKeyManager.CreateKeyContext .

2.  Call FaceProxy/EdgeProxy.GetReferenceKey to get reference keys with the same key context.

3. Call ReferenceKeyManager.KeyToString to convert reference keys to strings and save them somewhere.

4. Call ReferenceKeyManager.SaveContextToArray to convert the key context to an array and save the array somewhere(usually the same place with the keys' strings).

5. Close your document.

6. Reopen your document, and use ReferenceKeyManager.LoadContextFromArray to load the key context(now this Long value should not be the same as the one when you created it, but they indicates the same reference key table in the same document).

7. Use the ReferenceKeyManager.StringToKey to load a reference key from saved string, and you can use the ReferenceKeyManager.CanBindKeyToObject or BindKeyToObject to retrieve the BRep entity using the loaded key context and key.

So for @CattabianiI 's code, you can declare the keyContext variable as global variable in the class, and then call the 

keyContext = refKeyMgr.CreateKeyContext()

to generate the key context in the "Public Shared Sub Run()" , then you need not to create a key context for each and every element, please let me if this will improve the capacity there.

[CattabianiI]

Moved the key context creation to one per document and now it works like a charm.
Thank you @YuhanZhang and hope this could fix @amitabhVA4SD's real world case.

[MuzammilI]

Thankyou @CattabianiI, @YuhanZhang and @MatthiasMinich for your contribution.

We have tried to implement the API in different ways but all leading to out of memory exception at some point.

We have already implemented the method which you mentioned which is in the version 1 in the table below; along with out of memory exception the size of key context also increases.

I have found that the memory fills faster when reference key manager is invoked every time and the value of integer key context is around 32761 every time the exception is thrown. Please correct me if I am wrong.

Version	Reference key manager invoked	Key context table created	Observation	Possible reason for exception
1	Once per document	Once per document	1. Size of key  context increases. 2. Out of memory exception.	Lot of data was being stored in a single context table.
2	Multiple times (i.e. for every entity)	Multiple times ( i.e. for every entity key context is created and released)	1. Out of memory exception.	Inventor was not able to release the reference key managers even though the key context was released.
3	Once per document	Multiple times ( i.e. for every entity key context is created and released)	1. Out of memory exception.	unknown

I have attached an image of the exception caught in the implementation of version 3.

@CattabianiI please let me know how many keys are you able to retrieve (did the integer key context exceed 32764)? 

@YuhanZhangplease let me know if I have missed something.

Thanks!

[amitabhVA4SD]

@MuzammilI is from my team

[CattabianiI]

> @CattabianiI please let me know how many keys are you able to retrieve (did the integer key context exceed 32764)? 
I tryed on an assembly and created more than 100000
Fixed rule:

' https://forums.autodesk.com/t5/inventor-ilogic-api-vba-forum/out-of-memory-error-while-working-with-referencekey/td-p/10590755
Sub main

	TraverseAssemblyRule.InvApp = ThisApplication
	TraverseAssemblyRule.Run()

End Sub

Public Class TraverseAssemblyRule
	
	
	Public Shared InvApp As Inventor.Application
	Public Shared ThisDoc As Document
	Public Shared count As Integer = 0
	Public Shared RefKeyMgr As ReferenceKeyManager 
	Public Shared KeyContext As Long 
	
	Public Shared Sub Run() 
		
		ThisDoc = InvApp.ActiveDocument 
		
		' Set a reference to the ReferenceKeyManager object.
	    RefKeyMgr = ThisDoc.ReferenceKeyManager()
		
	    ' Create a key context.
	    KeyContext = RefKeyMgr.CreateKeyContext()
						
		Dim contextArray() As Byte = New Byte() {}
	    RefKeyMgr.SaveContextToArray(keyContext, contextArray)
	    Dim strContext As String
	    strContext = RefKeyMgr.KeyToString(contextArray)
		
		TraverseAssembly(ThisDoc.ComponentDefinition.Occurrences) 
		msgbox(count)
				
	End Sub 
	
	

			
	Private Shared Sub TraverseAssembly(Occurrences As ComponentOccurrences) 
		Dim oOcc As ComponentOccurrence 
		
		For Each oOcc In Occurrences 

			For Each sb As SurfaceBody In oOcc.SurfaceBodies 

				For Each ff As Object In sb.Faces
					ff = TryCast(ff, Face)
					If ff Is Nothing Then
						ff = ff.Parent
					End If 
					For Each ej As Object In ff.Edges
						ej = TryCast(ej, Edge)
						If ej Is Nothing Then
							ej = ej.Parent
						End If 
						ManageRefKey(ej)
					Next
				Next
			
				If oOcc.DefinitionDocumentType = kAssemblyDocumentObject Then 
					TraverseAssembly(oOcc.SubOccurrences) 
				End If 		
					
			Next
		Next 
	End Sub
	
	Private Shared Sub ManageRefKey(element As Object)
		Try		
		    ' Get a reference key from the selected entity. This will
		    ' fail if the selected object doesn't support reference keys.
		    Dim elemRefKey() As Byte = New Byte() {}
			
		  	element.GetReferenceKey(elemRefKey, KeyContext)
		   
		    ' Convert the reference keys to strings to make saving it easier.
		    Dim strElemKey As String
		    strElemKey = RefKeyMgr.KeyToString(elemRefKey)
			count = count + 1
			
		Catch ex As OutOfMemoryException
			MsgBox(count)
			MsgBox(ex.ToString)
			Throw
		Catch ex As Exception

		End Try
	End Sub

End Class

[MjDeck]

@amitabhVA4SD and @MuzammilI , please try the attached C# project. It will attach to a running Inventor session with an active assembly document, and get reference keys for all face and edge proxies in the assembly. I tested it on a fair-sized assembly and it got 47000 keys with no error.  This is with a single key context.
Note : this takes a long time to run. I think it would take a significantly shorter time if it were converted to an addin.

[MuzammilI]

Thankyou @MjDeck and  @CattabianiI.
I was able to retrieve more than 100,000 keys without any exception and retrieved only one key context string at the end. But I am facing an issue binding the key context back to the entity. Are you also facing this issue at your end?

Please let me know. Thanks!

@YuhanZhang the exception is thrown if we collect the key context string for every entity even though a single  ReferenceKeyManager.CreateKeyContext is called at the beginning.
Please let me know if it is possible to retrieve key context string for every entity if the quantity exceeds 32764. Thanks!

[YuhanZhang]

Hi @MuzammilI , is there any reason that you have to load the key context for each and every BRep entity? Both the ReferenceKeyManager.CreateKeyContext and LoadContextFromArray methods will generate a new key context actually(and so a reference key table will be generated along with it), so for first time when you use reference key for BRep you need to create a key context, and from next time when you reopen the document you just need to load the key context using the saved key context array, and use this key context for all entities in same open session. So if currently your code load the key context too many times you should optimize it to make sure in a document open session you just create/load it once.

Hope this makes it clear.

[MuzammilI]

Hi @YuhanZhang, I have tried by calling the respective APIs ( CreateKeyContext & SaveContextToArray) only once and I was able to generate more than 100,000 keys without any exception, the key context string generated at the end was huge (no issues with that).
The issue is that I am not able to bind the key context back to the entity, when I collect it only once (as you said) at the end but I am able to bind the key context back to the entity, when I collect it after every entity (i.e. by calling SaveContextToArray for every entity keeping the CreateKeyContext int value same; which is causing the out of memory exception).

Please refer the image attached. I have called this API only once to check if a reference key (random) I have previously collected binds back to the entity using the key context collected only once after all the reference keys were collected. There is an exception while loading the context from the Array (is it because of the size of the key context).

Please let me know if I am able to explain it clearly or if you have any questions.
If you understand the issue, please let me know the way forward.
Thanks!

[YuhanZhang]

Hi @MuzammilI,

From your code the function GetEntityFromReferenceKey will call the LoadContextFromArray for each time you call this function, and if you call this function for many times the reference key context will be loaded many times too, this will also cause the problem. To solve this issue you need to optimize your code to make sure for an open document you just call the CreateKeyContext or LoadContextFromArray only once and then pass the key context to the GetEntityFromReferenceKey.

[amitabhVA4SD]

Hi @YuhanZhang,

@MuzammilIis able to generate the topology tagging data but is having difficulty binding the retrieved data with the associated topology in a separate plugin program. We are looking into this issue.

The different Plugins work in the following manner:

1. The Topology tag data extractor plugin generates the data on the entire assembly (for the existing iam file)
2. A separate Data Binder Plugin works on the extracted data to bind the data to read the associated topology.

In between step 1 and Step 2, we process the geometry data of the assembly to build a mathematical model. This process runs on AWS and simply makes use of the Excel data.

All the Plugins are running in batch mode (no user interaction)

Is there any best practice that we can follow for such disconnected plugins (w.r.t working with the topology tagging data)?

Thanks,

Amitabh Mukherjee

[MjDeck]

@amitabhVA4SD , does your process make any modifications to the assembly (or components within it) in between step 1 and step 2?
Just for testing purposes, can you run only step 1 and 2, without any processing in between?
What kind of difficulty are you having in binding the keys back to the geometry? Is it out-of-memory or is it a different error?

[amitabhVA4SD]

@MjDeckIt is out-of-memory error. My teammate @MuzammilI has created a small application that shows this issue when the count of the entities goes beyond ~32000. We are able to successfully bind the entity back when the entity count is below this number. He will share this utility along with a walkthrough video by Monday.

[CattabianiI · Accepted solution]

@amitabhVA4SD @MuzammilI may I point out some things? I'm aware that your program is not an addin nor iLogic.

I initially reproduced your error which has been fixed after @YuhanZhang told us to use one key context per document.
And then a similar error came up binding back ref key to entity, I can also reproduce that error and is always a matter of doing some operation once per document, let me write down these steps:

1. Create a reference key context. - CreateKeyContext
2. Ask an object for its reference key and also providing the key context. - GetReferenceKey
3. Continue to get reference keys from other objects, using the same key context. - GetReferenceKey
4. Save the key context to an array of byte and store it as string. - SaveContextToArray + KeyToString
5. Store the ref key as string. - KeyToString

Now bind back ref key to object:

1. Convert the stored as string key context to an array of byte. - StringToKey
2. Load the key context array of byte retrieving the ref key context. - LoadContextFromArray
3. Convert the stored ref key to arrays of byte - StringToKey
4. Iterate the ref key and bind back to object using the same key context. - BindKeyToObject

The bold API has to be called once per document.
I hope to make clear what has to be managed once per document and also the order in which the api has to be called.

My rule tested on very big assembly:

' https://forums.autodesk.com/t5/inventor-ilogic-api-vba-forum/out-of-memory-error-while-working-with-referencekey/td-p/10590755
Sub Main()
	TraverseAssemblyRule.InvApp = ThisApplication
	TraverseAssemblyRule.iLogicLogger = Logger
	
	TraverseAssemblyRule.GetBRepRefKey()
	
	TraverseAssemblyRule.BindBRepRefKey()
End Sub

Public Class TraverseAssemblyRule	
	
	Const refKeysFfn As String = "c:\temp\RefKeys.txt"
	
	Const notBindedBackRefKeysFfn As String = "c:\temp\NBRefKeys.txt"
	Public Shared InvApp As Inventor.Application
	Public Shared ThisDoc As Document
	Public Shared Count As Integer = 0
	Public Shared RefKeyMgr As ReferenceKeyManager 
	Public Shared KeyContext As Long 
	
	Public Shared iLogicLogger As IRuleLogger
	
	Public Shared StrContext As String
	
	Public Shared Sub GetBRepRefKey() 
		
       	Using sw As IO.StreamWriter = IO.File.CreateText(refKeysFfn)
			sw.WriteLine("START")
		End Using
		ThisDoc = InvApp.ActiveDocument 
		
		RefKeyMgr = ThisDoc.ReferenceKeyManager()
		
	    KeyContext = RefKeyMgr.CreateKeyContext()
						
		TraverseAssembly(ThisDoc.ComponentDefinition.Occurrences) 
		
		Dim contextArray() As Byte = New Byte() {}
	    RefKeyMgr.SaveContextToArray(KeyContext, contextArray)	    
	    StrContext = RefKeyMgr.KeyToString(contextArray)
		iLogicLogger.Info(Count)				
	End Sub 	

			
	Private Shared Sub TraverseAssembly(Occurrences As ComponentOccurrences) 
		Dim oOcc As ComponentOccurrence 
		
		For Each oOcc In Occurrences 
			For Each sb As SurfaceBody In oOcc.SurfaceBodies 
				For Each ff As Object In sb.Faces
					ff = TryCast(ff, Face)
					If ff Is Nothing Then
						ff = ff.Parent
					End If 
					For Each ej As Object In ff.Edges
						ej = TryCast(ej, Edge)
						If ej Is Nothing Then
							ej = ej.Parent
						End If 
						ManageRefKey(ej)
						
					Next
					ManageRefKey(ff)
				Next
			
				If oOcc.DefinitionDocumentType = kAssemblyDocumentObject Then 
					TraverseAssembly(oOcc.SubOccurrences) 
				End If 		
					
			Next
		Next 
	End Sub
	
	Private Shared Sub ManageRefKey(element As Object)
		Try		
		    Dim elemRefKey() As Byte = New Byte() {}
			
		  	element.GetReferenceKey(elemRefKey, KeyContext)
		   	Dim curElemStrKey As String = RefKeyMgr.KeyToString(elemRefKey) 
			Using sw As IO.StreamWriter = IO.File.AppendText(refKeysFfn)
				sw.WriteLine(curElemStrKey)
			End Using
			
			Count = Count + 1
			
		Catch ex As OutOfMemoryException
			MsgBox(count)
			MsgBox(ex.ToString)
			Throw
		Catch ex As Exception

		End Try
	End Sub
	
Public Shared Sub BindBRepRefKey()
	Using sw As IO.StreamWriter = IO.File.CreateText(notBindedBackRefKeysFfn)
		sw.WriteLine("START")
	End Using	   
	
	Dim contextArray() As Byte = New Byte() {}
    RefKeyMgr.StringToKey(StrContext, contextArray)

	iLogicLogger.Info("contextArray size: " &contextArray.Length)

    Dim refKeyContext As Long
    refKeyContext = RefKeyMgr.LoadContextFromArray(contextArray)   

	For Each elemStrKey As String In IO.File.ReadLines(refKeysFfn)
		If elemStrKey = "START" Then
			Continue For
		End If
	    Dim elemRefKey() As Byte = New Byte() {}
	    RefKeyMgr.StringToKey(elemStrKey, elemRefKey)
	 
	    Dim foundElem = RefKeyMgr.BindKeyToObject(elemRefKey, refKeyContext)
		If foundElem Is Nothing Then
		   	Using sw As IO.StreamWriter = IO.File.AppendText(notBindedBackRefKeysFfn)
				sw.WriteLine(elemStrKey)
			End Using
		End If	   	
	Next    
End Sub

End Class

[amitabhVA4SD]

Thank you for the detailed code snippet @CattabianiI 

We have a question on the below implementation. Why are we collecting the face as an object  (generic) instead of the Face Interface in For Each loop. Under what circumstance will the face object be nothing since we are trying to collect all the faces that belong to a given Surface Body? They should all be of Type Face

Why are we collecting the parent (i.e Surface Body) and reporting that information instead?

ff = TryCast(ff, Face)
If ff Is Nothing Then
   ff = ff.Parent
End If 

 A similar explanation would be helpful for the edge as well.

Thanks,
Amitabh Mukherjee