ModelItemCollection performance issues

ModelItemCollection performance issues

luca_vezza
Enthusiast Enthusiast
39,481 Views
56 Replies
Message 1 of 57

ModelItemCollection performance issues

luca_vezza
Enthusiast
Enthusiast

Hi all,

I'm hitting serious performance issues with the ModelItemCollection class. I have a collection of about 270k items (so, big but not tremendously huge...). Just looping over the collection (with foreach) takes forever (several minutes!). I have tried converting the collection to a hashedSet and looping just flies (~1 millisec). Of course, converting takes ages too, so that is not a real solution...

Is there something big that I am missing? Or is it really the kind of performance I should expect from that type of collection?

Thanks,

 

   Luca

39,482 Views
56 Replies
Replies (56)
Message 21 of 57

luca_vezza
Enthusiast
Enthusiast

 


@Anonymous wrote:
...

There is one critical semantic difference between .NET and ComAPI categories:

- .NET API merge the "static" and the "custom GUI" properties

- ComAPI has 2 different APIs to access these

...

About different semantics, I have one more doubt on this. In .NET API, given a property you can retrieve its data type with "prop.Value.DataType". Is there an equivalent way to do that with ComAPI?

Thanks,

   Luca

 

0 Likes
Message 22 of 57

Anonymous
Not applicable

After you get the InwOaProperty, for example:

nwOaProperty prop = (InwOaProperty)props[j];

you can access its value as an object: prop.value, which seems to just be a boxed type.

 

In my case, for SP3D properties, I know they are all just strings so for expediency, I directly do 

prop.value as string

I guess, but have not investigated, you can use prop.value.GetType() and all of the Type properties and methods.

Or just things like

if (prop.value is string)
;
...

 

0 Likes
Message 23 of 57

luca_vezza
Enthusiast
Enthusiast

@Anonymous wrote:

After you get the InwOaProperty, for example:

nwOaProperty prop = (InwOaProperty)props[j];

you can access its value as an object: prop.value, which seems to just be a boxed type.

 

In my case, for SP3D properties, I know they are all just strings so for expediency, I directly do 

prop.value as string

I guess, but have not investigated, you can use prop.value.GetType() and all of the Type properties and methods.

Or just things like

if (prop.value is string)
;
...

 


Ok, thanks.

Yes, with SP3D all props are strings; that is not not true for example with AVEVA PDMS, where props are typed. I will try the way you suggest anyway.

   Luca

Message 24 of 57

luca_vezza
Enthusiast
Enthusiast

@Anonymous wrote:

I would suggest to use the ComAPI, converting your ModelItemCollection once.

Using this method I'm able to parse depth first a model with around half million objects in less than a second on a typical laptop, including accessing each object name.

 

 


Back to my performance quest...

I have a problem with the ComAPI. Suppose you have two items under the same parent group; if I get the paths for them, how can I detect that they belong to the same parent node? The two paths DO NOT include the same nodes apparently (or I may have missed something crucial there...).

 

   Luca

Message 25 of 57

Anonymous
Not applicable

@luca_vezza wrote:

@Anonymous wrote:

I would suggest to use the ComAPI, converting your ModelItemCollection once.

Using this method I'm able to parse depth first a model with around half million objects in less than a second on a typical laptop, including accessing each object name.

 

 


Back to my performance quest...

I have a problem with the ComAPI. Suppose you have two items under the same parent group; if I get the paths for them, how can I detect that they belong to the same parent node? The two paths DO NOT include the same nodes apparently (or I may have missed something crucial there...).

 

   Luca


Hi Luca,

Not sure how you get your paths or why you need to check the parent, but let me guess

Imagine you get the paths based on the current selection, something like this:

 

InwOpSelection nwSel = ComApiBridge.ToInwOpSelection(NWApp.ActiveDocument.CurrentSelection.SelectedItems);
InwSelectionPathsColl nwItems = nwSel.Paths()

And you want to get access to their parents but only once per parent?
for each node, you can access the parent (provided it's not already a root, I just keep the code simple):

int c = nwItems.Count;
for (int i=1;i<=c;i++)
{
     InwOaPath3 path = (InwOaPath3)nwItems[i];
     InwOaNode parent = path.Nodes()[path.Nodes().Count - 1] as InwOaNode;
     // check here if already in a list you build on the fly
}
				

Is it what you are looking at?

BTW, I was intrigued by your comment 


The two paths DO NOT include the same nodes apparently (or I may have missed something crucial there...).

Thus I decided to implement it and I ended up being "flabbergasted".

As you stated the nodes objects in different paths, and worst even when extracted from the same path are different and both == and .Equals() return false !!! Even their HashCodes are different !

 

And since it's not possible to access their nwHandle, the only workaround I found was to build the parent full path as a string and store it in a HashSet to check for already "found" parent

 

I include the code below but I strongly recommend you report this as a bug in the ComAPI to Autodesk team, hopefully they could fix the Equals method or worst case to avoid compatibility issues provides a way to check for equality and thus avoid such ugly and inefficient workaround.

 

InwOpSelection nwSel = ComApiBridge.ToInwOpSelection(NWApp.ActiveDocument.CurrentSelection.SelectedItems);
InwSelectionPathsColl nwItems = nwSel.Paths();

int pc = nwItems.Count;
HashSet<string> parentsPaths = new HashSet<string>(pc);

for (int i = 1; i <= pc; i++)
{
InwOaPath3 path = (InwOaPath3)nwItems[i];

// Check for parent
InwPathNodesColl nodes = path.Nodes();
int nc = nodes.Count;
if (nc > 1)
{
InwOaNode parent = path.Nodes()[nc - 1] as InwOaNode;
InwOaNode parent2 = path.Nodes()[nc - 1] as InwOaNode;

Debug.Print($"== {(parent == parent2).ToString()}");
Debug.Print($"Equals = {parent.Equals(parent2).ToString()}");
Debug.Print($"Equal HashCodes = {(parent.GetHashCode()==parent2.GetHashCode()).ToString()}");

System.Array adp = (System.Array)path.ArrayData;
string adpstr = adp.GetValue(1).ToString();

for (int j = 1; j != adp.Length-1; j++)
adpstr = adpstr + "/" + adp.GetValue(j).ToString();

if (parentsPaths.Contains(adpstr))
Debug.Print($"Already found parent: {parent.UserName}");
else
parentsPaths.Add(adpstr);
}

PPS: I hacked this quickly, I guess saving the sub SystemArray and comparing it would probably be more efficient than concatenating strings and comparing their hash code via a HashSet 😉

PS: Another option would be to convert the parent path back into a ModelItem but this would defeat the purpose of voiding ModelItems in the first place.

0 Likes
Message 26 of 57

luca_vezza
Enthusiast
Enthusiast

@Anonymous wrote:

BTW, I was intrigued by your comment 


The two paths DO NOT include the same nodes apparently (or I may have missed something crucial there...).

Thus I decided to implement it and I ended up being "flabbergasted".

As you stated the nodes objects in different paths, and worst even when extracted from the same path are different and both == and .Equals() return false !!! Even their HashCodes are different !

 

And since it's not possible to access their nwHandle, the only workaround I found was to build the parent full path as a string and store it in a HashSet to check for already "found" parent

 

I include the code below but I strongly recommend you report this as a bug in the ComAPI to Autodesk team, hopefully they could fix the Equals method or worst case to avoid compatibility issues provides a way to check for equality and thus avoid such ugly and inefficient workaround.


 

Yes, that was my main issue and I'm glad you confirm it was not just me... I will definitely post the bug, as working with string paths will certainly kill performance (doing string compares with a 1M-array is probably not so efficient...).

I will keep you updated on what the Autodesk team replies on this.

Thanks!

   Luca

0 Likes
Message 27 of 57

Anonymous
Not applicable

Great, Good to hear this is helpful.

 

There is probably one or two of the use case I'm working on for our plug-in that will need something similar, so if I end up writing [and instrumenting] various solutions, I will revert with the best performing approach and supporting numbers.

 

Cheers.

0 Likes
Message 28 of 57

luca_vezza
Enthusiast
Enthusiast

Ok, thanks.

In the meantime, I've posted to the bug; for reference, it is Case ID: 14848858

 

   Luca

0 Likes
Message 29 of 57

rade_tomovic
Advocate
Advocate

 

@Anonymous My problem is not in skipping, but rather in going through every single category, property, and value and caching for later (data structure is like dictionary within dictionary to be sure that there are no duplicates). I've tried with COM API, but I can't notice performance advancements, so I guess I'm doing something wrong.  Here is my .NET API code

 

 

 

private void IterateProperties(NW.ModelItemCollection modelItemCollection)
        {
            foreach (var modelItem in modelItemCollection)
                this.CheckModelItemForProperty(modelItem);
        }

private void CheckModelItemForProperty(NW.ModelItem modelItem)
        {
            foreach (var propertyCategory in modelItem.PropertyCategories)
                if (propertyCategory.DisplayName.Equals(this._groupRule.Category.DisplayName))
                    foreach (var property in propertyCategory.Properties)
                        if (property.DisplayName.Equals(this._groupRule.Property.DisplayName))
                            this._propertyValues.Add(property);
        }

 

and before this code, there is:

 

 

 

foreach (var model in doc.Models)
                {
                    foreach (var modelItem in model.RootItem.DescendantsAndSelf)
                        if (modelItem.InstanceGuid != Guid.Empty && !modelItem.IsCollection && !modelItem.IsLayer && !modelItem.HasModel)
                            itemsForGrouping.Add(modelItem);
                }

 

in order to get only items with GUID.

 

Any suggestion or help will be appreciated.

 

Regards,

 

Rade

 

0 Likes
Message 30 of 57

luca_vezza
Enthusiast
Enthusiast

Hi Rade,

I would suggest using a Search to get all the items with your target category and/or property, rather than looping and testing directly each one. In my experience this is way faster.

Somethings like:

// filter to keep only items that have the target property category
Search tmpSearch = new Search();
tmpSearch.Selection.CopyFrom(mySelection);
tmpSearch.PruneBelowMatch = false;
SearchCondition sc = SearchCondition.HasCategoryByDisplayName(myTargetPropCategory);
tmpSearch.SearchConditions.Add(sc);
ModelItemCollection targetItems = new ModelItemCollection();

try
{
   targetItems = tmpSearch.FindAll(oDoc, true);
   logFile.WriteLine("      Items with target property: " + targetItems.Count.ToString());
   logFile.Flush();
}
catch (CanceledOperationException)
{
   logFile.WriteLine("   Operation canceled!");
   logFile.Flush();
   MessageBox.Show("Operation canceled!");
   return false;
}

Hope this helps,

 

  Luca

Message 31 of 57

xiaodong_liang
Autodesk Support
Autodesk Support

Hi @luca_vezza,

 

Sorry, it has been a while until I got chance to check this question. The comparison way you used is actually compare the session variables of C#.  Internally, to compare two InwNode is by their nwHandle (of InwBase), but this is internal use only. Fortunately, InwOpState10 provides a method PtrEquals which can compare two pointers of Navisworks objects. 

 

 Assumes a model structure is like:

 

            //Model Item 1 
            //      Model Item 1 - 1
            //          Model Item 1 - 1 - 1
            //          Model Item 1 - 1 - 2
            //      Model Item 1 - 2 
            //          Model Item 1 - 2 - 1
            //          Model Item 1 - 2 - 2
           Document doc = Nw.Application.ActiveDocument;
 
            //assume the first selected item is Model Item 1 - 1 - 1
            COMApi.InwOaPath p1_1_1 = ComBridge.ToInwOaPath(doc.CurrentSelection.SelectedItems[0]); 
            InwPathNodesColl nodes_1_1_1 = p1_1_1.Nodes();  
            InwOaNode parent_1_1_by_p1_1_1 = p1_1_1.Nodes()[nodes_1_1_1.Count - 1] as InwOaNode;
            InwOaNode parent_1_by_p1_1_1 = p1_1_1.Nodes()[nodes_1_1_1.Count - 2] as InwOaNode;

            //assume the second selected item is Model Item 1 - 1 - 2
            COMApi.InwOaPath p1_1_2 = ComBridge.ToInwOaPath(doc.CurrentSelection.SelectedItems[1]); 
            InwPathNodesColl nodes_1_1_2 = p1_1_2.Nodes(); 
            InwOaNode parent_1_1_by_p1_1_2 = p1_1_2.Nodes()[nodes_1_1_2.Count - 1] as InwOaNode;
            InwOaNode parent_1_by_p1_1_2 = p1_1_2.Nodes()[nodes_1_1_2.Count - 2] as InwOaNode;

            //assume the third selected item is Model Item 1 - 2 - 1 
            COMApi.InwOaPath p1_2_1 = ComBridge.ToInwOaPath(doc.CurrentSelection.SelectedItems[2]);
            InwPathNodesColl nodes_1_2_1 = p1_2_1.Nodes();
            InwOaNode parent_1_2_by_p1_2_1 = p1_2_1.Nodes()[nodes_1_2_1.Count - 1] as InwOaNode;
            InwOaNode parent_1_by_p1_2_1 = p1_2_1.Nodes()[nodes_1_2_1.Count - 2] as InwOaNode;


            // parents of item_1_1_1 and item_1_1_2 are same
            Debug.Print($"== { ComBridge.State.PtrEquals(parent_1_1_by_p1_1_1, parent_1_1_by_p1_1_2).ToString()}");  //TRUE
            Debug.Print($"== { ComBridge.State.PtrEquals(parent_1_by_p1_1_1, parent_1_by_p1_1_2).ToString()}"); //TRUE

            // one parent of item_1_1_1 and item_1_2_1 are different
            // while the parent's parent are same
            Debug.Print($"== { ComBridge.State.PtrEquals(parent_1_1_by_p1_1_1, parent_1_2_by_p1_2_1).ToString()}");//FALSE
            Debug.Print($"== { ComBridge.State.PtrEquals(parent_1_by_p1_1_1, parent_1_by_p1_2_1).ToString()}");//TRUE

 

Message 32 of 57

Anonymous
Not applicable

Excellent !
This will be useful in multiple use cases.

Thanks !

0 Likes
Message 33 of 57

luca_vezza
Enthusiast
Enthusiast

Hello,

after a long time, we are back on this issue, still having serious performance problems when dealing with large (>1M) item collections. We have tried the COM approach, which seams faster for looping, indeed, But then we found that problem in comparing nodes, addressed by your solution relying on PtrEquals.

I have just done a test which basically does the following (all in COM):

- get the selected nodes

- loop on them. For each one:

  - loop on all parent and them to a list, using PtrEquals to check that they are not already in that list

Of course this is a quite heavy approach, but we did not find any possible/better alternative.

 

Now, this does not scale either: looks like PtrEquals is also quite heavy. So, for small collections it's fine; but once you get hundreds of thousand items, it's again unmanageable...

 

So, we're back to the initial problem. How can we loop through a huge list of select items and collect all their parent nodes (each one only once)?

 

Any suggestion is welcome of course.

Thanks,

 

        Luca

0 Likes
Message 34 of 57

ngombault
Enthusiast
Enthusiast

Hello 2018, 19 and 21.

Wwritting from 2022 here - soon to be 23 to provide further evidence or findings that may help others, or debuggers. The issue with an empty iteration loop is very noticeable even with 'smaller' collections, around 80k take already several seconds to iterate. A couple other things I have noticed:

1. If one runs the iteration several time (iterate once, then again, then again) the time increases dramatically at each iteration

2. Because of #1, I tried using the first iteration to copy the collection to a standard list<modelitem> (CopyTo, AddRange and foreach all have the same delay), then iterate on that list instead. The copy obviously takes a long time still, but looping on the list after that is a couple milliseconds. Yes that's right, milli-seconds vs full-seconds.

3. On rare and seemingly random occasions, the loop will run in 1/10th of the time, still slow but certainly better (And I measure my words carefully here after literrally days of testing and only a few occurances of that; on the same model, same search, same loop code (empty), clean run with navisworks restart each time).

4. That issue doesn't impact only iterations, native navisworks method using collection also become very slow, e.g. corruptedCollection.CopyTo(freshNewCollection) etc.

 

This led me to conclude that (1) there is a very big overhead on the modelitemcollection, and strangely enough it impacts the read (some marshalling issue?), and (2) that this overhead has some 'leak' that makes it worse and worse after each read. I am no further along making this more efficient but at least realising that the issue compounds on collection re-read, allowed me to switch to lists, run all the operations, and go back to collection only for Navisworks native operation that require collections, e.g. setting the Selection.

@xiaodong_liang@naveen.kumar.t - has anyone from the dev team looked at this yet considering the number of people who experienced it and the time span of the issue?

 

0 Likes
Message 35 of 57

ulski1
Collaborator
Collaborator
Hi,
If you can reproduce this type of issues I recommend that you open a support ticket. Attach sample code to the ticket and it is helpful for them if it can be reproduced using one of the sample models.
It is less likely that you will get bugs logged to development by posting them here.
Br
Ulrik
0 Likes
Message 36 of 57

ngombault
Enthusiast
Enthusiast

Thanks @ulski1, would you mind sharing a link? Somehow, scouting through the forums and knowledge network, couldn't land on the place where ticket need to be raised. Always got my ticket/issue addressed though this forum in the past.

0 Likes
Message 37 of 57

ulski1
Collaborator
Collaborator
0 Likes
Message 38 of 57

ngombault
Enthusiast
Enthusiast

Thanks, doesn't seem like I have access to this portal, after several attemps, redirects and more browsing. One might need to be a partner fo some sort for the access to work...

I still think it is a worthy bug to investigate, with reproducible code in this very forum. Any medium-large collection would do (I am trying 80k but the effect is noticeable before), and any loop run multiple time will magnify the issue.

Message 39 of 57

ulski1
Collaborator
Collaborator
Your company should have at least one person who got the permit to open support tickets
0 Likes
Message 40 of 57

ngombault
Enthusiast
Enthusiast

As it turns out we don't, being a small software dev group. Anyway that is digressing from the original purpose of this post, maybe someone from Autodesk can weight in on the issue and raise a ticket to that effect on our behalf as they have in the past, if it is considered relevant, as I expect bugs/issues with multiple independant report would be. Thanks for the help

0 Likes