Search efficiency and How to do it kind like a Regex

[Marauj0]

Hi,

I'm learning to program a plugin for Navisworks, I started by grabbing the ModelItens from the Current Selection and doing a foreach to see each ModelItem display name and assign it to the corresponding classification accordingly to its string pattern and then paint them, but now I wanted to expand, and try to do it to the entire file that have hundreds of thousands of itens, and i got the following error: 

"Managed Debugging Assistant 'ContextSwitchDeadlock' : 'The CLR has been unable to transition from COM context 0x774a81f0 to COM context 0x774a80c8 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. how to solve this error in C#?"

I believe that i'm getting this error because of the ineficiency of foreach, i tryed to use Threadding but I get other COM errors then, so I did a research and i'm thinking of using the native Search + SearchConditions for efficiency, I think that I will not have a problem with the first half of ModelItens I look as it's just those that the display name contains "/FRMW", but I don't now how to do a SearchCondition with the conditions that I used for the other half, as I used Regex to find those itens

the Regex pattern I used for the .IsMatch is : string primPattern = @"^(\/S-)\d{4}(-)\d{4}";

and here is how the DisplayName I wanted to select is  "/S-2020-0245" the numbers change, but it is always /S-XXXX-XXXX where X is numbers

right now this is what I'm thinking to do to replace the foreach:

        Search s = new Search();

         s.Selection.SelectAll();

         s.Locations = SearchLocations.DescendantsAndSelf;

/*

As the Model Itens I'm seraching are not at the botton of the tree but in the middle I think this Prune will make the code more efficient so as an exemple of the tree below I believe it will ge to the /S-2010-2212 and then it will stop going into the brance and them go to the next at the same level

*/

         s.PruneBelowMatch = true;

//I believe the best way to mimic the regex will be by a list of Search Conditions with wildcards but I don't know exacly how

List<SearchCondition> oG1 = new List<SearchCondition>();   

   SearchCondition oGroup1_SC1 = … …   

   SearchCondition oGroup1_SC2 = … …    

oG1.Add(oGroup1_SC1);

oG1.Add(oGroup1_SC2);

   s.SearchConditions.AddGroup(oG1);

// And them repeat this process to the simpler search condition of Contains "/FRMW"

  So recapping, I want to know if for what I want to do and the volumen of it the use of the Search and how I'm thinking will solve the DeadLock I'm getting, if it is really the fastest way for doing so and How to make it mimic the regex.ismatch I use in the foreach.

[alexisDVJML]

Hi @Marauj0 ,

No regex support in Search BUT

you can use SearchConditionComparison.DisplayStringWildcard

with @"/S-????-????" as wildcard pattern.

This will be less strict then your regex so you will have to post filter the result of the search.

Ideally you would use FindIncremental to avoid building the full list in memory.

I use similar technique in our plugin with great success: fast, low memory usage.

Can even show progress 😉

Hope this helps.

[alexisDVJML]

BTW, you can try this first using Navisworks Find Items panel 🙂

[Marauj0]

Thank you for the answer,   I will try it tommorow and than give you the feedback of how it went, researching i found about this wildcard but there is not that much about it on the internet

[alexisDVJML]

Let me know if you have issue implement this, I will try to find time to write and test something similar that is independent.
The code I have for our own plug-in rely heavily on classes we implement to simplify such stuff, so I can't just extract an example for this 😞

[alexisDVJML · Accepted solution]

OK, I was interested in seeing how such thing would actually work in real case, aka with a 2GB model 😉
So I quickly implemented and tested exactly that.
Code included below.
Not fully commented but sure you will understand it.

Just call SearchMatchingItemName with your regex pattern, wildcard pattern and wanted options.
Then do what you want with the returned result, ex Select matching items:

Application.ActiveDocument.CurrentSelection.CopyFrom(matchings);

It works really nice and fast, like less than 3s in a huge model using a pattern similar to yours, with ONE exception: combining FindIncremental and ShowProgress make it run like 50x time slower with the mouse cursor flickering 😲.

Hope this will work as nicely on your side.

Here the code:

[Flags]
public enum SearchMatchingNameOptions
{
	None = 0,
	Incremental = 0x01,
	ShowProgress = 0x02
}

// These NamedConstant has often used, so declare them static to
static private readonly NamedConstant cItemCategory = new NamedConstant(@"LcOaNode");
static private readonly NamedConstant cItemNameProperty = new NamedConstant(@"LcOaSceneBaseUserName");

static ModelItemCollection SearchMatchingItemName(Document doc, string aRegexPattern, string aWildCardPattern, SearchMatchingNameOptions options)
{
	// Create a compiled regex for efficiency.
	// If aRegexPattern is always the same, should be put in a static field
	var regex = new Regex(aRegexPattern, RegexOptions.Compiled);

	// Create a Search with aWildCardPatten as
	var search = new Search();
	var condition = new SearchCondition(cItemCategory, cItemNameProperty,
							SearchConditionOptions.IgnoreDisplayNames,
							SearchConditionComparison.DisplayStringWildcard, new VariantData(aWildCardPattern));

	search.SearchConditions.Add(condition);
	// PruneBelowMatch is set by default, just put here for reference
	search.PruneBelowMatch = true;
	search.Selection.SelectAll();

	bool fShowProgress = (options & SearchMatchingNameOptions.ShowProgress)!=0;
	bool fIncremental = (options & SearchMatchingNameOptions.Incremental)!=0;
	var matchingsWildPattern = fIncremental ? search.FindIncremental(doc, fShowProgress) : search.FindAll(doc, fShowProgress);

	var matchings = new ModelItemCollection();
	foreach(var modelItem in matchingsWildPattern)
	{
		if (regex.IsMatch(modelItem.DisplayName))
			matchings.Add(modelItem);
	}

	return matchings;
}

[alexisDVJML]

2 comments:

- I almost always use InternalName instead of DisplayName for categories and properties so no issue if running in a localized version of Navisworks. Exception is for some properties generated when exporting for some 3D software, where all properties have the same InternalName and thus must use the DisplayName!!!

- I tend to put common NamedConstants in static fields, to avoid recreating them, but I have not profiled it. Creating on the spot will probably ensure better locality so can't confirm which option is better...

[Marauj0]

Just sit down here after a cup of coffee do start doing the work and saw you response, I'm greatly appretiated, and you are right about the names the model i'm used to test is exported from a engineering program so the internal stuff are shared among them all, and the best way i found to differentiate one from another is by the display name

[Marauj0]

I created a new folder to test your suggestion and it worked just like my previous build, and took the same several minutes to do so, then I realized that when I copied the folder, the output bin was not carried on and returned to the default at the VisualStudio paste my mistake, when I updated the output to the plugin page of navisworks to change the dll there I got some impressive results:

Doing just what I asked you and you gave as a answer the reduction in time was 90%+ If I want to remain within what you made me, istead of minutes it takes just seconds to select all the itens needed, painting them with the .OverridePermanentColor just a couple second more but then I wanted to paint all the rest grey to highlight what I want to see, and boy it took a good 2 minutes to do so.

Then I was able efficiently to filter to about 2000 itens, from the entire list in seconds, by your method, but then I went with a foreach + LINQ in this list of 2000 itens to compare it with another list with 1000 itens to see what is missing and again a good 1-2 minutes is lost in this process, so foreach efficiency seems to be really arround 10% of the usage of the native code , SO again Thank You for your prompt, precise and great answer, it worked like a glove, now I will study it and test if using something similar in the comparasion list instead of using the foreach + Linq will improve things just to see,

[alexisDVJML]

Great to hear it worked.
Please mark my post as an answer, easier for other forum users to directly go to it.

Not sure about your LINQ search, was it just to debug/test that it works?
Maybe you can just use 2 HashSets build with each list and then check the difference with HashSet.Except?
That should take just a few seconds at max.

[alexisDVJML · Accepted solution]

Re: your speed issue with OverridePermanentColor.
Not sure how you do it.
But below a method I just wrote and tested that color ALL except the passed list in less than a second.
In addition you can cancel with a single Undo 😉

However there is a catch, this will work provided none of your model items have user overriden transform.
You can probably improved the sample code to also support this by using the same trick I suggested re: HashCodeSet.Except. Lazy to do it right now haha;

Quick explanations:
1/ OverridePermanentColor need a full ModelItemCollection.

Building it by regular means take a long time while using below code is near instantaneous:

Application.ActiveDocument.CurrentSelection.SelectAll();
var all = Application.ActiveDocument.CurrentSelection.SelectedItems;

2/ Putting multiple changes in a single transaction speed up things A LOT, up to x50 if you do lot of operations at once. And it allows to cancel all with a single Undo Ctrl+Z !

Here the promised code:

static private void ColorAllExcept(ModelItemCollection items)
{
	var d = Application.ActiveDocument;
	using (var t = new TransactionHandler(d, "Color All Except Some :-)"))
	{
		var currentSelection = d.CurrentSelection;
		currentSelection.SelectAll();
		Application.ActiveDocument.Models.OverridePermanentColor(currentSelection.SelectedItems, new Color(0.5,0.5,0.5));
		ComApiBridge.State.OverrideReset(ComApiBridge.ToInwOpSelection(ModelItemCollectionImpl.ToCollection(items)));
		t.ReadyToCommit = true;
	}
}

[Marauj0]

Already marked as a solution, thank you again!

What I did was similar to yours but without the TransactionHandler and the ComApiBridge, I already separated the function that color all the itens grey so I would click the button and use it only once. 

I will post here the results

I know that ComApiBridge is from Autodesk.Navisworks.Api.ComApi but the method 

TransactionHandler

is from what library? I though it was from some dll from navisworks, but the ones I tried didn't have, and searching the internet it seems it is from System.Data.Entity.Infrastructure of Microsoft, but when I asked chat GPT he said its wrong that it is actually from System.Web.Http.OData;

I'm actually new to programming, I have a graduation on Civil Engineering and start learning programing 3 months ago to solve some challenges on AutoCAD, and dived into naviswork only two weeks ago, the current company that I work put several people doing stuff by hand on those programns and I believed that it could be better done with plugins and automation, so I'm learing this stuff by myself at home, AutoCAD was easy, but navisworks is super hard, there is too few information online about programing on it, So I ask you more because I want to learn, the HashCodeSet.Except. form exaple is pretty cool I'm going to start using it as severaltimes what I'm doing is comparing lists, as i said the current method I'm using is already just 3 to 4 minutes, to do something that on my working hours take at least a week doing by hand.

[alexisDVJML · Accepted solution]

Oups, my wrong, TransactionHandler is one of the class I developped to simplify using Navisworks Transaction.

It also includes the option to disable undo etc. plus being an Idispoable, it is nicely release.

Full source code of the class below.

Licence granted to you for use in your projects, not for resale.

BTW, for a new guy in the block, you tackle really hard challenges.

I have more than 40 years of programming experience and he took me more than one year to get some confidence writing Navisworks plug-ins, so kuddo to you 👍

Keep in touch.

// Copyright Idigo Pte Ltd 2019-2023, All Right Reserved
// author: Alexis Martial, alexis@plixo.com.sg
//
// CLASS/STRUCT: TransactionHandler
// NAMESPACE: Idigo.Navisworks
//
// DESCRIPTION:
// 
// USAGES:
// 
// COMMENTS:
// - Navisworks Transaction, while having code that looks like Rollback and Commit actually ALWAYS commit even if Dispose() is called aka exception is raised
//		instead a Transaction just call the document BeginEdit when created and EndEdit when disposed !!!
//		so there is NO WAY for now to cancel already done operations in a transaction (except maybe for the embedded database provided we handle it)
//		STILL good to have a shcme that will directly support it WHEN available/supported in Navisworks
// - There is NO WAY to access a document ongoing transaction since there is no link maintained from document to transaction
//		this also mean we can end up with a transaction pointing to a zombie document! excpet by subscribing to d.TransactionBeginning and t.TransactionEnded
//		which should be done once for a document
//
// COMPATIBILITY:
//
// DEPENDENCIES:
//
// KNOWN BUGS:
// ...
// TODO:
// - more careful handling of Undo when multiple transactions are "stacked" ?


using System;
using System.Diagnostics;

using Autodesk.Navisworks.Api;

namespace Idigo.Navisworks
{
	//AMA, TODO? implement it as a struct IF small enough to avoid an extra object + expected to be used locally
	public class TransactionHandler : IDisposable
	{
		[Flags]
		protected enum Flags
		{
			None = 0,
			OwnTransaction = 0x01,
			ReenableUndo = 0x02,
			ReadyToCommit = 0x04,

			Disposed = 0x1000,
		}

		// need to keep it if we need to undo and we don't own our transaction
		// can get it from Transaction only IF we have created it
		protected Document pDoc;
		protected Transaction pTransaction;
		protected Flags pFlags;

		// NOTE: dummy transaction, pass d = null, this allows to use using (var t = TransactionHandler) WITHOUT creating an actual Navisworks transaction
		public TransactionHandler(string name = null)
		{
			pFlags = Flags.None;
		}

		//AMA, TODO, remove the fDisableUndo flag here, instead requires user to make a specific call
		// more explicit + can better handle the conflicts for stacked transaction
		// NOTE: for a dummy transaction using this method interface, pass d = null, this allows to use using (var t = TransactionHandler) WITHOUT creating an actual Navisworks transaction
		public TransactionHandler(Document d, string name, bool fDisableUndo = false) //bool start = true IS it really useful< worst case we create a transaction and do nothing, not a big deal
		{
			pFlags = Flags.None;

			if (d is null)
				return;

			//we disable undo ONLY if requested + not already disabled + not already in the transaction (so as to avoid inconsistent Undo
			if (fDisableUndo && !d.IsUndoDisabled)
			{
				if (d.IsActiveTransaction)
				{ } // Debug.Fail(@"Conflicting DisableUndo set while already in a transaction, Undo will NOT be disabled to avoid inconsistencies");
				else
				{
					d.StartDisableUndo();
					pFlags |= Flags.ReenableUndo;
					pDoc = d;
				}
			}

			if (!d.IsActiveTransaction)
			{
				pTransaction = d.BeginTransaction(name); 
				Debug.Assert(pTransaction != null);
				if (pTransaction != null)
					pFlags |= Flags.OwnTransaction;
			}
		}

		//public void Start() //string name = null)
		//{
		//	if (fSubTransaction || (transaction != null))
		//		return;

		//	transaction = doc.BeginTransaction(name); // name ?? this.name);
		//	Debug.Assert(transaction != null);
		//}

		public bool ReadyToCommit
		{
			get => (pFlags & Flags.ReadyToCommit) != 0;
			set
			{
				if (value)
					pFlags |= Flags.ReadyToCommit;
				else
					pFlags &= ~Flags.ReadyToCommit;
			}
		}

		~TransactionHandler()
		{
			Debug.Assert((pFlags & Flags.Disposed) != 0, @"Dispose NOT called on TransactionHandler or its subclasses");
			//Debug.Assert((pFlags & Flags.Disposed) != 0, @"Dispose NOT called on TransactionHandler or its subclasses");
			//Debug.Assert(pFlags == 0);
		}

		public virtual void Dispose()
		{
			if ((pFlags & Flags.OwnTransaction) != 0)
			{
				if ((pFlags & Flags.ReadyToCommit) != 0)
					pTransaction.Commit();
				pTransaction.Dispose();
			}

			if ((pFlags & Flags.ReenableUndo) != 0)
			{
				Debug.Assert(pDoc != null);
				Debug.Assert(!pDoc.IsDisposed);
				pDoc.EndDisableUndo();
			}

			pDoc = null;
			pTransaction = null;
			pFlags = Flags.Disposed;

			GC.SuppressFinalize(this);
		}

	}
}

[Marauj0]

Its the dispair, it is a great incentive, sitting in my chair working like a machine for 9 hours a day doing nothing more copying information from one file and pasting it in a excel sheet, made me think, if I'm working like a robot, why not make a robot to make my work, the information is already on the file, let me take out of it, so I can do something more human like think.

I Copied the code to Visual studio to study it already, I want to learn more special how you use Flags that is something new to me, so if you dont want it hanging on here you can delete.

But when I worked with Autocad it looked kind easy to do stuff on it, so I thought that it would also be like it with naviswork, if I new what navisworks was, I probably would not have got half the way where I got, not knowing how hard it was I simply keept insisting until I did it, but then I realize that it was not efficient so I came here for help.

Again, I'm Greatly appretiated by the class you gave me.

[alexisDVJML]

I'm a fan of [Flags] enum:
- allow to pass multiple "bool" at once
- easier to read and understand than many bool as parameters, less error prone.

Just C# is very verbose to use them...