Parallel.ForEach() Iteration on Revit Data

[sroswurm4GJQU]

I'm working on an aspect of my program that executes a number of geometry comparisons that run fairly slowly.  I'm looking at ways to shorten response times and one of the options that shows a great deal of promise is allowing parallel iterations via .NET's Parallel.ForEach().  

Now clearly, you're playing with fire if you attempt to get data from a document or modify a document during a parallel iteration.  That would have the same dire consequences as trying to mutli thread without proper use of ExternalEvents, etc.  What I'm interested in instead is the concept of extracting data in bulk and then performing demanding parallel iterations on a list of objects with no requests for new data and no changes to the Document during iteration.

For example, extract all Element location curves or geometry solids from the Document and store them in a list.  Then, use parallel iteration over that list of curves or solids in order to accelerate demanding computation.  Clearly, you would want to painstakingly avoid handling any kind of data that contains an explicit reference to the Document, such as Element, ElementId, etc.  

In testing, these processes appear very sound, but I want to do my due diligence to make sure this approach squares with the intended use and limitations of the API.  Jeremy Tammik answered a Stackoverflow post on a related topic in 2017 (linked here for your reference) .

His response read in part  "My recommendation would be to reduce the Revit API interaction and processing to an absolute minimimum, collect all the data required for processing, terminate the Revit API interaction after collecting the data, and then run the pure calculations with no further Revit API interaction in a separate thread or several threads at a later point after leaving the Revit API context.".

So here is my big-picture question:  Do geometry objects of type XZY, Line, Plane, Curve, Solid, and similar contain data that inherently references the originating document in a way that creates a crashing or corruption risk during parallel iteration?  Or does the approach I'm describing here accurately reflect Jeremy's concept of separating API calls from the mathematical operations?

[jeremytammik]

Thank you for a very interesting and pertinent question, and also for the very apt quote.

I would tend to assume that the simple pure in-memory geometry objects that you list can safely be used in multiple threads. However, as Scott Conover pointed out in the recent Revit API roundtable, there is no guarantee for locking and immutability etc.:

https://thebuildingcoder.typepad.com/blog/2020/11/revit-api-roundtable-notes.html 

Q: I have a few questions. Geometric calculations on multiple threads?

A You can use multiple threads in your own code; the Revit API is single threaded. Certain touch points enforce the main thread, e.g., transactions; some utility classes might allow multithreading, but none of them are guaranteed to be thread safe; they may seem immutable and not really be so.

Personally, I would tend to err on caution and just copy all of the information required for your parallel processing to your own custom objects. After all, the heavy lifting of marshalling it out of Revit memory over to your .NET add-in code has already been completed.

Looking forward to hearing how you fare.

Good luck!

[sroswurm4GJQU]

Jeremy,

Thanks so much for your prompt reply and for the further info from the roundtable.  One of the biggest hurdles I've been facing is time-efficient testing of physical proximity and intersection.  That's why I've been experimenting with bulk collection of solids and then testing them in parallel iteration using BooleanOperationsUtils.  The initial tests are very promising, both in speed and accuracy, but I worry about the unseen risk of crashing on any given run.  Since I've been leveraging Revit classes and methods such as BooleanOperationsUtils, ClosestPointsPairBetweenTwoCurves, Curve.Project(), etc., it doesn't seem feasible to replicate all of this functionality with my own custom classes and methods.

Using ElementIntersectsSolidFilter is a good brute force solution and the FilteredElementCollector itself computes surprisingly fast in most cases.  Unfortunately, actually obtaining the contents that pass the collector using .ToElements() or .ToElementIds() causes a huge hit to performance which definitely becomes noticeable when testing high volumes of elements.  I know this is a topic that has been much discussed in years past.  Any thoughts on this?

[RPTHOMAS108]

I've used this method in the past unsuccessfully, I put it down to lack of thread safety. 

When you call those API helper methods most of the time they will be asking Revit to do the work. So doing that in a multi-threaded way I'd assume would trip you up at some point (given the message we have). 

Sounds like you've done testing so really only need to satisfy yourself to a statistical level in terms of what probability of success counts as certainty. You will likely not get better assurances than the tests you conduct yourself, I doubt anyone will tell you it is ok. This is the burden of being a pioneer.

[sroswurm4GJQU]

Thanks for your feedback.  I think you nailed the question that's on my mind. 

When I use one of these methods, is it asking Revit to do the work (i.e., a 'call to the API'), or is it simply a public method of the class being executed like any other method would in .NET? 

It's not any easy thing to know from our perspective and most certainly requires a great deal of knowledge about what goes on behind the curtain.

[jeremytammik · Accepted solution]

> Any thoughts on this?

Yes, definitely. Just as Thomas says, anything complex is done by Revit internals and is not thread safe.

> obtaining the contents that pass the collector using .ToElements() or .ToElementIds() causes a huge hit to performance

If you are retrieving that much data and marshalling it over to .NET, there is no way to achieve high performance.

Furthermore, the calls to those two methods are often superfluous anyway, and themselves cost significant time and space. Red flags for me, those two.

> FilteredElementCollector itself computes surprisingly fast

Conclusion: find a way to make use of the built-in Revit functionality without marshalling out any data whatsoever, or only the final results, to .NET.

Here is a recent example of achieving exactly that:

https://thebuildingcoder.typepad.com/blog/2020/10/high-performance-outline-and-line-plane-intersecti...

Alexey Ovchinnikov presents a solution to find an outline for many elements (>100'000 items). Target elements come from a FilteredElementCollector. As usual, I'm looking for the fastest possible way...

I look forward to seeing a similar solution for your case.

[sroswurm4GJQU]

I should have replied to this thread some time ago.  I plan to continue evaluating this algorithm over time to find other ways to improve efficiency, but for now I was able to make a huge leap in efficiency with a simple change inspired by the example that Jeremy linked above.  

I added an additional filter using "BoundingBoxIntersectsFilter".  The outline used by this quick filter drastically reduces the number of elements which are subsequently tested using the ElementIntersectsSolidFilter.  As time permits, I will also seek out a way to completely avoid marshalling out to .NET lists using .ToElements() or .ToElementIds().

[jeremy_tammik]

Glad to hear it helped.

Please note that this huge leap in performance is achieved by using a quick filter up front, and not by multi-threading.

Afaik, that is the only way to achieve this kind of performance when filtering for Revit elements in a large BIM.

Good luck and much success continuing on this path and taking it even further.