Multi-threaded import

[JeradNJ]

Ever since learning of Decap, I have been testing the limits of what my computer is capable of for pointcloud processing. My work flow is as follows:

I open a command line (as administrator) then open Windows Task Manager > Details and check for the conhost.exe with the larger memory set. I "Set Affinity" of that exe as well as the cmd.exe to a single thread. Then I replicate this task 7 more times (utilizing a different thread for each instance) as I have 8 threads available. Next I make sure all of my FLS files are in a single folder and I create 8 separate List control files essentially dividing my overall load by 8 (72 scan... List1 = Scans1-9, List 2 = Scans10-18, etc). Then I set up my syntax something like:

decap.exe --importWithLicense "C:\ScanFolder" Project1 --controlFile "C:\ScanFolder\List1.txt" --decimation 1

Where I can copy this syntax once, then paste it into each command line and simply increment the number in each instance.

Once all command lines are rolling my CPU ramps up to nearly 100% for the duration and I get the entire project's scans processed in no time flat. I have noticed this creates 8 separate instances of the AdskFAROConverter.exe program which is normally my bottleneck when using Recap. After all of the Decap's are complete I compile all of the Project Support folders into one folder (I call CompiledSupport), then I start Recap fresh. I create a new Project.rcp, then when it asks for the folder location to import, I reference the CompiledSupport folder and everything loads in very quickly, ready for registration. I will also note that once registered, whether Manual or Automatic, the Indexing process is completely gone... again, major time savings. 

I should also note I utilize a SSD for this procedure, as I'm sure a standard spinning hard drive would choke on all of this. And even with 8 instances running, the only bottleneck is my 3.3 GHz i7 CPU. My RAM never seems to top 12GB, and my SSD barely notices the effort. All of which I am OK with, just interesting to see the biggest bottleneck in everything has to be the AdskFAROConverter.exe operation's CPU load.

I have found this is streamlining my overall process and saving me major hours of sit-and-wait time. While I recognize that this has already saved me a lot of time, I am curious if there is a way to integrate this method into Recap and make my life easier? Possibly by adding in certain options that the user has to adjust so that this doesn't default to "taking over" a machine without the user's consent (and keeps everyone happy). I would imagine it would be something like, "You have x CPU Threads available, how many would you like Recap to use?" So that way if a user has 8 threads available, but wants to keep working on something else while Recap grinds, they can set it to 6 (or 2, or 4, or whatever) and still have the remaining threads for other tasks. This would also open up major opportunities to improve registration times with hardware. A quick search of NewEgg shows there is an i9 processor with 18 Cores/ 36 Threads. If I knew I could work on other tasks, while importing 34 scans at once, utilizing 34 Threads directly in Recap (through the computer's programming, not my own "Set Affinity" efforts), I would buy that CPU in a heartbeat... and probably a new mobo with an M2 drive and some ridiculous RAM as well... but I'm veering off topic.

TLDNR - Based on this user's tested and utilized methods, can Recap's programming be modified to multi-thread certain tasks such as Import and Indexing?

[dgorsman]

If they are working with third-party tools to support other formats, they may not have the opportunity to make many modifications.  If you have any FARO contacts, you might want to give them a poke to see what modules they are providing to other developers and whether those are in fact capable of native multi-threading.  Otherwise Autodesk has to fiddle around with some kind of scripting/scheduling front end.

And you don't want an i9 for a desktop CPU - most of the time all those (expensive and slow) cores are going to be sitting idle which isn't very cost effective.  If you're open to spending that much money on a box, create a secondary "render box" that sits beside your main computer that you can offload computationally expensive tasks on an as-needed basis.

[JeradNJ]

@dgorsman What I am doing does not alter the FARO package in one bit, it simply compartmentalizes each task to run it through multiple threads. That's why when I use my 8 Threads I get 8 separate instances of the AdskFAROConverter.exe in my task manager. It isn't multithreading the individual task, it is just running multiple instances of one task simultaneously. My statement is that by utilizing this method, Recap as a whole will be a multithreading program, the FARO package would not. An analogy... this process is like a pitcher of beer. To fill 8 glasses of beer, I'm not trying to pour the beer into 8 glasses from one pitcher, I am taking 8 pitchers and pouring them into once glass each, all at the same time.

In my current set-up I keep Recap on a laptop that's sole function is Recap, so when I max out it's CPU, I couldn't care less. I keep working on Navis/CAD on my desktop and experience no slowdowns. I do realize that some people don't have this option, so for flexibility sake, having a user controlled option would prevent people from experiencing lock-ups on machines they use for Recap with other programs. Or if someone wasn't comfortable using all of their resources... just in general a user-level of control.

The i9 statement was more to illustrate the trend in hardware is towards more cores for increasing work load, as the ability to perform per core hasn't increased that tremendously in the past 5 years (and I'm talking MIPS, not the gigahertz myth). It is a truth that computer programming needs to focus heavily on opportunities to utilize multiple threads whenever possible, because this can make such a vast overall impact on the time involved in processing data heavy operations, such as pointclouds.

[dgorsman]

My point was, that it may not be possible for *Autodesk* to change the process for third-party components.  Ultimately it would be best, as you indicate, if that module itself is multi-threaded so the necessary development and maintenance is being done by the third party.  That way Autodesk doesn't have to devote development resources to creating some kind of scripted/scheduled front end to manage how and when successive processes are spun off.

[JasonMeshach]

@dgorsman I believe you're missing the entire point of JeradNJ's post.  He wants to be able to run Decap which essentially calls "AdskFAROConverter.exe" on a single thread on every thread that is available on his machine.  If he has 8 cores, he wants to run 16 instances of "AdskFAROConverter.exe" because it's 15 more instances running than the current choice of only 1 without manually running and delegating resources to each instance of the executable.  But he wants to make the entire process MORE USER FRIENDLY!  He's already doing it, he just wants to make it way easier to do than his method he described.  Could he write some batch files to automate some of it?  Maybe.. but the point of this post is to see if the developers can add this into Decap as it would be a huge benefit for everyone.

Essentially the time it takes to import/index 1 raw scan would take the exact same time it would take to import/index 16 raw scans as there would be a dedicated thread simultaneously working and completing at close the same time.  This is an order of magnitude of time savings on scale with threads available on the hardware being used.

He's not asking for it to be added in as a feature.  He's already doing it.  He's asking for it to be way more user friendly than manually setting up all these instances into workable groups of scans for each command line instance to process.  This would benefit anyone who needs to get through a large amount of scans faster.

I don't know about you, but when I'm waiting for 80+ scans to import for hours and then finally get to register them only to be waiting hours again to index them, I would be completely on board with a way to make it X amount of times faster where X is the number of threads your processor has available for Decap.  If by some luck you have less scans than you have threads available, perhaps they could code it in a way where mulitple threads could work on a single import/index. 

[JeradNJ]

@dgorsman I think most definitely this falls onto Autodesk to change how its product utilizes a 3rd party's component, as there is no incentive for FARO to change anything (they have a competing product called SCENE). And the level of programming difficulty which I am proposing has to be incredibly low, as I am already doing this myself in a rudimentary manual fashion. If I had any advanced degree in computer programming I would definitely automate this for myself, but my background is actually mechanical engineering... I just have an aptitude for computers.

@JasonMeshach you are correct. The idea is to have some methodology of automating this procedure as I am already currently doing this. I have googled some about batch files and will explore this as an option for myself. Thank you for calling my attention to that. Also, the indexing being completed before registration is an amazing time saver, you should definitely check this out.

[klarinda.gunadi.widjanarko]

Hi all,

We have released a Beta where we implemented multi-threading import and indexing in ReCap.

We'd like to invite your participation to test the build.
See following post for more details: https://forums.autodesk.com/t5/recap-forum/recap-2023-1-beta-participation/td-p/11319694