Community
ReCap SDK
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

RCP Reading Performance and Thread Safety

3 REPLIES 3
Reply
Message 1 of 4
philip.holzmann
352 Views, 3 Replies

RCP Reading Performance and Thread Safety

We're interested in reading large ReCap projects quickly, i.e. extracting all points, for further processing.

However, we're getting very low performance so far (<50 MB/s). Is this expected?

 

Our reading code looks like this, which is the recommended way, as far as I know:

Autodesk::RealityComputing::Data::RCPointIteratorSettings iterSettings;
iterSettings.setIsVisiblePointsOnly(true);
iterSettings.setIsReadOnly(true);
auto point_iterator = project->createPointBatchIterator(iterSettings);

while (point_iterator->moveToNextBatch(1024))
{
    const auto &batch = point_iterator->getPoints();
    for (const auto &point : batch)
    {
        const auto &xyz = point->getPosition();
        const auto color = point->getColor();

        // use the point ...
    }
}
point_iterator->close();

 

Other things we have tried to improve the performance:

  1. Reading multiple scans in parallel (using the same RCProject in all threads)
    • Create an iterator for each scan, and spawn a thread for each iterator.
    • This ends up being much slower than using only one thread.
  2. Reading multiple scans in parallel (from multiple RCProject objects)
    • Open an RCProject (on the same file) for each thread
    • In each thread, create an iterator (for a distinct scan)
    • This, again, is slower than using one thread.
  3. Reading multiple scans in parallel, from multiple processes
    • Same as (2.), but we start full processes (not just threads). Each process reads one scan.
    • This is much faster!
    • However, this is not possible with unified scans (because they contain only one scan)

Now, our questions are:

  1. Thread Safety:
    • Is reading the multiple scans from multiple threads safe?
    • Is reading the same file from multiple processes safe?
  2. Performance in general:
    • Is there a way to speed up reading? Is there something we may be doing wrong.
3 REPLIES 3
Message 2 of 4

Hi @philip.holzmann ,

 

We do not guarantee thread safety. For accessing the same file from multiple processes, I am a bit curious about how the work will be split.

 

Regarding to the code you posted, may I know how fast is the following (which is modified based on your code)?

Autodesk::RealityComputing::Data::RCPointIteratorSettings iterSettings;
iterSettings.setIsVisiblePointsOnly(true);
iterSettings.setIsReadOnly(true);
auto point_iterator = project->createPointBatchIterator(iterSettings);

while (point_iterator->moveToNextBatch(1024))
{
    point_iterator->getPoints();
}
point_iterator->close();

If it is fast enough, could you check to parallelize the point access part in a way like this (which is modified based on your code. But you need to implement it properly to access points in parallel)?

Autodesk::RealityComputing::Data::RCPointIteratorSettings iterSettings;
iterSettings.setIsVisiblePointsOnly(true);
iterSettings.setIsReadOnly(true);
auto point_iterator = project->createPointBatchIterator(iterSettings);

while (point_iterator->moveToNextBatch(1024))
{
    const auto &batch = point_iterator->getPoints();
    // access points in parallel instead of one after one
}
point_iterator->close();

 

Two other suggestions:

  • You can change the batch size to be much bigger to get better performance.
  • If it is not really necessary to get VISIBLE points, could you call setIsVisiblePointsOnly with false? This will have better performance for some projects. You need to make this decision carefully depending on the requirement.
Message 3 of 4


For accessing the same file from multiple processes, I am a bit curious about how the work will be split.

 We run one process for each scan in the project. Each process gets a scan index that it should process. This, of course, won't work with unified projects (because they only contain one scan).

Do you know if that is safe? It should be, unless the ReCap SDK writes temporary files during reading, which could cause the processes to interfere with each other.

 

Your suggestion of benchmarking the outer loop only is interesting. However, it does not make significant difference:

Only running the outer loop takes 2min 1s; with the inner loop, the time is 2min 5s. So the expensive part is indeed the point_iterator->moveToNextBatch().

 

Varying the batch size is indeed useful. We found it improves the runtime from ~3min (at 1k batch size) to ~2min (at 32k batch size). Larger batch sizes degrade the performance again.

 

Not setting setIsVisiblePointsOnly, too, improves the performance a lot (to ~ 1min), however we rely on only processing the visible points, so we have to keep it in.

 

Still, running multiple processes is by far the fastest option (reading the whole project in 18s).

Message 4 of 4

good information

Can't find what you're looking for? Ask the community or share your knowledge.

Post to forums  

Rail Community


Autodesk Design & Make Report