Large Data Sets (revisited)

[mnorton]

How do you model surfaces with large data sets?  According to one Autodesk product manager, Civil 3D is not well suited for dealing with large data sets and I'm just looking for a little direction.  I'm not sure anymore where "large" begins so, for the sake of this discussion, let's just assume the number will end with at least six zeros.

[AllenJessup]

I don't have any personnel experience but most of it is covered here Working with Large Data Sets

I think what is meant is that C3D not well suited to working with LDS's in the standard methods that most of us are familiar with. C3D will handle it if the tools created to deal with the LDS's are used. I think most anyone would consider Point Cloud Data sets to be "Large".

Definition of Large Data Set from above source

• A surface with more than a million points
• A subdivision with more than 200 lots
• A corridor longer than 15 km (9.3 miles)

Allen

[mnorton]

Happy New Year, Allen.  I'm familiar with that document and the tools and methods for dealing with most normal data but I keep bumping my head on projects way beyond your bulleted items. A recent example, a 90 km pipeline route with over 7 million LIDAR points.  Data boundaries are a huge help but my client doesn't exactly embrace the surface simplification concept.  He struggles with the fact that data collection has so far outpaced civil software and thinks that surface simplification is tantamount to "dummying down" data that he has paid for.  I encounter the same mindset on SONAR projects.  Even if I can convince them that we're not digging dirt by the spoonful and they accept simplification I have to reduce the points by almost 90% to get under that one million point ceiling.  Now, we're into an area that I'm not real comfortable with which brings me back to my original question.  I'm a huge Civil 3D fan but its development seems to have stalled over the past couple of releases and I guess I'm looking for... more.

[AllenJessup]

I don't have any experience with projects of that size. A good question to ask is if there are programs that are built to handle that much data. If there are, do they also do design? Can they output an XML file that C3D can reasonably use?

I understand the data densificatioin problem. I've been at this long enough to remember when the only things that were located with better than 1'± tolerance were boundary control and traverse points or anything that you would need an exact offset to. The rest was done "Topo" style with a transit and level rod. Angles were measured to the nearest 15 minutes and distance was to the nearest foot from the Stadia hairs. We plotted that with a protractor and scale and did the contours by hand. Now we just use the Robotic Total Station to shoot everything is sight.

Maybe you should have you clients watch the "Dirty Jobs" episode that just aired. Mike was working with a paving crew. I wonder how close to the Design Value the cross slope was when he was operating the screed.

Allen

[mnorton]

Frankly, I've struggled with much smaller data sets attempting to delineate drainage subareas.  I can't be the only one having this problem and I even posted three time in hopes of eliciting some kind of response from Autodesk.  Anything would be better than silence - I don't know or I can't talk about it or anything.

[Neilw_05]

---

@AllenJessup wrote:

I don't have any experience with projects of that size. A good question to ask is if there are programs that are built to handle that much data. If there are, do they also do design? Can they output an XML file that C3D can reasonably use?

I know it doesn't go over well to mention Bentley's products here but since you raised the question I'll go ahead and put my neck out again. Their products tend to handle large datasets more efficiently than Civil 3D and even vanilla Autocad in my experience. They too have integrated point cloud technology(http://vector1media.com/spatialsustain/bentley-incorporates-point-clouds-as-an-integrated-data-type...., http://www.bentley.com/en-US/Products/Bentley+CloudWorx/) and I have seen claims of billions of points. In my direct personal experience with Power Civil I have succesfully built DTM's from nearly 30 million points. One advantage is the surface can be stored in a database and utilized by the design tools without having to load any of the surface graphics at all. Of course C3D can too if the surface style is set to no display but even so, Microstation utilizes fewer resources and supports 64 bit processes for some of it's inner workings.

However even if you could generate an XML file for imprt into C3D I think you'd run into the same limitations.

[mnorton]

Thanks, Neil.  To be perfectly honest, I tend to dismiss their products purely out of habit.  I have an excellent local resource.  I'll check with him and post my results.  Ha!  I said all that without mentioning the "M" word!

[AllenJessup]

So that may be the answer for people who work with insane amounts of point data.

I do remember working with one large data set where at one point I got a message that C3D was saving the Surface to an external file because of it's size. I didn't look in to it further because the drawing was working fine. That was in R2009. I was working with a County-Wide terrain model that was created from DEM's.

Allen

[Neilw_05]

---

@AllenJessup wrote:

So that may be the answer for people who work with insane amounts of point data.

---

Regretably I have run into performance limitations with datasets that would not typically be considered "insanely large". Aerial imagery and DEM files have been issues for me and others. However since I am limited to a 32 bit environment those problems may not occur so readily in a 64 bit environment.

If I may interject some insight I may have in regards to the performance issues with C3D and Autocad, I've noticed that when I load some of our Land Desktop drawings into Microstation, performance in Microstation is incredibly snappy by orders of magnatude. Inquiring into why this might be I was told by a member of Bentley's development team that the difference is likely due to ARX technology. When a file with ARX components is opened in Autocad, Autocad is constantly scanning and updating those objects for or when any action my cause the objects to change (i.e. changing to a 3D view orientation of a surface object requires a different display state for the object). Since Microstation does not support the dynamic state of ARX objects (they are loaded as ProxyGraphics), it is not affected by the refresh process.

I have noticed significant performance degredation with each release of Autocad in recent years and I suspect this is due to an increase in the number of ARX components implemented in the format. If I am correct then I expect the performance issues with C3D and Autocad are going to continue to be a plague until there is a change in the paradigm.

[AllenJessup]

Well. You can't go by me. I'm still giddy from being allowed to upgrade to Win 7 64 bit. I haven't worked with any large sets of data yet. So I can't comment on that. But I've seen a great improvement in the overall operation of the program. No more long coffee breaks 😞

Allen

[mathewkol]

I have a client who routinely works with 5 million point surfaces.  1 million should be a breeze for Civil 3D and there is no reason to decimate 7 million points down to 1 million, in one of your examples.  He doesn't work with 90km roads though so I can't vouch for performance of that.  You may need to have multiple alignments to get smooth performance.

You WILL need Civil 3D 2011, Windows 7 64 bit and tons of ram.  If you don't have those things then there's nothing we can do for you.

Your best bet is to use your reseller or another suitable resource who can come to your office and help you with these things.  it's tough to do this on a discussioun group other than to give you a few tidbits of advice.  Someone in your office beside you should help immencely as long as that person is skilled enough.  This will not be a free call but in the long run it will save you.

[mnorton]

Thanks, Matt, but I think my hardware is adequate and I spent the last eleven years as the Civil AE for a local reseller.  I'm encountering the same issues on multiple machines, multiple drawings, and multiple projects.  So far, nobody has stepped up with a viable solution.  Can you identify anything different in your client's setup?  Is there anything about it unique?

Windows 7 (Ultimate 64)
AMD Phenom X2 550 Processor @ 3.10 GHz
ATI Radeon HD 5700 Series
8 GB RAM
AutoCAD Civil 3D 2011 (64)

[Matt.Anderson]

Mike - What type of watershed issues are you having? Can you contact me? Matt dot Anderson at autodesk dot com

[mathewkol]

So what kind of performance issues are we talking here?  Can you be fairly specific with what you're not happy with?

How are you creating surfaces with that LiDAR data?  Create surface, Add Point File?  Any other data besides the LiDAR files?

Roughly, how far apart are the LiDAR points?  If it's 7 million, and you have 90km I would think the spacing is not too terribly close.  Sounds like a reasonably sized LiDAR file for 90km.

Have you considered splitting up the EG surface into more manageable chunks?

You mentioned development has stalled; I would disagree.  Autodesk has very much improved efficiency specifically aimed at surfaces and large data sets.  You certainly couldn't do with 2009 what you can do now.  Can you be more specific?

[b.raynorS4QZX]

Well, it's good to know that 11 years on and Autodesk still isn't taking this seriously.

[mathewkol]

Can you be more specific?

[morris_calda]

Hi Mike, 

It's a complete pain in the ass to be honest. I find that 2 million points is about the limit that civil 3d can handle when comes to a point cloud. Sure the developers will say it can handle more but it depends what your patience level is and how much coffee do you like to get up and make during the day. 

Typically when I get large data sets which is nearly always these days. I have to strip out classified data like trees using cloud compare (third party software). Then recap the las and import into civil 3d. 

With regards to striping out data I only ever strip data heavily out of areas I do not intend on working in. If the eg is still huge the only other option you currently have is to break eg into chunks. 

But to be honest data collection has out stripped what civil 3d can comfortably handle and In my opinion Autodesk has done stuff all to keep up with that and left it to users to try and workout work flows to stay efficient. It's a joke really 

It also dosent help that people who conduct lidar surveys think that in collecting data that I need to see someone's lunch box on an embankment. The data collection is getting ridiculous. There seems to a huge disconnect between what is be collected and what is really needed to complete the job. 

Anyway that's my 5cents