Civil 3D PC for Large Data Sets

[craigs1]

Hi All,

I am currently working on a Project that has an extremely large(derived from Lidar) data set.  I am currently trying to weed out some of the vertices on the contours in these files and then export the data to a point file to be used to create my surface with.  I have exported a point file for a portion (10%) of the project area and the point file alone is 90 megs and can only be partially opened in Excel!!

I would like your opinions on what are the optimum choices for the Hard Drive, RAM and Video Card for dealing with extremely large data sets.

This is what I am currently running:

Civil 3D 2011

Intel (R) Xeon (R) CPU Quad Core  W3530@2.80GHz  64-bit  12 Gigs Ram

NVIDIA Quadro 4000 video card

Any help will be greatly appreciated!

Regards,

Craig

[Alfred.NESWADBA]

Hi,

I don't see your machine as really bad. And when you have opened your projects you can look into the taskmanager and decide yourself, if memory is going out or not.

I would not see the necessarity to have more than quad-core, also some MHz more will be fine, but that does not make the machine twice as fast.

>> I have exported a point file for a portion (10%) of the project area

That would be the place to start: Why don't you use the point-cloud-feature within Civil3D? You can load and display a huge amount of layerscan-data and convert them directly to a surface using Civil3D. Look >>>here<<<!

- alfred -

[jmayo-EE]

You didn't mention your version. The proccessor speed could be faster. The current min. system requirement for Win 7 is 3 GHz or higher.

[craigs1]

Sorry John!

My O/S is Windows 7.

Craig

[Alfred.NESWADBA]

Hi,

>> The current min. system requirement for Win 7 is 3 GHz or higher.

Does that really make sense? Well for an old DualCore I would say ok, but e.g. if using an i5 or i7 - and now having turbo-boost - I would say faster memory is more essential than 200MHz primary processor clock rate.

Even using an SSD instead of standard harddisks makes more sense for large projects than just the clock rate.

It's a matter of opinion, and the above words are just my opinion.

- alfred -

[jmayo-EE]

http://usa.autodesk.com/civil-3d/system-requirements/

If I got a new generation proccessor I would rather have the base core speed meet the min requirement. The turbo boost would be a bonus and give me better performance.

[jmayo-EE]

I meant C3D. 🙂

The newer versions require a bit more umph from the PC.

[mathewkol]

Alfred mentioned the point cloud.  I'm not really a fan since that cloud can only display 1.5 million points...and your surface will have only that many too.

Another bit of advice is to do away with contour data if you can.  Can you get your hands on the original point data?  Contour data is a lousy way to define a surface and often far too dense.

[Sinc]

Those minimum specs are terrible.  They specifically state a 3GHz minimum Pentium 4.  You can't compare clock speeds across architectures in any meaningful way.  Every Core 2 or Core i3/i5/i7 machine will outperform a 3GHz Pentium 4.

You get no benefit from having more than 2 cores.  It doesn't hurt, but speed of a single core is the critical factor, which means that you can get a good boost from the new Intel chips that have the turbo boost.

You get no meaningful boost from going with faster RAM.  Since C3D can only use a single core, 1333MHz RAM will be fast enough that the bottleneck is actually the CPU.  You get the most benefit from faster RAM in multiple-CPU configurations, and to a lesser degree in applications that use multiple cores.  But RAM speed is not a bottleneck for single-threaded apps like C3D.  This is actually where the 2nd Gen i3/i5/i7 chips start to shine...  They have an improved memory manager that can get data between CPU and RAM significantly faster.  That's a good chunk of why the 2nd Gen i3/i5/i7 provide such significant improvement for C3D performance over the older i3/i5/i7.

And of course, I think those minimum specs are absolutely ridiculous.  We just replaced our 2.67 GHz Core 2 Duos, because they were too painful to use with C3D 2010.  And a 2.67 GHz Core 2 Duo runs circles around a 3.4 GHz Pentium 4.

[craigs1]

Matt,

I have the request gone in for the original point data. In the interim I am reducing down the physical size of the contour area and will try extracting points from the contour data.  Painful but hopefully worth it in the long run!

Sinc,

In your opinion, what would be your dream machine for use with C3D?

Thanks to everyone for their replies!  Very much appreciate the input!

Craig

[OMCUSNR]

"I would like your opinions on what are the optimum choices for the Hard Drive, RAM and Video Card for dealing with extremely large data sets."

Start with a new MoBo capable of handening SATA 6 thruput.  If the pocketbook is large enough, then get the biggest SSD you can find.  For a hard drive, the Western Digital Velociraptor is about the fastest, most stable available right now.  Hard drives are reaching their physical limits now and the next step in speed of data transfer will come with the SSD's.  (Sidebar -my brother works designing machines that test and assemble hard drive heads & arms.  They've gotten about as small and as fast as stepper moters can physically go.  The steps between data points are now just about as close together as they can physically can be, so now the next step is away from moving machinery).

For very large data sets 12 gig is a good start.  Match the speed with the NoBo.  Video cards with a lot of Vram are good for rendering and refresh rates in animation.  A "good" card, balanced for price / performance in C3d right now would have about 1 gig of Vram.  I like Nvidia drivers,  seem to have less problems with them.

Acad is not a multi threading program, so it uses only one core.  Fastest speed wins here.  The new I7 / I5 "sandy Bridge" processors are dual core and have been designed for about the best data throughput available now, even better than some of the Xeon chips.  Some of the higer end I7 chips can match the new ones, but they can't beat them, especially in price.

Take a look at my sig.  That's a good start for a large data set drawing.  I put together the best components I could find in a budget under $2k.  With a case & 650W ps, DVD burner & card reader the parts from newegg came in just under $1400.

Reid

[Alfred.NESWADBA]

Hi,

@mathewk:

>> I'm not really a fan since that cloud can only display 1.5 million points...and your surface will have only that many too.

One step further, if you like to convert a pointcloud to a Civil3D-surface ... you will not be able to create a surface with more DTM-points (than 1.5mio)!

I see that the reduction from points in pointcloud-database to points displayed on screen is really intelligent (the algorithm which of the points are display, which are filtered, I did a few tests and was surprised - in a positve way) and so I see it as a good base to get a Civil-surface.

However, having trillions of points in a pointcloud and having to use them all (which makes not sense in 99%) then I think, Civil3D will not be the product you need ..... but what product will handle that (now) ?

- alfred -

[rl_jackson]

"I am currently trying to weed out some of the vertices on the contours in these files and then export the data to a point file to be used to create my surface with"

If you are using the contour lines privided within the LIDAR topo drawing to do this you are going about building the surface the wrong way.

If it is a LIDAR, that your firm (client) paid to have flown - they should have provided you with several drawing files (at least in my experience anyway).

• One with the planemtrics and contours (which can be used as an xref)
• One with the DTM points (which should be used to create a point file from the point blocks)
• One with breaklines (which should be used to create your breakline) Note sometimes the breaklines are in the DTM file with the points)

If you did not recieve this information you need to request it.

If you are creating a large surface from contours you are simply wasting your time, you will never weed the surface enough - and that comes from experience.

Attached below is an excerp of a 1400 acrea EG Surface created from a LIDAR drawing that was 14mb in size using similar methods above. (one thing i did not do here was create point/point file for the points since my PC at the time using 2010 came to a screeching hult)

General
    Revision number    0
    Number of points    145209
    Minimum X coordinate    1766488.463'
    Minimum Y coordinate    178153.149'
    Maximum X coordinate    1777877.373'
    Maximum Y coordinate    188893.515'
    Minimum elevation    0.295'
    Maximum elevation    46.137'
    Mean elevation    22.984'

Extended
    2D surface area    60843024.53 Sq. Ft.
    3D surface area    60932173.12 Sq. Ft.
    Minimum grade/slope    0.00%
    Maximum grade/slope    2286992.11%
    Mean grade/slope    5.10%

TIN
    Number of triangles    289536
    Maximum triangle area    6176.29 Sq. Ft.
    Minimum triangle area    0.00 Sq. Ft.
    Minimum triangle length    0.001'
    Maximum triangle length    198.675'

[mathewkol]

RL, better put than my reply. Nicely said.

[rl_jackson]

It can be done - its the knowing

[Neilw_05]

With regard to point cloud density, after investigating that very issue I was told by ADESK that the way to get a higher density of points into a surface is to add the points in a tiling fashion by zooming to portions of the cloud. The number of points added to a surface is dependent on both the zoom scale and the cloud display density setting, so the closer you zoom the more points will be added, up to 1.5 million if display density is set to 100 and there are that many points in the cloud for the area displayed. However in my experiments I was not able to get this technique to work but I may have been overlooking something in my workflow.

As was mentioned, the best way to get the points into a surface is to get the LIDAR points in a text file. You will certainly not be able to open such a large points file in Excel or even a common text editor. There are some editors out there that can handle large files, but unless you need to edit the file you should be able to get by with the point cloud tools using the point import format tool. That tool will parse a few lines from the text file so you can view the format and content. I agree that trying to convert the contours to a point cloud is not a good solution.