Simplifying TIN surfaces created from USGS 1M DEM data

[Anonymous]

I have a colleague who has been dealing with 1M DEM data at our office. The typical workflow is to bring the raw data into Infraworks, generate an IMX model, import that model into Civil3D, crop the surface, then simplify the surface using edge contraction as many times as he sees fit. Recently I went to UAV conference and learned quite a bit about the modern accuracy standards for digital elevation models. That being said I challenged that he could remove points from the original data set and still certify 1 foot or possibly even 2 foot contours. The ASPRS 2014 standards are quite clear and I don't understand how you can removed points and still say surface accuracy is maintained. Furthermore, without getting very technical, one can see dramatic changes in the contour lines when performing surface simplification. How then can the surface accuracy be maintained when the model changes so much between simplifications. Common sense would tell you that too very different contours can not both be correct. Any thoughts or experiences would be appreciated.  

[Civil3DReminders_com]

I think your assumption that any of the contours are correct is erroneous. If I go out and survey a site today and then do the same thing tomorrow. I can guarantee the contours wouldn't match exactly. Which one is correct? Aren't they both wrong?

The difference would come from me selecting different points to sample, each sample would give different results. I haven't researched it much from a technical point, but from a practical sense when I worked in construction was that if the site averaged out correctly if I compared the engineer's surface with my survey of the site before construction then everything would work out. If it didn't then we'd have a problem. So if you compare the simplified surface to the original surface are there large differences or small differences? I'm guessing they'd be small and wouldn't result in much difference in quantities.

The next question would be how are you calculating volumes for payment? Is that method accurate enough to worry about it? For instance for DOT projects that pay by cross section there wouldn't be much use worrying about it if the error averages out (ie the survey isn't a higher or lower distance all around). 

[AllenJessup]

To continue with what @Civil3DReminders_com says. I believe "Simplify" will reduce the precision of the surface. That's kind of intrinsic in the word. But it may not reduce the accuracy of the entire surface. As mentioned. You may get a reliable balance from a simplified surface. So in any particular location you're elevation from the DEM surface may be less likely to match a field shot at that location but overall the surface is still viable.

As far a design goes. I'd never rely on an imported DEM for precise work. We always shoot any points we have to match hard surfaces such as roads. We also shoot any area we need to know better that whatever the stated accuracy of the data is. If it's ± a foot then you need to shoot anything you can't miss by that much. The one exception is with some high accuracy LIDAR. We've has some amazing results matching into 2m LIDAR surfaces.

[tcorey]

I see you haven't discovered QGIS. This free program will open the DEM in seconds, no exaggeration, then allow you to crop and export. When you export the cropped area, you have the choice to reduce the resolution of the terrain, thus reducing its size. It does all this very quickly, nothing that Civil 3D and InfraWorks can even approach, speed-wise.

[Anonymous]

So I understand that for a given site you may have two different surfaces with two different sets of contours shot by two different field crews. Given the field crews are equally competent and using comparable equipment both sets of contours can be legitimate according to the National Map Accuracy Standards. This really has nothing to do with what I am saying. I shall be more specific. Digital data follows a different set of standards than traditional survey methods such as field topo or long range photogrammetry. The reason, as I understand it, is that the data is sampled randomly. There is no human element in the selection of points. The USGS DEMs are generated from raw LIDAR data here in New York. USGS essentially wrote the book on guidelines and standards on how to define your accuracy horizontally and vertically with this data. There are minimum point densities, maximum point sampling distances, individual point accuracy and point precision. According to the 2014 standards, all these must be at certain values in order to say a map generated from this data is of a certain accuracy, 1 foot or 2 foot contours, etc. My understanding is that this because the all the points are randomly sampled so overall accuracy is dictated by statistical mathematics, as it should be. The USGS DEMs are produced to meet this standard and so how can you "simplify surface" and essentially decimate that data and still say that you are compliant with the 2014 standards? I suppose if you went around the area and took a significant number of check shots on the site and they satisfied 95% confidence then it does not matter how much data you weed out. Without doing that how can Autodesk say, verbatim, "When you simplify, you reduce the number of points or triangles in a TIN surface, while preserving the surface accuracy." I see how this could be true in many cases but I do not see how you can guarantee such a thing. I can literally take a DEM generated surface with 500,000 points and simplify, simplify, simplify until only contains 1,000 points. The initial surface and the final surface can look characteristically different.    

[AllenJessup]

---

@Anonymous wrote:

. Without doing that how can Autodesk say, verbatim, "When you simplify, you reduce the number of points or triangles in a TIN surface, while preserving the surface accuracy." I see how this could be true in many cases but I do not see how you can guarantee such a thing. I can literally take a DEM generated surface with 500,000 points and simplify, simplify, simplify until only contains 1,000 points. The initial surface and the final surface can look characteristically different.    

---

It depends on how you define accuracy. You may not be degrading the accuracy of the surface as a whole. You will be degrading the precision. I'm no expert at all. But to my simple way of thinking the term accuracy really should be replaced with precision. Since the overall product could average out as accurate with many deviations from precision. In measuring accuracy deviation from the expected is not taken into account. Only the average of the whole. Precision take deviation from the expected into account.

So using those definitions Autodesk could make that statement and have it hold true. I totally agree that you can't simplify a surface greatly and meet the published accuracy standards.

I'd say a much better workflow is to download the IMG file from the GIS Clearinghouse and use a free utility such as OpenEV to convert the IMG file to a GeoTiff. Create the Surface from the Geotiff, crop the Surface and it should be manageable without simplifying.

[Civil3DReminders_com]

Considering your responses, I think it would be safe to say it doesn't matter what we reply to your question if the comments are counter to your position. It seems like you are more interested in getting validation of your point of view so you can then tell your co-worker that other people on the internet agree with your position and prove to your co-worker they are wrong.

USGS 1M DEM is garbage data for accurate and precise civil engineering roadway or grading design. Having more or less garbage data isn't going to improve the design. Yes, the USGS has improved their data collection methods, but in the end it's still garbage data. It's great data for planning or floodplain studies, but probably shouldn't be used for other precise purposes. 

By your own position, the USGS 1M DEM data is garbage. The USGS is simplifying a surface and deleting all of the awesome Lidar data to create garbage. They are taking an accurate surface and then making it a garbage surface that is only accurate to 1 or 2 feet and only reporting a point every ~3 feet that represents a composite of the surveyed data that in now way can accurately reflect the ground surface.

Why use this garbage data AND elevate it to the status of NOT garbage?