Projection trouble with MAPIMPORT and DEM Surface from QGIS data

[bhaeffner]

This may be a "user error" or lack of understanding issue, but...

Contours created in QGIS and imported with MAPIMPORT do not align with contours created from Civil3D surface.

Using ESRI:102697 (Missouri State Plane Central NAVD83 Foot) in QGIS

• Clip DEM in QGIS to .tif and import to Civil3D using Surface Definition DEM Files with MO83-CF project CRS. Display with 1'/5' contours.
• Create 1' contours in QGIS from same DEM clip.
• MAPIMPORT the contours with MO83-CF

The two sets of contours are nearly identical, except offset and scaled differently.

[jroot]

Check the units used for each, is one using International Feet and the other US Survey Foot?

[bhaeffner]

As near as I can tell, both are in US (Survey) Feet.

Regardless, the same units and CRS were used in the .TIF and the .SHP in QGIS. I can't see why/how they would be projected differently in Civil3D, except that the contours from the shape file are at elevation 0 instead of 7-hundred something.

[ChicagoLooper]

Where did you get the DEM? Can you post the link?

[bhaeffner]

The DEM came from the Missouri Spatial Data Information Service (MSDIS) as a County-wide DEM file:

msdis-archive.missouri.edu - /archive/Missouri_County_LiDAR/

And is about 10GB decompressed.

I did reproject the DEM to ESRI:102697 and convert elevations to feet in QGIS.

I have attached the clipped DEM file .TIF and a .ZIP with the SHP files, both created in QGIS. Also attached is a DWG with the DEM surface and imported contours.

[Pointdump · Accepted solution]

Brian,
Thanks for posting your drawing and data.
This is NOT a US Survey/International Foot problem. It has to do with Civil 3D choosing a corner point of each pixel instead of the center.
You can check this by using "Raster Pixels to Points" in QGIS. That places a shapefile point in the center of each pixel of your GeoTiff. I added a Northing and Easting Field to the Attribute Table of the Center Points and then Saved As a CSV File. The New Surface, created with the CSV File, looks like the QGIS Contours.

Dave

[ChicagoLooper]

To @Pointdump x’lent observation!! Glad you brought that up. I agree 100%, it’s NOT a international ft vs US survey ft. 

Does this middle vs. corner insertion point also apply when you bring in an image to modelspace?  If you use an FDO is the raster’s insertion point the corner or the middle? How about the DEM in question? If the DEM is exported to geotiff in QGIS, will C3D insert the newly exported DEM using the corner or middle? 

If there’s a difference using an FDO connection vs. MAPIINSERT, which is more geospatially accurate? 

[bhaeffner]

Dave,

Wow, thanks for the explanation.

I will try to figure out how to automate the creation and import the CSV rather than the DEM.

So, a step further...any idea why C3D chooses to place the elevation at the corner of the pixel rather than the center? I would have thought that the center provides a better correlation to the real world.

[bhaeffner]

For anyone (probably me) doing this in the future:

QGIS has an Vector Tool called Add Geometry Attributes. After using the Raster Pixels to Points noted by @Pointdump above, you can easily add the XY coordinates. Exporting this layer to a CSV will give a file with this format:

VALUE	xcoord	ycoord
760.6956	1871904	1267311

In Civil3d, under Surface Definition, choose Point Files/Add Points. The existing point file formats won't work, but you can copy an existing point file format then modify as follows:

1. Give a new name
2. Change default extension to .CSV 
3. Use "V" or "VALUE" as the comment tag - this eliminates the header row
4. Change the values in the column headings to "Point Elevation", "Easting", and "Northing" by clicking on the existing column heading. Note that if a column heading is already used, it's not available in the drop down list - you have to change one of the columns to "unused" first.

This will create a new point file format file that will directly read the output from QGIS.

[ChicagoLooper]

The question isn't, 'How to create the surface using the csv,' that's easily done by anyone with a rudimentary understanding of point file formats, the question should be what should the insertion point be? Should it be the corner or the middle? 'Pretty sure it's the middle' is not an acceptable answer.

DEMs, such as those downloaded from USGS, are referenced against UTM. Your uploaded file test.tif, however, is not. It's referenced against MO83-CF. Also, did you have the entire tif? Looks like you only uploaded a just a portion?  

Where did you get that tif? Or please provide the link so it can be downloaded in its entirety.

[Pointdump]

Brian,
"...any idea why C3D chooses to place the elevation at the corner of the pixel rather than the center?"
No idea. Makes you wonder what the thinking is there. I don't know how a GeoTiff DEM is created, but I think it's a "derivative" product from, say, a LAS Point Cloud or somesuch. If the original elevation data is available, I'd use it in preference to the GeoTiff.

Dave

[bhaeffner]

There is more than one question. At least I have/had more than one question.

My immediate need was/is to create a surface in Civil3D using the data I have available, thus my documentation of how to do so for my future reference as I GAINED a rudimentary understanding of point file formats. And for the reference of anyone who might face the same problem as me.

I described above that I reprojected the DEM from MSDIS (link provided above) to 102697/MO83-CF. I did this because I wanted to be able to extract data from the DEM in that coordinate system (and feet) for other uses (specifically, using HY-8 for culvert design). Yes, I only uploaded a portion of the 10GB .TIF. I am not attempting to load the entire DEM into Civil3D - I clip a portion for each project. And I clipped an even smaller portion for demonstration of what I was seeing.

Perhaps I should have left the original DEM alone, and/or used the original UTM referenced DEM to import to Civil3D?

But, I didn't, and therefore the issue I originally asked about appeared. I now know why and how to resolve it thanks to Dave. I will also spend some time investigating whether it was necessary to reproject in the first place- it's been a while since I did that and I don't recall the exact reason.

As for the question of what should be the insertion point - I'll leave that to those with a deeper understanding of the issue - I certainly am not qualified to speculate. I was just curious why Civil3D "chooses" a corner. If you have an acceptable answer to what it should be, I guess I'm also interested in that.

Thanks for the responses.

[bhaeffner]

Dave,

I see your point about using the original LiDAR point data; however, the County-wide DEM is much more convenient for my purposes, only one of which is creating surfaces in Civil3D.

The point cloud data is available for download from the same source but in 1-sq mi (or smaller) .ZIP files. I'll take a look at using that for my C3D work rather than the DEM. The DEM is simply better for what I want to do in QGIS.

Thanks - you've been extremely helpful.

[Pointdump]

Brian,
No need to re-project. Civil 3D is more than capable of re-projecting ANY data upon import into your drawings. But sometimes it's easier to clip your data in a program like QGIS before bringing it into C3D.
Dave

[ChicagoLooper]

Based on your small target area, it's roughly a square measuring 150' X 150', using that specific DEM could be an issue. Find and use more detailed data.

The DEM is evaluating the ground's surface by looking at something like a 10-meter X 10-meter grid then determining the 'average elevation' inside each grid cell. (The color of the pixel represents elevation.) A smaller grid, something like 3-meter X 3-meter, then your resolution improves. 

You don't mention the resolution of your original DEM and resolution is critical . Your target area is only 150' X 150' so things like a 6' wide swale and a 30' wide road can easily get under-represented when your TINN's surface elevations are calculated.

 Your surface will only be as accurate as your input data, not just how well you manipulate that data in C3D. I'd look at better resolution DEMs, if any exist to begin with, or look at alternate sources of data such as LiDAR.

[bhaeffner]

Well, I'm back with yet another question. I'm trying to bring in a surface from the original (not reprojected) DEM data.

My Civil3D drawing is in MO83-CF State Plane Coordinates.

I clip the DEM surface in QGIS which has CRS: EPSG:26915 - NAD83 / UTM zone 15N - Projected, assign points to the raster pixels, add geometry values, export to CSV and create a point file format as above.

But since the points are in a different coordinate system, I also have to check the box for Coordinate Zone Transfer in the point file format dialog and select the appropriate Coordinate Zone which I assume is: UTM with NAD83 datum, Zone 15, Meter; Central Meridian 93d W

When I import the point file, I check the "Do coordinate transformation if possible." 

But the new surface is not aligned with the rest of my drawing - the transformation is going wrong somewhere.

If I clip from the reprojected DEM which has CRS ESRI:102697 and go through the process above (minus the coordinate transformation parts) it works great.

What do I have wrong?

[ChicagoLooper]

So you are clipping the DEM in QGIS, Correct?

Go back and verify the QGIS coordinate system prior to clipping. What is that coordinate system. DO NOT ASSUME.......Physically perform verification using mouse clicks and eye sight.  

[Pointdump]

Brian,
When you create the new Point File Format, you need to use "Grid" Northing and "Grid" Easting instead of Northing and Easting.

Dave

[ChicagoLooper]

---

@bhaeffner wrote:

Well, I'm back with yet another question. I'm trying to bring in a surface from the original (not reprojected) DEM data.

My Civil3D drawing is in MO83-CF State Plane Coordinates.

I clip the DEM surface in QGIS which has CRS: EPSG:26915 - NAD83 / UTM zone 15N - Projected, assign points to the raster pixels, add geometry values, export to CSV and create a point file format as above.

But since the points are in a different coordinate system, I also have to check the box for Coordinate Zone Transfer in the point file format dialog and select the appropriate Coordinate Zone which I assume is: UTM with NAD83 datum, Zone 15, Meter; Central Meridian 93d W

When I import the point file, I check the "Do coordinate transformation if possible." 

But the new surface is not aligned with the rest of my drawing - the transformation is going wrong somewhere.

If I clip from the reprojected DEM which has CRS ESRI:102697 and go through the process above (minus the coordinate transformation parts) it works great.

What do I have wrong?

---

Your second to last sentence highlighted in blue, indicates your QGIS has Missouri State Plane, NAD83, Central Zone, US feet. Even though the DEM itself has a native coordinate system of UTM83-15 meters, you have inserted that DEM into a drawing environment with MO83-CF. When you export points, lines or polygons from QGIS, the exported entities will adopt, or inherit, the QGIS coordinates, the coordinates in which the DEM now resides, not the native DEM coordinates.  

Start over and setup your QGIS coordinates to UTM83-15 meters before you insert the DEM and before exporting and your 'transform' procedure from UTM to State Plane will work.