testing local elevation variability within a point field

[autoMick]

Hi, Please see attached drawing and snip.

I have a field of points which were collected as part of an echo sounder test (actually this is the test for the real test which would be larger scale). The objective was to collect numerous overlapping points so that the point elevations could be compared to determine the error/variability within the echo footprint during multiple passes of the same area. Lets say the echo footprint at the depths being surveyed is around 1.0m radius

What I would like is to be able to analyse the neighbours within the footprint of each recording point to determine the range elevation differences. So for instance, the circled point in the snip, has two neighbours and the deviations in elevation from the point being analysed are +0.007 and -0.010m. I would like to somehow record these values. Some points won't have any close neighbours so they would have no records against them.

Ideally I would also love to be able to vary the radius of the nominated footprint based on the depths recorded as the echo sounder has a 9 degree footprint and the footprint radius on the seabed changes with depth (tricky I know - answer this one for bonus points).

Any suggestions on an automated way to do this?

Thanks

- Mick

[Anonymous]

Hi,

I'm not sure if this can be done using "out of the box" Map.

What you have:
- points and elevation for each point

What you want:
- for each point find all neighbouring points within 1m distance
- calculate the difference between elevation of those points

The first requirement is a typical use case for a spatial query. Therefore I would use a spatial database to get that solved. As you mentioned the next test would be on larger scale - databases are a good option for that as well.

If you have spatial databases in house - you can export your points and import them into your database (Oracel, SQL Server, PostGIS, ...).

If you do not have a spatial database at hand - you could export your points to SQLite (_mapexport). SQLite is a file based database and there are tools (freely available) to connect to and perform analysises (Spatialite GUI SQL). You need to be familiar with basic SQL and basic SQL spatial queries.

If you want to vary the distance for neighbouring points - that might be a bit tricky, don't know if there is an easy solution for that.

Rob

[autoMick]

@Anonymous Thanks for the reply.

I have some SQL experience.

I use the word "experience" in the loosest possible way, in that I call recall once or twice that I may have written some SQL code using some handy wizard or another. 

OK.... let's just say I have none and never mention it again 

If I had the time to invest in this I'd take to the time to learn the right tools, but unfortunately, I'm starting to think the most time efficient approach will be to just manually pick 50-100 points, but ideally I wanted a procedure that I could repeat automatically on new surveys as they were done.

Another strategy I was considering was along the lines of creating a surface in Civil3d from the points, smooth by Kriging then looking at point deviation from the surface. Not really the right analysis for my question but thought I'd mention it in case it sparked some other ideas.

Cheers

- Mick

[Pointdump]

Mick,

I consider "code" to be a 4-letter word.

I have nothing to add here. I just find your post interesting.

Can you add a little more information to the need for comparison of points in proximity?

Dave

[braudpat]

Hello

1) As Dave, your post is interesting !

2) Your DWG seems to be an ACAD CIVIL drawing not an ACAD ou ACAD MAP drawing !

3) I think that you need a specific routine retrieving and analysing AECC_COGO CIVIL points !?

Regards, Patrice

[Anonymous]

Hi,

I did a test with your drawing - screenshot and SDF file are attached.

At least for the point you referred to I get the same result.

The whole workflow shouldn't take longer than 10-15min. But in reality it took longer as I'm not overly familiar with SQLite - SQL and the peculiarities of SQLite Spatial. Let me know if the result is correct or not. If correct I can write down the steps required to get to the result.

The approach is flexible - once you have prepared SQL then you can feed new data in easily and get results back in no time.

Rob

[autoMick]

@braudpat Yes you are right - civil3d is my preferred platform. I've posted the question here as it really has more of a GIS solution than a civil design focus.

@Pointdump The question arises as there are standards for hydrographic surveying that specify cross-checks and allowable errors, but in my experience as a recipient of survey information, I have never seen any evidence that this sort of data that would allow verification has been collected, let alone the appropriate analysis conducted. As we are doing more and more survey work ourselves I wanted to at least have an initial understanding of the variability in our surveys, and ideally a repeatable methodology where I could determine a metric for each survey conducted. I use broadcast RTK corrections which have greater elevation error with distance from the base station (up to 50mm), GPS error (typically 10mm) and then echo sounder error (10-20mm). Adding those together seems very acceptable for most of the work we target, however there are variations on the bed of the river, latency between the GPS and sonar signals, boat speed, which all play a part - so it's more about what is the real error with all things considered in a typical field situation. Part of it is pure curiosity on my part.

@Anonymous this is looking very promising and more than I dared hope for. I would be very grateful if you could spell out the process for me. Although i use civil3d daily, with some MAP commands, I'm really a newcomer to MAP3d and will need some hints! It doesn't worry me about having to go through a process (rather than just running a script).

Thanks

- Mick

[autoMick]

Oops... posted the same response twice. Edited to delete. It would nice in this forum if it was possible to delete duplicate posts

- Mick

[Anonymous · Accepted solution]

Hi,

I have seen similar questions before and therefore took the effort to write a short description of the process for further reference:

http://raumpatrouille3d.blogspot.ch/2016/08/spatial-queries-and-map.html

With regards to your second requirement:

"Ideally I would also love to be able to vary the radius of the nominated footprint based on the depths recorded as the echo sounder has a 9 degree footprint and the footprint radius on the seabed changes with depth (tricky I know - answer this one for bonus points)."

Instead of using a fixed value (1m according to your description) for all points in your SQL statement you can use a dynamic value depending on the point itself. I don't know how you would calculate the search-perimeter / radius for a specific point - could you explain that?

Rob

[autoMick]

@Anonymous Rob, thanks for putting this together - it's going to have to wait to the weekend when I've got some time to work through it properly, but I'm confident given your description and results that this is the way forward. I'll provide some more feedback when I get to spend some time on it.

Regarding the search radius: the sonar puts out a vertical cone of sound waves and records the first echo return that a sound wave encounters within that zone, from which a depth is calculated. The shallowest depth could be anywhere within the base of that cone, but is assumed to be in the centre of it, vertically under the sonar transducer. The cone has an angle of 9 degrees, so the radius that the echo sounder could detect a return from is R=depth*tan(4.5deg). The points included in my example drawing have raw descriptions included, which is actually the depth value below the transducer which would be used in this expression.

Thanks for your detailed responses.

- Mick

[Anonymous · Accepted solution]

Hi Mick,

TAN doesn't seem to be part of core SQLite SQL but SpatiaLite_GUI does seem to have its on TAN function.
SpatiaLite help says, that TAN takes radians as input (in SpatiaLite GUI click on Help button to get further information).

I created a new SQLite file as I did not take the depth column into account with my first attempt.
The column name is "desc1". SQL I came up with is:

select 
p1.number as p1_number, 
p2.number as p2_number,
p1.elevation as p1_elevation,
p2.elevation as p2_elevation,
ST_Distance(p1.geom, p2.geom) as dist_points,
p1.elevation-p2.elevation as diff_pointheight,
p1.desc1 as p1_depth,
p2.desc1 as p2_depth,
p1.geom as p1_geom,
p1.desc1 * tan(Radians(4.5)) as distance_filter
from fdo_points_2_copy as p1 cross join fdo_points_2_copy  as p2
where dist_points < distance_filter -- returns only 36 rows
--where dist_points < 1  --returns 936 rows
and p1.number <> p2.number
order by p1_number

Please note that the table name is slightly different as I changed it during export into SQLite.

I don't know if the result is ok  - it returns only 36 rows. When I use 1 (meter) in my filter I get nearly 1000 rows back.

Rob

[autoMick]

Wow Rob, you love a challenge I see !

Still strapped for time this week, but keen to look through it properly on the weekend.

thanks again for your help

- Mick

[parkr4st]

AutoMick

Trying to make sure I understand your project.  You are charting water depth in a tidal ? river in Balina?

your set up is a boat with transponder, gps, and recording equipment.

for each x,y point there are 4 elevations

BD = Base datum presumed to be dead high or low tide or actual elevation if the water surface elevation never changes
WS = water surface elevation varying with tide?
TD = transponder depth
RB = river bed

Therefor do some data work

Z1 = tide differential is   BD - WS   Top of water as you work

Z2 = Water depth is WS-(TD + RB)  the water surface above the bottom aka too shallow the boat hits bottom

Z3 = Bottom elevation is  BD-(Z1 + Z2) bottom elevation relative to base datum

Z4 = TD to RB  TD-RB

all of these can be calculated in the data table keeping in mind to multiply by -1 when needed to produce positive number for calculations

4.5 degrees = 0.0785398 radians

so the radius formula is Z4*tan(0.0785398)  this is a radius of 0.07870169 meters per meter of water depth?
(3 inches +/-)  is it really enough to matter?

In the data table add a calculation for field titled radius  Z2 * Tan ( 0.0785398 )

given any x,y the river bottom is at x,y,z3 compared to BD

new dwg set CS

mapimport the points with data and save as dwg1

new dwg set CS

attach dwg1, query in with a alteration setting ELEVATION from Z3.

the result is 3d points of the bottom.

make a C3D surface

graphically

If you need a visual of the overlap, buffer your sdf points with 0.14 buffers. 

do an analysis of points and buffer using identity

the result is empty as there are no overlaps at 2 meters down.

buffer at 1 meter (attached) and the data will show overlaps in the FID column (sort the column and it shows plainly)  attached file

look at the map it shows plainly as overlapping circles

not automated, but your data can be used to get to a 3d surface.  someone who enjoys coding can probably encode any of this. 

delta of the water surface and the bottom surface is the water body.  c3d can draw that as a 3d object?

knowing the tide high and lows you can change the water surface for any given tide situation without having to measure at all tide variations

have fun make charts galore from the data

querying on water depth i.e. all points where Z2 (WD) exceeds 1.5 meters you can map a channel in 3d if there is a channel

for reruns, use the sdf schema with an empty table, adding your newly collected data should so all the calculations automatically.

IMHO this is what map3d can do.

dave

[autoMick]

@Anonymous Rob, this works very well. Yes the actual overlap is much less than the 1m I posed as my example. So the 39 returns is spot on. I quoted 1m as that provides multiple neighbours for a given point, whereas it looks like the smaller radius given by the depth formula only has found single pairs. In deeper water there will be more overlaps.

I found I had to change a few of the field codes to suit what I actually exported (FeatId instead of number, etc.) but it all worked very nicely thanks to your detailed description.

I'm now considering what to do with the point-pairs. Probably leave them I think and just report overall mean and 95 confidence limits.

I'm interested in thoughts on how to handle the pairs? exclude the duplicates or not?

Is there an easy way to exclude the duplicates within the select command?

Regards

- Mick

[autoMick]

@parkr4st Hi Dave, yes tidal waters with some minor wave surge. The GPS takes the sonar signal input at a frequency 20Hz and records an elevation of the river bed taking into account the GPS antenna elevation, the fixed offset to the sonar transducer and the sonar depth data feed. So the point Elevation is the result of this hardware recording and represents RB. I also record depth separately (which was in the raw description) for error checking and also with the view to calculating the sonar footprint as per the discussion above. Apart from the footprint aspect, the actually water level at the time is basically irrelevant, which is lucky since it varies constantly and erratically.

I've tried to follow your explanation, but it's been a long day and I'm just not following it completely - I'll re-read tomorrow and respond more intelligently then I hope.

Regards

- Mick

[autoMick]

Hi Dave, I've just re-read your post but isn't quite what I was after.

I'm confident with charting the bed elevation and civil3d surfaces - my goal is more to give some sort of repeatable indication of measurement error in the vertical plane between points taken on different sample runs. Ideally, Run A would overlap exactly with Run B then I would have an exact comparison of my depth readings, but as you can imagine - getting a boat to drive over exactly the same piece of water multiple times with winds, currents and varying water levels is quite a challenge - hence the need to bring in the footprint of the sonar (which as you point out is very small) to give some leeway as to which points would be a valid comparison and which ones. It's still not perfect as the natural river bed has sand waves, etc. but that's just part of the accepted error.

thanks for your input . It's led me to learn a few new things in map3d which is always a good thing.

Regards

- Mick

[Anonymous]

Hi Mick,

with regards to "Is there an easy way to exclude the duplicates within the select command?

No, I don't think so. At least I cant think of an easy solution. Why not filter them out in Excel - it has a "remove duplicates" function (although I have used it in a different context where the rows in question where identical over all columns - so Im not 100% sure if it will work for your data set).

Rob

[autoMick]

---

@Anonymous_Fritz wrote:

Why not filter them out in Excel - it has a "remove duplicates" function

---

Hi Rob, this is exactly what I did - it does work with a few steps in excel - I thought maybe there was a quick way to achieve the same in a more elegant way.

thanks again for your help

- Mick

[parkr4st]

your  welcome

interesting project to challenge what GIS can be used for

what you may need is       http://diydrones.com/profiles/blogs/ardupilot-goes-into-the-water-3

just to add to the interesting aspects of what can be done.

kudos to R Fritz are well deserved.

dave

[autoMick]

---

@parkr4st wrote:

what you may need is       http://diydrones.com/profiles/blogs/ardupilot-goes-into-the-water-3 

---

So true.... There are so many cool gadgets out there these days - in fact, the traditional echo sounder for hydrographic survey is really a bit of a dinosaur now. The swath style side scan sonars are the equivalent of underwater lidar which give accurate bathymetric point clouds - still too pricey for me, but amazing technology which I'm confident will be mainstream in the coming years.