ANGTOS Accuracy Problem

[Anonymous]

I work with ANGTOS and ANGTOF quite a bit.  I'm working with latitude and longitude conversions in MAP 3D for an application where accuracy is top priority.  Most of the time ANGTOS (and RTOS, for that matter) return an accurate result for the number if decimal places requested, but then I run into a situation like this:

Command:

(angtos (angtof "120.0") 1 16)

"119d59'59.999999999949\""

I got it to return properly by specifying only 13 decimal places, but the result makes conversion between coordinate systems inaccurate.

Is there a technique to remedy this problem?  I know its been around for quite a while.  Somebody must have solved it by now.

Thank you,

J Leavitt

[Lee_Mac · Accepted solution]

Since AutoLISP stores the 'Real' data type as Doubles, there is a accuracy limit of 53 bits, equivalent to approximately 16 decimal places. As a result of this precision limitation, infinitesimal errors will accumulate across calculations.