AutoLisp doesn't play nice with certain numbers

[Kent1Cooper]

I have reason [it's complicated] to want to know the greatest common denominator of a pair of numbers, but when they are not integers which the (gcd) function requires.  [For example, the greatest number that divides into both 4.0 and 2.5 is 0.5 -- that's what I'm looking for.]

The number of decimal places varies.  So I'm finding the number of decimal places of the one with the most of them, raising 10 to that power as a factor [e.g. if one of them has 3 decimal places and the other no more than 3, use 10^3 = 1000], multiplying both numbers by that and (fix)ing them so they'll be integers, using (gcd) on those, and dividing the result back down by the same factor to get a decimal end result.

In trying things, I've come across a freakish thing.  AutoLisp somehow doesn't like working in certain ways with numbers with 3 decimal places within a certain range, when the last decimal place is odd.  I just coincidentally seem to have stumbled into that certain range by accident, but it may be [probably is?] true within other ranges of numbers, and with other numbers of decimal places.

The first few functions here return what you would expect, but note where it (fix)es to the next number down:

(fix (* 0.997 (expt 10 3)))
  997
(fix (* 0.998 (expt 10 3)))
  998
(fix (* 0.999 (expt 10 3)))
  999
(fix (* 1.000 (expt 10 3)))
  1000
(fix (* 1.001 (expt 10 3)))
  1000 <-- first wrong one
(fix (* 1.002 (expt 10 3)))
  1002
(fix (* 1.003 (expt 10 3)))
  1002
(fix (* 1.004 (expt 10 3)))
  1004
(fix (* 1.005 (expt 10 3)))
  1004
.... same down-one error with all odd-third-decimal-place numbers 1.007 through 1.021 [omitted] ....
(fix (* 1.022 (expt 10 3)))
  1022
(fix (* 1.023 (expt 10 3)))
  1022 <-- last wrong one
(fix (* 1.024 (expt 10 3)))
  1024
(fix (* 1.025 (expt 10 3)))
  1025 <-- right again
(fix (* 1.026 (expt 10 3)))
  1026
(fix (* 1.027 (expt 10 3)))
  1027

Wondering about the (expt) function and possibly tiny processing errors, I checked way down the decimal places for it come up a hair short, but that didn't reveal anything:

(rtos (expt 10 3) 2 17)
  "1000.000000000000"

Wondering about whether integer values could be the issue, I tried decimal values, but the same happens if I use (expt 10.0 3.0).  Even forgetting (expt) entirely, it happens if I just straight-out use 1000 [and whether that's an integer here or a real number 1000.0]:

(fix (* 1.019 1000))
  1018

And these parts of that work right:

(* 1.019 1000)
  1019.0

(fix 1019.0)
  1019

But there's something about the particular combination of functions....

It works right whenever that third decimal place is even, as apparent above, and when it's odd outside that particular range [within the short distances outside that I tried].

Does this happen for other people?  Does anyone have any idea what could be causing such a thing, or whether or where other danger ranges might be, or how to prevent or compensate for the error?

[rkmcswain]

@Anonymous 

@Anonymous 

@Anonymous 

Maybe one of those will tag the real Lee Ambrosius.

This forum has the worst @ tagging I've ever seen.

smh

[Kent1Cooper]

---

@rkmcswain wrote:

.... Maybe one of those will tag the real Lee Ambrosius. ....

---

Thanks for trying, but I'm not hopeful -- the latest contribution listed from any of those was from 12 years ago....

But in any case, do you also get the same error with those same functions/numbers?

[rkmcswain]

Yeah, I'm trying. But of course after I typed in VLIDE, I had to then tell it AGAIN, that I want to use the VLIDE, and then, of course shut down AutoCAD and restart it.

smh again....

So anyway.... (in AutoCAD 2023)

[cadffm · Accepted solution]

I would search outside of AutoCAD, I think that is the general issue with floats

math and data type / rounding error float

https://floating-point-gui.de/errors/rounding/

[Kent1Cooper]

I'm hesitant to say I'm glad you get the same error, but you know what I mean.  [No need for the VLIDE -- just paste into the Command line.]  In any case, this approach seems to be a way around it:

(atoi (rtos (* 1.009 (expt 10 3)) 2))
  1009

which works right on all those values that had the problem in the other method.

[leeminardi]

A wild guess. 0.1 is an irrational number in binary.   This could present an issue in high precision calculations but I would think not with your examples.

[CADaSchtroumpf · Accepted solution]

Hi,

My explanation

for exemple for understand:

(rtos (- (* 1.023 (expt 10 3)) 1022.0) 2 16) ->"0.9999999999998863"

so

(fix (+ 1022.0 0.9999999999998863)) -> 1022  fix is rigth.

see:  Why 0.1 Does Not Exist In Floating-Point 

[Kent1Cooper]

Taking it down to around 1 so you see the maximum number of decimal places reveals the off-ness in a way that my earlier (rtos) test [which got the maximum number of significant figures, but some of them were before the decimal point] didn't.

Raising an integer to an integer power should get an integer result, and I think (expt) really does, because even replacing the (expt) function with the 1000 directly [see above] gets it wrong, and so does substituting a simple multiplication:

Command: (fix (* 1.005 (* 10 10 10)))
1004

So it's got to be in the nature of the decimal number and how it works with that.  This kind of result would wreak havoc on what I need to do with it, so I guess I'll have to go with the (atoi (rtos)) approach.

[dbroad]

Not that it helps but I offer this explanation:

The calculations are done to 14 decimal places and the representation of decimal numbers in binary have some roundoff error.  Consider that the number may be actually represented as something extremely close to the number.

(fix 1000.9999999999)

Perhaps, if you need the rounding to be nearest, you need to be using a rounding function rather than the fix function or add the likely error.

(fix  (* (+ 1.001 1e-13) (expt 10 3)))

[Kent1Cooper]

---

@dbroad wrote:

.... Perhaps... you need to be using a rounding function rather than the fix function ....

---

Since I need an integer result to use in (gcd), it's going to involve either (fix) or (atoi) -- rounding functions typically include (fix) somewhere.  Rounding to the nearest whole number typically uses this clunkier approach [no need to assume how big the likely error might be], which works in this case [the green part being where the error can creep in]:

(fix (+ (* 1.003 (expt 10 3)) 0.5))

[rkmcswain]

@Kent1Cooper - Lee is with Autodesk and an autolisp guru. He has seen this thread now. See below.

Twitter link

[ВeekeeCZ]

Well... ok, the conclusion is that it's a double-precision issue. We all see the same issue.

Now, MS Excel also calculates in double-precision. But... does not suffer from this problem, at least not for the given examples. So I would wonder why.

[cadffm]

The source problem is everywhere the same, but there are different ways to handle it (with different wrong values as result)

https://www.microsoft.com/en-us/microsoft-365/blog/2008/04/10/understanding-floating-point-precision...

I am not an Excel expert and I didn't test it, but generally:

Perhaps the function you used(int?) uses another rounding method (see my Link above, there are 3 of more what are common)

In theory also possibly

Or it is rounding the multiply result first and Int calculates with the rounded number

[ВeekeeCZ]

It was TRUNC actually.

One, one who has no advanced education in math, might think that Math is just one. Obviously, even if both SW are built on the same foundations, they are built differently.