FIX function returning incorrect integer

[c_sansom]

Could anybody tell me why the following lisp line returns 9 and not 10 ?

(FIX (* (- 70.1 70) 100))

If you increase the multiplication factor it keeps returning 9 up till 1.0+e10 

It appears that the minus function calculates this to be 0.099999999999 not 0.1

It is doing this in a short routine that I have been using for over 10 years and I do not recall it being a problem. It also gives the same answer if the line is typed at the command prompt.

I have precision set to 0.0000

Any ideas appreciated

Thanks

Chris

[ВeekeeCZ · Accepted solution]

Look HERE at the recent thread and possible fix.

[leeminardi · Accepted solution]

A few things to keep in mind

1. The precision you set in AutoCAD has nothing to do with the precision of the resulting AutoCAD calculations.  AutoCAD's precision affects "what you see" for precision but not the actual number.  
2. Keep in mind the some decimal fractions, for example 0.1, are irrational numbers in binary.  Just as 1/3 decimal is an irrational number in base 10 (0.333333...), 0.1 base 10 is an irrational number in binary. In the process of converting a base 10 number some normalization takes place.  See this link for an example.  At the referenced link M= manitssa, e = exponent and s = sign.
3. There is a big differnce in the meaning of decimal places and siginificant digits.  Decimal places refer to the number of digits to the right of the decimal point whereas significant digits refers to the total number of digits for a number.  AutoCAD uses 64 bit floating point representation for real numbers (numbers with a decimal point).  This yield about 16 significant digits.
4. 16 significant digits will be more precision than you will ever need.
5. A rule-of-thumb when measring something is to have a ruler that is at least an order of maginitue more precise than what you are measuring. I suggest rounding the final value of any calculation that you will need to at leat 1 more digit than the precision you want.  So, if you want a 4 decimal place result, round the number to 5 decimal places and then use the 4 digit rounded number.

[c_sansom]

Thanks BeekeeCZ

Interesting read - the vagaries of programming !

[c_sansom]

Thanks leeminardi
I had no idea there was such an issue with floating point binary. Also highlights the difference between rounding a number and truncating a number.