topic Re: Decimal number in AutoCAD Forum

Decimal number

johnkelly60 — Tue, 19 Sep 2023 00:17:16 GMT

It's not that big of a deal, but I just want to know why it's happening...

I have my data cell as 0.00 precision decimal number. When I enter a number that's ending in 5 in the 10th decimal place, it rounds up so it'll show up as a number I want. (For example 1.05) However, when I click the cell to edit the number, it shows up as .04999999...1 Why it's happening like this? (Attached is the reference image I'm talking about)

Re: Decimal number

illusionistNUGXG — Tue, 19 Sep 2023 08:47:24 GMT

That is nothing to worry about.

AutoCAD (and basically all vector-based programs) don't use regular numbers.

They work with floating point variables (double float to be precise). These give odd results at about ~12 places after the point. Usually it get rounded to let us see regular numbers.

This is also the reason why drawings loose precision, if the content is placed very far away from 0,0,0 coordinates. The odd results get then shifted closer to the point.

Re: Decimal number

Valentin_CAD — Tue, 19 Sep 2023 10:06:52 GMT

@johnkelly60 ,

The highest precision is to 8 decimal places. Limit your decimal precision no more than 8 decimal.

In your case you may want to display it to two.

You currently show 14 decimals.

Re: Decimal number

johnkelly60 — Tue, 19 Sep 2023 16:30:17 GMT

Thanks for the reply, but I already know this settings.
Like I said, I have mine set up as 0.00 precision and showing up as the image I included when I want to edit the number.

Re: Decimal number

johnkelly60 — Tue, 19 Sep 2023 16:32:20 GMT

Yeah I know it's not that big of a deal since it does round up, but I'm just wondering why it does that to only the number ending in 5... it doesn't show up like that when I enter any other number (like 0.02, 0.03, et.)

Re: Decimal number

Valentin_CAD — Tue, 19 Sep 2023 16:51:55 GMT

@johnkelly60 ,

If you typed 45.84999999999999 in the cell, then you will see what you typed when you edit.

The 45.85 is just displaying the rounding you specified.

If you wanted to see the exact number, then you will have to make the precision as in the cell (with an awareness that it only goes out to 8 decimal places).

Re: Decimal number

leeminardi — Tue, 19 Sep 2023 17:44:27 GMT

As noted, AutoCAD uses a floating point data base of 64 bits for coordinates and table values. The mantissa is 52 of the bits, the + or - 1 bit, and the exponent 11 bits. This yields about 15 significant digits of precision which is from the fact that 2^52 = 4.5035... x 10^15.

It is not correct to say that AutoCAD precision is limited to 8 decimal places. The display of coordinates is limited to 8 decimal places. For coordinates of less than 1 the precision could be 15 or more decimal places. For coordinates of a billion the precision would be less than 6 decimal places (15-9) with the rest of the precision to the left of the decimal point.

Just as in a base 10 system 1/3 is an irrational number (0.33333...). In base 2 (binary) the value of base 10 for 1/10 (0.1) is an irrational number approximately equal to 0.00011001100110011... an irrational number! This is why some "nice" numbers in base 10 cannot be cleanly represented in base 2.