topic Re: Is this a bug . strange values in Fusion API and Scripts Forum

Is this a bug . strange values

StephenCim-001 — Tue, 02 Apr 2019 05:53:01 GMT

Hi All

My Add-in has a couple of Value inputs, one for Shank Width (ShankWidth) and one for Stone Diameter (StoneDia), both are in mm

The values inputed are stored has User Parameters .

here are a few screen shots both set to 4mm, and you can see the user parameters showing 4mm

now with settings set to 2mm

and user parameters

But if I use any number 3.x number I get this

If I entre 3.5 , it will get stored has

3.5000000000000004

3.708 and above is store has it's entered

3.707 becomes 3.7070000000000003 mm

2.999 is fine if I enter 2.9999 it then becomes 2.9999000000000002 mm

Which is does not look so good when I read the value back into the input

Has anyone come across this before?

Cheers, Stephen

Re: Is this a bug . strange values

StephenCim-001 — Mon, 08 Apr 2019 12:25:39 GMT

Am I right in saying no one has seen this sort of error before?

Cheers, Stephen

Re: Is this a bug . strange values

BrianEkins — Mon, 08 Apr 2019 18:21:29 GMT

Welcome to the world of floating point numbers and computers. It's not a bug but is just a reflection of how floating point numbers are stored in a computer. I wrote a post about this a few years ago illustrating it using Visual Basic. I also just found a good write-up in the Python documentation. It's important to note that it's not a programming language issue but just a reflection of the internals of how computers work with floating point numbers.

Re: Is this a bug . strange values

StephenCim-001 — Tue, 09 Apr 2019 10:15:42 GMT

Thanks Brian I've added some code to truncate figures to 2 decimal places.... 🙂 Cheers, Stephen

Re: Is this a bug . strange values

JeromeBriot — Wed, 10 Apr 2019 05:01:54 GMT

Hello,

As @BrianEkins already said, this is related to how floating-point numbers are encoded into memory. See IEEE 754 for more information. Most of the time, these numbers are encoded with 64 bits and are represented with 16 decimals.

The only problem here is that the last digit of the numbers are incorrect. It's obvious for 3.5 that should be exactly represented as 3.5000000000000000

Here is a small python code:

X = [2.99, 3.5, 3.707, 3.708]
['{:.16f}'.format(x) for x in X]

And the results:

['2.9900000000000002',
'3.5000000000000000',
'3.7069999999999999',
'3.7080000000000002']

As you can see, these values are different from the ones you reported in the first post.