Adding Decimals

[stevecasper]

Accidentally posted this to the ME3 board originally (though ME3 also displays this same behavior).

 

For reasons I won't go into right now, I'm using a repeat command to increment a decimal's value.

 

Because the "Modify Decimal" command doesn't offer an Increment option, I am using a very basic addition routine.

 

%D[1]% starts with a value of .001 and on each run-through it adds .001 to the total.

 

It works fine... until it gets to .072

 

.071 + .001 (according to MEP) = 0.0720000000000001

 

This continues until %D[1]%=.1, after that, it gets back to normal: .101, .102, etc.

 

I built the same macro in ME 3.x and got the same result.

 

Here's the relevant MEP Code:

 

Variable Set Decimal %D[1]% to .001
Repeat Start (Repeat 75 times)
 If Variable %n[1]% Is Greater Than "69"
   Text Box Display: 1
 End If
 Variable Modify Decimal: %d[1]% = %D[1]% + .001
 If Variable %n[1]% Is Greater Than "69"
   Text Box Display: 2
 End If
End Repeat

 

 

<VARIABLE SET DECIMAL Option="\x00" Destination="%D[1]%" Value=".001"/>
<REPEAT START Start="1" Step="1" Count="75" Save="TRUE" Variable="%n[1]%"/>
<IF VARIABLE Variable="%n[1]%" Condition="\x03" Value="69" IgnoreCase="FALSE"/>
<TEXT BOX DISPLAY Title="1" Content="{\\rtf1\\ansi\\ansicpg1252\\deff0\\deflang1033{\\fonttbl{\\f0\\fnil\\fcharset0 Tahoma;}{\\f1\\fnil Tahoma;}}\r\n\\viewkind4\\uc1\\pard\\f0\\fs16 %d[1]%\r\n\\par \\f1 \r\n\\par }\r\n" Left="Center" Top="Center" Width="278" Height="200" Monitor="0" OnTop="FALSE" Keep_Focus="TRUE" Mode="\x00" Delay="0"/>
<END IF/>
<VARIABLE MODIFY DECIMAL Option="\x00" Destination="%d[1]%" Value1="%D[1]%" Value2=".001"/>
<IF VARIABLE Variable="%n[1]%" Condition="\x03" Value="69" IgnoreCase="FALSE"/>
<TEXT BOX DISPLAY Title="2" Content="{\\rtf1\\ansi\\ansicpg1252\\deff0\\deflang1033{\\fonttbl{\\f0\\fnil\\fcharset0 Tahoma;}{\\f1\\fnil Tahoma;}}\r\n\\viewkind4\\uc1\\pard\\f0\\fs16 %d[1]%\r\n\\par \\f1 \r\n\\par }\r\n" Left="Center" Top="Center" Width="278" Height="200" Monitor="0" OnTop="FALSE" Keep_Focus="TRUE" Mode="\x00" Delay="0"/>
<END IF/>
<END REPEAT/>

 

Very strange. Any explanation?

 

I did figure out a simple work-around: Add a second Modify Decimal line: Round to 3 decimal points. That seems to have fixed the issue.

 

 

Follow-up

Oh, and one other thing:

 

If I start with %D[1]% = .018 or higher, the bug doesn't happen.

 

Weird.

[Cory]

There have been several posts on this matter hear in the forum and one very recently. Do a search here for floating point accuracy and read up on it there.

 

Astronomers and scientists have had to deal Whit this fundamental problem in computers for a very long time and as a general rule they find ways to do the math differently to not have problems like base 10 math in a base 2 environment. I'm guessing they may often actually compute in base 2 but there's an easier one you can use. Consider your largest need for decimal places and multiply all values by that order. A machinist for instance rarely needs less than a mil (1/1000) of an inch. So multiply al your values by 1000 initially, convert to integer or large integer, do integer calculations, and divide by it in the end. Saves a lot of rounding. And essentially that’s what floating point is. It’s like the exponent we used to use in scientific math. It keeps the meaty part of the number and then remembers where the decimal point is. Problem is with two values with largely different decimal points these little errors creep in. At least that’s how I understand it. But then again I’m no math whiz. But by multiplying the integer value yourself you’re essentially locking down where the decimal point is and avoiding problems.

 

My two cents is that I think it's a short coming of MEP and there should be a smarter routine to deal with this. If you’re a computer math geek it’s all understood but to the layman it’s not only mysterious it can cause serious problems. Especially when making logical comparison like “Is equal to”. Hopefully one day they will have some time to address this.

 

 

 

In the past I found this problem in the date math functions as well but I think they may have that one fixed now. <BR style="mso-special-character: line-break"><BR style="mso-special-character: line-break">

 

 

 

 

[paul]

My two cents is that I think it's a short coming of MEP and there should be a smarter routine to deal with this. If you’re a computer math geek it’s all understood but to the layman it’s not only mysterious it can cause serious problems. Especially when making logical comparison like “Is equal to”. Hopefully one day they will have some time to address this.

But surely the kind of problem described here is common among [almost?] all languages? I am entirely unsurprised by the difficulties perhaps most ME users have when dealing with FP numbers (and I certainly don't exclude myself from their number) precisely because ME allows laymen users to do various things that some people would prefer restricted to the kingdom of the "computer geek"! When doing FP stuff with computers, rounding is a real problem, and cannot be swept under the carpet simply to protect the innocents.

[Cory]

I largely agree with you Paul. But most of the users here are not programmers and have never even heard the term "floating point". And I ran into the problem doing a date comparison using whole days once! I'm just saying it's going to frustrate a lot of ISS's target market. No good solution really I suppose. But I wonder if it wouldn't be too difficult to identify the greatest number of significant digits in certain operations and automatically do some rounding based on that. Yeah, maybe not huh? I suppose you’re right.

 

 

[paul]

I largely agree with you Paul. But most of the users here are not programmers and have never even heard the term "floating point". And I ran into the problem doing a date comparison using whole days once! I'm just saying it's going to frustrate a lot of ISS's target market. No good solution really I suppose. But I wonder if it wouldn't be too difficult to identify the greatest number of significant digits in certain operations and automatically do some rounding based on that. Yeah, maybe not huh? I suppose you’re right.

I suspect that any kind of "solution" is going to produce the wrong answer one day, thus simply replacing one, perceived, "wrong" answer with another, real, wrong answer!

[Cory]

You're 100% correct. I offered ISS a suggestion to put a little warning link in any of the decimal calculation dialogs. Kind of a "Learn more..." kind of thing.