Xojo Conferences

« Tip of the day: Start… | Home | MBS Xojo / Real Studi… »

Tip of the day: Double comparison

Just run into the problem this week that double comparison often works, but also fails unexpected in an application. This remembers again to use comparison with tolerance for double variables. Like here with abs() and a 0.01 tolerance:

dim d as Double = 10 dim e as Double = 1.19 dim f as Double = d * e // checking exact fails if 11.90 = f then Break else Break end if // better check with tolerance if abs(11.90-f) < 0.01 then Break else Break end if
11 09 14 - 10:34
two comments

Wouldn’t it make sense to use an extension method?

Function IsEqualTo(Extends d1 as double, d2 as double) as Boolean
return abs( d1 – d2 ) < 0.0001
End Function

Then you would just call it as

if d1.IsEqualTo( d2 ) then
Markus Wnter - 12 09 14 - 19:07

Xojo has actually introduced an Equals method which does exactly that (I still mainly work with RS2012R2 so hadn’t noticed).

Will Shank had another tip: In the menu go to Project > Analysis Warnings…

make sure “Performing a Item1 comparison on floating-point values can yield unexpected results due to their inexact binary representation.” is checked and they’ll appear in Analyze Project results.
Markus Wnter - 14 09 14 - 22:06

Remember personal info?

Emoticons / Textile

Hide email:

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.