.to_d, or not .to_d


Okay, short story time:

As I mentioned last time, Renegade had a problem. And it was driving me insane.

I had just solved my first issue involving lack of precision with floating point numbers using the Big Decimal method .to_d. Things were finally running smoothly….or so I thought.

After using it for a week or so, I discovered that one particular calculation would sometimes be $0.01 off. For example, if given $530.04 - $500.02, the system would output $30.01 instead of $30.02.

It only occurred about 10% of the time. I couldn't even figure out the exact conditions under which it happened. Regardless, Renegade deals with money. It can't afford to be off, even by $0.01.

Exasperated, I turned to my good old friend Google (and the lovely, brave folks at Stack Overflow) to figure out what to do. I ran in circles for a while before I realized what had happened.

There are two ways to invoke Big Decimal – one way is to use the .to_d method. The other way is to create a new Big Decimal class using BigDecimal.new().

There's a big difference between the two options. When you use BigDecimal.new(), you put a number in the Big Decimal class before you manipulate it. This means that the number is precise from the very beginning. It's like using waterproof materials to make a boot.

You invoke the .to_d method after the number has already been represented as a floating point number – which, as explained here, is not very precise. It's like spraying a regular boot with waterproofer. This is what I had been doing.

If you know the difference between a waterproof boot and a boot that's been sprayed with waterproofer, you'll see where my problem was.

When I realized this (and partly because I didn't know what else to do), I put all of my numbers under the Big Decimal class. New inputs would have the precision of Big Decimal from the beginning.

It changed everything. Suddenly Renegade was so precise that it gave out more accurate results than I got when I was using a manual 10 key calculator. It was beautiful. I almost cried.

Moral of the story: Use the .to_d method with caution. If you're dealing with currency, BigDecimal.new() may be a better option for you.

Have you ever had an unexpected issue with the Big Decimal method or class?

* You can find the source code for Renegade (which is written in Ruby), as well as the commits that inspired this post, here on Github. Try not to laugh too hard at the GUI, or lack thereof. *

Copyright © 2016. Demiera Harris.
Design by Herdiansyah Hamzah. & Distributed by Free Blogger Templates
Creative Commons License