3
$\begingroup$

I made the following observation following a suggestion from a friend. This question is about the Microsoft calculator program implemented on Windows. Since I don't have access to any other calculator at the moment, I cannot confirm whether this is a general feature.

If you enter an integer in the calculator program, say $N$, hit the minus button followed by the same integer again, you get zero as expected.

However, if you input $N^2$, then hit square root button, you get $N$, then subtract $N$, you get a very small number, but not zero.

This has been my observation for several integers that I have tried.

For example, with $N=5$, you get $1.232\times 10^{-24}$ in the final step.

Now, I understand computers use finite precision (and hence the square root function must be working under finite precision). However, I don't understand why this shows up in the final step only? Is it only because this is a small number in comparison to any integer, but when you get $0$, you can no longer make that approximation?

  • 0
    @AD: The division is a likely culprit. Adding and multiplying are no longer as troublesome as they were (especially now that hardware nowadays support things like FMA), but division remains to trap the unwary (anybody remember that famous division bug a while back?).2010-11-26

2 Answers 2

8

You'll find your answer and much more in the articles below:

What Every Computer Scientist Should Know About Floating-Point Arithmetic

Also see this article on Calculator Forensics, esp. "Hints and Tricks For Obtaining Guard Digits".

  • 0
    +1 for the beautiful article, which should be read after perusing the IEEE guidelines. Velvel Kahan's miscellaneous papers in his site, like [this one](http://www.cs.berkeley.edu/~wkahan/SQRTest.ps) are also an excellent read.2010-11-26
0

J.M., since when is 24 decimal digits accuracy considered a bad approximation?

Original poster, they could show sqrt(5)**2 as

5.000000000000000000000001232

but since the display probably has fewer display digits than this, they chop it to "5" as that is the closest representation in the allowed number of digits.

BTW, I just tried this in "calc" in windows 7 and the 5-sqrt(5)**2 is 1.7599e-37

If you want exact math, you will need to use a symbolic math package.

  • 0
    First, my apologies for posting what I intended to be a comment as an answer, which it isn't.2010-11-26