×

Discussion Board

Results 1 to 11 of 11
  1. #1
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Strange issue with precision on N9

    I'm testing a calculator application and apparently got to a strange precision issue with simple operations on occasional operands. For example, operation
    55*0.45
    returns
    24.750000000000004
    The operations are executed through JS, not C++.
    Issue present on both physical N9 (PR 1.1) and Qemu.
    Does anyone know how to avoid this besides artificially limiting the precision?
    Thanks!
    Last edited by conxt; 2011-12-06 at 20:21.

  2. #2
    Super Contributor
    Join Date
    Mar 2009
    Posts
    1,024

    Re: Strange issue with precision on N9

    Hi,
    You can round flat numbers by using toFixed() JS core function.
    Usually 2 decimals are enough.

    http://www.w3schools.com/jsref/jsref_tofixed.asp

  3. #3
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    That would really be the solution if the calculator were financial, but it's engineering with sines, logarithms etc., where it's not fully appropriate to limit the precision.
    Anyway, I wonder how this issue is possible at all.

  4. #4
    Super Contributor
    Join Date
    Mar 2009
    Posts
    1,024

    Re: Strange issue with precision on N9

    Well, even scientific calcultors round their results.
    No idea if there is a function to identify significant figures in JS.. but I don't think so.
    You maybe need to write them, if you are not able to find it googling.

    http://en.wikipedia.org/wiki/Signifi...ificant_digits

  5. #5
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    What really bothers me, is that the result in the given example is basically incorrect.
    Say, I do sqr(1.0000001). The correct answer here is 1.00000020000001, it's computed correctly, and the tailing 1 does matter.
    However, 0.555*25 produces 13.875000000000002 which is false, and there is no way to determine this computational mistake, the whole concept of which seems very weird.

  6. #6
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    By the way, I just downloaded one of the calculators from the Store and it produces exactly the same result.
    So, I guess, this might be at the OS core level...

  7. #7
    Super Contributor
    Join Date
    Mar 2009
    Posts
    1,024

    Re: Strange issue with precision on N9

    Hi,
    These are the result of your operations, calculated by Wolframe Alpha. http://www.wolframalpha.com
    * 0.555*25 = 13.875
    * sqr(1.0000001)= 1.0000000499999987500000624999960937502734374794921891113279...

    If you get 13.875000000000002, I can see the error there, but to be honest I see a bigger mistake in the second one.

    Do you have the same results if you run this calculations in your PC?
    In any case you can file this as bug in Qt bug reporting tool https://bugreports.qt.nokia.com/secure/Dashboard.jspa

  8. #8
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    Wolframalpha interprets "sqr" as square root, not as ^2, so that result is correct :)
    Thank you for the link! - I wanted to post this issue but I couldn't find the correct place to do it
    I guess I will have to live with this problem (luckily, the set of erroneous operands is very limited) until the fix.
    Thanks once again!

  9. #9
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    Before posting the bug I will try to compile a small sample app for different platforms (Qt Desktop and emulated Symbian) to be able to be more specific there.
    Thanks!

  10. #10
    Super Contributor
    Join Date
    Mar 2009
    Posts
    1,024

    Re: Strange issue with precision on N9

    You are welcome,
    BTW please if you file the bug there, paste here the URL to the bug report for reference.
    Other people may run in the same issue.

    Anyway it's interesting to understand if this is a platform problem or not.
    If the bug is related to Harmattan itself the bug could be solved quickly.
    In case this error happens on other platforms too (PC for instance), then
    the problem is in the JS engine and it could take more time!

  11. #11
    Registered User
    Join Date
    Nov 2011
    Posts
    48

    Re: Strange issue with precision on N9

    After some tests it came down to this:
    https://bugreports.qt.nokia.com/browse/QTBUG-23074

Similar Threads

  1. Need help to the strange issue
    By SecondInnings in forum Symbian
    Replies: 7
    Last Post: 2011-08-26, 15:07
  2. Strange issue please anybody help ?
    By aamitgupta in forum Symbian
    Replies: 11
    Last Post: 2009-05-04, 19:05
  3. Strange issue with TSoInetInterfaceInfo
    By patrickfrei in forum Symbian Networking & Messaging (Closed)
    Replies: 1
    Last Post: 2008-01-21, 09:37
  4. Strange issue in N73, pleez help...
    By mailtomandy2000 in forum Symbian
    Replies: 1
    Last Post: 2007-05-01, 19:38

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •