5 ms·

Be careful with JS numbers

124 points by gren 14y ago

0x0 14y ago
This bit a lot of twitter integrations hard, when tweet id's exceeded the non-lossy integer range of javascript floats. All of a sudden, tweet ids got rounded off to point to completely different entries! https://dev.twitter.com/docs/twitter-ids-json-and-snowflake https://dev.twitter.com/docs/twitter-ids-json-and-snowflake Other JSON parsers may also be affected.
- gren 14y ago
  Woo I didn't know this! When I mention twitter ids it was just an example. So it can happen to anyone and it's a real issue for any JSON API.
- nness 14y ago
  For quite some time there were API calls in Facebook that would return the ID's as numbers, not strings. It wasn't specific to the entire API though, only some calls would show the behaviour. As you can imagine, PHP did not like it. It was rather irritating...
prophetjohn 14y ago
I have determined by experience that 9007199254740995 (which is 2^53+3) is the smallest not representable integer in Javascript. Here's something to drop in your browser console as an amusing illustration of this 9007199254740994 === (9007199254740995 - 1) and 9007199254740995 === (9007199254740995 - 1)
- gren 14y ago
  Hehe fun one! and also: 9007199254740995 - 1 - 1 - 1 - 1
- evincarofautumn 14y ago
  He’dn’t’ve had to “determine” it, had he known the basics of IEEE-754.
jamestnz 14y ago
This is not the only area in JS where it pays to be careful with numbers. For example, the parseInt function mentioned in the article actually does magic base determination. Your string will be parsed into a base-10 number, unless it begins with '0x' in which case base-16 is used, or if it begins with a leading '0' then it is treated as base-8. This last case has stung me on several occasions when parsing user input into numbers: User puts a leading-zero, and you magically end up with octal conversions. (I believe this whole octal thing has been deprecated in recent JS implementations). In any case it's sensible to specify the 'radix' whenever using the parseInt function, as in parseInt(numberString, 10);
- Dylan16807 14y ago
  If the user deliberately puts in 0x it's probably because the number is more convenient in hex. Definitely get rid of octal though. The octal designation should have had a letter in it right from the start.
- frewsxcv 14y ago
  Yet another reason to use JSLint (it makes the second parameter of parseInt not optional)
- VMG 14y ago
  you can also just use Number() to be safe
  - gren 14y ago
    How is it different?
    - Hovertruck 14y ago
      Think of parseInt() as what it says- parsing out the number value. Number() can be considered as a cast. parseInt('12', 10); => 12 Number('12'); => 12 parseInt('12x', 10); => 12 Number('12x'); => NaN
martin-adams 14y ago
I got hit by JavaScript rounding on a project once. Funny thing is, IE8 was okay with it and Chrome was the one that caused issues. Turns out that 1000000 * 8.2 = 8199999.999999999 Sometimes we learn the hard way. The bug was getting written into an XML document and pushed to an embedded device over HTTP which was expecting an int and caused the device to crash. We fixed both bugs.
- deleted 14y ago
  [deleted]
- ricardobeat 14y ago
  // Scheme (print (* 1000000 8.2)) > 8199999.999999999 // Python >>> 1000000 * 8.2 8199999.999999999 //PHP > 1000000 * 8.2 > 8199999.999999999
  - ehsanu1 14y ago
    # Ruby 1.8.7 1.8.7 :022 > 1000000 * 8.2 => 8200000.0 # Ruby 1.9.2 1.9.2p320 :001 > 1000000 * 8.2 => 8199999.999999999
  - CamperBob2 14y ago
    C/C++: printf("%lf\n", 1000000 * 8.2); 8200000.000000 Here, printf() is doing the necessary rounding to hide the fact that 8.2 isn't exactly representable. It's interesting that other languages don't do something similar when outputting a float to text.
    - pindi 14y ago
      They do, you just have to request it: //Python >>> repr(1000000 * 8.2) '8199999.999999999' >>> "%lf" % (1000000 * 8.2) '8200000.000000'
  - jabiko 14y ago
    I can't reproduce the PHP example: $ uname -p x86_64 $ php -v PHP 5.4.6-1ubuntu1 (cli) (built: Aug 22 2012 21:13:52) Copyright (c) 1997-2012 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies $ php -r 'var_dump(1000000 * 8.2);' float(8200000) $
Dylan16807 14y ago
1. IDs are strings, not numbers. This is the real problem twitter had. The ID size was only vaguely related to the number of tweets. 2. An event counter is not going to reach 2^50.
crazygringo 14y ago
Essential rule: ALWAYS treat large ID numbers in JSON as strings. Produce them as strings on the server, so the browser treats them as strings during the JSON decoding process. (Fortunately, if they're ID's where they need to stay exact, then you probably don't need to do any math on them! Although once a coworker of mine had to write a bitwise XOR function in JavaScript that operated on large ID numbers represented as strings. That was fun...)
- Joeri 14y ago
  Been burned by this myself, because while the person that created the ID knew it was a string, the person using it down the line thought it looked like a number and treated it like one. My advice is to store those ID's with a leading "$" character or something similar. Removes the temptation to treat them like numbers.
  - ewang1 14y ago
    Or convert them to hex before sending to client. And you can convert them back to decimal on the server side.
wallflower 14y ago
"Eich: No one expects the rounding errors you get - powers of five aren't representable well. They round poorly in base-two. So dollars and cents, sums and differences, will get you strange long zeros with a nine at the end in JavaScript. There was a blog about this that blamed Safari and Mac for doing math wrong and it's IEEE double -- it's in everything, Java and C" -Peter Seibel Interview with Brendan Eich "who under intense time pressure, created JavaScript", Coders at Work, p.136
- jrabone 14y ago
  Some languages have arbitrary precision types or use BCD. Next time you see someone using floating point types for money, please stop them. There's too many financial libraries out there like OFX4J which get this wrong (or at least did the last time I looked).
  - derleth 14y ago
    > Next time you see someone using floating point types for money ... apply that logic to their paycheck and let them deal with being underpaid by a bizarre amount because of rounding. Or apply it to their investment accounts so the errors accumulate for a few decades. Nothing like an object lesson to remedy bad behavior.
  - icebraining 14y ago
    Python has Decimal for that. Unfortunately, the people who wrote the platform I use - which deals with money - decided they're "too slow".
    - masklinn 14y ago
      > Unfortunately, the people who wrote the platform I use - which deals with money - decided they're "too slow". I've had the same problem (that platform wouldn't ha. It's… annoying (also bullshit: in the worst case — CPython <= 3.2 — basic arithmetic operations will take ~20µs on my machine aside from divisions which take a bit more and if that shows up in profiling use cdecimal, CPython 3.3 [integrated cdecimal] or pypy)
meaty 14y ago
This is one reason I stick to statically typed languages. I do a lot of work with cash values and without an explicit decimal type, the shit hits the fan when you hit a float issue. I get a lot of flak on here for that opinion which is odd.
- mmahemoff 14y ago
  Dynamically typed languages still have types. Such as BigNum.
  - meaty 14y ago
    Yes and they are cumbersome.
- jre 14y ago
  It's not really about static typing or not, it's really about JS having some really strange and counter-intuitive behaviours in a lot of cases (numbers, default method scope, truthy values, ...). Part of it is probably due to the fact that the syntax looks so much like C and Java that you'd except it to behave the same. Part of it seem to be that it's hard to fix things without breaking existing codebases.
  - ricardobeat 14y ago
    Every language has it's own quirks, some even weirder than JS. The floating point "errors" are common to a hundred programming languages, that's why BigNum classes/extensions exist.
    - jre 14y ago
      The problem in this case is not floating point approximations. It is that some functionalities (parseFloat vs parseInt, ++, 45.0 being printed 45 in most browsers) makes the user believe that JS has an integer type.
      - 14y ago
RyanZAG 14y ago
Interestingly, GWT gets around this problem entirely by having the 'long' datatype be created as two 32 bit numbers. Slows down calculation a little bit, but you get the correct values always. https://developers.google.com/web-toolkit/doc/latest/DevGuideCodingBasicsCompatibility https://developers.google.com/web-toolkit/doc/latest/DevGuid...
- niggler 14y ago
  Emscripten does the same thing for uint64_t etc
malkia 14y ago
Is this in the JS standard, or is this in just some implementations?
- VMG 14y ago
  http://interglacial.com/javascript_spec/a-8.html http://interglacial.com/javascript_spec/a-8.html The Number type has exactly 18437736874454810627 (that is, 264-253 +3) values, representing the double-precision 64-bit format IEEE 754 values as specified in the IEEE Standard for Binary Floating-Point Arithmetic, except that the 9007199254740990 (that is, 253-2) distinct "Not-a-Number" values of the IEEE Standard are represented in ECMAScript as a single special NaN value.
shtylman 14y ago
For people using npm, check out the 'int' and 'num' modules.
rorrr 14y ago
> Javascript doesn’t have integer type but lets you think it has Actually, JS has integer typed arrays. new Uint8Array([1,100,300]) [1, 100, 44] new Uint16Array([-1,2,300]) [65535, 2, 300] new Uint32Array([-1,2,300]) [4294967295, 2, 300] Here's the spec: https://www.khronos.org/registry/typedarray/specs/latest/#7 https://www.khronos.org/registry/typedarray/specs/latest/#7
pcwalton 14y ago
A lot of these issues are equally applicable to integer overflow (termination of loops and the "real web application disaster"). As a result, this is a deeper issue that extends beyond JavaScript into most languages—in general, programmers have to be aware that numbers in the runtime model of a language may not always work like you would expect theoretical, mathematical numbers to work. Of course, floating point imprecision results in more surprising behaviors than integer overflow on the whole, but the danger is still there no matter what language you use (not counting the languages with full numeric towers, like Scheme).
afhof 14y ago
Minor nit: isn't smallest integer not representable in JS -1000000000000000000000000000000000000 or something?

124 points by gren 14y ago

0x0 14y ago

This bit a lot of twitter integrations hard, when tweet id's exceeded the non-lossy integer range of javascript floats. All of a sudden, tweet ids got rounded off to point to completely different entries! https://dev.twitter.com/docs/twitter-ids-json-and-snowflake https://dev.twitter.com/docs/twitter-ids-json-and-snowflake Other JSON parsers may also be affected.

gren 14y ago
Woo I didn't know this! When I mention twitter ids it was just an example. So it can happen to anyone and it's a real issue for any JSON API.
nness 14y ago
For quite some time there were API calls in Facebook that would return the ID's as numbers, not strings. It wasn't specific to the entire API though, only some calls would show the behaviour. As you can imagine, PHP did not like it. It was rather irritating...

prophetjohn 14y ago

I have determined by experience that 9007199254740995 (which is 2^53+3) is the smallest not representable integer in Javascript. Here's something to drop in your browser console as an amusing illustration of this 9007199254740994 === (9007199254740995 - 1) and 9007199254740995 === (9007199254740995 - 1)

gren 14y ago
Hehe fun one! and also: 9007199254740995 - 1 - 1 - 1 - 1
evincarofautumn 14y ago
He’dn’t’ve had to “determine” it, had he known the basics of IEEE-754.

jamestnz 14y ago

This is not the only area in JS where it pays to be careful with numbers. For example, the parseInt function mentioned in the article actually does magic base determination. Your string will be parsed into a base-10 number, unless it begins with '0x' in which case base-16 is used, or if it begins with a leading '0' then it is treated as base-8. This last case has stung me on several occasions when parsing user input into numbers: User puts a leading-zero, and you magically end up with octal conversions. (I believe this whole octal thing has been deprecated in recent JS implementations). In any case it's sensible to specify the 'radix' whenever using the parseInt function, as in parseInt(numberString, 10);

Dylan16807 14y ago
If the user deliberately puts in 0x it's probably because the number is more convenient in hex. Definitely get rid of octal though. The octal designation should have had a letter in it right from the start.
frewsxcv 14y ago
Yet another reason to use JSLint (it makes the second parameter of parseInt not optional)
VMG 14y ago
you can also just use Number() to be safe
- gren 14y ago
  How is it different?
  - Hovertruck 14y ago
    Think of parseInt() as what it says- parsing out the number value. Number() can be considered as a cast. parseInt('12', 10); => 12 Number('12'); => 12 parseInt('12x', 10); => 12 Number('12x'); => NaN

martin-adams 14y ago

I got hit by JavaScript rounding on a project once. Funny thing is, IE8 was okay with it and Chrome was the one that caused issues. Turns out that 1000000 * 8.2 = 8199999.999999999 Sometimes we learn the hard way. The bug was getting written into an XML document and pushed to an embedded device over HTTP which was expecting an int and caused the device to crash. We fixed both bugs.

deleted 14y ago
[deleted]
ricardobeat 14y ago
// Scheme (print (* 1000000 8.2)) > 8199999.999999999 // Python >>> 1000000 * 8.2 8199999.999999999 //PHP > 1000000 * 8.2 > 8199999.999999999
- ehsanu1 14y ago
  # Ruby 1.8.7 1.8.7 :022 > 1000000 * 8.2 => 8200000.0 # Ruby 1.9.2 1.9.2p320 :001 > 1000000 * 8.2 => 8199999.999999999
- CamperBob2 14y ago
  C/C++: printf("%lf\n", 1000000 * 8.2); 8200000.000000 Here, printf() is doing the necessary rounding to hide the fact that 8.2 isn't exactly representable. It's interesting that other languages don't do something similar when outputting a float to text.
  - pindi 14y ago
    They do, you just have to request it: //Python >>> repr(1000000 * 8.2) '8199999.999999999' >>> "%lf" % (1000000 * 8.2) '8200000.000000'
- jabiko 14y ago
  I can't reproduce the PHP example: $ uname -p x86_64 $ php -v PHP 5.4.6-1ubuntu1 (cli) (built: Aug 22 2012 21:13:52) Copyright (c) 1997-2012 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies $ php -r 'var_dump(1000000 * 8.2);' float(8200000) $

Dylan16807 14y ago

1. IDs are strings, not numbers. This is the real problem twitter had. The ID size was only vaguely related to the number of tweets. 2. An event counter is not going to reach 2^50.

crazygringo 14y ago

Essential rule: ALWAYS treat large ID numbers in JSON as strings. Produce them as strings on the server, so the browser treats them as strings during the JSON decoding process. (Fortunately, if they're ID's where they need to stay exact, then you probably don't need to do any math on them! Although once a coworker of mine had to write a bitwise XOR function in JavaScript that operated on large ID numbers represented as strings. That was fun...)

Joeri 14y ago
Been burned by this myself, because while the person that created the ID knew it was a string, the person using it down the line thought it looked like a number and treated it like one. My advice is to store those ID's with a leading "$" character or something similar. Removes the temptation to treat them like numbers.
- ewang1 14y ago
  Or convert them to hex before sending to client. And you can convert them back to decimal on the server side.

wallflower 14y ago

"Eich: No one expects the rounding errors you get - powers of five aren't representable well. They round poorly in base-two. So dollars and cents, sums and differences, will get you strange long zeros with a nine at the end in JavaScript. There was a blog about this that blamed Safari and Mac for doing math wrong and it's IEEE double -- it's in everything, Java and C" -Peter Seibel Interview with Brendan Eich "who under intense time pressure, created JavaScript", Coders at Work, p.136

jrabone 14y ago
Some languages have arbitrary precision types or use BCD. Next time you see someone using floating point types for money, please stop them. There's too many financial libraries out there like OFX4J which get this wrong (or at least did the last time I looked).
- derleth 14y ago
  > Next time you see someone using floating point types for money ... apply that logic to their paycheck and let them deal with being underpaid by a bizarre amount because of rounding. Or apply it to their investment accounts so the errors accumulate for a few decades. Nothing like an object lesson to remedy bad behavior.
- icebraining 14y ago
  Python has Decimal for that. Unfortunately, the people who wrote the platform I use - which deals with money - decided they're "too slow".
  - masklinn 14y ago
    > Unfortunately, the people who wrote the platform I use - which deals with money - decided they're "too slow". I've had the same problem (that platform wouldn't ha. It's… annoying (also bullshit: in the worst case — CPython <= 3.2 — basic arithmetic operations will take ~20µs on my machine aside from divisions which take a bit more and if that shows up in profiling use cdecimal, CPython 3.3 [integrated cdecimal] or pypy)

meaty 14y ago

This is one reason I stick to statically typed languages. I do a lot of work with cash values and without an explicit decimal type, the shit hits the fan when you hit a float issue. I get a lot of flak on here for that opinion which is odd.

mmahemoff 14y ago
Dynamically typed languages still have types. Such as BigNum.
- meaty 14y ago
  Yes and they are cumbersome.
jre 14y ago
It's not really about static typing or not, it's really about JS having some really strange and counter-intuitive behaviours in a lot of cases (numbers, default method scope, truthy values, ...). Part of it is probably due to the fact that the syntax looks so much like C and Java that you'd except it to behave the same. Part of it seem to be that it's hard to fix things without breaking existing codebases.
- ricardobeat 14y ago
  Every language has it's own quirks, some even weirder than JS. The floating point "errors" are common to a hundred programming languages, that's why BigNum classes/extensions exist.
  - jre 14y ago
    The problem in this case is not floating point approximations. It is that some functionalities (parseFloat vs parseInt, ++, 45.0 being printed 45 in most browsers) makes the user believe that JS has an integer type.
    - 14y ago

RyanZAG 14y ago

Interestingly, GWT gets around this problem entirely by having the 'long' datatype be created as two 32 bit numbers. Slows down calculation a little bit, but you get the correct values always. https://developers.google.com/web-toolkit/doc/latest/DevGuideCodingBasicsCompatibility https://developers.google.com/web-toolkit/doc/latest/DevGuid...

niggler 14y ago
Emscripten does the same thing for uint64_t etc

malkia 14y ago

Is this in the JS standard, or is this in just some implementations?

VMG 14y ago
http://interglacial.com/javascript_spec/a-8.html http://interglacial.com/javascript_spec/a-8.html The Number type has exactly 18437736874454810627 (that is, 264-253 +3) values, representing the double-precision 64-bit format IEEE 754 values as specified in the IEEE Standard for Binary Floating-Point Arithmetic, except that the 9007199254740990 (that is, 253-2) distinct "Not-a-Number" values of the IEEE Standard are represented in ECMAScript as a single special NaN value.

shtylman 14y ago

For people using npm, check out the 'int' and 'num' modules.

rorrr 14y ago

> Javascript doesn’t have integer type but lets you think it has Actually, JS has integer typed arrays. new Uint8Array([1,100,300]) [1, 100, 44] new Uint16Array([-1,2,300]) [65535, 2, 300] new Uint32Array([-1,2,300]) [4294967295, 2, 300] Here's the spec: https://www.khronos.org/registry/typedarray/specs/latest/#7 https://www.khronos.org/registry/typedarray/specs/latest/#7

pcwalton 14y ago

A lot of these issues are equally applicable to integer overflow (termination of loops and the "real web application disaster"). As a result, this is a deeper issue that extends beyond JavaScript into most languages—in general, programmers have to be aware that numbers in the runtime model of a language may not always work like you would expect theoretical, mathematical numbers to work. Of course, floating point imprecision results in more surprising behaviors than integer overflow on the whole, but the danger is still there no matter what language you use (not counting the languages with full numeric towers, like Scheme).

afhof 14y ago

Minor nit: isn't smallest integer not representable in JS -1000000000000000000000000000000000000 or something?