7 ms·
On JSON: "if you only need 53 bits of your 64 bit numbers" JSON numbers are arbitrary precision. "blowing CPU on ridiculously inefficient marshaling and unma
by mahkoh 7y ago
On JSON:
"if you only need 53 bits of your 64 bit numbers"
JSON numbers are arbitrary precision.
"blowing CPU on ridiculously inefficient marshaling and unmarshaling steps"
On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor.
- mgraczyk 7y agoThat is not true of JSON numbers either practically or technically. This specification allows implementations to set limits on the range and precision of numbers accepted. https://tools.ietf.org/html/rfc7159#section-6 https://tools.ietf.org/html/rfc7159#section-6
- SahAssar 7y agoYes it is. All that part of the spec says is that it allows implementations to set/have limits on ranges and precision, which is obvious because for example javascript only has 53 bit numbers. Other implementations can have other limits, but that does not mean that JSON itself or all implementations have the specific limits of javascript.
- megous 7y agoNot anymore. Try writing 10000000000000000000000000000000000000000000000000000000000000000000000000000000n + 1n into your browser console.
- majewsky 7y agoWell yeah, those are bigints, but JSON does not support bigints, so the point still stands.
- nemetroid 7y agoIf by "JSON" you mean "JSON parsers in web browsers", then yes.
- howdydoo 7y agoFirefox: >JSON.stringify(1n) TypeError: BigInt value can't be serialized in JSON Chrome: >JSON.stringify(1n) Uncaught TypeError: Do not know how to serialize a BigInt at JSON.stringify (<anonymous>) at <anonymous>:1:6 I don't think we're quite there yet.
- megous 7y agoIt was counterpoint to "that does not mean that JSON itself or all implementations have the specific limits of javascript." Javascript can do arbitrary precision integers now. So the last part is not true. You can generate your JSON strings with string concatenation in JS if you like. Just because JSON.stringify has some limit doesn't mean JS has.
- apaprocki 7y agoSee my comment elsewhere in the thread that shows how to properly use BigInts with JSON.stringify, if you choose to support them in your environment.
- raverbashing 7y agoI'm assuming the percentage of problems where the difference between 64 bits and 53 bits precision matter are really, really small (and you should be paying attention then, because you're bound to lose precision in other parts and not only in JSON) For everything else, it is really a non-issue.
- mgraczyk 7y agoIt is not the case that JSON numbers are "arbitrary precision". JSON numbers are in fact "implementation defined" precision. The technical and practical implication is exactly what was described in the article: Since precision is implementation defined (not uniform and with no lower-bound), JSON numbers make a poor vessel for inter-service communication.
- jcoffland 7y agoJavascript cannot handle 64-bit unsigned integers and the largest exact integral value is 2^53-1, but yeah JSON itself has no such restrictions.
- kristiandupont 7y agoI don't understand why you are being downvoted. This struck me not as something that starts to "emerge after you've dealt with enough failures in a given domain" as the author claims, but more like a pet peeve. The fact that JS has 53 bit precision will be a JS problem whether you use protobuf or anything else. On the other hand, if you are not using JS, it will almost certainly parse numbers of the precision that your language offers.
- TeMPOraL 7y agoThe original article's complaint wasn't (just) about 53 bits, but about using inefficient formats requiring constant serialization and deserialization for backend-to-backend communication.
- kristiandupont 7y agoProtobuf or Thrift would require serialization as well? I still wouldn't do that (or, in fact, I am currently not doing that) before I have an incentive, i.e. performance problems. Of course, my services are all Nodejs at the moment, so JSON feels more natural there than it does in other environments.
- mr_crankypants 7y agoOne potentially nice thing about formats like Cap'n Proto is that they don't eagerly parse the message. You can just lazily grab the values you care about, which can be a big performance win if you typically only care about a few fields out of a larger message. That's impossible with text-based formats like XML and JSON, because everything is fractally variable-length.
- bibabaloo 7y agoIt seems like its more "problems you deal with when you work at scale" like the author has at Google & FB. At smaller scales, I think the (machine) benefits of using something like protobuf don't nearly outweigh the human benefits of just using JSON.
- TeMPOraL 7y ago> On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. Because learning how to use correct tool for the job is so difficult and so outside of dev/qa job descriptions, that it's much better to waste compute on your servers and deal with performance and scalability problems.
- rimliu 7y agoSo the old "computer time is cheap, human time is not" is not true anymore?
- joshuamorton 7y agoStrongly typed, schema'd data interchange leads to less human time in the long run.
- EdwardDiego 7y agoYep, you just have to put the time into ensuring that convenient <Serialisation X> utilities exist that offer the (near) same functionality of curl/jq etc. It's a pain, but it's a short term pain, and necessary.
- Piskvorrr 7y agoIt is. OTOH, "these tools have been sufficient ten years ago, so I'm sticking to them forever" leads to wasting human time (by ignoring newer, potentially more efficient, tools).
- nvarsj 7y agoI think that line is mostly propaganda from defenders of poor performing language runtimes. It’s not a dichotomy - both are important. Eg AWS bills can easily eclipse engineer salaries if you use it inefficiently.
- taffer 7y ago> propaganda from defenders of poor performing language runtimes Your comment is unnecessarily inflammatory. > AWS bills can easily eclipse engineer salaries if you use it inefficiently. If this happens, it is more due to architectural and algorithms problems than to the runtime of the language. There are many large applications written in Python, Ruby and PHP that run just fine.
- patrec 7y ago> JSON numbers are arbitrary precision. > On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. What a pleasant surprise it will be for you when you find out that jq silently corrupts integers with more than 53 bits.
- mr_crankypants 7y agoIt's perhaps less a limitation of JSON and more a limitation of JavaScript itself[1]. But still, given that easy consumption from JavaScript is the ultimate primordial reason for choosing JSON over other formats, it seems like trying to transmit integers with more than 53 bits of precision over JSON is asking for trouble. Because it's only a matter of time until someone will want to do something like write a new service in Node, and the JavaScript parsers for other formats are at least somewhat more likely to guide people toward using BigInt for large integers. [1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- apaprocki 7y agoIn TC39, we specified BigInt to not participate in JSON by default, precisely because emitting it as a native JSON number would not be able to be read back by JSON.parse() and many other environments would also not be able to parse it without extra logic if they simply use IEEE754 double. I explicitly asked for and achieved step 2. in the modified SerializeJSONProperty algorithm[1] so that users could decide and opt-in to serializing BigInts as strings if they so choose, with or without some sigil that could be interpreted by a reviver function. e.g.: > JSON.stringify(BigInt(1)) TypeError: Do not know how to serialize a BigInt ... > BigInt.prototype.toJSON = function() { return this.toString(); } > JSON.stringify(BigInt(1)) '"1"' [1]: https://tc39.es/proposal-bigint/#sec-serializejsonproperty https://tc39.es/proposal-bigint/#sec-serializejsonproperty
- awinder 7y agoOr they'd just wrap as a string and move on with life, much as people do when they send uuids which should consume 128-bit as wide strings. The cases I've encountered that actually fundamentally required 64-bit ints are completely overshadowed by the cases where some well-intentioned developer decides to pack 64-bit ints with information to turn it into a uuid as well, so I really hope the consternation over this is from people who work a lot with mathematical apis/systems and not devs doing more heinous things with 64bit ints.
- LaGrange 7y ago> On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. Is your turnover so high that learning a single tool is a significant portion of employee costs?
- raxxorrax 7y agoWhy not just use SOAP? I think recent experience should tell us that the easiest thing plainly wins out. JSON can be terrible, Javascript can be terrible, but limitations on large integers isn't the worst hurdle to take.