6 ms·

I don't understand why you are being downvoted. This struck me not as something that starts to "emerge after you've dealt with enough failures in a given domain

I don't understand why you are being downvoted. This struck me not as something that starts to "emerge after you've dealt with enough failures in a given domain" as the author claims, but more like a pet peeve. The fact that JS has 53 bit precision will be a JS problem whether you use protobuf or anything else. On the other hand, if you are not using JS, it will almost certainly parse numbers of the precision that your language offers.

TeMPOraL 7y ago
The original article's complaint wasn't (just) about 53 bits, but about using inefficient formats requiring constant serialization and deserialization for backend-to-backend communication.
- kristiandupont 7y ago
  Protobuf or Thrift would require serialization as well? I still wouldn't do that (or, in fact, I am currently not doing that) before I have an incentive, i.e. performance problems. Of course, my services are all Nodejs at the moment, so JSON feels more natural there than it does in other environments.
  - mr_crankypants 7y ago
    One potentially nice thing about formats like Cap'n Proto is that they don't eagerly parse the message. You can just lazily grab the values you care about, which can be a big performance win if you typically only care about a few fields out of a larger message. That's impossible with text-based formats like XML and JSON, because everything is fractally variable-length.
bibabaloo 7y ago
It seems like its more "problems you deal with when you work at scale" like the author has at Google & FB. At smaller scales, I think the (machine) benefits of using something like protobuf don't nearly outweigh the human benefits of just using JSON.
- jlouis 7y ago
  They both exist for a reason. JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system. Up to a point, this also scales well. On the other hand, if you have many developers, and the problem space starts converging toward scale, then certainly the problem space is different. Encoding of data cannot follow the language with the weakest type zoo, and efficiency starts to matter. The hard part is to know when to make the switch, or when to anticipate growth in advance, such that you pick the right tool for the job. It isn't just a question of the machine. For complex messaging, the added value of a well-defined message typing, provided by protobuf, will help. It will also remove a lot of problems if you have multiple different languages in the stack, talking to each other.
  - wil421 7y ago
    Once you hit the wall with JSON what format do you use? At a past job we were stuck using SOAP/XML middleware. We hit walls but I’m not sure if a different format would work. We ended up rate limiting lower priority traffic to get past our issues.
    - q3k 7y ago
      Protobuf/gRPC.
    - juliusmusseau 7y ago
      CSV. Just kidding, but I suspect it would be a lot faster.
  - parliament32 7y ago
    >JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system. This is really, really bad thinking and will always cause you a timebomb that will 100% explode on you in the long run. "just a quick POC" will always end up being the actual product, and once you start down this path you end up with "let's just use mongo, it's schema-less, it'll be faster for devs and we can use a proper db later"... "let's just use nodejs for the POC, all the devs already know JS so it'll be great"... a few years later you end up with some gargantuan monstrosity that nobody wants to maintain, your "quick" language/db-du-jour is dead and unmaintained, you're EOL on four different software fronts and you now get to explain to the bosses why you need to rewrite the last four years of dev work.
Strilanc 7y ago
It's not limited to javascript: http://twistedoakstudios.com/blog/Post4428_unfathomable-bugs-6-pretend-precision http://twistedoakstudios.com/blog/Post4428_unfathomable-bugs...

TeMPOraL 7y ago

The original article's complaint wasn't (just) about 53 bits, but about using inefficient formats requiring constant serialization and deserialization for backend-to-backend communication.

kristiandupont 7y ago
Protobuf or Thrift would require serialization as well? I still wouldn't do that (or, in fact, I am currently not doing that) before I have an incentive, i.e. performance problems. Of course, my services are all Nodejs at the moment, so JSON feels more natural there than it does in other environments.
- mr_crankypants 7y ago
  One potentially nice thing about formats like Cap'n Proto is that they don't eagerly parse the message. You can just lazily grab the values you care about, which can be a big performance win if you typically only care about a few fields out of a larger message. That's impossible with text-based formats like XML and JSON, because everything is fractally variable-length.

bibabaloo 7y ago

It seems like its more "problems you deal with when you work at scale" like the author has at Google & FB. At smaller scales, I think the (machine) benefits of using something like protobuf don't nearly outweigh the human benefits of just using JSON.

jlouis 7y ago
They both exist for a reason. JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system. Up to a point, this also scales well. On the other hand, if you have many developers, and the problem space starts converging toward scale, then certainly the problem space is different. Encoding of data cannot follow the language with the weakest type zoo, and efficiency starts to matter. The hard part is to know when to make the switch, or when to anticipate growth in advance, such that you pick the right tool for the job. It isn't just a question of the machine. For complex messaging, the added value of a well-defined message typing, provided by protobuf, will help. It will also remove a lot of problems if you have multiple different languages in the stack, talking to each other.
- wil421 7y ago
  Once you hit the wall with JSON what format do you use? At a past job we were stuck using SOAP/XML middleware. We hit walls but I’m not sure if a different format would work. We ended up rate limiting lower priority traffic to get past our issues.
  - q3k 7y ago
    Protobuf/gRPC.
  - juliusmusseau 7y ago
    CSV. Just kidding, but I suspect it would be a lot faster.
- parliament32 7y ago
  >JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system. This is really, really bad thinking and will always cause you a timebomb that will 100% explode on you in the long run. "just a quick POC" will always end up being the actual product, and once you start down this path you end up with "let's just use mongo, it's schema-less, it'll be faster for devs and we can use a proper db later"... "let's just use nodejs for the POC, all the devs already know JS so it'll be great"... a few years later you end up with some gargantuan monstrosity that nobody wants to maintain, your "quick" language/db-du-jour is dead and unmaintained, you're EOL on four different software fronts and you now get to explain to the bosses why you need to rewrite the last four years of dev work.

Strilanc 7y ago

It's not limited to javascript: http://twistedoakstudios.com/blog/Post4428_unfathomable-bugs-6-pretend-precision http://twistedoakstudios.com/blog/Post4428_unfathomable-bugs...