8 ms·

I can't help but wonder if it isn't simpler just to use gzipped JSON. I'd be interested to see a size comparison of the two. It seems like they're going to an a

by kayamon 11y ago

I can't help but wonder if it isn't simpler just to use gzipped JSON. I'd be interested to see a size comparison of the two. It seems like they're going to an awful lot of work to hand-roll a suboptimal text compression scheme here.

easytiger 11y ago
It's the whole FAST/FIX clusterfuck all over again. Use a binary format and stop cocking around. You can't serialise/deserialise it faster and JSON is a string based mess. No low latency path can absorb it. Even influxdb have removed JSON support because it was their slowest part of their critical path that could not be optimised. Complete lack of mechanical sympathy + laziness is why JSON is popular outside of the web world
- moomin 11y ago
  Well, I think you need to look at the intended use cases for transit. On the browser, transit serialisation is faster than binary precisely because it uses the fast paths. For server to server communication, it can use msgpack. There's also a) it's fully schemaless and b) you can add new primitives. Which you may or may not need. But if you're looking for something that does a) and b) while being fast for browser communication, transit's currently your best option.
- unimpressive 11y ago
  >Complete lack of mechanical sympathy + laziness is why JSON is popular outside of the web world I generally prefer JSON as a serialization format for a variety of reasons beyond what you cited: 1. Simplicity/plain text. Even a total newb could read a JSON file and probably puzzle out how the data works. This is important if I'm providing an export function or otherwise expect my users might want to shop somewhere else to deal with this data. 2. Forward compatibility/future proofing. If somebody finds my JSON file in twenty years with no documentation, (1) ensures that it's not going to be lost to them and they'll be able to extract the data with minimal difficulty. 3. Cross platform. It might not be the best format, but it's going to be supported in pretty much any language I would ever want to add support for the data to in a new system.
- waxjar 11y ago
  I don't think I would enjoy debugging a service that talks in a binary format.
  - djKianoosh 11y ago
    debugging the data, no. debugging the code, depending on how the representation looks like, probably wouldn't be too bad...
  - to3m 11y ago
    I've never done it with JS. It's not really a problem in C, though. If there's a lot of data, you're no better off with text; if there's not much, reading binary data isn't too bad. You've usually got other problems on top of decoding the actual data - like, why is this data coming in the first place? Why isn't the code accepting it properly? Why isn't the other end listening? The data encoding is just the tip of the iceberg. Either way, the advantages of binary often outweigh the difficulty of having a human read the data. As the saying goes, code is executed far more often than it's read. (Most binary formats have some fairly obvious possible text representations, at least for key fields (they're just encoding ints, floats, strings, bitfields, etc.). When you're having difficulty, or you're simply curious, you can print them out. If the format is any good, malformed messages are easy to check for in code - this code is not harder to debug than any other. If anything it's actually easier than with some kind of text format.)
    - waxjar 11y ago
      Sure, but it's not very convenient. I regularly MITM a connection between a mobile app and a HTTP server, when something isn't going quite right. A look at the JSON they exchange exposes the problem in under a minute more often than not. If I want to test something out quickly, I simply modify the incoming / outgoing JSON by hand. It's rare to get the syntax wrong and involves no context switches. I can then go back to the code, find the relevant section and make the changes I need to make. I find this a very convenient way of debugging and I don't think it would be as nice to do this with a binary format.
  - falcolas 11y ago
- zapov 11y ago
  Easy there tiger ;) What does it even mean that something is string based? Here: https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki a JVM bench where JSON is pretty competitive with binary formats [and results would be event better if project owners actually merged PRs ;( ]
dantiberian 11y ago
From the announcement blog post: Transit is self describing using tags, and encourages transmitting information using maps, which have named keys/fields which will repeat often in data. These overheads, typical of self-describing formats, are mitigated in Transit by an integrated cache code system, which replaces repetitive data with small codes. This yields not only a reduction in size, but also an increase in performance and memory utilization. Contrast this with gzipping, which, while it may reduce size on the wire, takes time and doesn't reduce the amount of text to be parsed or the number of objects generated in memory after parsing. - http://blog.cognitect.com/blog/2014/7/22/transit http://blog.cognitect.com/blog/2014/7/22/transit

by kayamon 11y ago

easytiger 11y ago

It's the whole FAST/FIX clusterfuck all over again. Use a binary format and stop cocking around. You can't serialise/deserialise it faster and JSON is a string based mess. No low latency path can absorb it. Even influxdb have removed JSON support because it was their slowest part of their critical path that could not be optimised. Complete lack of mechanical sympathy + laziness is why JSON is popular outside of the web world

moomin 11y ago
Well, I think you need to look at the intended use cases for transit. On the browser, transit serialisation is faster than binary precisely because it uses the fast paths. For server to server communication, it can use msgpack. There's also a) it's fully schemaless and b) you can add new primitives. Which you may or may not need. But if you're looking for something that does a) and b) while being fast for browser communication, transit's currently your best option.
unimpressive 11y ago
>Complete lack of mechanical sympathy + laziness is why JSON is popular outside of the web world I generally prefer JSON as a serialization format for a variety of reasons beyond what you cited: 1. Simplicity/plain text. Even a total newb could read a JSON file and probably puzzle out how the data works. This is important if I'm providing an export function or otherwise expect my users might want to shop somewhere else to deal with this data. 2. Forward compatibility/future proofing. If somebody finds my JSON file in twenty years with no documentation, (1) ensures that it's not going to be lost to them and they'll be able to extract the data with minimal difficulty. 3. Cross platform. It might not be the best format, but it's going to be supported in pretty much any language I would ever want to add support for the data to in a new system.
waxjar 11y ago
I don't think I would enjoy debugging a service that talks in a binary format.
- djKianoosh 11y ago
  debugging the data, no. debugging the code, depending on how the representation looks like, probably wouldn't be too bad...
- to3m 11y ago
  I've never done it with JS. It's not really a problem in C, though. If there's a lot of data, you're no better off with text; if there's not much, reading binary data isn't too bad. You've usually got other problems on top of decoding the actual data - like, why is this data coming in the first place? Why isn't the code accepting it properly? Why isn't the other end listening? The data encoding is just the tip of the iceberg. Either way, the advantages of binary often outweigh the difficulty of having a human read the data. As the saying goes, code is executed far more often than it's read. (Most binary formats have some fairly obvious possible text representations, at least for key fields (they're just encoding ints, floats, strings, bitfields, etc.). When you're having difficulty, or you're simply curious, you can print them out. If the format is any good, malformed messages are easy to check for in code - this code is not harder to debug than any other. If anything it's actually easier than with some kind of text format.)
  - waxjar 11y ago
    Sure, but it's not very convenient. I regularly MITM a connection between a mobile app and a HTTP server, when something isn't going quite right. A look at the JSON they exchange exposes the problem in under a minute more often than not. If I want to test something out quickly, I simply modify the incoming / outgoing JSON by hand. It's rare to get the syntax wrong and involves no context switches. I can then go back to the code, find the relevant section and make the changes I need to make. I find this a very convenient way of debugging and I don't think it would be as nice to do this with a binary format.
- falcolas 11y ago
zapov 11y ago
Easy there tiger ;) What does it even mean that something is string based? Here: https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki a JVM bench where JSON is pretty competitive with binary formats [and results would be event better if project owners actually merged PRs ;( ]

dantiberian 11y ago

From the announcement blog post: Transit is self describing using tags, and encourages transmitting information using maps, which have named keys/fields which will repeat often in data. These overheads, typical of self-describing formats, are mitigated in Transit by an integrated cache code system, which replaces repetitive data with small codes. This yields not only a reduction in size, but also an increase in performance and memory utilization. Contrast this with gzipping, which, while it may reduce size on the wire, takes time and doesn't reduce the amount of text to be parsed or the number of objects generated in memory after parsing. - http://blog.cognitect.com/blog/2014/7/22/transit http://blog.cognitect.com/blog/2014/7/22/transit