8 ms·

Parsing gigabytes of JSON per second

139 points by signa11 5y ago

implying 5y ago
(2019)
loloquwowndueo 5y ago
The project described in the paper is https://simdjson.org/ https://simdjson.org/.
- pickledish 5y ago
  Thank you for the link! Much nicer to browse this than an arxiv paper
- westurner 5y ago
  Source: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson PyPI: https://pypi.org/project/pysimdjson/ https://pypi.org/project/pysimdjson/ There's a rust port: https://github.com/simd-lite/simd-json https://github.com/simd-lite/simd-json ... From ijson https://pypi.org/project/ijson/#id3 https://pypi.org/project/ijson/#id3 which supports streaming JSON: > Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends: [yajl2_c, yajl2_cffi, yajl2, yajl, python]
stkdump 5y ago
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of the 100s of binary variants of json-similar formats has the same capabilities and unite around it.
- EwanToo 5y ago
  I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
  - lifthrasiir 5y ago
    JSON doesn't allow any custom type, so it is not "flexible" per se. Therefore you only need a format that supports the JSON data model and pretty much nothing else; CBOR [1] for example almost surely fits the bill. [1] https://cbor.io/ https://cbor.io/
  - Waterluvian 5y ago
    That’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.
    - maccard 5y ago
      Datasets are not always provided by you; if your data source outputs json it doesn't matter why. Also just because your volume of data is measured in gigabytes, that doesn't mean it's a singular stream. As an example you could be handling tens of thousands of small requests.
  - the8472 5y ago
    Converting a text format to a compact in-memory data structure takes extra CPU cycles. (de)compression takes extra cycles. The point of using a binary format is to achieve the same result while avoiding that overhead. For some data formats compression also has the downside that it prevents seeking.
  - kevin_thibedeau 5y ago
    Serialized binary encodings don't require a schema. They can have all the expressability of JSON and more.
- 5y ago
otrahuevada 5y ago
Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used a hand-rolled analogue back then but JSONStream is actually pretty cool, check it out- What is the use case here? data lakes?
- jamal-kumar 5y ago
  I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds I bet this method works even faster though somehow if they felt compelled to throw research money at all at it
- Zababa 5y ago
  If you're at a very large scale, maybe the reduced server cost by parsing JSON this way can "finance" an engineer? If you're starting to reach the limits of JSON, maybe it could also help you avoid a transition for a while longer? At least those are the obvious cases for me. If it's a drop-in replacement for other JSON parsers, I think this could have a huge impact. There is a lot of value to gain by optimizing the fundamentals
- suction 5y ago
  I worked for small(ish) music supplier of Spotify, not a major label like Universal etc. at all - they were receiving a daily 3-4 Gb JSON file (uncompressed) of yesterday's activity. Things get big, fast, nowadays.
- estebarb 5y ago
  I worked on a database engine that uses JSON as communication protocol. Sure, most messages are small, but for a system processing JSONs 24/7 it quickly sums a lot. The main issues see had with our codec was memory allocation and the big number of "if" in the code. I tried to use simdjson, but there were concerns at that time about portability and long term support. It's nice to see that in retrospective we were wrong.
throwawaygal7 5y ago
This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither of those use protobuf... Most importantly, it's business critical to be able to stream this data to data lakes where it can be read by humans. None of the options are going to support protocol buffers, you're either going to have to write a parser (impossible for some) or transform to JSON before ingest (fairly expensive due to some poor choices in how to represent various fields). It was a sound technical decision to use protoc , and a terrible choice for the business. I think it would have been much better to use avro, still benefit from schemas but push the work of marshaling JSON back to clients...
- wodenokoto 5y ago
  I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.
  - MayeulC 5y ago
    IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.html https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...
    - morelisp 5y ago
      Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.
kabber 5y ago
Fast JSON parsers are all fine and good, but take such benchmarks with a grain of salt. JSON, like any format, is useless unless you do something meaningful with it - populate a database, verify the result, or transform it in some useful way for the task at hand. Many of these faster JSON DOM-style parsing libraries store key/value pairs in lists or arrays because it's the most efficient way to do so. But they aren't great at lookup speed. If the JSON parse library is event based, i.e. SAX-style, you have to store it somewhere, and this takes CPU as well.
Ginden 5y ago
I think there is some room for "object predictor" optimization. In real world cases, 99% of objects in array will have the same keys over and over. Smart parser can potentially exploit that.

139 points by signa11 5y ago

implying 5y ago

(2019)

loloquwowndueo 5y ago

The project described in the paper is https://simdjson.org/ https://simdjson.org/.

pickledish 5y ago
Thank you for the link! Much nicer to browse this than an arxiv paper
westurner 5y ago
Source: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson PyPI: https://pypi.org/project/pysimdjson/ https://pypi.org/project/pysimdjson/ There's a rust port: https://github.com/simd-lite/simd-json https://github.com/simd-lite/simd-json ... From ijson https://pypi.org/project/ijson/#id3 https://pypi.org/project/ijson/#id3 which supports streaming JSON: > Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends: [yajl2_c, yajl2_cffi, yajl2, yajl, python]

stkdump 5y ago

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of the 100s of binary variants of json-similar formats has the same capabilities and unite around it.

EwanToo 5y ago
I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
- lifthrasiir 5y ago
  JSON doesn't allow any custom type, so it is not "flexible" per se. Therefore you only need a format that supports the JSON data model and pretty much nothing else; CBOR [1] for example almost surely fits the bill. [1] https://cbor.io/ https://cbor.io/
- Waterluvian 5y ago
  That’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.
  - maccard 5y ago
    Datasets are not always provided by you; if your data source outputs json it doesn't matter why. Also just because your volume of data is measured in gigabytes, that doesn't mean it's a singular stream. As an example you could be handling tens of thousands of small requests.
- the8472 5y ago
  Converting a text format to a compact in-memory data structure takes extra CPU cycles. (de)compression takes extra cycles. The point of using a binary format is to achieve the same result while avoiding that overhead. For some data formats compression also has the downside that it prevents seeking.
- kevin_thibedeau 5y ago
  Serialized binary encodings don't require a schema. They can have all the expressability of JSON and more.
5y ago

otrahuevada 5y ago

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used a hand-rolled analogue back then but JSONStream is actually pretty cool, check it out- What is the use case here? data lakes?

jamal-kumar 5y ago
I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds I bet this method works even faster though somehow if they felt compelled to throw research money at all at it
Zababa 5y ago
If you're at a very large scale, maybe the reduced server cost by parsing JSON this way can "finance" an engineer? If you're starting to reach the limits of JSON, maybe it could also help you avoid a transition for a while longer? At least those are the obvious cases for me. If it's a drop-in replacement for other JSON parsers, I think this could have a huge impact. There is a lot of value to gain by optimizing the fundamentals
suction 5y ago
I worked for small(ish) music supplier of Spotify, not a major label like Universal etc. at all - they were receiving a daily 3-4 Gb JSON file (uncompressed) of yesterday's activity. Things get big, fast, nowadays.
estebarb 5y ago
I worked on a database engine that uses JSON as communication protocol. Sure, most messages are small, but for a system processing JSONs 24/7 it quickly sums a lot. The main issues see had with our codec was memory allocation and the big number of "if" in the code. I tried to use simdjson, but there were concerns at that time about portability and long term support. It's nice to see that in retrospective we were wrong.

throwawaygal7 5y ago

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither of those use protobuf... Most importantly, it's business critical to be able to stream this data to data lakes where it can be read by humans. None of the options are going to support protocol buffers, you're either going to have to write a parser (impossible for some) or transform to JSON before ingest (fairly expensive due to some poor choices in how to represent various fields). It was a sound technical decision to use protoc , and a terrible choice for the business. I think it would have been much better to use avro, still benefit from schemas but push the work of marshaling JSON back to clients...

wodenokoto 5y ago
I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.
- MayeulC 5y ago
  IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.html https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...
  - morelisp 5y ago
    Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.

kabber 5y ago

Fast JSON parsers are all fine and good, but take such benchmarks with a grain of salt. JSON, like any format, is useless unless you do something meaningful with it - populate a database, verify the result, or transform it in some useful way for the task at hand. Many of these faster JSON DOM-style parsing libraries store key/value pairs in lists or arrays because it's the most efficient way to do so. But they aren't great at lookup speed. If the JSON parse library is event based, i.e. SAX-style, you have to store it somewhere, and this takes CPU as well.

Ginden 5y ago

I think there is some room for "object predictor" optimization. In real world cases, 99% of objects in array will have the same keys over and over. Smart parser can potentially exploit that.