4 ms·

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-ha

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used a hand-rolled analogue back then but JSONStream is actually pretty cool, check it out- What is the use case here? data lakes?

jamal-kumar 5y ago
I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds I bet this method works even faster though somehow if they felt compelled to throw research money at all at it
Zababa 5y ago
If you're at a very large scale, maybe the reduced server cost by parsing JSON this way can "finance" an engineer? If you're starting to reach the limits of JSON, maybe it could also help you avoid a transition for a while longer? At least those are the obvious cases for me. If it's a drop-in replacement for other JSON parsers, I think this could have a huge impact. There is a lot of value to gain by optimizing the fundamentals
suction 5y ago
I worked for small(ish) music supplier of Spotify, not a major label like Universal etc. at all - they were receiving a daily 3-4 Gb JSON file (uncompressed) of yesterday's activity. Things get big, fast, nowadays.
estebarb 5y ago
I worked on a database engine that uses JSON as communication protocol. Sure, most messages are small, but for a system processing JSONs 24/7 it quickly sums a lot. The main issues see had with our codec was memory allocation and the big number of "if" in the code. I tried to use simdjson, but there were concerns at that time about portability and long term support. It's nice to see that in retrospective we were wrong.
llamajams 5y ago
I'm working on an embedded vision/navigation system. The core app itself generates about 50mb a minute of metadata alone in the simplest scenario. It's supposed to be long running system. There is one case that generates just over a TB. Now you may say we should optimize this or that but I'm not an architect and I have no say, and it's just what I gotta deal with:(
IshKebab 5y ago
Yeah if you have a situation where JSON gets into the megabytes you should definitely stop using JSON! I recommend SQLite for large datasets that you need to put in a file. I guess if you don't have a choice then it might become a bottleneck. Parsing 300 MB of data can be very slow. But even with small JSON messages it can become a bottleneck, e.g. check out the Xi editor's issues. They thought it wouldn't be a bottleneck and then found that sound languages don't have insanely optimised JSON parsers like this. Boom. Bottleneck.
anonymoushn 5y ago
We receive market data formatted as JSON and use it to make trading decisions. So we would like to make those decisions quickly.
sethev 5y ago
The use-case for a more efficient JSON parser is exactly the same as the use-case for a less efficient one. They're doing the same thing. Daniel Lemire does a ton of performance research like this. He's an unusual case of a professor who publishes fully working source code. There's a good chance that your database of choice uses some of his stuff or was influenced by his research.

jamal-kumar 5y ago

I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds I bet this method works even faster though somehow if they felt compelled to throw research money at all at it

Zababa 5y ago

If you're at a very large scale, maybe the reduced server cost by parsing JSON this way can "finance" an engineer? If you're starting to reach the limits of JSON, maybe it could also help you avoid a transition for a while longer? At least those are the obvious cases for me. If it's a drop-in replacement for other JSON parsers, I think this could have a huge impact. There is a lot of value to gain by optimizing the fundamentals

suction 5y ago

I worked for small(ish) music supplier of Spotify, not a major label like Universal etc. at all - they were receiving a daily 3-4 Gb JSON file (uncompressed) of yesterday's activity. Things get big, fast, nowadays.

estebarb 5y ago

I worked on a database engine that uses JSON as communication protocol. Sure, most messages are small, but for a system processing JSONs 24/7 it quickly sums a lot. The main issues see had with our codec was memory allocation and the big number of "if" in the code. I tried to use simdjson, but there were concerns at that time about portability and long term support. It's nice to see that in retrospective we were wrong.

llamajams 5y ago

I'm working on an embedded vision/navigation system. The core app itself generates about 50mb a minute of metadata alone in the simplest scenario. It's supposed to be long running system. There is one case that generates just over a TB. Now you may say we should optimize this or that but I'm not an architect and I have no say, and it's just what I gotta deal with:(

IshKebab 5y ago

Yeah if you have a situation where JSON gets into the megabytes you should definitely stop using JSON! I recommend SQLite for large datasets that you need to put in a file. I guess if you don't have a choice then it might become a bottleneck. Parsing 300 MB of data can be very slow. But even with small JSON messages it can become a bottleneck, e.g. check out the Xi editor's issues. They thought it wouldn't be a bottleneck and then found that sound languages don't have insanely optimised JSON parsers like this. Boom. Bottleneck.

anonymoushn 5y ago

We receive market data formatted as JSON and use it to make trading decisions. So we would like to make those decisions quickly.

sethev 5y ago

The use-case for a more efficient JSON parser is exactly the same as the use-case for a less efficient one. They're doing the same thing. Daniel Lemire does a ton of performance research like this. He's an unusual case of a professor who publishes fully working source code. There's a good chance that your database of choice uses some of his stuff or was influenced by his research.