3 ms·
Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.
by craigching 3y ago
Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.
- ahoka 3y agoNot necessarily, for example Newtonsoft is fine with multiple hundreds of megabyes if you use it correctly. But of course depends on how large we are talking about.
- beached_whale 3y agoThere are several that are into the GB/s of performance with various interfaces. Most are just trash for large documents and sit in the allocators far too long, but that's not required either
- zlg_codes 3y agoWhat on Earth are you storing in JSON that this sort of performance issue becomes an issue? How big is 'large' here? I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware. I'm having difficulty picturing a JSON dataset big enough to slow down modern hardware. Maybe Gentoo's portage tree if it were JSON encoded?
- EMM_386 3y ago> What on Earth are you storing in JSON that this sort of performance issue becomes an issue? I've been in the industry for a while. I've probably left more than one client site muttering "I've seen some things ...". If it can be done, it will be done. And often in a way that shouldn't have even been considered at all. Many times, "it works" is all that is needed. Not exactly the pinnacle of software design. But hey, it does indeed "work"!
- Edwinr95 3y agoI've seen people dump and share entire databases in JSON format at my job....
- nly 3y agoI've seen tens of millions of market data events from a single day of trading encoded in JSON and used in various post-trade pipelines.
- zlg_codes 3y agoAh, that's a dataset with a size certainly intimidating, and in an environment where performance means money. Thanks for pointing that out!
- mxmlnkn 3y agoChrome trace format files also use JSON and can also become large and are a pain to work with.
- craigching 3y agoIn my case, sentry events that represent crash logs for Adobe Digital Video applications. I’m trying to remember off the top of my head, but I think it was in the gigabytes for a single event.
- pbrumm 3y agoInsurance price transparency can have 16gb of compressed JSON that represents a single object. Here is the anthem page. The toc link is 16gb https://www.anthem.com/machine-readable-file/search/ https://www.anthem.com/machine-readable-file/search/ They are complying with the mandate. But not optimizing for the parsers