4 ms·
"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications
by mgaunard 3y ago
"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications
- isuckatcoding 3y agoYes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy. Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.
- dh2022 3y agoIf performance is important why would you keep large datasets in JSON format?
- querulous 3y agosometimes it's not your data
- isuckatcoding 3y agoUsually because the downstream service or store needs it
- Maxion 3y agoBecause you work at or for some bureaucratic MegaCorp, that does weird things with no real logic behind it other than clueless Dilbert managers making decisions based on LinkedIn blogs. Alternatively desperate IT consultants trying to get something to work with too low of a budget and/or no access to do things the right way. Be glad you have JSON to parse, and not EDI, some custom deliminated data format (with no or old documentation) - or shudders you work in the airline industry with SABRE.
- capableweb 3y agohttps://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- mannyv 3y agoIf you're building a library you either need to explicitly call out your limits or do streaming. I've pumped gigs of jaon data, so a streaming parser is appreciated. Plus streaming shows the author is better at engineering and is aware of the various use cases. Memory is not cheap or free except in theory.
- jjeaff 3y agoI guess it's all relative. Memory is significantly cheaper if you get it anywhere but on loan from a cloud provider.
- mannyv 3y agoRAM is always expensive no matter where you get it from. Would you rather do two hours of work or force thousands of people to buy more RAM because your library is a memory hog? And on embedded systems RAM is a premium. More RAM = most cost.
- crabbone 3y agoHere people confidently keep repeating "streaming JSON". What do you mean by that? I'm genuinely curios. Do you mean XML SAX-like interface? If so, how do you deal with repeated keys in "hash tables"? Do you first translate JSON into intermediate objects (i.e. arrays, hash-tables) and then transform them into application-specific structures, or do you try to skip the intermediate step? I mean, streaming tokens is kind of worthless on its own. If you are going for SAX-like interface, you want to be able to go all the way with streaming (i.e. in no layer of the code that reads JSON you don't "accumulate" data (esp. not possibly indefinitely) until it can be sent to the layer above that).
- chii 3y ago> If so, how do you deal with repeated keys in "hash tables"? depending on the parser, behaviour might differ. But looking at https://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object https://stackoverflow.com/questions/21832701/does-json-synta... , it seems like the "best" option is to have 'last key wins' as the resolution. This works fine under a SAX like interface in a streaming JSON parser - your 'event handler' code will execute for a given key, and a 2nd time for the duplicate.
- e12e 3y agoIf you can live with "fits on disk" mmap() is a viable option? Unless you truly need streaming (early handling of early data, like a stream of transactions/operations from a single JSON file?)
- mannyv 3y agoIn general, JSON comes over the network, so MMAP won't really work unless you save to a file. But then you'll run out of disk space. I mean, you have a 1k, 2k, 4k buffer. Why use more, because it's too much work?
- ahoka 3y agoMost applications read JSONs from networks, where you have a stream. Buffering and fiddling with the whole request in memory increases latency by a lot, even if your JSON is smallish.
- kiitos 3y agoIs the HTTP request body part of the input?