5 ms·

I have to ask - why? I understand if you're dealing with an external resource that you can't change, but I'd still like to ask them the same question. JSON's st

I have to ask - why? I understand if you're dealing with an external resource that you can't change, but I'd still like to ask them the same question. JSON's strength, it seems to me, is that it's great for human consumption. It's a very readable data structure when serialized! But who is ever going to read 90GB of JSON?

nly 7y ago
Not OP, but I deal with a variety of proprietary binary/encoded files in the 10s of MB to the 10s of GB range at work, and we have started decoding to JSON as the first step in our normalization, indexing and analytics pipeline (To be honest we essentially just throw everything in to MongoDB). Daily we're in the terabyte range. It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.
- e12e 7y ago
  A question - do know of any good resources for jq? I haven't quite been "angry enough" to use jq much beyond using it for formatting json - but that's also partly because I can't seem to quite grasp the syntax. It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool? I feel like I'm missing a piece - and missing out...
  - chucky_z 7y ago
    The manual is pretty dang good, tbh. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/
    - e12e 7y ago
      I agree that it is clear, but I still struggle to figure out (by myself) how to do (in an awk sense) simple things like: Filter out empty or null values: https://stackoverflow.com/questions/39500608/remove-all-null-values https://stackoverflow.com/questions/39500608/remove-all-null... I seem to either end up with unwieldy, long/verbose solutions - or shorter ones that does not work :)
      - chucky_z 7y ago
        Oh wow, that's terrible! I tend to get 90% of the way with jq and then awk/sed/grep my way to the desired result (jq -r).
orf 7y ago
It's not necessarily that it's great for human consumption, more that it's super interchangeable and ridiculously easy to get up and running, and for the most part 90Gb of space is so cheap that it's not worth the overhead of using a binary format. I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects. Plus the human element helps.
- deleted 7y ago
  [deleted]
- weberc2 7y ago
  Especially considering that JSON compresses well.
- nicoburns 7y ago
  > I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects. I wish that were true. Alas it really was a 90gb object. Actually, the first thing I did was split it by top-level key before processing further.
- deleted 7y ago
  [deleted]
nicoburns 7y ago
Not out of choice! It was a backup of a firebase database (also not my choice). I totally agree that JSON is not a good format for files this large, but it was the only format the data was available in so what can you do.

nly 7y ago

Not OP, but I deal with a variety of proprietary binary/encoded files in the 10s of MB to the 10s of GB range at work, and we have started decoding to JSON as the first step in our normalization, indexing and analytics pipeline (To be honest we essentially just throw everything in to MongoDB). Daily we're in the terabyte range. It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.

e12e 7y ago
A question - do know of any good resources for jq? I haven't quite been "angry enough" to use jq much beyond using it for formatting json - but that's also partly because I can't seem to quite grasp the syntax. It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool? I feel like I'm missing a piece - and missing out...
- chucky_z 7y ago
  The manual is pretty dang good, tbh. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/
  - e12e 7y ago
    I agree that it is clear, but I still struggle to figure out (by myself) how to do (in an awk sense) simple things like: Filter out empty or null values: https://stackoverflow.com/questions/39500608/remove-all-null-values https://stackoverflow.com/questions/39500608/remove-all-null... I seem to either end up with unwieldy, long/verbose solutions - or shorter ones that does not work :)
    - chucky_z 7y ago
      Oh wow, that's terrible! I tend to get 90% of the way with jq and then awk/sed/grep my way to the desired result (jq -r).

orf 7y ago

It's not necessarily that it's great for human consumption, more that it's super interchangeable and ridiculously easy to get up and running, and for the most part 90Gb of space is so cheap that it's not worth the overhead of using a binary format. I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects. Plus the human element helps.

deleted 7y ago
[deleted]
weberc2 7y ago
Especially considering that JSON compresses well.
nicoburns 7y ago
> I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects. I wish that were true. Alas it really was a 90gb object. Actually, the first thing I did was split it by top-level key before processing further.
deleted 7y ago
[deleted]

nicoburns 7y ago

Not out of choice! It was a backup of a firebase database (also not my choice). I totally agree that JSON is not a good format for files this large, but it was the only format the data was available in so what can you do.