7 ms·

Some random comments: * A few GBs of data isn't really that much. Even /considering/ the use of cloud services just for this sounds crazy to me... but I'm sure

by jmmv 4y ago

Some random comments: * A few GBs of data isn't really that much. Even /considering/ the use of cloud services just for this sounds crazy to me... but I'm sure there are people out there that believe it's the only way to do this (not the author, fortunately). * "You might find out that the data doesn’t fit into RAM (which it well might, JSON is a human-readable format after all)" -- if I'm reading this right, the author is saying that the parsed data takes _more_ space than the JSON version? JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. * "When you’re ~trial-and-error~iteratively building jq commands as I do, you’ll quickly grow tired of having to wait about a minute for your command to succeed" -- well, change your workflow then. When tackling new queries, it's usually a good idea to reduce the data set. Operate on a few records until you have the right query so that you can iterate as fast as possible. Only once you are confident with the query, run it on the full data. * Importing the data into a SQLite database may be better overall for exploration. Again, JSON is slow to operate on because it's text. Pay the cost of parsing only once. * Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language.

bastawhiz 4y ago
> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. If you're parsing to structs, yes. Otherwise, no. Each object key is going to be a short string, which is going to have some amount of overhead. You're probably storing the objects as hash tables, which will necessarily be larger than the two bytes needed to represent them as text (and probably far more than you expect, so they have enough free space for there to be sufficiently few hash collisions). JSON numbers are also 64-bit floats, which will almost universally take up more bytes per number than their serialized format for most JSON data.
- vlovich123 4y ago
  I think even structs have this problem because typically you heap allocate all the structs/arrays. You could try to arena allocate contiguous objects in place, but that sounds hard enough that I doubt that anyone bothers. Using a SAX parser is almost certainly the tool you want to use.
- matja 4y ago
  > JSON numbers are also 64-bit floats In common implementations they are, but RFC 8259 and ECMA-404 do not specify the range, precision, or underlying implementation for the storage of numbers in JSON. A implementation that guarantees interoperability between all implementations of JSON would use an arbitrary-sized number format, but they seldom do. No idea what ISO/IEC 21778:2017 says because it's not free.
  - bastawhiz 4y ago
    I mean, I suppose you could use a numeric format that takes more than 64 bits, but you certainly couldn't use one that uses fewer. You have no way to know whether a number should be an int or a float. And since JavaScript (the JS in JSON) encodes with 64 bits, you'd be losing precision if you choose to use any less. The point is that an integer written as one byte in the JSON will take up more space (at least seven bytes more) when decoded, all the way up to numbers that take seven characters.
    - gpderetta 4y ago
      (subsets of )ASN.1, LEB128, and of course UTF-8 are all examples of encoding binary integers with variable length.
      - bastawhiz 4y ago
        Sure, and JSON is as well. 0 takes three fewer bytes to encode than 1234. But if I parse JSON and my numbers come back as a type backed by UTF-8, I'm going to be left confused and angry. Nobody expects their JSON parser to give them back numeric values that their CPU can't natively perform operations on.
        gpderetta 4y ago
        After you have spent some time confused and angry you can then read the documentation of whatever library you are using and verify whether it overloads numerical operations for its special number type or it has a function to convert them to whatever numerical type you need. In any case you wouldn't of course use UTF-8 to represent JSON numbers as it can only encode 21 bit intgers.
taeric 4y ago
It still surprises me how many have the intuition that loading the data will take more space than the file. Even more annoying when it is. (Compressed or binary formats not withstanding.)
- vlovich123 4y ago
  Isn’t that almost always true unless the format is specifically designed to take the same amount on disk and in RAM? If that’s the case I think typically it means the file on disk is larger than it needs to be. That’s because you have to convert the file to some kind of in-memory representation that’s easier to work with to manipulate it and what not which requires structure that isn’t necessary at rest. Can you give an example where the file on disk is going to be larger than what it is in memory? Provided you’re not just reading it and working with it as an opaque binary blob.
  - hnlmorg 4y ago
    XML would probably fall into that category. But I do agree with your point more generally speaking.
  - taeric 4y ago
    Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory. So, my examples are largely csv files. JSON and xml are the same, in many ways. The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory. In large, this is why parquet files are a ridiculous win in space.
    - coldtea 4y ago
      My favorite example is the meg's of commas and quotes that just aren't needed in memory. Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...
vlovich123 4y ago
> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. ~~Yeah but parsing it can require ~2x the RAM available and push you into swap / make it not possible.~~ > Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language. What is the state of SAX JSON parsing? I used yajl a long time ago but not sure if that’s still the state of the art (and it’s C interface was not the easiest to work with). EDIT: Actually, I think the reason is that you typically will have pointers (8 bytes) in place of 2 byte demarcations in the text version (eg “”, {}, [] become pointers). It’s very hard to avoid that (maybe impossible? Not sure) and no surprise that Python has a problem with this.
- saidinesh5 4y ago
  There's libraries like ijson. At my last job, i wrote a wrapper around this C++ library called rapidjson to parse streaming json. It worked quite well. Here's the write up about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ And here's the code: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet The API isn't the best. I'd have preferred an iterator based solution as opposed to this callback based one. But we worked with what rapidjson gave us for the proof of concept. The reason for this specific implementation was we wanted to build an index to query the server directly about it's huge json files (compressed size of 20+GB per file) using http range queries.
deleted 4y ago
[deleted]
the_duke 4y ago
> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. That depends a lot on the language and the json library. Lets take `{"ab": 22}` as an example. That's 10 bytes. In a language like Rust and using the serde library, this could be deserialized directly into a struct with one integer, let's pick a u32. So that would only be four bytes. But if it was deserialized to serdes dynamic Value type, this would be : a HashMap<String, u32>, which has a constant size of 48 bytes, plus an allocation of I don't know how much (first allocation will cover more than one entry), plus 16 bytes overhead for the string, plus 2 bytes for the actual string contents, plus the 4 bytes for the u32. So that's already over ~90 bytes, a lot more than the JSON. Dynamic languages like Python also have a lot of overhead for all the objects. Keys can of course be interned, but not that many default JSON parser libraries do that afaik.
- DonHopkins 4y ago
  The key names of all the identically shaped objects are repeated and can all be interned into sharing the same strings, saving a huge amount of memory. That's why csv files can be much more compact that the equivalent json files: they only list the keys once in the first row.
  - mindslight 4y ago
    "ab" is 4 bytes, while a 64 bit pointer to an interned string is 8 bytes. It would seem that the savings of CSV would be better analogized to static typing - the type is defined once, then each record only contains data. I had the same intuition as the original comment. But no, the relative sizes of data formats aren't that straightforward. One could intern symbols to say 16 bit values, or one could infer structural types and compress the data that way. But those are both creating additional assumptions and processing that likely aren't done by commonly available tools.
    - DonHopkins 4y ago
      You still have to pay for a 64 bit pointer to an un-interned string though. It's the fact that the keys are repeated for each record that overwhelms the size of the keys or pointers themselves, when you have millions of records. If you are using fixed structures, then the keys are implicit and don't need to be repeated or even represented as pointers, just known fixed offsets into the structures. I wrote this about "Representing and Editing JSON with Spreadsheets": https://news.ycombinator.com/item?id=21109798 https://news.ycombinator.com/item?id=21109798 https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... SpreadOn: Apply Directly to the Forehead. https://www.youtube.com/watch?v=f_SwD7RveNE https://www.youtube.com/watch?v=f_SwD7RveNE
saidinesh5 4y ago
> * "You might find out that the data doesn’t fit into RAM (which it well might, JSON is a human-readable format after all)" -- if I'm reading this right, the author is saying that the parsed data takes _more_ space than the JSON version? JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Not to mention, even when using bad data structures (eg. hashmap of hashmaps..), One can just add a large enough swapfile and brute force their way through it no?
- samus 4y ago
  This will fail if the algorithm is not O(n). Even if only a bounded number of passes is required, there will be trouble.
zigzag312 4y ago
> if I'm reading this right, the author is saying that the parsed data takes _more_ space than the JSON version? JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Utf8 json strings will get converted to utf16 strings in some languages, doubling the size of strings in memory compared to the size on disk.
ot 4y ago
> Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language. Several years ago I wrote a paper [1] on representing the parse tree of a JSON document in a tiny fraction of the JSON size itself, using succinct data structures. The representation could be built with a single pass of the JSON, and basically constant additional memory. The idea was to pre-process the JSON and then save the parse tree, so it could be kept in memory over several passes of the JSON data (which may not fit in memory), avoiding to re-do the parsing work on each pass. I don't think I've seen this idea used anywhere, but I still wonder if it could have applications :) [1] http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pdf http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pd...
- aardvarkr 4y ago
  Have you implemented this idea in code? Seems interesting and worth playing around with
  - ot 4y ago
    Yes, the code is linked in the paper: https://github.com/ot/semi_index https://github.com/ot/semi_index
coldtea 4y ago
>JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Wanna bet? >>> import sys >>> import json >>> data_as_json_string = '{"a": 5000, "b": 1000}' >>> len(data_as_json_string) 22 >>> data_as_native_structure = json.loads(data_as_json_string) >>> sys.getsizeof(data_as_native_structure) 232 That's not even the whole story, as what's 232 bytes is not the contents of the dict, but just the Python object with the dict metadata. So the total for the struct inside is much bigger than 232 bytes. A single int wrapped as a Python object can be quite a lot by itself: >>> sys.getsizeof(1) 28 A binary int64 would be 8 bytes for comparison.
- berkle4455 4y ago
  Maybe 2023 is the year people stop using python.
  - IshKebab 4y ago
    We can dream but I seriously doubt it. Not only are there very few nice alternatives (Dart is great but unpopular; Typescript is great but has a terrible standard library, etc.) but very few of them have a decent REPL which I think is probably Python's most underrated feature. I mean... it's REPL is extremely basic, but anything you can do in Python you can do in the REPL. Very useful especially for beginners.
- killingtime74 4y ago
  Python is my favorite language but you literally chose one of the least efficient. This would probably be different in Java or Rust.
  - bufferoverflow 4y ago
    I thought Java stores strings as 16-bit character arrays internally. Which means it's very inefficient when it comes to memory consumption.
    - 4y ago
walrus01 4y ago
> A few GBs of data isn't really that much. The entire FCC radio license database (the ULS) is about 14GB in text CSV format and can be imported into a sqlite or sql db and easily queried in RAM on a local workstation...
nerdponx 4y ago
If the JSON data has a regular structure, you probably want a database and/or a "data frame" library and Parquet as the storage file format. SQLite, DuckDB, Polars, plenty of options nowadays that are usable from several different programming languages.
ZephyrBlu 4y ago
Have you done much data analysis? In my experience adding more steps to your pipeline (E.g. database, deserializing, etc) is a pain when you are figuring things out because nothing has solidified yet, so you're literally just adding overhead that requires even more work to remove/alter later on. If you're not careful you end up with something unmaintainable extremely quickly. Only analyzing a subset of your data is usually not a magic bullet either. Unless your data is extremely well cleaned and standardized you're probably going to run into edge cases on the full dataset that were not in your subset. Being able to run your full pipeline on the entire dataset in a short period of time is very useful for testing on the full dataset and seeing realistic analysis results. If you're doing any sort of aggregate analysis it becomes even more important, if not required. I now believe a relatively fast clean run is one of the most important things for performing data analysis. It increases your velocity tremendously.
fdgddggfddfg 4y ago
so on this note, unless you have massive json lists... having a json object per line (jsonl) can really help this process... I use jq when necessary .. but I usually use `cat file.json | grep foo | grep bar | jq -r .thing` since it's a string, use string comparison tools rather than jq :)

by jmmv 4y ago

bastawhiz 4y ago

> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. If you're parsing to structs, yes. Otherwise, no. Each object key is going to be a short string, which is going to have some amount of overhead. You're probably storing the objects as hash tables, which will necessarily be larger than the two bytes needed to represent them as text (and probably far more than you expect, so they have enough free space for there to be sufficiently few hash collisions). JSON numbers are also 64-bit floats, which will almost universally take up more bytes per number than their serialized format for most JSON data.

vlovich123 4y ago
I think even structs have this problem because typically you heap allocate all the structs/arrays. You could try to arena allocate contiguous objects in place, but that sounds hard enough that I doubt that anyone bothers. Using a SAX parser is almost certainly the tool you want to use.
matja 4y ago
> JSON numbers are also 64-bit floats In common implementations they are, but RFC 8259 and ECMA-404 do not specify the range, precision, or underlying implementation for the storage of numbers in JSON. A implementation that guarantees interoperability between all implementations of JSON would use an arbitrary-sized number format, but they seldom do. No idea what ISO/IEC 21778:2017 says because it's not free.
- bastawhiz 4y ago
  I mean, I suppose you could use a numeric format that takes more than 64 bits, but you certainly couldn't use one that uses fewer. You have no way to know whether a number should be an int or a float. And since JavaScript (the JS in JSON) encodes with 64 bits, you'd be losing precision if you choose to use any less. The point is that an integer written as one byte in the JSON will take up more space (at least seven bytes more) when decoded, all the way up to numbers that take seven characters.
  - gpderetta 4y ago
    (subsets of )ASN.1, LEB128, and of course UTF-8 are all examples of encoding binary integers with variable length.
    - bastawhiz 4y ago
      Sure, and JSON is as well. 0 takes three fewer bytes to encode than 1234. But if I parse JSON and my numbers come back as a type backed by UTF-8, I'm going to be left confused and angry. Nobody expects their JSON parser to give them back numeric values that their CPU can't natively perform operations on.
      - gpderetta 4y ago
        After you have spent some time confused and angry you can then read the documentation of whatever library you are using and verify whether it overloads numerical operations for its special number type or it has a function to convert them to whatever numerical type you need. In any case you wouldn't of course use UTF-8 to represent JSON numbers as it can only encode 21 bit intgers.

taeric 4y ago

It still surprises me how many have the intuition that loading the data will take more space than the file. Even more annoying when it is. (Compressed or binary formats not withstanding.)

vlovich123 4y ago
Isn’t that almost always true unless the format is specifically designed to take the same amount on disk and in RAM? If that’s the case I think typically it means the file on disk is larger than it needs to be. That’s because you have to convert the file to some kind of in-memory representation that’s easier to work with to manipulate it and what not which requires structure that isn’t necessary at rest. Can you give an example where the file on disk is going to be larger than what it is in memory? Provided you’re not just reading it and working with it as an opaque binary blob.
- hnlmorg 4y ago
  XML would probably fall into that category. But I do agree with your point more generally speaking.
- taeric 4y ago
  Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory. So, my examples are largely csv files. JSON and xml are the same, in many ways. The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory. In large, this is why parquet files are a ridiculous win in space.
  - coldtea 4y ago
    My favorite example is the meg's of commas and quotes that just aren't needed in memory. Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...

vlovich123 4y ago

> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. ~~Yeah but parsing it can require ~2x the RAM available and push you into swap / make it not possible.~~ > Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language. What is the state of SAX JSON parsing? I used yajl a long time ago but not sure if that’s still the state of the art (and it’s C interface was not the easiest to work with). EDIT: Actually, I think the reason is that you typically will have pointers (8 bytes) in place of 2 byte demarcations in the text version (eg “”, {}, [] become pointers). It’s very hard to avoid that (maybe impossible? Not sure) and no surprise that Python has a problem with this.

saidinesh5 4y ago
There's libraries like ijson. At my last job, i wrote a wrapper around this C++ library called rapidjson to parse streaming json. It worked quite well. Here's the write up about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ And here's the code: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet The API isn't the best. I'd have preferred an iterator based solution as opposed to this callback based one. But we worked with what rapidjson gave us for the proof of concept. The reason for this specific implementation was we wanted to build an index to query the server directly about it's huge json files (compressed size of 20+GB per file) using http range queries.

deleted 4y ago

[deleted]

the_duke 4y ago

> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. That depends a lot on the language and the json library. Lets take `{"ab": 22}` as an example. That's 10 bytes. In a language like Rust and using the serde library, this could be deserialized directly into a struct with one integer, let's pick a u32. So that would only be four bytes. But if it was deserialized to serdes dynamic Value type, this would be : a HashMap<String, u32>, which has a constant size of 48 bytes, plus an allocation of I don't know how much (first allocation will cover more than one entry), plus 16 bytes overhead for the string, plus 2 bytes for the actual string contents, plus the 4 bytes for the u32. So that's already over ~90 bytes, a lot more than the JSON. Dynamic languages like Python also have a lot of overhead for all the objects. Keys can of course be interned, but not that many default JSON parser libraries do that afaik.

DonHopkins 4y ago
The key names of all the identically shaped objects are repeated and can all be interned into sharing the same strings, saving a huge amount of memory. That's why csv files can be much more compact that the equivalent json files: they only list the keys once in the first row.
- mindslight 4y ago
  "ab" is 4 bytes, while a 64 bit pointer to an interned string is 8 bytes. It would seem that the savings of CSV would be better analogized to static typing - the type is defined once, then each record only contains data. I had the same intuition as the original comment. But no, the relative sizes of data formats aren't that straightforward. One could intern symbols to say 16 bit values, or one could infer structural types and compress the data that way. But those are both creating additional assumptions and processing that likely aren't done by commonly available tools.
  - DonHopkins 4y ago
    You still have to pay for a 64 bit pointer to an un-interned string though. It's the fact that the keys are repeated for each record that overwhelms the size of the keys or pointers themselves, when you have millions of records. If you are using fixed structures, then the keys are implicit and don't need to be repeated or even represented as pointers, just known fixed offsets into the structures. I wrote this about "Representing and Editing JSON with Spreadsheets": https://news.ycombinator.com/item?id=21109798 https://news.ycombinator.com/item?id=21109798 https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... SpreadOn: Apply Directly to the Forehead. https://www.youtube.com/watch?v=f_SwD7RveNE https://www.youtube.com/watch?v=f_SwD7RveNE

saidinesh5 4y ago

> * "You might find out that the data doesn’t fit into RAM (which it well might, JSON is a human-readable format after all)" -- if I'm reading this right, the author is saying that the parsed data takes _more_ space than the JSON version? JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Not to mention, even when using bad data structures (eg. hashmap of hashmaps..), One can just add a large enough swapfile and brute force their way through it no?

samus 4y ago
This will fail if the algorithm is not O(n). Even if only a bounded number of passes is required, there will be trouble.

zigzag312 4y ago

> if I'm reading this right, the author is saying that the parsed data takes _more_ space than the JSON version? JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Utf8 json strings will get converted to utf16 strings in some languages, doubling the size of strings in memory compared to the size on disk.

ot 4y ago

> Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language. Several years ago I wrote a paper [1] on representing the parse tree of a JSON document in a tiny fraction of the JSON size itself, using succinct data structures. The representation could be built with a single pass of the JSON, and basically constant additional memory. The idea was to pre-process the JSON and then save the parse tree, so it could be kept in memory over several passes of the JSON data (which may not fit in memory), avoiding to re-do the parsing work on each pass. I don't think I've seen this idea used anywhere, but I still wonder if it could have applications :) [1] http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pdf http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pd...

aardvarkr 4y ago
Have you implemented this idea in code? Seems interesting and worth playing around with
- ot 4y ago
  Yes, the code is linked in the paper: https://github.com/ot/semi_index https://github.com/ot/semi_index

coldtea 4y ago

>JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. Wanna bet? >>> import sys >>> import json >>> data_as_json_string = '{"a": 5000, "b": 1000}' >>> len(data_as_json_string) 22 >>> data_as_native_structure = json.loads(data_as_json_string) >>> sys.getsizeof(data_as_native_structure) 232 That's not even the whole story, as what's 232 bytes is not the contents of the dict, but just the Python object with the dict metadata. So the total for the struct inside is much bigger than 232 bytes. A single int wrapped as a Python object can be quite a lot by itself: >>> sys.getsizeof(1) 28 A binary int64 would be 8 bytes for comparison.

berkle4455 4y ago
Maybe 2023 is the year people stop using python.
- IshKebab 4y ago
  We can dream but I seriously doubt it. Not only are there very few nice alternatives (Dart is great but unpopular; Typescript is great but has a terrible standard library, etc.) but very few of them have a decent REPL which I think is probably Python's most underrated feature. I mean... it's REPL is extremely basic, but anything you can do in Python you can do in the REPL. Very useful especially for beginners.
killingtime74 4y ago
Python is my favorite language but you literally chose one of the least efficient. This would probably be different in Java or Rust.
- bufferoverflow 4y ago
  I thought Java stores strings as 16-bit character arrays internally. Which means it's very inefficient when it comes to memory consumption.
  - 4y ago

walrus01 4y ago

> A few GBs of data isn't really that much. The entire FCC radio license database (the ULS) is about 14GB in text CSV format and can be imported into a sqlite or sql db and easily queried in RAM on a local workstation...

nerdponx 4y ago

If the JSON data has a regular structure, you probably want a database and/or a "data frame" library and Parquet as the storage file format. SQLite, DuckDB, Polars, plenty of options nowadays that are usable from several different programming languages.

ZephyrBlu 4y ago

Have you done much data analysis? In my experience adding more steps to your pipeline (E.g. database, deserializing, etc) is a pain when you are figuring things out because nothing has solidified yet, so you're literally just adding overhead that requires even more work to remove/alter later on. If you're not careful you end up with something unmaintainable extremely quickly. Only analyzing a subset of your data is usually not a magic bullet either. Unless your data is extremely well cleaned and standardized you're probably going to run into edge cases on the full dataset that were not in your subset. Being able to run your full pipeline on the entire dataset in a short period of time is very useful for testing on the full dataset and seeing realistic analysis results. If you're doing any sort of aggregate analysis it becomes even more important, if not required. I now believe a relatively fast clean run is one of the most important things for performing data analysis. It increases your velocity tremendously.

fdgddggfddfg 4y ago

so on this note, unless you have massive json lists... having a json object per line (jsonl) can really help this process... I use jq when necessary .. but I usually use `cat file.json | grep foo | grep bar | jq -r .thing` since it's a string, use string comparison tools rather than jq :)