4 ms·
I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so sl
by n_e 6mo ago
I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.
- eru 6mo agoThis reminds me of someone who wrote a regex tool that matches by compiling regexes (at runtime of the tool) via LLVM to native code. You could probably do something similar for a faster jq.
- nchmy 6mo agoThis isn't for you then > The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation. Mind me asking what sorts of TB json files you work with? Seems excessively immense.
- szundi 6mo ago[dead]
- rennokki 6mo ago> Uses jq for TB json files > Hadoop: bro > Spark: bro > hive: bro > data team: bro
- deleted 6mo ago[deleted]
- anonymoushn 6mo agoare those tools known for their fast json parsers?
- rennokki 6mo agoIf we talk about TB or PB+ scales, then yes.
- anonymoushn 6mo agoOh, can you post some benchmarks? I didn't know that parser throughput per core would change with the amount of data like that.
- f311a 6mo agoJQ is very convenient, even if your files are more than 100GB. I often need to extract one field from huge JSON line files, I just pipe jq to it to get results. It's slower, but implementing proper data processing will take more time.
- rennokki 6mo agoMore than 100GB can be 101GB, 500GB or 1TB+. I was speaking about 1TB+ files. I'm not sure you can get it faster unless you have a parallel processor.
- eevmanu 6mo agomade me remember this article <https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...> Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.
- rennokki 6mo agoThis article is good for new programmers to understand why certain solutions are better at scale, there is no silver bullet. And also, this is from 2014, and the dataset is < 4GB. No reason to use hadoop. The discussion we had here was involving TB of data, so I'm curious how this is faster with CLIs rather than parallel processing...
- messe 6mo agoNow I'm really curious. What field are you in that ndjson files of that size are common? I'm sure there are reasons against switching to something more efficient–we've all been there–I'm just surprised.
- overfeed 6mo ago> Now I'm really curious. What field are you in that ndjson files of that size are common? I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.
- messe 6mo agoSo what's the use case for keeping them in that format rather than something more easily indexed and queryable? I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.
- paavope 6mo agoThe use case could be e.g. exactly processing an old trove of logs into something more easily indexed and queryable, and you might want to use jq as part of that processing pipeline
- messe 6mo agoFair, but for a once-off thing performance isn't usually a major factor. The comment I was replying to implied this was something more regular. EDIT: why is this being downvoted? I didn't think I was rude. The person I responded to made a good point, I was just clarifying that it wasn't quite the situation I was asking about.
- adastra22 6mo agoAt scale, low performance can very easily mean "longer than the lifetime of the universe to execute." The question isn't how quickly something will get done, but whether it can be done at all.
- deleted 6mo ago[deleted]
- loxias 6mo agoI would love, _love_ to know more about your data formats, your tools, what the JSON looks like, basically as much as you're willing to share. :) For about a month now I've been working on a suite of tools for dealing with JSON specifically written for the imagined audience of "for people who like CLIs or TUIs and have to deal with PILES AND PILES of JSON and care deeply about performance". For me, I've been writing them just because it's an "itch". I like writing high performance/efficient software, and there's a few gaps that it bugged me they existed, that I knew I could fill. I'm having fun and will be happy when I finish, regardless, but it would be so cool if it happened to solve a problem for someone else.
- landr0id 6mo agoI maintain some tools for the videogame World of Warships. The developer has a file called GameParams.bin which is Python-pickled data (their scripting language is Python). Working with this is pretty painful, so I convert the Pickled structure to other formats including JSON. The file has always been prettified around ~500MB but as of recently expands to about 3GB I think because they’ve added extra regional parameters. The file inflates to a large size because Pickle refcounts objects for deduping, whereas obviously that’s lost in JSON. I care about speed and tools not choking on the large inputs so I use jaq for querying and instruction LLMs operating on the data to do the same.