Nice! > However, for reasons unknown to me, they wrap these neatly separated rows with brackets ([ and ]) and add a comma to each line Well, the reason (misgu

Nice! > However, for reasons unknown to me, they wrap these neatly separated rows with brackets ([ and ]) and add a comma to each line Well, the reason (misguided or not) is as you say, I imagine: > so it’s a valid, JSON array containing 100+ million items. > We are not going to attempt to load a this massive array. Instead, we’re running this command: zcat ../latest-all.json.gz | sed 's/,$//' | split -l 100000 - wd_items_cw --filter='gzip > $FILE.gz' That's one approach - I'm always a little wary of treating a rich format like JSON as <something> deliminated text - I'd be curious if using jq in streaming mode is much different in run-time. I believe this snippet, the core of which we lifted from stack overflow or somewhere does the same thing; split a valid JSON array into ndjson (with tweaks to hopefully generate similar splits: gunzip -c ../latest-all.json.gz \ | jq -cn --stream \ 'fromstream(inputs|(.[0] |= .[1:]) | select(. != [[]]) )' \ | split -l 100000 - wd_items_cw --filter='gzip > $FILE.gz Note on MacOS zcat might not be gunzip, hence the change.