3 ms·
20 GB JSON is surprising to me. I have an sqlite file of all HN data that is 20 GB, it would be much larger as JSON.
by xnx 2y ago
20 GB JSON is surprising to me. I have an sqlite file of all HN data that is 20 GB, it would be much larger as JSON.
- wolfgang42 2y ago20 GB of JSON is correct; here’s the entire dump straight from the API up to last Monday: $ du -c ~/feepsearch-prod/datasource/hacker-news/data/dump/*.jsonl | tail -n1 19428360 total Not sure how your sqlite file is structured but my intuition is that the sizes being roughly the same sounds plausible: JSON has a lot of overhead from redundant structure and ASCII-formatted values; but sqlite has indexes, btrees, ptrmaps, overflow pages, freelists, and so on.
- elcritch 2y agoSqlite also doesn’t have fixed types, but uses a tagged value system to store data. Well according to what I’ve read on the topic.
- kortilla 2y agoSQLite files are optimized for fast querying, not size.