2 ms·
I have messaged you on Slack. The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. P
by StreamBright 6y ago
I have messaged you on Slack.
The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. Parquet is much better for analytical use.
- 2.9G Mar 8 08:37 snippets-dev.db
- 427M Mar 8 14:05 test1.parquet
(venv) snippets-dev sqlite3 -header -csv -readonly snippets-dev.db 'SELECT COUNT() FROM snippets;'
COUNT()
4850000
(venv) snippets-dev python test.py test1.parquet
4850000
I can share the 3 lines of Python and 1 line for SQL I used to convert the dataset.
There was only weird thing that I encountered, you somehow have a new line character in the commit_hash field for every value.