3 ms·
That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.
by StreamBright 6y ago
That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.
- zomglings 6y agoHi StreamBright - just published the development version of the dataset also to Kaggle: https://www.kaggle.com/simiotic/github-code-snippets-development-sample https://www.kaggle.com/simiotic/github-code-snippets-develop... Compressed, it's 471 MB. Uncompressed, just a little more than 3 GB. If you want to get in touch with me in a better way than HN comments two good options: 1. My email is in my profile 2. You can direct message me (@zomglings) on the Bugout community Slack: https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt87-5XcJLy0iu702SO_hMFKNhQ https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt8... Looking forward to collaborating with you. :)
- StreamBright 6y agoI have messaged you on Slack. The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. Parquet is much better for analytical use. - 2.9G Mar 8 08:37 snippets-dev.db - 427M Mar 8 14:05 test1.parquet (venv) snippets-dev sqlite3 -header -csv -readonly snippets-dev.db 'SELECT COUNT() FROM snippets;' COUNT() 4850000 (venv) snippets-dev python test.py test1.parquet 4850000 I can share the 3 lines of Python and 1 line for SQL I used to convert the dataset. There was only weird thing that I encountered, you somehow have a new line character in the commit_hash field for every value.