4 ms·

You have other options too. If I have time i can try to reduce the size with a columnar format that is designed for this use case (repeated values, static data

You have other options too. If I have time i can try to reduce the size with a columnar format that is designed for this use case (repeated values, static dataset).

zomglings 6y ago
That would be really great. Let me know if there's any way we can help. Maybe if we released a small version of the dataset for testing/benchmarking and then I could take care of running the final processing on the full dataset?
- StreamBright 6y ago
  That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.
  - zomglings 6y ago
    Hi StreamBright - just published the development version of the dataset also to Kaggle: https://www.kaggle.com/simiotic/github-code-snippets-development-sample https://www.kaggle.com/simiotic/github-code-snippets-develop... Compressed, it's 471 MB. Uncompressed, just a little more than 3 GB. If you want to get in touch with me in a better way than HN comments two good options: 1. My email is in my profile 2. You can direct message me (@zomglings) on the Bugout community Slack: https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt87-5XcJLy0iu702SO_hMFKNhQ https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt8... Looking forward to collaborating with you. :)
    - StreamBright 6y ago
      I have messaged you on Slack. The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. Parquet is much better for analytical use. - 2.9G Mar 8 08:37 snippets-dev.db - 427M Mar 8 14:05 test1.parquet (venv) snippets-dev sqlite3 -header -csv -readonly snippets-dev.db 'SELECT COUNT() FROM snippets;' COUNT() 4850000 (venv) snippets-dev python test.py test1.parquet 4850000 I can share the 3 lines of Python and 1 line for SQL I used to convert the dataset. There was only weird thing that I encountered, you somehow have a new line character in the commit_hash field for every value.

You have other options too. If I have time i can try to reduce the size with a columnar format that is designed for this use case (repeated values, static dataset).

zomglings 6y ago

That would be really great. Let me know if there's any way we can help. Maybe if we released a small version of the dataset for testing/benchmarking and then I could take care of running the final processing on the full dataset?

StreamBright 6y ago
That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.
- zomglings 6y ago
  Hi StreamBright - just published the development version of the dataset also to Kaggle: https://www.kaggle.com/simiotic/github-code-snippets-development-sample https://www.kaggle.com/simiotic/github-code-snippets-develop... Compressed, it's 471 MB. Uncompressed, just a little more than 3 GB. If you want to get in touch with me in a better way than HN comments two good options: 1. My email is in my profile 2. You can direct message me (@zomglings) on the Bugout community Slack: https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt87-5XcJLy0iu702SO_hMFKNhQ https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt8... Looking forward to collaborating with you. :)
  - StreamBright 6y ago
    I have messaged you on Slack. The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. Parquet is much better for analytical use. - 2.9G Mar 8 08:37 snippets-dev.db - 427M Mar 8 14:05 test1.parquet (venv) snippets-dev sqlite3 -header -csv -readonly snippets-dev.db 'SELECT COUNT() FROM snippets;' COUNT() 4850000 (venv) snippets-dev python test.py test1.parquet 4850000 I can share the 3 lines of Python and 1 line for SQL I used to convert the dataset. There was only weird thing that I encountered, you somehow have a new line character in the commit_hash field for every value.