4 ms·
Ask HN: Python package for managing science datasets
I was looking for a Python package to manage science datasets, mainly description, download, extraction, on-disk handling, etc... ML frameworks tend to roll their own:
- https://github.com/nilearn/nilearn/blob/master/nilearn/datasets/utils.py
- https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/download/download_manager.py
- https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/base.py
- https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py
I was hoping that somebody had made a generic package for this purpose so that I don't roll yet another one.
- swaroop 7y agoHow about https://dvc.org/ https://dvc.org/ ?
- kelsolaar 7y agoInteresting, I did not think about this way at all, I will take a look.
- amirouche 7y agoI started a list of such software https://github.com/awesome-data-distribution/awesome-data-distribution#awesome-data-distribution https://github.com/awesome-data-distribution/awesome-data-di...
- kelsolaar 7y agoThat is awesome (pun intended), thanks!