3 ms·
I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ https://drivendata.github
by paulgb 10y ago
I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ https://drivendata.github.io/cookiecutter-data-science/
I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.
- in9 10y agoI really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually
- paulgb 10y agoYes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though. (I don't actually use their template, I just use the directory layout as guidance)