dmpetrov

8 ms·

61.
Show HN: DVC 1.0 release. 5 lessons from 3 years of building open-source ML tool
19 points by dmpetrov 6y ago|4 comments
62.
by dmpetrov 7y ago
I really believe that "end-to-end AI platforms" of the future should be built on top of open-source tools like these two. I cannot wait to see when other parts of AI platforms will be integrated with the existing ones: model perfo
63.
by dmpetrov 8y ago
"From a very high level perspective - Pachyderm is a data engineering tool designed with ML in mind, DVC.org is a tool to organize and version control an ML project. Probably, one way to think would be Spark/Hadoop or Airflow vs G
64.
by dmpetrov 8y ago
Content-defined chunks - very interesting. I'd suggest you ask this question in DVC issue tracker or DVC channel https://dvc.org/chat
65.
by dmpetrov 8y ago
The major pitfall - it depends on your goal. Your approach looks good if you just need to retrain an existing model\code in production. However, this approach is not perfect in the ML modeling\development stage. Let me explain... `--entrypo
66.
by dmpetrov 8y ago
Exactly. And it is flexible as Git. You can define your own workflow. For example, some data scientists avoid using git-branches for experiments - they use directories.
67.
by dmpetrov 8y ago
The sweet spot for LFS is a few 100Mb files with less than a couple Gb overall. While DVC was designed with 10Gb-100Gb scenarios in mind.
68.
by dmpetrov 8y ago
Great point! `dvc pull\push mydataset` actually exists in DVC. But `dvc tag` is missing and now we (DVC team) understand that it prevent DVC from a proper datasets and ML models tracking. It will be implemented soon. There is an ongoing dis
69.
by dmpetrov 8y ago
Docker can help only if there is a single step in your project. In ML projects you usually have many steps - Preprocess, Train. Each of the steps can be divided: extract Evaluate step from Train etc. Also, Docker has an overhead - copy of d
70.
by dmpetrov 8y ago
Example. A query to DB gives you different results since the data\table evolves over time. So, you just store the query output (let say a couple GBs) in DVC to make your research reproducible. This is like assigning a random-seed to DB :) S
71.
by dmpetrov 8y ago
@mcncfie your understanding is correct. #3 might include output data\models as well and intermediate results like preprocessed data. DVC also handles dependencies between all of these. Answers: 1. DVC does dependency tracking in addition to
72.
by dmpetrov 9y ago
'Kinda putting the "science" back in data science. ' Exactly! This is the primary goal in DVC project.
73.
How to Version Control Your Machine Learning and Deep Learning Tasks(medium.com)
3 points by dmpetrov 9y ago|0 comments
74.
Git for data science projects(dataversioncontrol.com)
3 points by dmpetrov 9y ago|0 comments