4 ms·

I have a similar harness going for my recent experiments, except instead of hosting with huggingface I have a dataframe with pointers to the files on S3 and the

by icyfox 3y ago

I have a similar harness going for my recent experiments, except instead of hosting with huggingface I have a dataframe with pointers to the files on S3 and then just download them during local preprocessing. Every time I see DVC mentioned I always feel like the idea was so close (and perhaps right in intuition to use git for everything) but the execution had just enough friction that I looked elsewhere. Small DX improvements really do cascade pretty far.

nerdponx 3y ago
DVC is great for medium-scale projects in small teams, but that's where I'd stop with it. It only really makes sense for work that you're doing on your own machine, or an old-school Linux server type of setup, not something you'd use for modern-day ML work in a cloud environment. Also I always thought the idea of using Git branches to track experiments was a bad idea. I would never want to only have one experiment "active" at a time. Even if I'm only running one process at a time, I still want to be able to look at outputs and such all side-by-side. Maybe there's some magic tooling they created that makes it workable.
- unsynced 3y ago
  FYI, you can use git worktrees [1] to work on multiple branches simultaneously [1] https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree
  - nerdponx 3y ago
    Yeah, I know and love that feature for software projects, especially if I need to switch over to a bugfix while I'm deep in a topic branch. But for a data project it would be a big pain to have separate worktrees just to work around what IMO is a usage anti-pattern to begin with!
- shcheklein 3y ago
  DVC has `dvc exp` that doesn't require creating commits or branches. It's utilizing git custom references (technical details [1]). And it can be visualized in CLI or VS Code. [1] https://iterative.ai/blog/experiment-refs https://iterative.ai/blog/experiment-refs [2] https://marketplace.visualstudio.com/items?itemName=Iterative.dvc https://marketplace.visualstudio.com/items?itemName=Iterativ...
  - nerdponx 3y ago
    Thanks! I've been using DVC solely for tracking data, and had basically ignored all of its other features. I'll have to take a look at this. Most/all of my projects use small or medium scale data, and I consider DVC indispensable for tracking data therein. I wouldn't mind having a good system for tracking experiment results, although admittedly I find that a spreadsheet or text file does a pretty good job for what I need to do.
a_bonobo 3y ago
I've really liked the idea of scidataflow in this context: https://github.com/vsbuffalo/scidataflow https://github.com/vsbuffalo/scidataflow It's neat for research as it stores the data on scientific data repositories like Zenodo and you get DOIs.

by icyfox 3y ago

nerdponx 3y ago

DVC is great for medium-scale projects in small teams, but that's where I'd stop with it. It only really makes sense for work that you're doing on your own machine, or an old-school Linux server type of setup, not something you'd use for modern-day ML work in a cloud environment. Also I always thought the idea of using Git branches to track experiments was a bad idea. I would never want to only have one experiment "active" at a time. Even if I'm only running one process at a time, I still want to be able to look at outputs and such all side-by-side. Maybe there's some magic tooling they created that makes it workable.

unsynced 3y ago
FYI, you can use git worktrees [1] to work on multiple branches simultaneously [1] https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree
- nerdponx 3y ago
  Yeah, I know and love that feature for software projects, especially if I need to switch over to a bugfix while I'm deep in a topic branch. But for a data project it would be a big pain to have separate worktrees just to work around what IMO is a usage anti-pattern to begin with!
shcheklein 3y ago
DVC has `dvc exp` that doesn't require creating commits or branches. It's utilizing git custom references (technical details [1]). And it can be visualized in CLI or VS Code. [1] https://iterative.ai/blog/experiment-refs https://iterative.ai/blog/experiment-refs [2] https://marketplace.visualstudio.com/items?itemName=Iterative.dvc https://marketplace.visualstudio.com/items?itemName=Iterativ...
- nerdponx 3y ago
  Thanks! I've been using DVC solely for tracking data, and had basically ignored all of its other features. I'll have to take a look at this. Most/all of my projects use small or medium scale data, and I consider DVC indispensable for tracking data therein. I wouldn't mind having a good system for tracking experiment results, although admittedly I find that a spreadsheet or text file does a pretty good job for what I need to do.

a_bonobo 3y ago

I've really liked the idea of scidataflow in this context: https://github.com/vsbuffalo/scidataflow https://github.com/vsbuffalo/scidataflow It's neat for research as it stores the data on scientific data repositories like Zenodo and you get DOIs.