4 ms·

I'd think that Pachyderm [0] would be mentioned in the article, but it wasn't. It uses Docker under the hood for the data pipelines. I've always wanted an excus

I'd think that Pachyderm [0] would be mentioned in the article, but it wasn't. It uses Docker under the hood for the data pipelines. I've always wanted an excuse to do some data science... [0]: https://www.pachyderm.io/ https://www.pachyderm.io/

raab 9y ago
Cool! I had never heard of that. Starred...
haraball 9y ago
Singularity [0] is a related alternative. [0]: http://singularity.lbl.gov/ http://singularity.lbl.gov/
jdoliner 9y ago
Thanks for the plug wyattjoh, I'm one of the founders of Pachyderm so I'd like to expand on what you said just a bit to clarify. Pachyderm does use Docker under the hood, but we don't obfuscate it away, so Data Scientists get the full power of Docker (and most of the power of Kubernetes) at their fingertips. This means you can easily grab prefabricated environments such as Jupyter or Tensorflow container images and deploy them directly. Docker is so good for packaging environments we didn't want to conceal it. On the other hand, we felt the data orchestration capabilities of Docker were pretty lacking for Data Science use cases, so that's where we've focused our energy. Volumes are a good basic tool for getting data to your code, but they're a pretty blunt instrument. How do you split data up to parallelize over it? How do you make sure you've got the right version of data? How do you schedule new computations when data becomes available? Those are some of the use cases we solve with our distributed file system PFS.
claytonjy 9y ago
+1 for pachyderm; my team and I are using it to build backtesting platforms for financial modeling, and it's been really nice to use. Also great in combination with Gitlab's built-in CI/CD stuff. Product aside, we've appreciated the team's availability on both Slack and Github too.

4 ms·

I'd think that Pachyderm [0] would be mentioned in the article, but it wasn't. It uses Docker under the hood for the data pipelines. I've always wanted an excus

by wyattjoh 9y ago

raab 9y ago
Cool! I had never heard of that. Starred...
haraball 9y ago
Singularity [0] is a related alternative. [0]: http://singularity.lbl.gov/ http://singularity.lbl.gov/
jdoliner 9y ago
Thanks for the plug wyattjoh, I'm one of the founders of Pachyderm so I'd like to expand on what you said just a bit to clarify. Pachyderm does use Docker under the hood, but we don't obfuscate it away, so Data Scientists get the full power of Docker (and most of the power of Kubernetes) at their fingertips. This means you can easily grab prefabricated environments such as Jupyter or Tensorflow container images and deploy them directly. Docker is so good for packaging environments we didn't want to conceal it. On the other hand, we felt the data orchestration capabilities of Docker were pretty lacking for Data Science use cases, so that's where we've focused our energy. Volumes are a good basic tool for getting data to your code, but they're a pretty blunt instrument. How do you split data up to parallelize over it? How do you make sure you've got the right version of data? How do you schedule new computations when data becomes available? Those are some of the use cases we solve with our distributed file system PFS.
claytonjy 9y ago
+1 for pachyderm; my team and I are using it to build backtesting platforms for financial modeling, and it's been really nice to use. Also great in combination with Gitlab's built-in CI/CD stuff. Product aside, we've appreciated the team's availability on both Slack and Github too.