5 ms·

Pachyderm Raises $10M to Bring Data Provenance to the Enterprise

102 points by jaz46 8y ago

otoolep 8y ago
Congrats to the team at Pachyderm!
atav1k 8y ago
nice!
coolhand1 8y ago
Love this project! Git for data is such a brilliant concept
mmasters 8y ago
I was part of an experimental neuroimaging group that tested Pachyderm OSS years ago and at the time we were really impressed with the versioning capabilities it provided. For us at the time it made it easy for each researcher to grab and change data as needed for their own development without requiring support from eng.
- marmaduke 8y ago
  How well does that work when you datasets are a sizeable percentage of available storage capacity, though? Is there some sort of deduplication at work?
  - jaz46 8y ago
    Pachyderm does a ton of data deduplication, both for input data that's added to pachyderm repos as well as for output files. Pachyderm's pipelines are also smart enough to know what data has changed and what hasn't and only process the incremental data "diffs" as needed. If your pipeline is just one giant reduce or training job that's can't be broken up at all, then this isn't valuable, but most workloads include lots of Map steps where only processing diffs can be incredibly powerful
    - marmaduke 8y ago
      This is super cool, thanks for pointing that out. Is the hard part done by Pachyderm or as some layer over container file systems?
      - ztjio 8y ago
        Pachyderm does it. It's like half of what pachyderm does, manage the versioned data, and schedule workers to run your containered processes against them. FYI, it's ridiculously easy to get going playing with Pachyderm if you just want to check it out. You can run it on Minikube.
        marmaduke 8y ago
        > You can run it on Minikub Thanks for the tip. I just started down the k8s path from bare metal cluster and will try this.
mb4 8y ago
Congrats guys! Data provenance is only becoming more important
_drFaust 8y ago
much deserved, been following this project for sometime and have continued to be impressed.
alex_lfw 8y ago
He was holding these two pieces of pizza...
- jdoliner 8y ago
  Hi, Pachyderm CEO here. We have been discussing this comment internally for the last 15 minutes and still don't understand it. But we really enjoy it. Would you mind explaining a bit more what you are saying?
  - Liru 8y ago
    Not the person you are responding to, but the comment you're replying to is referencing Seinfeld, due to the name of your company. https://en.wikipedia.org/wiki/The_Stand_In_(Seinfeld) https://en.wikipedia.org/wiki/The_Stand_In_(Seinfeld)
    - jaz46 8y ago
      What a wonderfully obscure reference.
    - jdoliner 8y ago
      This is lovely, thank you.
marmaduke 8y ago
I have a “data science pipeline” coordinated with a Makefile and run on CI/CD (GitLab) with reports generated as build artifacts. Big stuff checked in with Git LFS. Why would I use Pachyderm?
- claytonjy 8y ago
  for a job where that workflow is sufficient, you probably shouldn't. But if you need an always-on, always-ready pipeline that can scale horizontally on an as-needed basis, and perhaps mixes many different modules/languages, than Pachyderm might be a good fit.
  - emkt 8y ago
    Well you can use https://github.com/argoproj/argo https://github.com/argoproj/argo for workflow needs without need to version everything. For data versioning if you need, you can use https://dvc.org/ https://dvc.org/ As this gives me freedom to choose as if I want data versioning and/or workflow rather than one bug bundle.
- IanCal 8y ago
  I think it'd be overkill if what you have is working for you. At least, replacing what you have may not be worth it. For me, there's various properties I like about pachyderm (though they're missing some key workflows I'd like). Looking at some output data, I can see exactly what versions of everything was run against what data, and the data is all version controlled. This should also allow experimentation by individual people and easier approval of updates. I work with multiple teams with many components, where each one may impact others - being able to look at some final result that's weird and go back to figure out what's going on is hard. Tools like pachyderm help with that.
- jaz46 8y ago
  Pachyderm pipelines can be run in a massively distributed fashion with data being sharded across many workers. Pachyderm also offers much better failure semantics than Make + CI. For example, if one shard of data of your pipeline fails such as a node or container dies, Pachyderm will automatically make sure that the data get rescheduled to another worker. Each pipeline can have separate resource requirements (e.g. GPUs, lots of memory, etc) and gets scheduled by Kubernetes. Finally, Pachyderm is versioning all of the intermediate steps in your data pipeline so if a downstream step fails, you don't have to restart from scratch, you can pick up right where it left off.
akircher 8y ago
This is great news. An amazing team with a good solution to a huge problem. I look forward to following your progress.
grbno 8y ago
I've also wondered why I should use Pachyderm. Decided to give it a try, and wrote the following blog about it : https://medium.com/bigdatarepublic/pachyderm-for-data-scientists-d1d1dff3a2fa https://medium.com/bigdatarepublic/pachyderm-for-data-scient... " Finally, version control for your data "
koolhead17 8y ago
Congrats Pachyderm team.

102 points by jaz46 8y ago

otoolep 8y ago

Congrats to the team at Pachyderm!

atav1k 8y ago

nice!

coolhand1 8y ago

Love this project! Git for data is such a brilliant concept

mmasters 8y ago

I was part of an experimental neuroimaging group that tested Pachyderm OSS years ago and at the time we were really impressed with the versioning capabilities it provided. For us at the time it made it easy for each researcher to grab and change data as needed for their own development without requiring support from eng.

marmaduke 8y ago
How well does that work when you datasets are a sizeable percentage of available storage capacity, though? Is there some sort of deduplication at work?
- jaz46 8y ago
  Pachyderm does a ton of data deduplication, both for input data that's added to pachyderm repos as well as for output files. Pachyderm's pipelines are also smart enough to know what data has changed and what hasn't and only process the incremental data "diffs" as needed. If your pipeline is just one giant reduce or training job that's can't be broken up at all, then this isn't valuable, but most workloads include lots of Map steps where only processing diffs can be incredibly powerful
  - marmaduke 8y ago
    This is super cool, thanks for pointing that out. Is the hard part done by Pachyderm or as some layer over container file systems?
    - ztjio 8y ago
      Pachyderm does it. It's like half of what pachyderm does, manage the versioned data, and schedule workers to run your containered processes against them. FYI, it's ridiculously easy to get going playing with Pachyderm if you just want to check it out. You can run it on Minikube.
      - marmaduke 8y ago
        > You can run it on Minikub Thanks for the tip. I just started down the k8s path from bare metal cluster and will try this.

mb4 8y ago

Congrats guys! Data provenance is only becoming more important

_drFaust 8y ago

much deserved, been following this project for sometime and have continued to be impressed.

alex_lfw 8y ago

He was holding these two pieces of pizza...

jdoliner 8y ago
Hi, Pachyderm CEO here. We have been discussing this comment internally for the last 15 minutes and still don't understand it. But we really enjoy it. Would you mind explaining a bit more what you are saying?
- Liru 8y ago
  Not the person you are responding to, but the comment you're replying to is referencing Seinfeld, due to the name of your company. https://en.wikipedia.org/wiki/The_Stand_In_(Seinfeld) https://en.wikipedia.org/wiki/The_Stand_In_(Seinfeld)
  - jaz46 8y ago
    What a wonderfully obscure reference.
  - jdoliner 8y ago
    This is lovely, thank you.

marmaduke 8y ago

I have a “data science pipeline” coordinated with a Makefile and run on CI/CD (GitLab) with reports generated as build artifacts. Big stuff checked in with Git LFS. Why would I use Pachyderm?

claytonjy 8y ago
for a job where that workflow is sufficient, you probably shouldn't. But if you need an always-on, always-ready pipeline that can scale horizontally on an as-needed basis, and perhaps mixes many different modules/languages, than Pachyderm might be a good fit.
- emkt 8y ago
  Well you can use https://github.com/argoproj/argo https://github.com/argoproj/argo for workflow needs without need to version everything. For data versioning if you need, you can use https://dvc.org/ https://dvc.org/ As this gives me freedom to choose as if I want data versioning and/or workflow rather than one bug bundle.
IanCal 8y ago
I think it'd be overkill if what you have is working for you. At least, replacing what you have may not be worth it. For me, there's various properties I like about pachyderm (though they're missing some key workflows I'd like). Looking at some output data, I can see exactly what versions of everything was run against what data, and the data is all version controlled. This should also allow experimentation by individual people and easier approval of updates. I work with multiple teams with many components, where each one may impact others - being able to look at some final result that's weird and go back to figure out what's going on is hard. Tools like pachyderm help with that.
jaz46 8y ago
Pachyderm pipelines can be run in a massively distributed fashion with data being sharded across many workers. Pachyderm also offers much better failure semantics than Make + CI. For example, if one shard of data of your pipeline fails such as a node or container dies, Pachyderm will automatically make sure that the data get rescheduled to another worker. Each pipeline can have separate resource requirements (e.g. GPUs, lots of memory, etc) and gets scheduled by Kubernetes. Finally, Pachyderm is versioning all of the intermediate steps in your data pipeline so if a downstream step fails, you don't have to restart from scratch, you can pick up right where it left off.

akircher 8y ago

This is great news. An amazing team with a good solution to a huge problem. I look forward to following your progress.

grbno 8y ago

I've also wondered why I should use Pachyderm. Decided to give it a try, and wrote the following blog about it : https://medium.com/bigdatarepublic/pachyderm-for-data-scientists-d1d1dff3a2fa https://medium.com/bigdatarepublic/pachyderm-for-data-scient... " Finally, version control for your data "

koolhead17 8y ago

Congrats Pachyderm team.