4 ms·

Pachyderm pipelines can be run in a massively distributed fashion with data being sharded across many workers. Pachyderm also offers much better failure semanti

by jaz46 8y ago

Pachyderm pipelines can be run in a massively distributed fashion with data being sharded across many workers. Pachyderm also offers much better failure semantics than Make + CI. For example, if one shard of data of your pipeline fails such as a node or container dies, Pachyderm will automatically make sure that the data get rescheduled to another worker. Each pipeline can have separate resource requirements (e.g. GPUs, lots of memory, etc) and gets scheduled by Kubernetes. Finally, Pachyderm is versioning all of the intermediate steps in your data pipeline so if a downstream step fails, you don't have to restart from scratch, you can pick up right where it left off.

agibsonccc 8y ago
Congrats on your round! FWIW, (me being in a space adjacent to yours) I see these kinds of comments all the time about the "my 1 off pipeline seems good enough". Most people don't need "industrial strength" till you hit a certain scale. They tend to optimize for ease of use/simplicity. It's one thing if you don't have to change their workflow, it's another if you have to not only have people change their workflow but also teach something new. Convenience matters more. There's a pain threshold of "new tool" vs "this costs me x amount of time". How are you guys overcoming this? Even though we're also in the infra space, I've never seen you guys out in the wild. Where would I bump in to you and what is the scale I would want to add the complexity of k8s + pachyderm + whatever other deps you guys have over just using an S3 bucket? Another question: Why hasn't AWS just packaged this up and offered it as an extension of their k8s service? How are you guys going to overcome that? Usually I see "hybrid cloud" or "on prem" as the response. If it is on prem, are you guys relying on the presence of k8s at customers? Do you guys use something like gravitational?
- marmaduke 8y ago
  > my 1 off pipeline seems good enough Not what I was describing. We’ve been using the setup for a while with multiple projects and for the data science end of a clinical trial. I like building new setups incrementally out of tools I already know, so the question is obviously why is this new tool a radical improvement
marmaduke 8y ago
Thanks for the response. I forgot to mention that the Make/CI jobs are run on a Slurm HPC cluster (actually multiple since the CI file uses runners running in multiple Slurm clusters for different resources). Make ensures we don’t redo stuff unless needed. Point taken about restarts, but I usually want to see why a thing fails before retrying. It seems like Pachyderm mainly offers better granularity, especially for the data management.

by jaz46 8y ago

agibsonccc 8y ago

Congrats on your round! FWIW, (me being in a space adjacent to yours) I see these kinds of comments all the time about the "my 1 off pipeline seems good enough". Most people don't need "industrial strength" till you hit a certain scale. They tend to optimize for ease of use/simplicity. It's one thing if you don't have to change their workflow, it's another if you have to not only have people change their workflow but also teach something new. Convenience matters more. There's a pain threshold of "new tool" vs "this costs me x amount of time". How are you guys overcoming this? Even though we're also in the infra space, I've never seen you guys out in the wild. Where would I bump in to you and what is the scale I would want to add the complexity of k8s + pachyderm + whatever other deps you guys have over just using an S3 bucket? Another question: Why hasn't AWS just packaged this up and offered it as an extension of their k8s service? How are you guys going to overcome that? Usually I see "hybrid cloud" or "on prem" as the response. If it is on prem, are you guys relying on the presence of k8s at customers? Do you guys use something like gravitational?

marmaduke 8y ago
> my 1 off pipeline seems good enough Not what I was describing. We’ve been using the setup for a while with multiple projects and for the data science end of a clinical trial. I like building new setups incrementally out of tools I already know, so the question is obviously why is this new tool a radical improvement

marmaduke 8y ago

Thanks for the response. I forgot to mention that the Make/CI jobs are run on a Slurm HPC cluster (actually multiple since the CI file uses runners running in multiple Slurm clusters for different resources). Make ensures we don’t redo stuff unless needed. Point taken about restarts, but I usually want to see why a thing fails before retrying. It seems like Pachyderm mainly offers better granularity, especially for the data management.