3 ms·

Hi Guys, co-founder here. We'll be hanging out on this thread all day to answer any questions.

by jdoliner 11y ago

TallGuyShort 11y ago
I'm curious about the decision to target Docker so specificially. Is that just how you plan to manage resources between different users / workloads? Or is there another benefit? I work in the Hadoop space (although I like the stack as a whole, I do see the downside of having evolved the stack over time with a bunch of independent projects over time, so although I suspect Pachyderm will have a long way to go, I watch with interest). We mainly use Docker containers for testing / training, etc. A distributed system allows you to scale, but you still want each node to be as big as possible, so in production I'd never use Docker. YARN of course puts things in to "containers" to manage resources, but each container is still running in the same OS. Targeting a system that partitions the resources of a physical machine so completely seems counter-intuitive for Big Data, so what's the thinking there? edit: or was the focus on Docker added by TC to attract more clicks? :)
- jdoliner 11y ago
  Containers solve a few problems for us. They allow us to provide strong isolation guarantees for jobs and offer nice resource quota semantics (via cgroups.) They also serve as a good format for specifying jobs in because they can be easily passed around between nodes via docker pull and push. Our use of Docker is very similar to how you describe YARN's use of containers. We deploy on a cluster of CoreOS machines and each machine is generally as big as possible. We then pack several containers on to that same machine so that we can get as much performance as possible out of it. This also gives us an opportunity to do smart things like sharing data volumes between docker containers that are running different workloads over the same data.

3 ms·

Hi Guys, co-founder here. We'll be hanging out on this thread all day to answer any questions.

by jdoliner 11y ago

Hi Guys, co-founder here. We'll be hanging out on this thread all day to answer any questions.

TallGuyShort 11y ago
I'm curious about the decision to target Docker so specificially. Is that just how you plan to manage resources between different users / workloads? Or is there another benefit? I work in the Hadoop space (although I like the stack as a whole, I do see the downside of having evolved the stack over time with a bunch of independent projects over time, so although I suspect Pachyderm will have a long way to go, I watch with interest). We mainly use Docker containers for testing / training, etc. A distributed system allows you to scale, but you still want each node to be as big as possible, so in production I'd never use Docker. YARN of course puts things in to "containers" to manage resources, but each container is still running in the same OS. Targeting a system that partitions the resources of a physical machine so completely seems counter-intuitive for Big Data, so what's the thinking there? edit: or was the focus on Docker added by TC to attract more clicks? :)
- jdoliner 11y ago
  Containers solve a few problems for us. They allow us to provide strong isolation guarantees for jobs and offer nice resource quota semantics (via cgroups.) They also serve as a good format for specifying jobs in because they can be easily passed around between nodes via docker pull and push. Our use of Docker is very similar to how you describe YARN's use of containers. We deploy on a cluster of CoreOS machines and each machine is generally as big as possible. We then pack several containers on to that same machine so that we can get as much performance as possible out of it. This also gives us an opportunity to do smart things like sharing data volumes between docker containers that are running different workloads over the same data.