3 ms·
Uses Kubernetes underneath, which is pretty interesting. Whole premise of containers as MR primitives is also good. It will boil down to how the filesystem, the
by manojlds 10y ago
Uses Kubernetes underneath, which is pretty interesting. Whole premise of containers as MR primitives is also good. It will boil down to how the filesystem, the pipeline service and the clustering is implemented and how well they work together.
- jdoliner 10y agoKubernetes has worked really well for us, it's given use distributed systems primitives that just work which has allowed us to focus on the parts of the system that are unique to what we're doing. We get a lot of questions about why we choose to implement our own filesystem and computation layer and the answer we gave is the one you have here, it allows us to make sure they work well together because they're built from the ground up with containerized workloads in mind. We could have tried to containerize Hadoop, but I doubt it would have resulted in the fully integrated system that we have today. Too much in the Hadoop ecosystem was written for a precontainer world for it to make sense to containerize it after the fact.