3 ms·
We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the
by tinco 11y ago
We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred lines of bash and Ruby does have?
- sagichmal 11y agoKubernetes is not a Java project.
- tinco 11y agoSorry got confused, Mesos is C++ and Java project I think. Probably got it swapped with that one.
- paukiatwee 11y ago* Kubernetes is not written in Java * There is no one size fit all, just like never ending Ruby vs JavaScript VS Java. * Kubernetes just hit v1.0 not long ago. * Seem like you already wrote the functionality you want, now what is the problem?
- tinco 11y ago* Alright, sorry Kubernetes is in Go I must've gotten it confused with some other project. * Well, alright, but they market it as a general purpose project. I'm just saying that to us it's strange that none of these frameworks out there deal with persistent storage, as the author of the article also observed. * Ok. * Well the problem is that it'd be much nicer if we could just use Kubernetes. Obviously having a homegrown orchestrator is not very nice. There's bounds to be lots of edgecases that our ops will run into that'll continuously cause us to perform maintenance on it, and maintaining an orchestration framework is not our core business. Anyway, this is not necessarily a critique of Kubernetes, it's great software. It's just an affirmation of the point the article is making, that it's curious that there's no interest in stateful containers from the maintainers of these frameworks.
- TheIronYuppie 11y agoDisclaimer: I work at Google on Kubernetes. We do care enormously about stateful solutions; I'll say what I've said elsewhere - how do you handle stateful services in your VMs?
- tinco 11y agoWe don't run anything in VMs at the moment. I guess if we would we'd do it like we're doing with the containers, mount the drives into them. Do you guys run the search engine in VMs?
- stephenitis 11y agoI'm curious what was the responsibilities of your own orchestrater? what are examples of situations it had to handle?
- tinco 11y agoWell basically the same things most frameworks do. Encode relations between containers, and allow use to spin containers up on machines, and automatically establishing the links between them. In addition to that, to encode persistent storage resources and link them to tasks. For example machine-1 has 1 HDD of 2TB and 3 SSD's of 250GB. On SSD1 the task postgres-1 has a data resource allocated called abc123. Now when we restart the machine, we can restart postgres-1 task and point it at its data resource and everything will work again. Note that the orchestrators do sort of give solutions to this issue. They might for example recommend doing the storage inside the docker container, and have the data disappear whenever the container disappears. This might be a little less reliable or transparent, but since you have a redundant cluster you can always restore the data from some other container. Our data is a little bit too big for restoration processes to happen during normal operations. And we're a bit too dependent on the speed to have no control over on which disk a resource is stored.