3 ms·
Kubernetes requires machines big enough to run all your containers. Those machines are the bins. Your containers are the packages. Fitting your containers in su
by eropple 9y ago
Kubernetes requires machines big enough to run all your containers. Those machines are the bins. Your containers are the packages. Fitting your containers in such that there is no criticality overlap (in AWS, that all instances of service X are spread across machines in different AZs) and that there is room for immediate scaling/emergency fault recovery (headroom on the machines running your containers) gets expensive. You're buying big and running little, and that comes with costs.
Meanwhile, in AWS, you already have pre-sized blobs of RAM and compute. They're called EC2 instances. And then AWS pays the cost of the extra inventory, not you. (To forestall the usual, "overhead" of a Linux OS is like fifty megs these days, it's not something I'd worry about--most of the folks I know who have gone down the container-fleet road have bins that are consistently around 20% empty, and that does add up.)
You may be the one percent of companies for whom immediate rollout, rather than 200-second rollout, is important, and for those companies a solution like Kubernetes or Mesos can make a lot of sense. Most aren't, and I think that they would be better served, in most cases, with a CloudFormation template, an autoscaling group with a cloud-init script to launch one container (if not chef-zero or whatever, that's my go-to but I'm also a devops guy by trade), and a Route 53 record.
You're basically paying overhead for the privilege of `kubectl` that, personally, I don't think is really that useful in a cloud environment. (I think it makes a lot of sense on-prem, where you've already bought the hardware and the alternatives are something like vSphere or the ongoing tire fire that is OpenStack.)
- andrewstuart2 9y agoI know you're answering the question of bin-packing, but after two years of experience with it, I can say that for me, bin-packing is one of the smallest benefits (though it sells very well with management), though perhaps a baseline requirement these days. The real benefits, in my experience, stem from the declarative nature of cluster management, and the automation of doing the sensible thing to enact changes to that declarative desired state.
- eropple 9y agoSure. CloudFormation exists for that, though, and both its difficulty and its complexity are way overstated while also letting you manage AWS resources on top of that. And it doesn't cost anything to use.
- andrewstuart2 9y agoEh, there are a lot of terrible things I'd rather put myself through than writing another CloudFormation template for any sort of complex infrastructure. It could have been made easier and more readable if my company had allowed the use of something like Monsanto's generator [1], but creating ASTs in JSON is not my idea of a good user experience. [1] https://github.com/MonsantoCo/cloudformation-template-generator https://github.com/MonsantoCo/cloudformation-template-genera...
- eropple 9y agoI maintain auster[1] and am a contributor to cfer[2] for exactly that purpose. ;) CloudFormation really isn't a rough time anymore, IMO. [1] - https://github.com/eropple/auster https://github.com/eropple/auster [2] - https://github.com/seanedwards/cfer https://github.com/seanedwards/cfer
- deleted 9y ago[deleted]
- dvlsg 9y agoIf you know those tools exist, maybe. I just put together a new project using cloudformation (technically serverless, but it turned into 90 percent cloudformation syntax anyways), and it was pretty rough.
- eropple 9y agoMaybe it's just me, but as a programmer the first thing I ever asked when looking at the wall of CloudFormation JSON was "so how do we make this not suck?". Our job is not just to automate servers, it's to automate processes, including stupid developer-facing ones.
- ttul 9y agoImmutability, in other words.