17 ms·

We don’t use Kubernetes

338 points by rocketpastsix 5y ago

100011_100001 5y ago
Using Docker on EC2 but not Kubernetes is like using a car but deciding you will never use the gas or break pedals. At that point might as well walk (simpler) or fly (use Kubernetes). They have essentially semi mocked Kubernetes without any of the benefits.
- cpach 5y ago
  I’m not sure I follow. Kubernetes is “just” an orchestrator for handling containers. This company use Docker Engine directly for handling their containers. They still get the benefit of containers even if they don’t happen to use Kubernetes. What’s wrong with that?
  - pm90 5y ago
    Docker Engine is not a scheduler. It’s a container runtime. It doesn’t have visibility over all your VMs, networking etc.
    - cpach 5y ago
      Indeed. But in the article he mentioned that they have other tools for doing that kind of stuff.
  - Fiahil 5y ago
    Kubernetes is not for orchestrating container workloads, it's for making orchestrated containers __portable__.
    - cpach 5y ago
      Portable between what?
- mrkurt 5y ago
  That's not really true. Running a service and orchestrating a fleet of services are two entirely separate problems. The orchestration requirements for many workloads are different than what k8s provides. If your orchestration needs aren't "use every last % of server capacity" you might not need k8s.
- darkwater 5y ago
  As someone working in a place where we did the same (and it was mainly my idea), I beg to differ. You don't have to learn/cope with all the new low level k8s plumbing, especially pre-EKS. Just plain old EC2, in an immutable fashion, just using docker as your package manager instead of baking a RPM or DEB. ANd even with the advent of EKS in AWS there are still a lot of overlapping functionalities and new failure modes (HPA and its metrics/limited rules WRT EC2 ASG rules, coreDNS etc).
- follownotrend 5y ago
  I might use a different analogy -- it might be more like using a bike versus a motorcycle. You can run a delivery business on a bike, but you'd be able to do it with much less tinkering, if you were using a motorcycle.
- pacofvf 5y ago
  They are using cloudformation which predates Kubernetes and it's meant to solve the same problem. CloudFormation still uses docker, originally it used AMIs. I've worked for a lot of Fortune 500 companies that don't use Kubernetes and use Docker.
- durnygbur 5y ago
  I'll take a walk most of the time (virtual instance provisioned with shell script). Flying yes, but only with pilots and the crew - no way in hell I'll start tinkering with it on my own as a sidetask.
voidfunc 5y ago
> We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason! This is not too surprising. Candidates want to join companies that are perceived to be hip and with it technology-wise in order to further their own resume.
- toomuchtodo 5y ago
  You want your skills to be transferrable (unsolicited career advice). Someone isn't going to work on your bespoke infra management system for below or at market rate when those skills won't transfer to another org (and they'll be behind the curve compared to others who have been "in the trenches" with k8s during the same window of time). Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically). Personally, you don't have to be hip, but I want to be able to have another job in hand based on the tech I was working with and the work I was working on in a matter of days when I decide to bounce (or the decision is made for me). This is just good risk and career management. (disclosure: infra roles previously)
  - Sevii 5y ago
    There is also the quality of work factor. What is easier to use a bespoke infra or kubernetes? For the same pay I'd rather work with k8s than deal with whatever gremlins are buried in 'custom' infra.
  - shadowgovt 5y ago
    > Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically). You left out "Their bespoke infra work tends to be the genesis of the commodity infrastructure other people are using." It's not hard for a Googler familiar with Borg to pick up Kubernetes. It's not hard for a Facebooker with some UI work under their belt to figure out React.
- jscheel 5y ago
  Counter-point, these candidates foresee constant frustration and struggles because the company has a culture of NIH. Not saying that's actually the case, but since their website is down, it's kinda hard to tell.
mjlee 5y ago
This page is returning a 500 for me. Perhaps they should use Kubernetes.
- chrisandchris 5y ago
  Probably a classic Hug of Death caused by being #1 on HN.
  - intev 5y ago
    I'm actually curious as to how big of an issue this actually is. HN is a relatively niche site. Do we actually have the power to take down a blog of a tech company?
    - deleted 5y ago
      [deleted]
    - codetrotter 5y ago
      Happens regularly. I am surprised you haven’t seen it happen before, given that you’ve been on HN since at least 2011.
      - intev 5y ago
        I moved and live in a different TZ now. Not in the American TZ. That's probably why.
  - IceWreck 5y ago
    Even a $5 VPS can easily handle like 10k page views per minute if you use a performant web server i.e. not PHP or Ruby. Or even those with caching.
    - _joel 5y ago
      Caching is the key here, it could still be making database requests which are blocked, unless it's using async.
deleted 5y ago
[deleted]
trhoad 5y ago
Site down. Perhaps they forgot to set autoscaling on the pods.
- conjectures 5y ago
  Read this as 'sit down'.
ep103 5y ago
Employers: We are looking for an applicant with 3/5/7/10/20 years of experience in the following technologies. Also Employers: Some of our applicants turned down our job offer, when we revealed that we don't use specific technologies!
imglorp 5y ago
https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...
pm90 5y ago
> This has been asked by current and potential customers, by developers interested in our platform, and by candidates interviewing for roles at Ably. We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason! I celebrate a diversity in opinion on infrastructure but… if I was a CTO/VP of engineering and I read that line, that would be enough to convince me to use kubernetes.
- nitrogen 5y ago
  Someone walking away from an offer because the company doesn't use unnecessary tool X is potentially a sign of a flaky engineer. I'd hesitate before working at any company with fewer than 100 engineers that does use k8s, because it's way more complexity and make-work than is appropriate for small services and teams.
- detaro 5y ago
  Would you use the same reasoning to adopt Java/Cobol/NodeJS/...?
andreineculau 5y ago
I'm all for not using Kubernetes (or any other tech) just because, but seeing their website giving 500.. I can't help but feel all the k8s' laughter. :( Google Cache doesn't work either https://webcache.googleusercontent.com/search?q=cache:YECd_IyHLZsJ:https://ably.com/blog/no-we-dont-use-kubernetes+&cd=1&hl=en&ct=clnk&gl=se&client=safari https://webcache.googleusercontent.com/search?q=cache:YECd_I... Luckily there's an Internet Archive https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...
- alanwreath 5y ago
  I think the laughter is mostly just fun. I don't think anyone laughing believes K8s would have auto prevented the issue. In fact there is probably a growing problem that the ways this could have been solved in Kubernetes is slowly approaching the number of ways that it could have been solved on good ol' baremetal/ec2's. Kubernetes != non-bespoke system. I like the fact, however, that there do at least exist standard primitives in Kubernetes that could approach horizontal scaling (if that was the issue as is insinuated by the aforementioned laughing) since the best thing Kubernetes does for me is helping me keep my cloud vendor (aws,gcp,azure,etc) knowledge shallow for typical web hosting solutions.
pictur 5y ago
The piss races like we use x or we don't use y in engineering sound hilarious for some reason. such discussions focus on the process, but few talk about the results ¯\_(ツ)_/¯
imstil3earning 5y ago
> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs
- cassianoleal 5y ago
  I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.
  - RamRodification 5y ago
    Do/Did you use some specific method or orchestration product to set it up?
    - Sahbak 5y ago
      Not OP, but similar workloads, team of 5 We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.
    - cassianoleal 5y ago
      We use GKE but there's nothing stopping most organisations to do the same. It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect. I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.
- theptip 5y ago
  This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster. Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.
smashed 5y ago
To others wondering what this company does: > Ably is a Pub/Sub messaging platform that companies can use to develop realtime features in their products. At this time, https://status.ably.com/ https://status.ably.com/ is reporting all green. Althought their entire website is returning 500 errors, including the blog. It is very hard not to point the irony of the situation. In general I would not be so critic, but this is a company claiming to run highly available, mission critical distributed computing systems. Yet, they publish a popular blog article and it brings down their enire web presence?
- konschubert 5y ago
  They're not selling a blogging platform, as long as pub/sub works it's fair to say that they're up.
  - rvz 5y ago
    Well said. Their main service and offering is still up. They are not hosting blogs or websites as their business. This reaction is a storm in a tea cup left inside of a sunny cottage. Downvoters: I don't see anyone screaming at this stock trading app that also stopped using Kubernetes [0]. [0] https://freetrade.io/blog/killing-kubernetes https://freetrade.io/blog/killing-kubernetes
  - mountainriver 5y ago
    This is the face of your company, if you can’t handle the increased load on your blog why should I trust your other systems. I run on k8s and it works really well
    - Nicksil 5y ago
      >This is the face of your company, if you can’t handle the increased load on your blog why should I trust your other systems. I suspect the sort of individual visiting this website and reading the blog will know that their services and blog do not run on the same machines.
JediPig 5y ago
typical not invented here syndrome. seems like a me too, i know more than you type of project. looks geared towards cryptos though.
- spmurrayzzz 5y ago
  is it really NIH if they made no attempts to build anything resembling a k8s clone themselves? They decided container orchestration is just not a major problem for them. Its a completely different model, they're embracing immutable infrastructure principles instead of the k8s model (container versions in their stack are bound to the lifetime of the ec2 instance they're on). This reads as simply a choice to have less complexity in their stack, which I think is admirable.
  - pilotpilot 5y ago
    If only everybody else had actually read the post. It's how I read it too. There's no need for k8s as far as I can see.
gtirloni 5y ago
Count how many times the words "custom $something" are mentioned in this article and you have a pretty strong case for using Kubernetes.
- advisedwang 5y ago
  Three*? That doesn't seem that much of an indictment * Well four, but one of the mentions of "custom x" is talking about k8s way of doing things.
lallysingh 5y ago
Ok, where to start... > Packing servers has the minor advantage of using spare resources on existing machines instead of additional machines for small-footprint services. It also has the major disadvantage of running heterogeneous services on the same machine, competing for resources. ... Have a look at your CPU/MEM resource distributions, specifically the tails. That 'spare' resource is often 25-50% of resource used for the last 5% of usage. Cost optimization on the cloud is a matter of raising utilization. Have a look at your pods' use covariance and you can find populations to stochastically 'take turns' on that extra CPU/RAM. > One possible approach is to attempt to preserve the “one VM, one service” model while using Kubernetes. The Kubernetes minions don’t have to be identical, they can be virtual machines of different sizes, and Kubernetes scheduling constraints can be used to run exactly one logical service on each minion. This raises the question, though: if you are running fixed sets of containers on specific groups of EC2 instances, why do you have a Kubernetes layer in there instead of just doing that? The real reason is your AWS bill. Remember that splitting up a large .metal into smaller VMs means that you're paying the CPU/RAM bill for a kernel + basic services multiple times for the same motherboard. Static allocation is inefficient when exposed to load variance. Allocating small VMs to reduce the sizes of your static allocations costs a lot more overhead than tuning your pod requests and scheduling prefs. Think of it like trucks for transporting packages. Yes you can pay AWS to rent you just the right truck, in the right number for each package you want to carry. Or you can just rent big-rigs and carry many, many packages. You'll have to figure out how to pack them in the trailer, and to make sure they survive the vibration of the trip, but you will almost certainly save money. EDIT: Formatting
- tjungblut 5y ago
  it's absolutely insane, they claim several thousands (!) of VMs and each autoscaling group having their own load balancer. Just the cost worth saving when they have only 10-20% wasted resource and an ingress controller instead of multiple hundred ELBs pays for several years of development (+ headcount to maintain).
- ec109685 5y ago
  Great point. By packing heterogeneous workloads on the same underlying VM, you can amortize the spare capacity against non-correlated workloads versus the one workload per VM they are pushing.
dadro 5y ago
We use AWS ElasticBeanstalk with Docker images for our app infrastructure and it has served us well. I think it ends up being similar in that it pushes docker images to EC2 containers. It may not be cutting edge but affords us all the conveniences of docker images for deps while not needing the deep knowledge (and team resources) Kubernetes often requires.
- pestkranker 5y ago
  Do you use the "Multi-container Docker running on 64bit Amazon Linux/2.26.2" platform?
- pantulis 5y ago
  Totally agree, it can scale to a lot of requests until you need to level up to container orchestration, but to be fair it is not competing on the same space as K8s.
codetrotter 5y ago
Off topic but: The chat icon obscures the x to close the cookie banner. Safari on iPhone X.
hkt 5y ago
Since it is down, wayback machine: https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...
pantulis 5y ago
I'm honestly curious if this is a prank from ably.com or what the content of the page was.
- pantulis 5y ago
  Well it seems a SEO farm caught their content before going down. https://hispanicbusinesstv.com/no-we-dont-use-kubernetes/ https://hispanicbusinesstv.com/no-we-dont-use-kubernetes/
mrweasel 5y ago
Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't. This is one of the first cases where I think that maybe Kubernetes would be the right solution, and it's an article about not using it. While there's a lot of information in the article, there might be some underlaying reason why this isn't a good fit for Kubernetes. One thing that is highlighted very well is that fact that Kubernetes is pretty much just viewed as orchestration now. It's no longer amount utilizing your hardware better (in fact it uses more hardware in a many cases).
- ojhughes 5y ago
  > Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't. I would have agreed with this statement 2 years ago but now I think K8s has been commoditized to the point where it makes sense for many organisations. The problem I have seen in the past is that every team rolls their own orchestration using something like Ansible, at least K8s brings some consistency across teams
  - mrweasel 5y ago
    > K8s has been commoditized to the point where it makes sense for many organisations Absolutely, but we still need to lose the control plane for it to make sense for small scale systems. Most of my customers run on something like two VMs, and that's only for redundancy. Asking them to pay for an additional three VMs for a control plane isn't feasible. I think we need something like kubectl, but which works directly on a VM.
    - oblio 5y ago
      A lot of this new cloud stuff doesn't really scale down, it reminds me of Oracle, back in the day. I had something like a desktop PC with 1GB of RAM, maybe 20 years ago, don't remember exactly how long ago. It was average, not too much but not too low either. Once I installed Oracle DB, it was idling at 512MB. Absolutely no DBs and no data set up, just the DB engine, idling at half the RAM of an average desktop PC.
orf 5y ago
"No, we don’t use Kubernetes - because we bodged together a homegrown version ourselves for some reason" Jokes aside, it sounds like they should just use ECS instead.
- throwaway894345 5y ago
  Agreed. I worked on a team that built out a solution like Ably described and we would run into lots of weird issues with ec2 lifecycle (often init script / configuration management issues) and deploys would take a long time (had to drain ec2 instances and bring new instances up) and there's just a lot more to manage (autoscaling groups, instance profiles, AMIs, configuration management, process management, log exfiltration, SSH keys, infra as code, etc). If you really don't want to use Kubernetes, you can get a lot of the benefits by using ECS/Fargate, but I really don't know why you wouldn't just go all-in on EKS at that point.
- nunez 5y ago
  They aren't using containers, though. They're using pure EC2 _but_ are using Docker images as the deployment artifact.
  - orf 5y ago
    That's.... containers. To quote the article: > A small custom boot service on each instance that is part of our boot image looks at the instance configuration, pulls the right container images, and starts the containers. > There are lightweight monitoring services on each instance that will respawn a required container if it dies, and self-terminate the instance if it is running a version of any software that is no longer the preferred version for that cluster. They've built a poor-mans Kubernetes that they won't be able to hire talent for, scales slower and costs more.
    - nunez 5y ago
      I didn't think they were using containerd; this phrase made me think that: > Functionally, we still do the same, as Docker images are just a bunch of tarballs bundled with a metadata JSON blob, but curl and tar have been replaced by docker pull. but, yes, I agree; it's a hand-made Kubernetes
andrewmcwatters 5y ago
> if you are still deploying to plain EC2 instances, you might as well be feeding punch cards. This type of language isn't something that should come out of a company, and may be a signal that there are other reasons developers refused to offer their services other than they just don't use K8s.
Ensorceled 5y ago
ably ceo: Our website is throwing 500 errors!!! ably cto: Go talk to the CMO, I can't help. ably ceo: What!! ably cto: Remember when you said the website was "Totally under the control of the CMO" and "I should mind my own business"? Well I don't even have a Wordpress login. I literally can't help.
kgraves 5y ago
I find it awfully sad and depressing how people on the internet pile onto a company that tells an opposing view without reading or discussing further into their choices. Instead it is easier to to critique the low hanging fruit point rather than discussing their actual reason for not using this 'kubernetes' software. So is their blog the main product? If not, then the 'their blog gone down lol' quips are irrelevant. I found their post rather interesting and didn't suffer any issues the rest of the 'commenters' are facing.
- deleted 5y ago
  [deleted]
ameyv 5y ago
I'm accessing this website/blog from southeast asia. Its working perfectly. Ably using ghost blogging platform (https://ghost.ably.com https://ghost.ably.com) seeing requests in network console.
ojhughes 5y ago
The risk with using Docker in production is that you can end up building your own bad version of Kubernetes over time. K8s is fairly complex but it solves a lot of useful problems such as zero downtime upgrades, service discovery, declarative config etc
- zzbzq 5y ago
  I was a big advocate for Docker-as-deployment-packaging for a long time, and cautious about Kubernetes. I imagined building something like Ably describes would be the most practical. I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging. If you're going to have infrastructure guys at all, might as well have them maintain k8s.
- dolni 5y ago
  You don't build your own Kubernetes, though. You run EC2 instances that are sized for the container they run and let autoscaling take care of it. Load balancers distribute the load. Kubernetes makes use of a lot of the same stuff, but with even more complexity. Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
  - gizdan 5y ago
    > Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide. Sure it's not needed! But cloud providers aren't needed in much the same way.
    - dolni 5y ago
      Uh... what? Apples and oranges. Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider. That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me. edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.
humbleMouse 5y ago
If people put as much effort into learning Kubernetes as they did into writing blog posts about “how complex” it is...... we’d all be better off.
Notanothertoo 5y ago
This reads as they don't have any experience with it and decided to roll their own. I had no experience with kube 3 months ago, we had rancher 1 cluster with 30-50 services, and I just migrated it, just me. Ended up on eks with CNI (pod networking) - Using the lb controller with ip target types and a targetgroupbinding for ingress and its great. Each pod gets its own secondary ip on the ec2 instance automatically. I'm deploying rancher as a management ui. I also now have a k3s cluster at home. The learning curve was insane, and I hated it all for about 8 weeks but then it all just clicked and it's working great. The arrogance to roll your own without assessing the standard fully speaks volumes. Candidates figured that out and saw the red flag.. Writing your own image bootstrapper... What about all the other features, plus the community and things h things like helm charts.
nunez 5y ago
I think many people new to Kubernetes get intimidated by its perceived complexity. It has so many resources, huge manifests, and a billion tools, with more coming online by the day. I was a huge Kubernetes hater for a while because of this, but I grew to love it and wouldn't recommend anything else now. I'm saying this because while their architecture seems reasonable, albeit crazy expensive (though I'd say it's small-scale if they use network CIDRs and tags for service discovery), it also seems like they wrote this without even trying to use Kubernetes. If they did, it isn't expressed clearly by this post. For instance, this: > Writing YAML files for Kubernetes is not the only way to manage Infrastructure as Code, and in many cases, not even the most appropriate way. and this: > There is a controller that will automatically create AWS load balancers and point them directly at the right set of pods when an Ingress or Service section is added to the Kubernetes specification for the service. Overall, this would not be more complicated than the way we expose our traffic routing instances now. > The hidden downside here, of course, is that this excellent level of integration is completely AWS-specific. For anyone trying to use Kubernetes as a way to go multi-cloud, it is therefore not very helpful. Sound like theoretical statements rather than ones driven by experience. Few would ever use raw YAMLs to deploy Kubernetes resources. Most would use tools like Helm or Kustomize for this purpose. These tools came online relatively soon after Kubernetes saw growth and are battle-tested. One would also know that while ingress controllers _can_ create cloud-provider-specific networking appliances, swapping them out for other ingress controllers is not only easy to do, but, in many cases, it can be done without affecting other Ingresses (unless they are using controller-specific functionality). I'd also ask them to reconsider is how they are using Docker images as a deployment package. They're using Docker images as a replacement for tarballs. This is evidenced by them using EC2 instances to run their services. I can see how they arrived at this (Docker images are just filesystem layers compressed as a gzipped tarball), but because images were meant to be used by containers, dealing with where Docker puts those images and moving things around must be a challenge. I would encourage them to try running their services on Docker containers. The lift is pretty small, but the amount of portability they can gain is massive. If containers legitimately won't work for them, then they should try something like Ansible for provisioning their machines.
ampdepolymerase 5y ago
They are not in a line of business where they might be deplatformed, shut down, or that AWS exists as a major liability. They also do not require any form of multi cloud redundancy that is not already being served by AWS.
ossusermivami 5y ago
sure you can do homegrown and probably be more efficient and cost savy, but you can't beat the amount of tooling and that new sysops you will have to train to those new tools compared to a pure k8 platform...
phendrenad2 5y ago
Good. Kubernetes is a swiss army knife, and I just need a fork and a spoon. Sure, the swiss army knife comes with a fold-out fork and a spoon, but then I have to build up institutional knowledge around where they are, and how to avoid stabbing myself with the adjacent blades.
deleted 5y ago
[deleted]
0xbadcafebee 5y ago
If you use AWS, just use Fargate. Fargate is the parts of Kubernetes you actually want, without all the unnecessary complexity, and with all the AWS ecosystem as an option. It even autoscales better than Kubernetes. It's cheaper, it's easier, it's better. If you can't use Fargate, use ECS. But for God's sake, don't poorly re-invent your own version of ECS. If you're on AWS, and not using the AWS ecosystem, you're probably wasting time and money. And if you eventually need to use Kubernetes, you can always spin up EKS. Just don't rush to ship on the Ever Given when an 18 wheeler works fine.
aiven 5y ago
"no, we don't use Kubernetes, we are stuck in AWS ecosystem"
pilotpilot 5y ago
Half the comments haven't read the post, and the other half are "should have used kubernetes". Funny isn't it when a company writes a post explaining why they don't use a technology that everyone who loves _said technology_ comes on and tells them why they should.
bdcravens 5y ago
ECS has continued to be great for us. I haven't run Kubernetes in production, but from my perspective, we have everything we would need from K8S with only a fraction of the effort. I've also been able to do some fun things via the AWS API that may have been challenging or even impossible with K8S (again, I may be naive here)
dijit 5y ago
I wanted to refrain from commenting because honestly I’m not the biggest fan of relatively opaque complexity and kubernetes tries its hardest to be this. (Cloud providers doing magic things with annotations for example) But, I have to say that kubernetes is not the devil. Lock-in, is the devil. I recently underwent the task of getting us off of AWS, which was not as painful as it could have been (I talk about it here[0]) But the thing is: I like auto healing, auto scaling and staggered rollouts. I had previously implemented/deployed this all myself using custom C++ code, salt and a lot of python glue. It worked super well but it was also many years of testing and trial and error. Doing all of that again is an insane effort. Kubernetes is 80% of the same stuff if your workload fits in it, but you have to learn the edge cases, which of course increases tremendously from the standard: python, Linux, terraform stuff most operators know. Anyway. I’m not saying go for it. But don’t replace it with lock-in. [0]: https://www.gcppodcast.com/post/episode-265-sharkmob-games-with-jan-harasym/ https://www.gcppodcast.com/post/episode-265-sharkmob-games-w...
- syntaxstic 5y ago
  Why did you leave AWS?
  - dijit 5y ago
    I made a relatively large list of reasons. Most are going to sound fickle but I consider some to be very problematic if you’re woken up at 3am and have to orient yourself- others I consider problematic because they cause an order of magnitude increase in complexity. Mostly it’s an issue of perception too, a cloud saves me time. If it doesn’t save me time it is not worth the premiums and for our case- it would not save time. (Due to the complexity mentioned before). But here’s part of list (with project specific items redacted): 3am topics: * Project name (impossible to see which project you're in, usually it's based on "account" but that gets messed up with SSO) * instance/object names (`i-987348ff`, `eip-7338971`, `sub-87326`) are hard to understand meaning of. * Terminated instances fill UI. * Resources in other regions may as well not exist, they're invisible- sometimes only found after checking the bill for that month. Time cost topics (stumbling things that make things slower): * Placements only supported on certain instances * EBS optimised only supported on certain instances Other: * Launch configurations (user_data) only 16KiB, life-cycling is hard also, user-data is a terrible name. * 58% more objects and relationships (239 -> 378 LoC after terraform graph) * networking model does not make best practice easy (Zonal based network, not regional) * Committed use (vs sustained use) discounts means you have to run cost projections _anyway_ (W.R.T. cost planning on-prem vs cloud) * no such thing as an unmanaged instance group (you need an ASG which can be provisioned exclusively with a user-data (launch script in real terms) * managed to create a VPC where nothing could talk to anything. Even cloud experts couldn't figure it out, not very transparent or easy to debug. Sticky topics (things that make you buy more AWS services or lock-in): * Use Managed ES! -> AWS ES Kibana requires usage of separately billed cognito service if you want SAML SSO. * Number of services brought in for a simple pipeline: https://aws.amazon.com/solutions/implementations/game-analytics-pipeline/ https://aws.amazon.com/solutions/implementations/game-analyt... * Simple things like auto-scaler instances being incrementally named requires lambda: https://stackoverflow.com/questions/53615049/method-to-provide-incremental-name-to-ec2-instances-created-using-auto-scaling-g https://stackoverflow.com/questions/53615049/method-to-provi... * CDK/Cloudformation is the only "simple" way to automatically provision infra.
motoboi 5y ago
"How we made our own container orchestration infrastructure over AWS. It doesn't have all Kubernetes features, but hey! we can then train our own employees on how it works. And we got to maintain the code all by ourselves too! It might take a bit too long to implement a new feature, but hey! its ours!" Really, Kubernetes is complex, but the problem it solves it even more complex. If you are ok solving a part of the problem, nice. You just built a competitor to google. Good luck hiring people who come in already knowing how to operate it. Good luck trying to keep it modern and useful too. But I totally understand the appeal.
flowerlad 5y ago
I wouldn't follow the path that this company took. I am a solo developer, use Kubernetes on Google cloud and I couldn't be happier. Parts of my application runs on AWS, taking advantage of what AWS does better, such as SES (Simple Email Service). All I had to learn is Docker and Kubernetes. If Kubernetes didn't exist I would have had to learn myriad tools and services, cloud-specific tools and services, and my application would be permanently wedded to one cloud. Thanks to Kubernetes my application can be moved to another cloud in whole or in part. Kubernetes is so well designed, it is the kind of thing you learn just because it is well designed. I am glad I invested the time. The knowledge I acquired is both durable and portable.
rcarmo 5y ago
"we would be doing mostly the same things, but in a more complicated way" is a pretty good summary considering their use cases seem to be well covered by autoscaling groups (which, incidentally, are a thing other clouds have) It's OK to not use k8s. We should normalize that.
icythere 5y ago
The hard thing of kubernetes, is that it sounds easy to do something/anything. It does, and because it's "easy" people tend to skip to understand how it's working: That's where the problem occurs. It _forces_ you to become a "yaml engineer" and to forget the other part of the systems. I was interviewed by a company and when I replied the next step I could do was to write some operators for the ops things, they simply rejected because I'm too experienced lolz
- pram 5y ago
  Exactly, now you have people making an EKS cluster and then deploying a vendor supplied Helm chart to it. When something breaks they literally have no idea how to start fixing it. It’s deceptively “easy”
exabrial 5y ago
Scaling is a sexy problem to have. Most places crate problems to solve just for the appeal. They're spending angel investor money, not their own, so prolonging the time to production is in their favor.
blacktriangle 5y ago
Um....are we getting trolled by some content marketing SEO hack here? https://www.sebastianbuza.com/2021/07/20/no-we-dont-use-kubernetes/ https://www.sebastianbuza.com/2021/07/20/no-we-dont-use-kube... Another pub/sub startup publishing almost the identical blog post also today, July 20, 2021.
- tester756 5y ago
  lol, what the hell is this
  - blacktriangle 5y ago
    Given that Ably looks like an actual product, I'm guessing the link I posted is some sketchy content repurposing farm. I don't know what recourse Ably has in this case, but that's unfortunate.
- matt_oriordan 5y ago
  Founder of Ably here That’s odd and clearly a rip off of our post :( Thanks for flagging
benlivengood 5y ago
I think that Kubernetes has advantages for many small services but a few large services are still worth managing directly on bare machines/VMs. Where I disagree with this article is on Kubernetes stability and manageability. The caveat is that GKE is easy to manage and EKS is straightforward but not quite easy. Terraform with a few flags for the google-gke module can manage dozens of clusters with helm_release resources making the clusters production-ready with very little human management overhead. EKS is still manageable but does require a bit more setup per cluster, but it all lives in the automation and can be standardized across clusters. Daily autoscaling is one of those things that some people can get away with, but most won't save money. For example, prices for reservations/commitments are ~65% of on-demand. Can a service really scale so low during off-hours that average utilization from peak machine count is under 35%? If so, then autoscale aggressively and it's totally worth it. Most services I've seen can't actually achieve that and instead would be ~60% utilized over a whole day (mostly global customer bases). The exception is if you can scale (or run entirely with loose enough SLOs) into spot or preemptible instances which should be about as cheap as committed instances at the risk of someday not being available.
knodi 5y ago
That great now live with your bad/good design patterns on your own for the life cycle of your product or service, you get non of the shared knowledge and advancement through the k8s community or platform.
kaydub 5y ago
I don't use K8s either, but we at least use an orchestration tool. ECS Fargate has been awesome. No reason to add the complexity of K8s/EKS. We're all in on AWS and everything works together. But this... you guys re-invented the wheel. You're probably going to find it's not round in certain spots too.
truthwhisperer 5y ago
typical circle of (IT) live. Start your own "solution" end up with a kubernetes sort a like clone which is better understood by you
emodendroket 5y ago
I think the killer feature of Kubernetes is really the infrastructure as code part -- it just makes it very easy to spin services up or down as desired without thinking too hard about it. But as the article alludes to, if you're comfortable with the lock-in, you can get that from your cloud provider with tighter integration.

338 points by rocketpastsix 5y ago

Using Docker on EC2 but not Kubernetes is like using a car but deciding you will never use the gas or break pedals. At that point might as well walk (simpler) or fly (use Kubernetes). They have essentially semi mocked Kubernetes without any of the benefits.

cpach 5y ago
I’m not sure I follow. Kubernetes is “just” an orchestrator for handling containers. This company use Docker Engine directly for handling their containers. They still get the benefit of containers even if they don’t happen to use Kubernetes. What’s wrong with that?
- pm90 5y ago
  Docker Engine is not a scheduler. It’s a container runtime. It doesn’t have visibility over all your VMs, networking etc.
  - cpach 5y ago
    Indeed. But in the article he mentioned that they have other tools for doing that kind of stuff.
- Fiahil 5y ago
  Kubernetes is not for orchestrating container workloads, it's for making orchestrated containers __portable__.
  - cpach 5y ago
    Portable between what?
mrkurt 5y ago
That's not really true. Running a service and orchestrating a fleet of services are two entirely separate problems. The orchestration requirements for many workloads are different than what k8s provides. If your orchestration needs aren't "use every last % of server capacity" you might not need k8s.
darkwater 5y ago
As someone working in a place where we did the same (and it was mainly my idea), I beg to differ. You don't have to learn/cope with all the new low level k8s plumbing, especially pre-EKS. Just plain old EC2, in an immutable fashion, just using docker as your package manager instead of baking a RPM or DEB. ANd even with the advent of EKS in AWS there are still a lot of overlapping functionalities and new failure modes (HPA and its metrics/limited rules WRT EC2 ASG rules, coreDNS etc).
follownotrend 5y ago
I might use a different analogy -- it might be more like using a bike versus a motorcycle. You can run a delivery business on a bike, but you'd be able to do it with much less tinkering, if you were using a motorcycle.
pacofvf 5y ago
They are using cloudformation which predates Kubernetes and it's meant to solve the same problem. CloudFormation still uses docker, originally it used AMIs. I've worked for a lot of Fortune 500 companies that don't use Kubernetes and use Docker.
durnygbur 5y ago
I'll take a walk most of the time (virtual instance provisioned with shell script). Flying yes, but only with pilots and the crew - no way in hell I'll start tinkering with it on my own as a sidetask.

voidfunc 5y ago

> We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason! This is not too surprising. Candidates want to join companies that are perceived to be hip and with it technology-wise in order to further their own resume.

toomuchtodo 5y ago
You want your skills to be transferrable (unsolicited career advice). Someone isn't going to work on your bespoke infra management system for below or at market rate when those skills won't transfer to another org (and they'll be behind the curve compared to others who have been "in the trenches" with k8s during the same window of time). Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically). Personally, you don't have to be hip, but I want to be able to have another job in hand based on the tech I was working with and the work I was working on in a matter of days when I decide to bounce (or the decision is made for me). This is just good risk and career management. (disclosure: infra roles previously)
- Sevii 5y ago
  There is also the quality of work factor. What is easier to use a bespoke infra or kubernetes? For the same pay I'd rather work with k8s than deal with whatever gremlins are buried in 'custom' infra.
- shadowgovt 5y ago
  > Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically). You left out "Their bespoke infra work tends to be the genesis of the commodity infrastructure other people are using." It's not hard for a Googler familiar with Borg to pick up Kubernetes. It's not hard for a Facebooker with some UI work under their belt to figure out React.
jscheel 5y ago
Counter-point, these candidates foresee constant frustration and struggles because the company has a culture of NIH. Not saying that's actually the case, but since their website is down, it's kinda hard to tell.

mjlee 5y ago

This page is returning a 500 for me. Perhaps they should use Kubernetes.

chrisandchris 5y ago
Probably a classic Hug of Death caused by being #1 on HN.
- intev 5y ago
  I'm actually curious as to how big of an issue this actually is. HN is a relatively niche site. Do we actually have the power to take down a blog of a tech company?
  - deleted 5y ago
    [deleted]
  - codetrotter 5y ago
    Happens regularly. I am surprised you haven’t seen it happen before, given that you’ve been on HN since at least 2011.
    - intev 5y ago
      I moved and live in a different TZ now. Not in the American TZ. That's probably why.
- IceWreck 5y ago
  Even a $5 VPS can easily handle like 10k page views per minute if you use a performant web server i.e. not PHP or Ruby. Or even those with caching.
  - _joel 5y ago
    Caching is the key here, it could still be making database requests which are blocked, unless it's using async.

deleted 5y ago

[deleted]

trhoad 5y ago

Site down. Perhaps they forgot to set autoscaling on the pods.

conjectures 5y ago
Read this as 'sit down'.

ep103 5y ago

Employers: We are looking for an applicant with 3/5/7/10/20 years of experience in the following technologies. Also Employers: Some of our applicants turned down our job offer, when we revealed that we don't use specific technologies!

imglorp 5y ago

https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...

pm90 5y ago

> This has been asked by current and potential customers, by developers interested in our platform, and by candidates interviewing for roles at Ably. We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason! I celebrate a diversity in opinion on infrastructure but… if I was a CTO/VP of engineering and I read that line, that would be enough to convince me to use kubernetes.

nitrogen 5y ago
Someone walking away from an offer because the company doesn't use unnecessary tool X is potentially a sign of a flaky engineer. I'd hesitate before working at any company with fewer than 100 engineers that does use k8s, because it's way more complexity and make-work than is appropriate for small services and teams.
detaro 5y ago
Would you use the same reasoning to adopt Java/Cobol/NodeJS/...?

andreineculau 5y ago

I'm all for not using Kubernetes (or any other tech) just because, but seeing their website giving 500.. I can't help but feel all the k8s' laughter. :( Google Cache doesn't work either https://webcache.googleusercontent.com/search?q=cache:YECd_IyHLZsJ:https://ably.com/blog/no-we-dont-use-kubernetes+&cd=1&hl=en&ct=clnk&gl=se&client=safari https://webcache.googleusercontent.com/search?q=cache:YECd_I... Luckily there's an Internet Archive https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...

alanwreath 5y ago
I think the laughter is mostly just fun. I don't think anyone laughing believes K8s would have auto prevented the issue. In fact there is probably a growing problem that the ways this could have been solved in Kubernetes is slowly approaching the number of ways that it could have been solved on good ol' baremetal/ec2's. Kubernetes != non-bespoke system. I like the fact, however, that there do at least exist standard primitives in Kubernetes that could approach horizontal scaling (if that was the issue as is insinuated by the aforementioned laughing) since the best thing Kubernetes does for me is helping me keep my cloud vendor (aws,gcp,azure,etc) knowledge shallow for typical web hosting solutions.

pictur 5y ago

The piss races like we use x or we don't use y in engineering sound hilarious for some reason. such discussions focus on the process, but few talk about the results ¯\_(ツ)_/¯

imstil3earning 5y ago

> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs

cassianoleal 5y ago
I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.
- RamRodification 5y ago
  Do/Did you use some specific method or orchestration product to set it up?
  - Sahbak 5y ago
    Not OP, but similar workloads, team of 5 We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.
  - cassianoleal 5y ago
    We use GKE but there's nothing stopping most organisations to do the same. It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect. I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.
theptip 5y ago
This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster. Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.

smashed 5y ago

To others wondering what this company does: > Ably is a Pub/Sub messaging platform that companies can use to develop realtime features in their products. At this time, https://status.ably.com/ https://status.ably.com/ is reporting all green. Althought their entire website is returning 500 errors, including the blog. It is very hard not to point the irony of the situation. In general I would not be so critic, but this is a company claiming to run highly available, mission critical distributed computing systems. Yet, they publish a popular blog article and it brings down their enire web presence?

konschubert 5y ago
They're not selling a blogging platform, as long as pub/sub works it's fair to say that they're up.
- rvz 5y ago
  Well said. Their main service and offering is still up. They are not hosting blogs or websites as their business. This reaction is a storm in a tea cup left inside of a sunny cottage. Downvoters: I don't see anyone screaming at this stock trading app that also stopped using Kubernetes [0]. [0] https://freetrade.io/blog/killing-kubernetes https://freetrade.io/blog/killing-kubernetes
- mountainriver 5y ago
  This is the face of your company, if you can’t handle the increased load on your blog why should I trust your other systems. I run on k8s and it works really well
  - Nicksil 5y ago
    >This is the face of your company, if you can’t handle the increased load on your blog why should I trust your other systems. I suspect the sort of individual visiting this website and reading the blog will know that their services and blog do not run on the same machines.

JediPig 5y ago

typical not invented here syndrome. seems like a me too, i know more than you type of project. looks geared towards cryptos though.

spmurrayzzz 5y ago
is it really NIH if they made no attempts to build anything resembling a k8s clone themselves? They decided container orchestration is just not a major problem for them. Its a completely different model, they're embracing immutable infrastructure principles instead of the k8s model (container versions in their stack are bound to the lifetime of the ec2 instance they're on). This reads as simply a choice to have less complexity in their stack, which I think is admirable.
- pilotpilot 5y ago
  If only everybody else had actually read the post. It's how I read it too. There's no need for k8s as far as I can see.

gtirloni 5y ago

Count how many times the words "custom $something" are mentioned in this article and you have a pretty strong case for using Kubernetes.

advisedwang 5y ago
Three*? That doesn't seem that much of an indictment * Well four, but one of the mentions of "custom x" is talking about k8s way of doing things.

lallysingh 5y ago

Ok, where to start... > Packing servers has the minor advantage of using spare resources on existing machines instead of additional machines for small-footprint services. It also has the major disadvantage of running heterogeneous services on the same machine, competing for resources. ... Have a look at your CPU/MEM resource distributions, specifically the tails. That 'spare' resource is often 25-50% of resource used for the last 5% of usage. Cost optimization on the cloud is a matter of raising utilization. Have a look at your pods' use covariance and you can find populations to stochastically 'take turns' on that extra CPU/RAM. > One possible approach is to attempt to preserve the “one VM, one service” model while using Kubernetes. The Kubernetes minions don’t have to be identical, they can be virtual machines of different sizes, and Kubernetes scheduling constraints can be used to run exactly one logical service on each minion. This raises the question, though: if you are running fixed sets of containers on specific groups of EC2 instances, why do you have a Kubernetes layer in there instead of just doing that? The real reason is your AWS bill. Remember that splitting up a large .metal into smaller VMs means that you're paying the CPU/RAM bill for a kernel + basic services multiple times for the same motherboard. Static allocation is inefficient when exposed to load variance. Allocating small VMs to reduce the sizes of your static allocations costs a lot more overhead than tuning your pod requests and scheduling prefs. Think of it like trucks for transporting packages. Yes you can pay AWS to rent you just the right truck, in the right number for each package you want to carry. Or you can just rent big-rigs and carry many, many packages. You'll have to figure out how to pack them in the trailer, and to make sure they survive the vibration of the trip, but you will almost certainly save money. EDIT: Formatting

tjungblut 5y ago
it's absolutely insane, they claim several thousands (!) of VMs and each autoscaling group having their own load balancer. Just the cost worth saving when they have only 10-20% wasted resource and an ingress controller instead of multiple hundred ELBs pays for several years of development (+ headcount to maintain).
ec109685 5y ago
Great point. By packing heterogeneous workloads on the same underlying VM, you can amortize the spare capacity against non-correlated workloads versus the one workload per VM they are pushing.

dadro 5y ago

We use AWS ElasticBeanstalk with Docker images for our app infrastructure and it has served us well. I think it ends up being similar in that it pushes docker images to EC2 containers. It may not be cutting edge but affords us all the conveniences of docker images for deps while not needing the deep knowledge (and team resources) Kubernetes often requires.

pestkranker 5y ago
Do you use the "Multi-container Docker running on 64bit Amazon Linux/2.26.2" platform?
pantulis 5y ago
Totally agree, it can scale to a lot of requests until you need to level up to container orchestration, but to be fair it is not competing on the same space as K8s.

codetrotter 5y ago

Off topic but: The chat icon obscures the x to close the cookie banner. Safari on iPhone X.

hkt 5y ago

Since it is down, wayback machine: https://web.archive.org/web/20210720134229/https://ably.com/blog/no-we-dont-use-kubernetes https://web.archive.org/web/20210720134229/https://ably.com/...

pantulis 5y ago

I'm honestly curious if this is a prank from ably.com or what the content of the page was.

pantulis 5y ago
Well it seems a SEO farm caught their content before going down. https://hispanicbusinesstv.com/no-we-dont-use-kubernetes/ https://hispanicbusinesstv.com/no-we-dont-use-kubernetes/

mrweasel 5y ago

Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't. This is one of the first cases where I think that maybe Kubernetes would be the right solution, and it's an article about not using it. While there's a lot of information in the article, there might be some underlaying reason why this isn't a good fit for Kubernetes. One thing that is highlighted very well is that fact that Kubernetes is pretty much just viewed as orchestration now. It's no longer amount utilizing your hardware better (in fact it uses more hardware in a many cases).

ojhughes 5y ago
> Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't. I would have agreed with this statement 2 years ago but now I think K8s has been commoditized to the point where it makes sense for many organisations. The problem I have seen in the past is that every team rolls their own orchestration using something like Ansible, at least K8s brings some consistency across teams
- mrweasel 5y ago
  > K8s has been commoditized to the point where it makes sense for many organisations Absolutely, but we still need to lose the control plane for it to make sense for small scale systems. Most of my customers run on something like two VMs, and that's only for redundancy. Asking them to pay for an additional three VMs for a control plane isn't feasible. I think we need something like kubectl, but which works directly on a VM.
  - oblio 5y ago
    A lot of this new cloud stuff doesn't really scale down, it reminds me of Oracle, back in the day. I had something like a desktop PC with 1GB of RAM, maybe 20 years ago, don't remember exactly how long ago. It was average, not too much but not too low either. Once I installed Oracle DB, it was idling at 512MB. Absolutely no DBs and no data set up, just the DB engine, idling at half the RAM of an average desktop PC.

orf 5y ago

"No, we don’t use Kubernetes - because we bodged together a homegrown version ourselves for some reason" Jokes aside, it sounds like they should just use ECS instead.

throwaway894345 5y ago
Agreed. I worked on a team that built out a solution like Ably described and we would run into lots of weird issues with ec2 lifecycle (often init script / configuration management issues) and deploys would take a long time (had to drain ec2 instances and bring new instances up) and there's just a lot more to manage (autoscaling groups, instance profiles, AMIs, configuration management, process management, log exfiltration, SSH keys, infra as code, etc). If you really don't want to use Kubernetes, you can get a lot of the benefits by using ECS/Fargate, but I really don't know why you wouldn't just go all-in on EKS at that point.
nunez 5y ago
They aren't using containers, though. They're using pure EC2 _but_ are using Docker images as the deployment artifact.
- orf 5y ago
  That's.... containers. To quote the article: > A small custom boot service on each instance that is part of our boot image looks at the instance configuration, pulls the right container images, and starts the containers. > There are lightweight monitoring services on each instance that will respawn a required container if it dies, and self-terminate the instance if it is running a version of any software that is no longer the preferred version for that cluster. They've built a poor-mans Kubernetes that they won't be able to hire talent for, scales slower and costs more.
  - nunez 5y ago
    I didn't think they were using containerd; this phrase made me think that: > Functionally, we still do the same, as Docker images are just a bunch of tarballs bundled with a metadata JSON blob, but curl and tar have been replaced by docker pull. but, yes, I agree; it's a hand-made Kubernetes

andrewmcwatters 5y ago

> if you are still deploying to plain EC2 instances, you might as well be feeding punch cards. This type of language isn't something that should come out of a company, and may be a signal that there are other reasons developers refused to offer their services other than they just don't use K8s.

Ensorceled 5y ago

ably ceo: Our website is throwing 500 errors!!! ably cto: Go talk to the CMO, I can't help. ably ceo: What!! ably cto: Remember when you said the website was "Totally under the control of the CMO" and "I should mind my own business"? Well I don't even have a Wordpress login. I literally can't help.

kgraves 5y ago

I find it awfully sad and depressing how people on the internet pile onto a company that tells an opposing view without reading or discussing further into their choices. Instead it is easier to to critique the low hanging fruit point rather than discussing their actual reason for not using this 'kubernetes' software. So is their blog the main product? If not, then the 'their blog gone down lol' quips are irrelevant. I found their post rather interesting and didn't suffer any issues the rest of the 'commenters' are facing.

deleted 5y ago
[deleted]

ameyv 5y ago

I'm accessing this website/blog from southeast asia. Its working perfectly. Ably using ghost blogging platform (https://ghost.ably.com https://ghost.ably.com) seeing requests in network console.

ojhughes 5y ago

The risk with using Docker in production is that you can end up building your own bad version of Kubernetes over time. K8s is fairly complex but it solves a lot of useful problems such as zero downtime upgrades, service discovery, declarative config etc

zzbzq 5y ago
I was a big advocate for Docker-as-deployment-packaging for a long time, and cautious about Kubernetes. I imagined building something like Ably describes would be the most practical. I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging. If you're going to have infrastructure guys at all, might as well have them maintain k8s.
dolni 5y ago
You don't build your own Kubernetes, though. You run EC2 instances that are sized for the container they run and let autoscaling take care of it. Load balancers distribute the load. Kubernetes makes use of a lot of the same stuff, but with even more complexity. Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
- gizdan 5y ago
  > Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide. Sure it's not needed! But cloud providers aren't needed in much the same way.
  - dolni 5y ago
    Uh... what? Apples and oranges. Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider. That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me. edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.

humbleMouse 5y ago

If people put as much effort into learning Kubernetes as they did into writing blog posts about “how complex” it is...... we’d all be better off.

Notanothertoo 5y ago

This reads as they don't have any experience with it and decided to roll their own. I had no experience with kube 3 months ago, we had rancher 1 cluster with 30-50 services, and I just migrated it, just me. Ended up on eks with CNI (pod networking) - Using the lb controller with ip target types and a targetgroupbinding for ingress and its great. Each pod gets its own secondary ip on the ec2 instance automatically. I'm deploying rancher as a management ui. I also now have a k3s cluster at home. The learning curve was insane, and I hated it all for about 8 weeks but then it all just clicked and it's working great. The arrogance to roll your own without assessing the standard fully speaks volumes. Candidates figured that out and saw the red flag.. Writing your own image bootstrapper... What about all the other features, plus the community and things h things like helm charts.

nunez 5y ago

I think many people new to Kubernetes get intimidated by its perceived complexity. It has so many resources, huge manifests, and a billion tools, with more coming online by the day. I was a huge Kubernetes hater for a while because of this, but I grew to love it and wouldn't recommend anything else now. I'm saying this because while their architecture seems reasonable, albeit crazy expensive (though I'd say it's small-scale if they use network CIDRs and tags for service discovery), it also seems like they wrote this without even trying to use Kubernetes. If they did, it isn't expressed clearly by this post. For instance, this: > Writing YAML files for Kubernetes is not the only way to manage Infrastructure as Code, and in many cases, not even the most appropriate way. and this: > There is a controller that will automatically create AWS load balancers and point them directly at the right set of pods when an Ingress or Service section is added to the Kubernetes specification for the service. Overall, this would not be more complicated than the way we expose our traffic routing instances now. > The hidden downside here, of course, is that this excellent level of integration is completely AWS-specific. For anyone trying to use Kubernetes as a way to go multi-cloud, it is therefore not very helpful. Sound like theoretical statements rather than ones driven by experience. Few would ever use raw YAMLs to deploy Kubernetes resources. Most would use tools like Helm or Kustomize for this purpose. These tools came online relatively soon after Kubernetes saw growth and are battle-tested. One would also know that while ingress controllers _can_ create cloud-provider-specific networking appliances, swapping them out for other ingress controllers is not only easy to do, but, in many cases, it can be done without affecting other Ingresses (unless they are using controller-specific functionality). I'd also ask them to reconsider is how they are using Docker images as a deployment package. They're using Docker images as a replacement for tarballs. This is evidenced by them using EC2 instances to run their services. I can see how they arrived at this (Docker images are just filesystem layers compressed as a gzipped tarball), but because images were meant to be used by containers, dealing with where Docker puts those images and moving things around must be a challenge. I would encourage them to try running their services on Docker containers. The lift is pretty small, but the amount of portability they can gain is massive. If containers legitimately won't work for them, then they should try something like Ansible for provisioning their machines.

ampdepolymerase 5y ago

They are not in a line of business where they might be deplatformed, shut down, or that AWS exists as a major liability. They also do not require any form of multi cloud redundancy that is not already being served by AWS.

ossusermivami 5y ago

sure you can do homegrown and probably be more efficient and cost savy, but you can't beat the amount of tooling and that new sysops you will have to train to those new tools compared to a pure k8 platform...

phendrenad2 5y ago

Good. Kubernetes is a swiss army knife, and I just need a fork and a spoon. Sure, the swiss army knife comes with a fold-out fork and a spoon, but then I have to build up institutional knowledge around where they are, and how to avoid stabbing myself with the adjacent blades.

deleted 5y ago

[deleted]

0xbadcafebee 5y ago

If you use AWS, just use Fargate. Fargate is the parts of Kubernetes you actually want, without all the unnecessary complexity, and with all the AWS ecosystem as an option. It even autoscales better than Kubernetes. It's cheaper, it's easier, it's better. If you can't use Fargate, use ECS. But for God's sake, don't poorly re-invent your own version of ECS. If you're on AWS, and not using the AWS ecosystem, you're probably wasting time and money. And if you eventually need to use Kubernetes, you can always spin up EKS. Just don't rush to ship on the Ever Given when an 18 wheeler works fine.

aiven 5y ago

"no, we don't use Kubernetes, we are stuck in AWS ecosystem"

pilotpilot 5y ago

Half the comments haven't read the post, and the other half are "should have used kubernetes". Funny isn't it when a company writes a post explaining why they don't use a technology that everyone who loves _said technology_ comes on and tells them why they should.

bdcravens 5y ago

ECS has continued to be great for us. I haven't run Kubernetes in production, but from my perspective, we have everything we would need from K8S with only a fraction of the effort. I've also been able to do some fun things via the AWS API that may have been challenging or even impossible with K8S (again, I may be naive here)

dijit 5y ago

I wanted to refrain from commenting because honestly I’m not the biggest fan of relatively opaque complexity and kubernetes tries its hardest to be this. (Cloud providers doing magic things with annotations for example) But, I have to say that kubernetes is not the devil. Lock-in, is the devil. I recently underwent the task of getting us off of AWS, which was not as painful as it could have been (I talk about it here[0]) But the thing is: I like auto healing, auto scaling and staggered rollouts. I had previously implemented/deployed this all myself using custom C++ code, salt and a lot of python glue. It worked super well but it was also many years of testing and trial and error. Doing all of that again is an insane effort. Kubernetes is 80% of the same stuff if your workload fits in it, but you have to learn the edge cases, which of course increases tremendously from the standard: python, Linux, terraform stuff most operators know. Anyway. I’m not saying go for it. But don’t replace it with lock-in. [0]: https://www.gcppodcast.com/post/episode-265-sharkmob-games-with-jan-harasym/ https://www.gcppodcast.com/post/episode-265-sharkmob-games-w...

syntaxstic 5y ago
Why did you leave AWS?
- dijit 5y ago
  I made a relatively large list of reasons. Most are going to sound fickle but I consider some to be very problematic if you’re woken up at 3am and have to orient yourself- others I consider problematic because they cause an order of magnitude increase in complexity. Mostly it’s an issue of perception too, a cloud saves me time. If it doesn’t save me time it is not worth the premiums and for our case- it would not save time. (Due to the complexity mentioned before). But here’s part of list (with project specific items redacted): 3am topics: * Project name (impossible to see which project you're in, usually it's based on "account" but that gets messed up with SSO) * instance/object names (`i-987348ff`, `eip-7338971`, `sub-87326`) are hard to understand meaning of. * Terminated instances fill UI. * Resources in other regions may as well not exist, they're invisible- sometimes only found after checking the bill for that month. Time cost topics (stumbling things that make things slower): * Placements only supported on certain instances * EBS optimised only supported on certain instances Other: * Launch configurations (user_data) only 16KiB, life-cycling is hard also, user-data is a terrible name. * 58% more objects and relationships (239 -> 378 LoC after terraform graph) * networking model does not make best practice easy (Zonal based network, not regional) * Committed use (vs sustained use) discounts means you have to run cost projections _anyway_ (W.R.T. cost planning on-prem vs cloud) * no such thing as an unmanaged instance group (you need an ASG which can be provisioned exclusively with a user-data (launch script in real terms) * managed to create a VPC where nothing could talk to anything. Even cloud experts couldn't figure it out, not very transparent or easy to debug. Sticky topics (things that make you buy more AWS services or lock-in): * Use Managed ES! -> AWS ES Kibana requires usage of separately billed cognito service if you want SAML SSO. * Number of services brought in for a simple pipeline: https://aws.amazon.com/solutions/implementations/game-analytics-pipeline/ https://aws.amazon.com/solutions/implementations/game-analyt... * Simple things like auto-scaler instances being incrementally named requires lambda: https://stackoverflow.com/questions/53615049/method-to-provide-incremental-name-to-ec2-instances-created-using-auto-scaling-g https://stackoverflow.com/questions/53615049/method-to-provi... * CDK/Cloudformation is the only "simple" way to automatically provision infra.

motoboi 5y ago

"How we made our own container orchestration infrastructure over AWS. It doesn't have all Kubernetes features, but hey! we can then train our own employees on how it works. And we got to maintain the code all by ourselves too! It might take a bit too long to implement a new feature, but hey! its ours!" Really, Kubernetes is complex, but the problem it solves it even more complex. If you are ok solving a part of the problem, nice. You just built a competitor to google. Good luck hiring people who come in already knowing how to operate it. Good luck trying to keep it modern and useful too. But I totally understand the appeal.

flowerlad 5y ago

I wouldn't follow the path that this company took. I am a solo developer, use Kubernetes on Google cloud and I couldn't be happier. Parts of my application runs on AWS, taking advantage of what AWS does better, such as SES (Simple Email Service). All I had to learn is Docker and Kubernetes. If Kubernetes didn't exist I would have had to learn myriad tools and services, cloud-specific tools and services, and my application would be permanently wedded to one cloud. Thanks to Kubernetes my application can be moved to another cloud in whole or in part. Kubernetes is so well designed, it is the kind of thing you learn just because it is well designed. I am glad I invested the time. The knowledge I acquired is both durable and portable.

rcarmo 5y ago

"we would be doing mostly the same things, but in a more complicated way" is a pretty good summary considering their use cases seem to be well covered by autoscaling groups (which, incidentally, are a thing other clouds have) It's OK to not use k8s. We should normalize that.

icythere 5y ago

The hard thing of kubernetes, is that it sounds easy to do something/anything. It does, and because it's "easy" people tend to skip to understand how it's working: That's where the problem occurs. It _forces_ you to become a "yaml engineer" and to forget the other part of the systems. I was interviewed by a company and when I replied the next step I could do was to write some operators for the ops things, they simply rejected because I'm too experienced lolz

pram 5y ago
Exactly, now you have people making an EKS cluster and then deploying a vendor supplied Helm chart to it. When something breaks they literally have no idea how to start fixing it. It’s deceptively “easy”

exabrial 5y ago

Scaling is a sexy problem to have. Most places crate problems to solve just for the appeal. They're spending angel investor money, not their own, so prolonging the time to production is in their favor.

blacktriangle 5y ago

Um....are we getting trolled by some content marketing SEO hack here? https://www.sebastianbuza.com/2021/07/20/no-we-dont-use-kubernetes/ https://www.sebastianbuza.com/2021/07/20/no-we-dont-use-kube... Another pub/sub startup publishing almost the identical blog post also today, July 20, 2021.

tester756 5y ago
lol, what the hell is this
- blacktriangle 5y ago
  Given that Ably looks like an actual product, I'm guessing the link I posted is some sketchy content repurposing farm. I don't know what recourse Ably has in this case, but that's unfortunate.
matt_oriordan 5y ago
Founder of Ably here That’s odd and clearly a rip off of our post :( Thanks for flagging

benlivengood 5y ago

I think that Kubernetes has advantages for many small services but a few large services are still worth managing directly on bare machines/VMs. Where I disagree with this article is on Kubernetes stability and manageability. The caveat is that GKE is easy to manage and EKS is straightforward but not quite easy. Terraform with a few flags for the google-gke module can manage dozens of clusters with helm_release resources making the clusters production-ready with very little human management overhead. EKS is still manageable but does require a bit more setup per cluster, but it all lives in the automation and can be standardized across clusters. Daily autoscaling is one of those things that some people can get away with, but most won't save money. For example, prices for reservations/commitments are ~65% of on-demand. Can a service really scale so low during off-hours that average utilization from peak machine count is under 35%? If so, then autoscale aggressively and it's totally worth it. Most services I've seen can't actually achieve that and instead would be ~60% utilized over a whole day (mostly global customer bases). The exception is if you can scale (or run entirely with loose enough SLOs) into spot or preemptible instances which should be about as cheap as committed instances at the risk of someday not being available.

knodi 5y ago

That great now live with your bad/good design patterns on your own for the life cycle of your product or service, you get non of the shared knowledge and advancement through the k8s community or platform.

kaydub 5y ago

I don't use K8s either, but we at least use an orchestration tool. ECS Fargate has been awesome. No reason to add the complexity of K8s/EKS. We're all in on AWS and everything works together. But this... you guys re-invented the wheel. You're probably going to find it's not round in certain spots too.

truthwhisperer 5y ago

typical circle of (IT) live. Start your own "solution" end up with a kubernetes sort a like clone which is better understood by you

emodendroket 5y ago

I think the killer feature of Kubernetes is really the infrastructure as code part -- it just makes it very easy to spin services up or down as desired without thinking too hard about it. But as the article alludes to, if you're comfortable with the lock-in, you can get that from your cloud provider with tighter integration.