10 ms·

I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they

I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they want to run an application (Node, Go, Ruby, Python, Java, whatever). The prevailing mythology is you should "containerize" everything and give it to a container orchestrator to run, but why? They had one problem, "Run an app". Now they have two, "Run a container that runs an app" and "maintain a container". Just give the app to a PAAS, and go home early. Most startups - most large companies - would be far better served with a real PAAS, rather than container orchestration. My encounters with container orchestrators is that ops teams spent inordinate amounts of time trying to bend them into a PAAS, rather than just starting with one. This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. My advice to almost every startup I speak to is "Just use Heroku; solve your business problems first". The article also mentions it enables "new set of distributed primitives and runtime for creating distributed systems that spread across multiple processes and nodes". I'll throw out my other assertion, which I always though was axiomatic - you want your system to be the least distributed you can make it at all times. Distributed systems are harder to reason about, harder to write, and harder to maintain. They fail in strange ways, and are so hard to get right, I'd bet I can find a hidden problem in yours within an hour of starting code review. Most teams running a non-trivial distributed system are coasting on luck rather than skill. This is not a reflection on them - just an inherent problem with building distributed logic. Computers are fast, and you are not Google. I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. I'm now helping a startup unpick it's 18 "service" containerised system back to something that can actually be maintained. TLDR; containers as production app deployment artefacts have, in the medium and long term, caused more problems than they've solved for almost every case I've seen.

merb 9y ago
since you mentioned Cloudfoundry... I think it's a thousand times easier to get up and running with k8s, than with Cloudfoundry on Bare-Metal (no Cloud). It's also a thousand times easier to maintain. (Thanks CoreOS) Basically if you want a managed simple no maintance, no cost bare-metal K8S installation you basically just use tectonic/kubeadm and you get something which is self-containing, or close to self-containing. and the only things you need to get it done is actually way easier than reading through cf docs (I'm pretty sure bare-metal isn't even supported that easily). running some services on top of it is than pretty simple, especially if you want to use a single ip, insteand of roundobin dns (https://github.com/kubernetes/contrib/tree/master/keepalived-vip https://github.com/kubernetes/contrib/tree/master/keepalived...) and if you have k8s running, adding some PaaS layer on top (openshift) can be pretty simple.
- jacques_chester 9y ago
  > I'm pretty sure bare-metal isn't even supported that easily BOSH with the RackHD CPI does this. It's the same basic operator experience across every platform with a CPI. Disclosure: I work for Pivotal, we work on this stuff.
  - merb 9y ago
    bare-metal without openstack.
    - jacques_chester 9y ago
      RackHD is not OpenStack. https://rackhd.github.io/ https://rackhd.github.io/
brightball 9y ago
I tend to agree with you and it's one of the biggest reasons that I'm a fan of Elixir. Here's the path that leads to K8s too early. 1. We think we need microservices 2. Look how much it will cost of we run ALL OF THESE microservices on Heroku 3. We should run it ourselves, let's use K8s One of the big "Elixir" perks is that it bypasses this conversation and lets you run a collection of small applications under a single monolith within the same runtime...efficiently. So you can built smaller services...like a monolith...with separate dependency trees...without needing to run a cluster of multiple nodes...and still just deploy to Heroku (or Gigalixir). Removes a lot of over-architectural hand-wringing so you can focus on getting your business problem out the door but will still allow you to separate things early enough that you don't have to worry about long term code-entanglement. And when you "need" to scale, clustering is already built in without needing to create API frontends for each application. It solves a combination of so many short term and long term issues at the same time.
- _asummers 9y ago
  > One of the big "Elixir" perks This is also true of Erlang, for those not aware that Elixir runs on the Erlang Virtual Machine (BEAM). You do get a lot of cool things with clustered nodes though (Node monitors are terrific) and tools like Observer and Wobserver have facilities for taking advantage of your network topology to give you more information.
- fokinsean 9y ago
  Interesting, I didn't know that about Elixir. Do you ever have to break them up into smaller Elixir apps or can you stick with that pseudo-monolith for good?
  - brightball 9y ago
    You break them into smaller apps. It’s little more than code rearranging though. You can still call the functions through the same Module.function() approach you’d use if they were in the same app. The $30 PragDave Elixir for Programmers course actually drills in this approach the whole way through if you’re looking for a good resource.
toomuchtodo 9y ago
> I'm now helping a startup unpick it's 18 "service " containerised system back to something that can actually be maintained. There's a lot of work (and money) out there to fix systems implemented on the hype train.
gaastonsr 9y ago
True, and that's why I think a managed Kubernetes service like GKE is the way to go. It's almost like a PaaS but you still have a lot of the control.
- pacala 9y ago
  Indeed, Kubernetes as a service is the way to go. https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/ https://azure.microsoft.com/en-us/services/container-service/ https://azure.microsoft.com/en-us/services/container-service... https://aws.amazon.com/eks/ https://aws.amazon.com/eks/ https://www.ibm.com/cloud/container-service https://www.ibm.com/cloud/container-service Or have someone knowledgeable build the service for you. https://heptio.com/products/kubernetes-subscription/ https://heptio.com/products/kubernetes-subscription/
  - ReidZB 9y ago
    Amazon's EKS is still in preview. I wouldn't expect it to be generally available (that is, stable) for several months at least. I've also heard reports that getting into the preview is really difficult at the moment. It's using a new networking model: https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s > Alpha This is an experimental release as part of the Amazon EKS Preview. Interfaces and functionality may change. Expect bugs (and please help us squash them). DO NOT use for production workloads.
- scprodigy 9y ago
  What control do you have?
jpswade 9y ago
Kubernetes takes you to serverless, where you don't care about the hardware. The next shift is what I've called "stackless" - why do you even care what platform it runs on? All you want to be able to do is have your application run somewhere. Kubernetes goes some way towards that, but there's another abstraction layer needed. Similar to how Docker was an abstraction further to Kubernetes and away from Vagrant. This is something I wrote about this not long ago[1]. 1. https://wade.be/development/sysadmin/2016/11/17/stackless.html https://wade.be/development/sysadmin/2016/11/17/stackless.ht...
- singularity2001 9y ago
  >> why do you even care what platform it runs on? Isn't that what the JVM/wasm solved?
  - zeveb 9y ago
    Well, yes — the problem is that the JVM was too big and too platform-independent. We don't want JVM everywhere; really, we want POSIX-everywhere. The JVM's also this weird level of statically-typed hyper-extensibility — it's Greenspun's Tenth Law in action, and the result is typically in really terrible taste. The end result is a JVM which is really, really impressive, but appallingly ugly.
    - pjmlp 9y ago
      Thanks to the JVM, I stopped caring about POSIX. Not everyone is found of it. Same applies to any other language with rich libraries.
  - pjmlp 9y ago
    Yes. JEE application servers already offer all the benefits of containers and OS independence.
- cnj 9y ago
  > Kubernetes takes you to serverless, where you don't care about the hardware. Serverless isn't a good name - but it doesn't stand for "don't care about the hardware". Devs are already not caring about hardware anymore since VMs. What serverless removes is the abstraction level of a server/vm/container. A simple example is scaling your stateless components. In a serverless FaaS, functions are scaled for you. You don't have to do anything to handle a peak in web traffic. You don't have to do anything to handle a peak of msgs in your MQ. In k8s, you still have to go and fumble around with CPU/memory limits and better get it right. k8s also doesn't scale your containers based on the msgs in your MQ out of the box. You have to build and run that service yourself (or ask GCP to whitelist you should you be running their MQ https://cloud.google.com/compute/docs/autoscaler/scaling-queue-based https://cloud.google.com/compute/docs/autoscaler/scaling-que... ). AWS Lambda had that since 2015...
IMTDb 9y ago
Kubernetes team foten says than one if it's goal is to be a "low level" project which should be the base additional tool/services/... are using under the hood. Helm (https://helm.sh/ https://helm.sh/) allows you to define an app as a collection of K8S components then to manage (=deploy, update, ...) your app as a standalone component
sytse 9y ago
I agree that most startups should work at a Heroku level of abstraction. You mention 18 microsevices, I think that small teams are better off with a monolith. I would see Kubernetes as a new machine level. We're moving from bare metal, to VMs, to container schedulers. Heroku was one of the first companies that ran a container scheduler internally. So I think we agree that is the future. But a small team probably doesn't need to work at that abstraction level. At GitLab we think most teams will want to work at a higher abstraction layer. Just push your code and have it deployed on Kubernetes. Without having to write a dockerfile or helm chart yourself.
bane 9y ago
There's even the higher-level desire, what users really want isn't a place to run their app, but to the function the app provides. e.g. in a more micro-services-like environment with a service that simply looks things up in a database, what they really want is just query access to the data. But now they have the data in some db, the db in some container, some API written using the latest API style, some software the provide the API (also in a container), some container orchestration to coordinate everything, load balancers, caches and so on. So there's all these layers of stuff that sit between the user and the data just to make the act of asking WHERE DATATHING="STUFF" convenient.
sandGorgon 9y ago
you should check out docker swarm. the UX of swarm is brilliant - use a 10 line compose.yml file to get a stack up and running. Let's you specify tons of stuff if you want to. The batteries included nature of swarm is a huge help as well - with k8s, you have to muck around overlay network, ingress, etc. However, I think the writing is clear on the wall - k8s has won. Probably even to Docker Inc, given the kubernetes integration they are building into swarm now. I think Docker Swarm can exist as an opinionated distro of k8s. I wouldnt mind paying it money for that.
pacala 9y ago
Containerization helps with one thing: end-to-end dependency hell management. You get the same executable artifact in prod and on every dev machine. You get to share arcane tricks required to bootstrap library X. You get to share the complete recipe of building your OS image. Hopefully, you pin versions so your build is not subject to the whims of upstream. Kubernetes helps with one thing: taking your container and running it on a fleet of machines. Building 18 services is an architectural choice made by the team. It has nothing to do with containerization or Kubernetes. For a single team, a monolith just works most of the time. You may consider multiple services if you have multiple [large] teams, think Search vs. Maps. Even then, consider the trade-offs carefully.
- scarface74 9y ago
  I deploy code with all of the dlls in separate folders. The executables/services don't share any dlls. I kept asking the "consultants" who are trying to push us to Docker, what is the business value over raw executables + Nomad. The build server creates one zip file that is stored as an artifact that gets decompressed and released in each environment - in a separate folder.
  - pjmlp 9y ago
    I look at these "solutions" and they don't seem to add much to how application servers work.
    - scarface74 9y ago
      We have a bunch of apps that run based on some type of external event - a time interval (Nomad supports cron like scheduling across an app pool), a file coming in, an external event, etc. We submit a job via the api and it runs the job on whatever server has available resources. We specify the mininimum amount of RAM and CPU needed to run a job. If too many jobs are queued on a regular basis, we can either add more RAM or CPU to an existing instance or add another instance and install a Nomad agent. Yes I know k8s can do the same thing but we don't have to use Docker, we can though.
    - 9y ago
scarface74 9y ago
I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they want to run an application (Node, Go, Ruby, Python, Java, whatever) I agree completely and your comment gives me the perfect opportunity to praise how much I love the flexibility of Hashicorp's Consul+Nomad. Nomad let's you run almost anything - Docker containers, executables (the raw_exec driver), jar files, etc. https://www.nomadproject.io/docs/drivers/index.html https://www.nomadproject.io/docs/drivers/index.html Dead simple to setup - one self contained < 20Mb executable that can be used in either client, server, or dev mode (client + server), configuration is basically automatic as either a single server or cluster of you are using Consul. The stock UI is weak but The third party HashiUI is great.
- jnsaff2 9y ago
  Don't forget that Nomad has awesome integration with Vault, possibly the best secrets handling out there.
  - scarface74 9y ago
    I played with Vault and it wasn't quite as simple as Consul and Nomad. My major issue was trying to figure out how I would "unlock" the Vault automatically in case of a system restart. I punted for now and just stored sensitive values directly in Consul encrypted.
deleted 9y ago
[deleted]
victor106 9y ago
This and This. If you are looking for “I just wanna run my app” I found CloudFoundry to be dope among all the other PAAS solutions out there.
jrs95 9y ago
The root of this is really people making distributed systems when they don't need to. This microservices trend really is a massive waste of resources for most smaller teams that get caught up in it.
polskibus 9y ago
Service fabric from Microsoft comes to mind but it is not open source.
- rraghur 9y ago
  I think it's overrated though - not open source, doesn't have an ecosystem.. the dev experience is sub par - services take too long to come up even with the one node cluster on a beefy laptop. Plus you cannot run the service outside of SF as an exe now. I migrated a decent sized solution still in dev from SF to .netcore and SF - 10/10 would do it again. Not to mention that you also end up saving 50% $$$ on vm costs with linux vms (not considering SF on Linux)
  - polskibus 9y ago
    Thanks for sharing your experience, I did not understand it in full. Do you recommend using SF or not? you mention that you would do it again - was that only about moving from Windows .NET to .NET Core on Linux (ie. NET Core rocks?) and the rest about SF is crap or would you recommend SF in general for any future work (instead of for example Akka.NET for service coordination in a cluster) ?
nickjj 9y ago
The funny thing is I have 3 courses on Docker and I'm a Docker Captain but I pretty much agree with what you wrote about container orchestration. A lot of people forget that you can just put your application up on 1 server and serve hundreds of thousands or millions of requests a month without breaking a sweat. For that type of use case (1 box deploys), Docker is still amazingly useful so I would 100% containerize your apps for that purpose, but I agree, Kubernetes and container orchestration in general is overkill for so many projects.
bryanlarsen 9y ago
Kubernetes has helped to make our app less distributed. Parts of the system were distributed not for capacity, but for HA reasons. So where before we had two instances of beanstalkd with their own storage and clients had logic to talk to both, we now have a single instance of beanstalkd backed by distributed storage and a Kubernetes service that points to it. And I think we get more benefit deploying dependencies than we do our own apps. If one of them is low volume and needs mysql, just `helm install mariadb`. No complicated HA setup, no worries about backups, we already know how to backup volumes.
daxfohl 9y ago
I agree with this for the most part, but wanted to point out that docker's first big success was as a dev tool. Solving the "it works on my machine" problem, or the "oh you forgot to install v13.1.2 of X and then uninstall it and then install v12.4 because that's the only way it works for some reason" problem. So, avoiding k8s in order to avoid docker seems odd. That said, a good number of projects don't require anything special about the environment other than a runtime for the app's language, where the remaining dependencies can be explicitly included in the build. For those, I agree, jumping on docker/k8s right away is overkill. An additional benefit of working with something like Heroku initially, is that it will help guide your devs to sticking with more tried and trusted stacks rather than everyone pulling in their own pet project into the business's critical path.
snuxoll 9y ago
This is why I primarily see Kubernetes as a set of low-level primitives for a PaaS to build upon. We don't use Kubernetes at my shop, we've begun to use OpenShift though which layers PaaS tooling on top of it and the developers on my team love it. They create a deployment, point it at the git repository containing their code, set their configuration and the app is live - the underlying primitives are available if we need them still, but that's for me to worry about as the DevOps guy and not the developers.
rollcat 9y ago
Each of the three digital production agencies I've worked with has the same problem: jobs come and go all the time, often have varied tech stacks (took over a project from a different company, resurrected 5yr old rotting dinosaur, one team prefers Node, another Django, etc), each project requires a dev/staging/live environment (and sometimes more than that, e.g. separate staging for code / content changes), and so on... In one shop we went thru 500 git repos in 4 years. One day I spun up a k8s cluster on GKE and just started putting all projects there. This cluster enabled huge cost savings (running a fleet of 3 VM's instead of ~50), allowed cheap per-feature dev/staging environments, forced developers to consider horizontal scaling BEFORE we needed to scale (read: when we missed our only shot), and overall reduced ops workload tenfold. It wasn't without a few challenges of its own, but I would never go back.
- bonesss 9y ago
  I think you've hit on the major issue with the "anti-hype" around kubernetes and related products: they're not something you need, per se, to develop an app. They are something you need to manage multiple parallel development processes. For devs stuck in a silo it's a little like putting margarine on butter. For DevOps looking at hundreds of little silos it's the foundation of operational sanity.
  - mikkergimenez 9y ago
    To sort of echo what you're saying, most of these articles seem to suggest that containers solve a technical problem. More often then not I've seen them as a solution to an organizational problem.
imsofuture 9y ago
Yeah its not an overblown generalization at all to suggest Heroku for '99.99%' of workloads.
- detaro 9y ago
  Luckily, the parent does not suggest that.
jacques_chester 9y ago
I agree with pretty much everything you said and it's very heartening to not be the token Cloud Foundry person in the comments. As a nitpick: > This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. In fairness, the reference was to Cloud Foundry Diego, which is the most analogical component to Kubernetes. And they are of comparable vintage. Diego never found any independent traction outside of CFAR. > I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. We have customers doing millions of payments per hour, billions of events per day. Running tens of thousands of apps, thousands of services, with thousands of developers, deploying thousands of times per week. CFAR doesn't get much press out of enterprise-land, but it works really well. Disclosure: I work for Pivotal. We have commercial distributions of both Cloud Foundry (PAS) and Kubernetes (PKS).
archgrove 9y ago
Clarification: 18 containerised services can absolutely be the right choice. It’s just my experience says the trade off between the costs of maintaining that versus a smaller PAASed system rarely come out in favour of it.

merb 9y ago

since you mentioned Cloudfoundry... I think it's a thousand times easier to get up and running with k8s, than with Cloudfoundry on Bare-Metal (no Cloud). It's also a thousand times easier to maintain. (Thanks CoreOS) Basically if you want a managed simple no maintance, no cost bare-metal K8S installation you basically just use tectonic/kubeadm and you get something which is self-containing, or close to self-containing. and the only things you need to get it done is actually way easier than reading through cf docs (I'm pretty sure bare-metal isn't even supported that easily). running some services on top of it is than pretty simple, especially if you want to use a single ip, insteand of roundobin dns (https://github.com/kubernetes/contrib/tree/master/keepalived-vip https://github.com/kubernetes/contrib/tree/master/keepalived...) and if you have k8s running, adding some PaaS layer on top (openshift) can be pretty simple.

jacques_chester 9y ago
> I'm pretty sure bare-metal isn't even supported that easily BOSH with the RackHD CPI does this. It's the same basic operator experience across every platform with a CPI. Disclosure: I work for Pivotal, we work on this stuff.
- merb 9y ago
  bare-metal without openstack.
  - jacques_chester 9y ago
    RackHD is not OpenStack. https://rackhd.github.io/ https://rackhd.github.io/

brightball 9y ago

I tend to agree with you and it's one of the biggest reasons that I'm a fan of Elixir. Here's the path that leads to K8s too early. 1. We think we need microservices 2. Look how much it will cost of we run ALL OF THESE microservices on Heroku 3. We should run it ourselves, let's use K8s One of the big "Elixir" perks is that it bypasses this conversation and lets you run a collection of small applications under a single monolith within the same runtime...efficiently. So you can built smaller services...like a monolith...with separate dependency trees...without needing to run a cluster of multiple nodes...and still just deploy to Heroku (or Gigalixir). Removes a lot of over-architectural hand-wringing so you can focus on getting your business problem out the door but will still allow you to separate things early enough that you don't have to worry about long term code-entanglement. And when you "need" to scale, clustering is already built in without needing to create API frontends for each application. It solves a combination of so many short term and long term issues at the same time.

_asummers 9y ago
> One of the big "Elixir" perks This is also true of Erlang, for those not aware that Elixir runs on the Erlang Virtual Machine (BEAM). You do get a lot of cool things with clustered nodes though (Node monitors are terrific) and tools like Observer and Wobserver have facilities for taking advantage of your network topology to give you more information.
fokinsean 9y ago
Interesting, I didn't know that about Elixir. Do you ever have to break them up into smaller Elixir apps or can you stick with that pseudo-monolith for good?
- brightball 9y ago
  You break them into smaller apps. It’s little more than code rearranging though. You can still call the functions through the same Module.function() approach you’d use if they were in the same app. The $30 PragDave Elixir for Programmers course actually drills in this approach the whole way through if you’re looking for a good resource.

toomuchtodo 9y ago

> I'm now helping a startup unpick it's 18 "service " containerised system back to something that can actually be maintained. There's a lot of work (and money) out there to fix systems implemented on the hype train.

gaastonsr 9y ago

True, and that's why I think a managed Kubernetes service like GKE is the way to go. It's almost like a PaaS but you still have a lot of the control.

pacala 9y ago
Indeed, Kubernetes as a service is the way to go. https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/ https://azure.microsoft.com/en-us/services/container-service/ https://azure.microsoft.com/en-us/services/container-service... https://aws.amazon.com/eks/ https://aws.amazon.com/eks/ https://www.ibm.com/cloud/container-service https://www.ibm.com/cloud/container-service Or have someone knowledgeable build the service for you. https://heptio.com/products/kubernetes-subscription/ https://heptio.com/products/kubernetes-subscription/
- ReidZB 9y ago
  Amazon's EKS is still in preview. I wouldn't expect it to be generally available (that is, stable) for several months at least. I've also heard reports that getting into the preview is really difficult at the moment. It's using a new networking model: https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s > Alpha This is an experimental release as part of the Amazon EKS Preview. Interfaces and functionality may change. Expect bugs (and please help us squash them). DO NOT use for production workloads.
scprodigy 9y ago
What control do you have?

jpswade 9y ago

Kubernetes takes you to serverless, where you don't care about the hardware. The next shift is what I've called "stackless" - why do you even care what platform it runs on? All you want to be able to do is have your application run somewhere. Kubernetes goes some way towards that, but there's another abstraction layer needed. Similar to how Docker was an abstraction further to Kubernetes and away from Vagrant. This is something I wrote about this not long ago[1]. 1. https://wade.be/development/sysadmin/2016/11/17/stackless.html https://wade.be/development/sysadmin/2016/11/17/stackless.ht...

singularity2001 9y ago
>> why do you even care what platform it runs on? Isn't that what the JVM/wasm solved?
- zeveb 9y ago
  Well, yes — the problem is that the JVM was too big and too platform-independent. We don't want JVM everywhere; really, we want POSIX-everywhere. The JVM's also this weird level of statically-typed hyper-extensibility — it's Greenspun's Tenth Law in action, and the result is typically in really terrible taste. The end result is a JVM which is really, really impressive, but appallingly ugly.
  - pjmlp 9y ago
    Thanks to the JVM, I stopped caring about POSIX. Not everyone is found of it. Same applies to any other language with rich libraries.
- pjmlp 9y ago
  Yes. JEE application servers already offer all the benefits of containers and OS independence.
cnj 9y ago
> Kubernetes takes you to serverless, where you don't care about the hardware. Serverless isn't a good name - but it doesn't stand for "don't care about the hardware". Devs are already not caring about hardware anymore since VMs. What serverless removes is the abstraction level of a server/vm/container. A simple example is scaling your stateless components. In a serverless FaaS, functions are scaled for you. You don't have to do anything to handle a peak in web traffic. You don't have to do anything to handle a peak of msgs in your MQ. In k8s, you still have to go and fumble around with CPU/memory limits and better get it right. k8s also doesn't scale your containers based on the msgs in your MQ out of the box. You have to build and run that service yourself (or ask GCP to whitelist you should you be running their MQ https://cloud.google.com/compute/docs/autoscaler/scaling-queue-based https://cloud.google.com/compute/docs/autoscaler/scaling-que... ). AWS Lambda had that since 2015...

IMTDb 9y ago

Kubernetes team foten says than one if it's goal is to be a "low level" project which should be the base additional tool/services/... are using under the hood. Helm (https://helm.sh/ https://helm.sh/) allows you to define an app as a collection of K8S components then to manage (=deploy, update, ...) your app as a standalone component

sytse 9y ago

I agree that most startups should work at a Heroku level of abstraction. You mention 18 microsevices, I think that small teams are better off with a monolith. I would see Kubernetes as a new machine level. We're moving from bare metal, to VMs, to container schedulers. Heroku was one of the first companies that ran a container scheduler internally. So I think we agree that is the future. But a small team probably doesn't need to work at that abstraction level. At GitLab we think most teams will want to work at a higher abstraction layer. Just push your code and have it deployed on Kubernetes. Without having to write a dockerfile or helm chart yourself.

bane 9y ago

There's even the higher-level desire, what users really want isn't a place to run their app, but to the function the app provides. e.g. in a more micro-services-like environment with a service that simply looks things up in a database, what they really want is just query access to the data. But now they have the data in some db, the db in some container, some API written using the latest API style, some software the provide the API (also in a container), some container orchestration to coordinate everything, load balancers, caches and so on. So there's all these layers of stuff that sit between the user and the data just to make the act of asking WHERE DATATHING="STUFF" convenient.

sandGorgon 9y ago

you should check out docker swarm. the UX of swarm is brilliant - use a 10 line compose.yml file to get a stack up and running. Let's you specify tons of stuff if you want to. The batteries included nature of swarm is a huge help as well - with k8s, you have to muck around overlay network, ingress, etc. However, I think the writing is clear on the wall - k8s has won. Probably even to Docker Inc, given the kubernetes integration they are building into swarm now. I think Docker Swarm can exist as an opinionated distro of k8s. I wouldnt mind paying it money for that.

pacala 9y ago

Containerization helps with one thing: end-to-end dependency hell management. You get the same executable artifact in prod and on every dev machine. You get to share arcane tricks required to bootstrap library X. You get to share the complete recipe of building your OS image. Hopefully, you pin versions so your build is not subject to the whims of upstream. Kubernetes helps with one thing: taking your container and running it on a fleet of machines. Building 18 services is an architectural choice made by the team. It has nothing to do with containerization or Kubernetes. For a single team, a monolith just works most of the time. You may consider multiple services if you have multiple [large] teams, think Search vs. Maps. Even then, consider the trade-offs carefully.

scarface74 9y ago
I deploy code with all of the dlls in separate folders. The executables/services don't share any dlls. I kept asking the "consultants" who are trying to push us to Docker, what is the business value over raw executables + Nomad. The build server creates one zip file that is stored as an artifact that gets decompressed and released in each environment - in a separate folder.
- pjmlp 9y ago
  I look at these "solutions" and they don't seem to add much to how application servers work.
  - scarface74 9y ago
    We have a bunch of apps that run based on some type of external event - a time interval (Nomad supports cron like scheduling across an app pool), a file coming in, an external event, etc. We submit a job via the api and it runs the job on whatever server has available resources. We specify the mininimum amount of RAM and CPU needed to run a job. If too many jobs are queued on a regular basis, we can either add more RAM or CPU to an existing instance or add another instance and install a Nomad agent. Yes I know k8s can do the same thing but we don't have to use Docker, we can though.
  - 9y ago

scarface74 9y ago

I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they want to run an application (Node, Go, Ruby, Python, Java, whatever) I agree completely and your comment gives me the perfect opportunity to praise how much I love the flexibility of Hashicorp's Consul+Nomad. Nomad let's you run almost anything - Docker containers, executables (the raw_exec driver), jar files, etc. https://www.nomadproject.io/docs/drivers/index.html https://www.nomadproject.io/docs/drivers/index.html Dead simple to setup - one self contained < 20Mb executable that can be used in either client, server, or dev mode (client + server), configuration is basically automatic as either a single server or cluster of you are using Consul. The stock UI is weak but The third party HashiUI is great.

jnsaff2 9y ago
Don't forget that Nomad has awesome integration with Vault, possibly the best secrets handling out there.
- scarface74 9y ago
  I played with Vault and it wasn't quite as simple as Consul and Nomad. My major issue was trying to figure out how I would "unlock" the Vault automatically in case of a system restart. I punted for now and just stored sensitive values directly in Consul encrypted.

deleted 9y ago

[deleted]

victor106 9y ago

This and This. If you are looking for “I just wanna run my app” I found CloudFoundry to be dope among all the other PAAS solutions out there.

jrs95 9y ago

The root of this is really people making distributed systems when they don't need to. This microservices trend really is a massive waste of resources for most smaller teams that get caught up in it.

polskibus 9y ago

Service fabric from Microsoft comes to mind but it is not open source.

rraghur 9y ago
I think it's overrated though - not open source, doesn't have an ecosystem.. the dev experience is sub par - services take too long to come up even with the one node cluster on a beefy laptop. Plus you cannot run the service outside of SF as an exe now. I migrated a decent sized solution still in dev from SF to .netcore and SF - 10/10 would do it again. Not to mention that you also end up saving 50% $$$ on vm costs with linux vms (not considering SF on Linux)
- polskibus 9y ago
  Thanks for sharing your experience, I did not understand it in full. Do you recommend using SF or not? you mention that you would do it again - was that only about moving from Windows .NET to .NET Core on Linux (ie. NET Core rocks?) and the rest about SF is crap or would you recommend SF in general for any future work (instead of for example Akka.NET for service coordination in a cluster) ?

nickjj 9y ago

The funny thing is I have 3 courses on Docker and I'm a Docker Captain but I pretty much agree with what you wrote about container orchestration. A lot of people forget that you can just put your application up on 1 server and serve hundreds of thousands or millions of requests a month without breaking a sweat. For that type of use case (1 box deploys), Docker is still amazingly useful so I would 100% containerize your apps for that purpose, but I agree, Kubernetes and container orchestration in general is overkill for so many projects.

bryanlarsen 9y ago

Kubernetes has helped to make our app less distributed. Parts of the system were distributed not for capacity, but for HA reasons. So where before we had two instances of beanstalkd with their own storage and clients had logic to talk to both, we now have a single instance of beanstalkd backed by distributed storage and a Kubernetes service that points to it. And I think we get more benefit deploying dependencies than we do our own apps. If one of them is low volume and needs mysql, just `helm install mariadb`. No complicated HA setup, no worries about backups, we already know how to backup volumes.

daxfohl 9y ago

I agree with this for the most part, but wanted to point out that docker's first big success was as a dev tool. Solving the "it works on my machine" problem, or the "oh you forgot to install v13.1.2 of X and then uninstall it and then install v12.4 because that's the only way it works for some reason" problem. So, avoiding k8s in order to avoid docker seems odd. That said, a good number of projects don't require anything special about the environment other than a runtime for the app's language, where the remaining dependencies can be explicitly included in the build. For those, I agree, jumping on docker/k8s right away is overkill. An additional benefit of working with something like Heroku initially, is that it will help guide your devs to sticking with more tried and trusted stacks rather than everyone pulling in their own pet project into the business's critical path.

snuxoll 9y ago

This is why I primarily see Kubernetes as a set of low-level primitives for a PaaS to build upon. We don't use Kubernetes at my shop, we've begun to use OpenShift though which layers PaaS tooling on top of it and the developers on my team love it. They create a deployment, point it at the git repository containing their code, set their configuration and the app is live - the underlying primitives are available if we need them still, but that's for me to worry about as the DevOps guy and not the developers.

rollcat 9y ago

Each of the three digital production agencies I've worked with has the same problem: jobs come and go all the time, often have varied tech stacks (took over a project from a different company, resurrected 5yr old rotting dinosaur, one team prefers Node, another Django, etc), each project requires a dev/staging/live environment (and sometimes more than that, e.g. separate staging for code / content changes), and so on... In one shop we went thru 500 git repos in 4 years. One day I spun up a k8s cluster on GKE and just started putting all projects there. This cluster enabled huge cost savings (running a fleet of 3 VM's instead of ~50), allowed cheap per-feature dev/staging environments, forced developers to consider horizontal scaling BEFORE we needed to scale (read: when we missed our only shot), and overall reduced ops workload tenfold. It wasn't without a few challenges of its own, but I would never go back.

bonesss 9y ago
I think you've hit on the major issue with the "anti-hype" around kubernetes and related products: they're not something you need, per se, to develop an app. They are something you need to manage multiple parallel development processes. For devs stuck in a silo it's a little like putting margarine on butter. For DevOps looking at hundreds of little silos it's the foundation of operational sanity.
- mikkergimenez 9y ago
  To sort of echo what you're saying, most of these articles seem to suggest that containers solve a technical problem. More often then not I've seen them as a solution to an organizational problem.

imsofuture 9y ago

Yeah its not an overblown generalization at all to suggest Heroku for '99.99%' of workloads.

detaro 9y ago
Luckily, the parent does not suggest that.

jacques_chester 9y ago

I agree with pretty much everything you said and it's very heartening to not be the token Cloud Foundry person in the comments. As a nitpick: > This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. In fairness, the reference was to Cloud Foundry Diego, which is the most analogical component to Kubernetes. And they are of comparable vintage. Diego never found any independent traction outside of CFAR. > I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. We have customers doing millions of payments per hour, billions of events per day. Running tens of thousands of apps, thousands of services, with thousands of developers, deploying thousands of times per week. CFAR doesn't get much press out of enterprise-land, but it works really well. Disclosure: I work for Pivotal. We have commercial distributions of both Cloud Foundry (PAS) and Kubernetes (PKS).

archgrove 9y ago

Clarification: 18 containerised services can absolutely be the right choice. It’s just my experience says the trade off between the costs of maintaining that versus a smaller PAASed system rarely come out in favour of it.