4 ms·

As a startup founder that's not VC funded, I would totally recommend you look into building with kubernetes from the get go. The biggest learning curve is for t

As a startup founder that's not VC funded, I would totally recommend you look into building with kubernetes from the get go. The biggest learning curve is for the person setting up the initial deployments, services, ingress etc Most other team members may just need to maybe change the image name and kubectl apply to roll things out. Knowing that rollouts won't bring down prod and that they can be tested in different environments consistently is really valuable. I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance. Once the costs were too high we moved to VMs. Doing deployments was fine but complicated enough that only seasoned team members could do it safely. We needed to build a lot of checks, monitoring etc to do this safely. A bunch of random scripts existed to set things up and migrating base operating system etc required a ton of time. Moving to kubernetes was a breath of fresh air and I wish we'd done it earlier. We now have an easy repeatable process . Infra is easier to understand. Rollouts are safer and honestly, the system is safer too. We know exactly what ports can allow ingress, what service boundaries exist. What cronjobs are configured, their state etc with simple kubectl commands. Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge

lbriner 4y ago
Totally agree! Kubernetes isn't just about global scale that most people will never need, which would agree with the article. It is about deploying new apps to an existing production system really quickly and easily. We can deploy a new app alongside an old app and proxy between them. Setting up a new application on IIS or a new web server to scale is a mare, doing the same on AKS (managed!) is a breeze. It is also really good value for money, because we can scale relatively quickly compared to dedicated servers. It is also harder to break something existing with a new deployment because of the container isolation. We might not need 1000 email services now but we could very quickly need that kind of scale and I don't want to be running up 100s of VMs at short notice as the business starts taking off when I can simply scale out the K8S deployment and add a few nodes. There is relatively little extra work (a dockerfile?) compared to hosting the same services on a web server.
- philliphaydon 4y ago
  > Setting up a new application on IIS or a new web server to scale is a mare. I disagree. With octopus deploy I can add a step, set a package and hostname, and press a button, and have a new deployment of a new api or website in IIS pushed out to how ever many servers currently exist in a few minutes. There are many ways to manage deploying services, and scaling, without K8S or containers in general. While I could setup a new Octopus Deploy quickly, it would be like you setting up something in AKS. We are both good at the tools we know. But saying my way or your way is wrong - is the thing thats wrong.
  - boloust 4y ago
    That is very interesting, I'd love to know more about how this works. - How does it decide which servers to deploy to? - Does it scale up the number of servers if the deployment is too large? - What happens when a server dies?
    - philliphaydon 4y ago
      Well I guess it all depends on your setup. Servers get have a tentacle which has an environment and tag. So you could say a server is test environment and tagged with main-web-app. While another server is production with the same tag. You promote a package from test to production. You configure your setup to say a package is deployed to a server tagged main-web-app. Octopus is an orchestration tool so isn’t responsible for scaling up. But in AWS with an autoscale group you can configure Octopus to auto detect a new server tag it and deploy. Part of the deployment would add it to the load balancer. As a side note you can deploy containers using octopus too. Tho I’ve never found a reason to use containers in production yet.
  - mountainriver 4y ago
    We’ll see if octopus deploy is around in 10 years
    - philliphaydon 4y ago
likortera 4y ago
> I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance Are you factoring in the salaries of the people setting up Kubernetes? And the cost of those people/salaries not working on the actual product? And the cost of those people leaving the company and leaving a ton of custom infrastructure code behind that the team can't quickly get up to speed with? > ton with huge bills especially from our managed db instance This doesn't have much to do with App Engine, right? Last time I used it, we were using a PostgresQL instance on AWS and had no problems with that. > Doing deployments was fine but complicated enough that only seasoned team members could do it safely I just plain don't believe this. I bet you were doing something wrong. How is it possible that the team find too difficult to do an App Engine deployment but then they're able to setup a full kubernetes cluster with all the stuff surrounding it? It's like saying I'm using React because JavaScript is too difficult. > Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge This is far, far, far from my own experience. Some other questions: How did you implement canary deployment? How much time are you investing in upgrading Kubernetes, and the underlying nodes operating systems? How did kubernetes solve the large database bills issue? How are you doing backups and restoration of the database now? If I were to found a company, specially not VC founded, dealing with kubernetes would be definitely far below on my list of priorities. But that's just me.
- nightpool 4y ago
  (I agree with your overall points, but when the GP said "Doing deployments was fine but complicated enough that only seasoned team members could do it safely" they were referring to their post-App Engine, pre-Kubernetes "manually managed VMs". I have no problem believing that deploys are very complicated in that scenario)
- ezekiel11 4y ago
  If setting up a PostgreSQL behind VPC with an EC2 in front of it was too difficult, there is now a serverless database product from AWS that costs 90% less what it used to. No more load balancers, no more VMs, no more scaling up or down to match demands.
hsn915 4y ago
> The biggest learning curve is for the person setting up the initial deployments, services, ingress etc Most other team members may just need to maybe change the image name and kubectl apply to roll things out. This is a huge redflag. It's basically admitting that you expect most later employees to not understand k8s or how its being used. You may think they don't need because it works, but you have to think about what happens when it doesn't work. The shops I've been to all had the same mindset: the docker/k8s infra was setup by one guy (or two) and no one else on the team understands what's going on, let alone have the ability to debug the configuration or fix problems with it. Another thing that might happen is some team members understand just barely enough to be able to "add" things to the config files. Over time the config files accumulate so much cruft, no one knows what configuration is used for what anymore.
- lima 4y ago
  You have this with literally every deployment mechanism, except with Kubernetes the boundary is standardized and you can easily find/hire/teach new team members to work on the complicated parts. Custom VM/cloud/$randomSaaS deployments are much worse when it comes to "the one guy who understands the intricate details is on vacation".
  - jrimbault 4y ago
    At some point, this becomes the dividing line between a "nu-ops" team and the dev team.
  - hsn915 4y ago
    Of course, if your deployment mechanism needs to be complicated, standarizing on something everyone knows is useful. The underlying assumption behind my comment is that you really really want to simplify your deployment as much as you can. Unfortunately this is nearly impossible with the currently accept set of standard best practices for developing web applications, where you use a scripting language and like six different database systems (one for source of truth on data, one for caching, one for full text search, and who knows what else everyone is using these days; I honestly can't keep track). There are many ways to make the deployment process simple. My rule of thumb is to use a compiled language (Go) and an embedded database engine. The levels.io guy famously did it even with php https://twitter.com/levelsio/status/1102487697220820994 https://twitter.com/levelsio/status/1102487697220820994
moomoo11 4y ago
Please don’t do this. I’m dealing with the mess caused by following this line of thinking. One guy (okay it was two guys) set up all the infrastructure and as soon as it was working bounced to new jobs with their newfound experience. The result is that dozens of engineers have no idea what the heck is going on and are lost navigating the numerous repos that hold various information related to deploying your feature. In my opinion (and I’m sure my opinion has flaws!), unless you have global customers and your actual users are pushing 1k requests per second load on your application servers/services, there is no reason to have these levels of abstractions. However once this becomes reality I think everyone working on that responsibility needs to learn k8s and whatever else. Otherwise you are screwed once the dude who set this up left for another job. And honestly.. I’ve built software using node for the application services and managed Postgres/cache instances with basic replication to handle heavy traffic (10-20k rps) within 100ms. It requires heavy use of YAGNI and a bit of creativity tho which engineers seem to hate because they may not get to use the latest and shiniest tech. Totally understand but if you want money printer to go brrr you need to use the right tool for the job at the right time.
- zinclozenge 4y ago
  The mistake people make thinking about Kubernetes is that it's about scale, when really it's just a provider for common utility, with a common interface, that you need anyway. You still need to ingress traffic, you still need to deploy your services, etc.
  - 0xCAP 4y ago
    This, so much...
  - moomoo11 4y ago
    Thank you for putting it so simply! I agree.
ezekiel11 4y ago
the only rationale to do what you described is if and only if you have outside capital. If you are spending your hard earned boostrapped cash on this, I'm sorry but its a poor business decision that won't really net you any technical dividends. Again, I really see this the result of VC money chasing large valuations, and business decisions influencing technical archietcture, a sign of our times, of exuberance and senselessness Engineering has to raise the cost of engineering to match it (280 character limit crud app on AWS lambda with 2 full stack developers vs 2000 devs in an expensive office).

lbriner 4y ago

Totally agree! Kubernetes isn't just about global scale that most people will never need, which would agree with the article. It is about deploying new apps to an existing production system really quickly and easily. We can deploy a new app alongside an old app and proxy between them. Setting up a new application on IIS or a new web server to scale is a mare, doing the same on AKS (managed!) is a breeze. It is also really good value for money, because we can scale relatively quickly compared to dedicated servers. It is also harder to break something existing with a new deployment because of the container isolation. We might not need 1000 email services now but we could very quickly need that kind of scale and I don't want to be running up 100s of VMs at short notice as the business starts taking off when I can simply scale out the K8S deployment and add a few nodes. There is relatively little extra work (a dockerfile?) compared to hosting the same services on a web server.

philliphaydon 4y ago
> Setting up a new application on IIS or a new web server to scale is a mare. I disagree. With octopus deploy I can add a step, set a package and hostname, and press a button, and have a new deployment of a new api or website in IIS pushed out to how ever many servers currently exist in a few minutes. There are many ways to manage deploying services, and scaling, without K8S or containers in general. While I could setup a new Octopus Deploy quickly, it would be like you setting up something in AKS. We are both good at the tools we know. But saying my way or your way is wrong - is the thing thats wrong.
- boloust 4y ago
  That is very interesting, I'd love to know more about how this works. - How does it decide which servers to deploy to? - Does it scale up the number of servers if the deployment is too large? - What happens when a server dies?
  - philliphaydon 4y ago
    Well I guess it all depends on your setup. Servers get have a tentacle which has an environment and tag. So you could say a server is test environment and tagged with main-web-app. While another server is production with the same tag. You promote a package from test to production. You configure your setup to say a package is deployed to a server tagged main-web-app. Octopus is an orchestration tool so isn’t responsible for scaling up. But in AWS with an autoscale group you can configure Octopus to auto detect a new server tag it and deploy. Part of the deployment would add it to the load balancer. As a side note you can deploy containers using octopus too. Tho I’ve never found a reason to use containers in production yet.
- mountainriver 4y ago
  We’ll see if octopus deploy is around in 10 years
  - philliphaydon 4y ago

likortera 4y ago

> I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance Are you factoring in the salaries of the people setting up Kubernetes? And the cost of those people/salaries not working on the actual product? And the cost of those people leaving the company and leaving a ton of custom infrastructure code behind that the team can't quickly get up to speed with? > ton with huge bills especially from our managed db instance This doesn't have much to do with App Engine, right? Last time I used it, we were using a PostgresQL instance on AWS and had no problems with that. > Doing deployments was fine but complicated enough that only seasoned team members could do it safely I just plain don't believe this. I bet you were doing something wrong. How is it possible that the team find too difficult to do an App Engine deployment but then they're able to setup a full kubernetes cluster with all the stuff surrounding it? It's like saying I'm using React because JavaScript is too difficult. > Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge This is far, far, far from my own experience. Some other questions: How did you implement canary deployment? How much time are you investing in upgrading Kubernetes, and the underlying nodes operating systems? How did kubernetes solve the large database bills issue? How are you doing backups and restoration of the database now? If I were to found a company, specially not VC founded, dealing with kubernetes would be definitely far below on my list of priorities. But that's just me.

nightpool 4y ago
(I agree with your overall points, but when the GP said "Doing deployments was fine but complicated enough that only seasoned team members could do it safely" they were referring to their post-App Engine, pre-Kubernetes "manually managed VMs". I have no problem believing that deploys are very complicated in that scenario)
ezekiel11 4y ago
If setting up a PostgreSQL behind VPC with an EC2 in front of it was too difficult, there is now a serverless database product from AWS that costs 90% less what it used to. No more load balancers, no more VMs, no more scaling up or down to match demands.

hsn915 4y ago

> The biggest learning curve is for the person setting up the initial deployments, services, ingress etc Most other team members may just need to maybe change the image name and kubectl apply to roll things out. This is a huge redflag. It's basically admitting that you expect most later employees to not understand k8s or how its being used. You may think they don't need because it works, but you have to think about what happens when it doesn't work. The shops I've been to all had the same mindset: the docker/k8s infra was setup by one guy (or two) and no one else on the team understands what's going on, let alone have the ability to debug the configuration or fix problems with it. Another thing that might happen is some team members understand just barely enough to be able to "add" things to the config files. Over time the config files accumulate so much cruft, no one knows what configuration is used for what anymore.

lima 4y ago
You have this with literally every deployment mechanism, except with Kubernetes the boundary is standardized and you can easily find/hire/teach new team members to work on the complicated parts. Custom VM/cloud/$randomSaaS deployments are much worse when it comes to "the one guy who understands the intricate details is on vacation".
- jrimbault 4y ago
  At some point, this becomes the dividing line between a "nu-ops" team and the dev team.
- hsn915 4y ago
  Of course, if your deployment mechanism needs to be complicated, standarizing on something everyone knows is useful. The underlying assumption behind my comment is that you really really want to simplify your deployment as much as you can. Unfortunately this is nearly impossible with the currently accept set of standard best practices for developing web applications, where you use a scripting language and like six different database systems (one for source of truth on data, one for caching, one for full text search, and who knows what else everyone is using these days; I honestly can't keep track). There are many ways to make the deployment process simple. My rule of thumb is to use a compiled language (Go) and an embedded database engine. The levels.io guy famously did it even with php https://twitter.com/levelsio/status/1102487697220820994 https://twitter.com/levelsio/status/1102487697220820994

moomoo11 4y ago

Please don’t do this. I’m dealing with the mess caused by following this line of thinking. One guy (okay it was two guys) set up all the infrastructure and as soon as it was working bounced to new jobs with their newfound experience. The result is that dozens of engineers have no idea what the heck is going on and are lost navigating the numerous repos that hold various information related to deploying your feature. In my opinion (and I’m sure my opinion has flaws!), unless you have global customers and your actual users are pushing 1k requests per second load on your application servers/services, there is no reason to have these levels of abstractions. However once this becomes reality I think everyone working on that responsibility needs to learn k8s and whatever else. Otherwise you are screwed once the dude who set this up left for another job. And honestly.. I’ve built software using node for the application services and managed Postgres/cache instances with basic replication to handle heavy traffic (10-20k rps) within 100ms. It requires heavy use of YAGNI and a bit of creativity tho which engineers seem to hate because they may not get to use the latest and shiniest tech. Totally understand but if you want money printer to go brrr you need to use the right tool for the job at the right time.

zinclozenge 4y ago
The mistake people make thinking about Kubernetes is that it's about scale, when really it's just a provider for common utility, with a common interface, that you need anyway. You still need to ingress traffic, you still need to deploy your services, etc.
- 0xCAP 4y ago
  This, so much...
- moomoo11 4y ago
  Thank you for putting it so simply! I agree.

ezekiel11 4y ago

the only rationale to do what you described is if and only if you have outside capital. If you are spending your hard earned boostrapped cash on this, I'm sorry but its a poor business decision that won't really net you any technical dividends. Again, I really see this the result of VC money chasing large valuations, and business decisions influencing technical archietcture, a sign of our times, of exuberance and senselessness Engineering has to raise the cost of engineering to match it (280 character limit crud app on AWS lambda with 2 full stack developers vs 2000 devs in an expensive office).