10 ms·

This, and the other articles like it, should be required reading on any "how to startup" list. I personally know startups for whom I believe drinking the k8s/go

by sho 7y ago

This, and the other articles like it, should be required reading on any "how to startup" list. I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time. For request loads one hundredth of what I was handling effortlessly with a monolithic Rails server in 2013. It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. k8s on day one at a startup is like a mom and pop grocery store buying SAP. It wouldn't be acceptable in any other industry, and can be a death sentence.

collyw 7y ago
Some seniors get exited by that crap as well. Tech lead in my team is full of common sense when it comes to not getting exited by JavaScripts framework of the season. But he loves to add new tools to the back end. He is choosing the "best tool" for the job, but we have a small team and already have way more things than we can comfortably manage. Good enough tools that are already in our system would be more pragmatic.
- valkon 7y ago
  I think it's because you don't see rapid changes on the back-end. We still consider Java 8 new.
- zentiggr 7y ago
  Hopefully that's not a case of "must be seen to be doing something" syndrome. If you can use pre-existing tools and they serve the purpose, how easy or difficult is it to resist the "shift to this" pressure?
marcinzm 7y ago
>It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. Then they might simply join another startup or a big tech company as competition for good engineers is fierce. Startups also famously underpay versus larger companies so you need to entice engineers with something.
- encoderer 7y ago
  It’s ok to let them go.
  - marcinzm 7y ago
    That depends on the situation and is the CTO's decision to make. Given the difficulty and time frames of hiring letting go could mean another 6 months of too few engineers. Acting like there is a single universal answer is the wrong world view I'd say.
    - wpietri 7y ago
      If you have a nominally full compliment of people who are over-building or playing around with shiny toys instead of delivering value, then you still have too few actual engineers, but are paying a lot more money for the privilege. You're also setting up a bill that will come due eventually. I've made some really good money going into companies and ripping out the hot-three-years-ago garbage that some long-gone goof put in. Last time this happened I looked up the responsible party. Turned out he was doing the work so he could give a conference talk on shiny tech. Not long after the talk was done, he took a job somewhere else, leaving his buzzword-compliant half-baked system to rot.
- ragebol 7y ago
  Maybe you need less engineers trained in the new hotness but just experience with, uh, older hotness. Which may be easier to find.
  - marcinzm 7y ago
    Engineers want what is best for them and not what is best for the company. That is the right choice for them. New hotness is one way to entice them. Far, far from the only one but it is a tool in the CTO's tool belt.
    - eeZah7Ux 7y ago
AznHisoka 7y ago
When I was younger, I was much more enticed with new technologies. But as I grow older, I've grown much more cynical. I just want to solve problems with ideally as little coding/thinking as possible.
- deleted 7y ago
  [deleted]
- ttflee 7y ago
  Old-man-yells-at-cloud.
  - hkt 7y ago
    Damn cloud.
    - missosoup 7y ago
      Cloud get off my lawn.
      - TeMPOraL 7y ago
        In late-stage capitalism, the cloud gets you off "your" lawn, when your Lawn-as-a-Service provider gets acquihired by an adtech company.
        dvtrn 7y ago
        "One of us...one of us..."
  - raxxorrax 7y ago
    I am not that old, develop most apps besides embedded firmware for the cloud and have been yelling at it from the beginning to absolutely no effect. In my experience it is often C-level customers who embrace the cloud with most enthusiasm. Interestingly the older generation often had the most reservations against hosting data on external systems. They are generally very big on everything surveillance though.
rogerkirkness 7y ago
The lack of type checking must have cost way more than standing up Golang services when the time came that you actually had customers though right?
- hkt 7y ago
  People managed with dynamically typed languages for a long, long time in building pretty vast services. Static typing has its merits but it really depends on what you're building - if you're a bank, it offers excellent guarantees. If you're building something smaller scale or lower impact, you can cope if you're logical and have good tests.
  - rogerkirkness 7y ago
    My understanding is that 1. Sooner or later you end up having to check all the types in your tests which means 2. You end up having the same number of lines of code in each, one where the type is declared in the model, which realistically is easier to document and to keep your data consistent, and the other where the only time you verify it is in the tests, or worse both tests and runtime, in which case by definition it's more lines of code. If lines of code is a proxy for bugs, and you build on something that doesn't have type safety, wouldn't that just increase your bugs across two very important immutable dimensions of computing? Stripe has a type checking team now...
TallGuyShort 7y ago
I've seen the pressure come from customers too, in the case of b2b between tech companies. My company embraced Kubernetes after years of customers asking us "what's your Kubernetes story?" and not liking the answer, "that doesn't solve any problems we have".
etxm 7y ago
I absolutely love Kubernetes, it makes my job so much easier... I can’t understand for the life of me why any start up uses it, it’s insane.
inkeddeveloper 7y ago
Sure, it is the job of the CTO to steer juniors in an appropriate direction. But, and I hate to say this, isn't it also the job of the CTO to help source funding and it seems VCs like to go after the buzzwords?
zeveb 7y ago
> I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time. If Kubernetes had only cost us a year and two hundred thousand dollars then we'd have been luckier than we actually are. It definitely has a place, but it is so not a good idea for a small team. You don't need K8s until you start to build a half-assed K8s.
- merb 7y ago
  > You don't need K8s until you start to build a half-assed K8s. which you start doing with more than 1 db server and more than one app server. until you realize you have a ansible script that is tied to your specific program. oh shit now you have two programs, and copied some stuff from ansible, but not everything is the same - damn. deployments occur a downtime (5 seconds), which some user notice - until you add like a thousand lines of ansible code. now you need monitoring and oh shit another ansible bloat, soon you outgrown k8s codebase with just ansible. (p.s. this does not account how you start your baremetal servers, etc. this would be another story)
  - imtringued 7y ago
    What are you talking about? Just create two more virtual machines. Copy some files over and run `docker-compose up`. After that you only need to configure a load balancer.
lowercased 7y ago
> It is the job of the CTO to steer excitable juniors away from the new hotness Sometimes it's the CTO who is the excitable one pursuing new hotness...
- rbanffy 7y ago
  Then the company is doomed.
theptip 7y ago
Anecdata: series B startup. First year was a single VM with two docker containers for api and nginx. Deploy was a one shot “pull new container, stop old, start new” shell command. Year 2 onwards, k8s. No regrets, we only needed to make our first dedicated ops hire after 15 engineers, due in large part to the infra being so easy to program. I used GKE and I was also very familiar with k8s ahead of time. I would not recommend someone in my shoes to learn k8s from scratch at the stage I rolled it out, but if you know it already, it’s a solid choice for the first point that you want a second instance.
- jiofih 7y ago
  When people talk about companies using k8s too soon, they are talking about deploying it on your own, not using a hosted service like GCE. That’s a whole new ball game and takes 100x the effort.
  - deleted 7y ago
    [deleted]
  - theptip 7y ago
    I read TFA as mostly complaining about the conceptual and operational complexity of running your code on k8s, not so much about operating a cluster itself. Lots of ink spilled on irrelevant concepts that most users don't need to know or care about like EndpointSlices. And, arguing against microservices is a reasonable position -- but IF you have made that architectural choice, then Docker-for-Mac + the built-in kubernetes cluster is the most developer-friendly way of working on microservices that I am aware of. So a bit of a non-sequiteur there.
    - jiofih 7y ago
      I don’t see when you’d need to understand what an EndpointSlice is, unless running k8s itself. The concept does not leak through any of the pod management interfaces.
Kaze404 7y ago
I worked at a startup where the CTO was the impressionable junior and it caused all of those issues you described and more. I ended up leaving when I decided I shouldn't be having panic attacks in the middle of the night due to ridiculously tight deadlines that I couldn't meet because of the overhead introduced by k8s and microservices.
mirko22 7y ago
1000 users, almost 0 request per second / 10000 euro a month to deploy a very simple software on GKE in 3 continents and have some development instances. Why? Because Docker and "scalability" it offered looked much better on the investor slides... How? Instead of actually hiring someone that at least has experience with docker he decided it was very easy thing to learn so he did it him self and we ended up with thing like running two application in a same container, having database on containers that disappear (container restarts when ram is full is the best way to run a database) etc... And after all that started talking about mirco-services and how cool are they for code decoupling... Of all the things... I don't work there anymore... And if challenged on these reasons some people give (that supposedly have more experience) blanket statements like: "Docker is more safe by default thus we should use it..." Maybe when you go thought these situations you get to write articles like this. Of course containers, docker, k8 etc have their place, but in reality you can find all kinds of stunning non-sense.
simplecto 7y ago
Exactly this. For most startups Heroku, Dokku, or plain docker is enough. The stack will certainly evolve as growth (and success) comes. Building with 12-factor principles make that transition effortless when the time comes.
- thu2111 7y ago
  What about raw systemd? It can do resource control, service monitoring and restart and seems pretty lightweight compared to Docker. Fewer ways to footgun with data being erased, no separate shells or finding a tool you need isn't available inside the container.
  - hedora 7y ago
    Or, you could use a distro with a sane init system, which eliminates all the systemd and reinvented wheel footguns. The last time I installed ubuntu 18.04, DNS queries took ~5 seconds. It’s a well known issue, with no diagnosed root cause. The solutions involved uninstalling the local dns stack, starting with systemd’s resolver. 2018 was well after DNS was reliable. How can stuff like that break in a long term support release?
    - montjoy 7y ago
      5 seconds is the amount of time Linux takes to fail over to the next dns resolver in resolv.conf. You likely had a bad/unreachable DNS server.
      - Spivak 7y ago
        Which is ironic because systemd got a lot of hate for doing the right thing and not just querying DNS servers in order but discovering the remembering the servers that are up/fastest. Turns out a lot of people just ignored the fact that all configured DNS servers are assumed to serve the same records and used DNS ordering to implement shitty split-horizon DNS.
  - wdroz 7y ago

by sho 7y ago

collyw 7y ago

Some seniors get exited by that crap as well. Tech lead in my team is full of common sense when it comes to not getting exited by JavaScripts framework of the season. But he loves to add new tools to the back end. He is choosing the "best tool" for the job, but we have a small team and already have way more things than we can comfortably manage. Good enough tools that are already in our system would be more pragmatic.

valkon 7y ago
I think it's because you don't see rapid changes on the back-end. We still consider Java 8 new.
zentiggr 7y ago
Hopefully that's not a case of "must be seen to be doing something" syndrome. If you can use pre-existing tools and they serve the purpose, how easy or difficult is it to resist the "shift to this" pressure?

marcinzm 7y ago

>It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. Then they might simply join another startup or a big tech company as competition for good engineers is fierce. Startups also famously underpay versus larger companies so you need to entice engineers with something.

encoderer 7y ago
It’s ok to let them go.
- marcinzm 7y ago
  That depends on the situation and is the CTO's decision to make. Given the difficulty and time frames of hiring letting go could mean another 6 months of too few engineers. Acting like there is a single universal answer is the wrong world view I'd say.
  - wpietri 7y ago
    If you have a nominally full compliment of people who are over-building or playing around with shiny toys instead of delivering value, then you still have too few actual engineers, but are paying a lot more money for the privilege. You're also setting up a bill that will come due eventually. I've made some really good money going into companies and ripping out the hot-three-years-ago garbage that some long-gone goof put in. Last time this happened I looked up the responsible party. Turned out he was doing the work so he could give a conference talk on shiny tech. Not long after the talk was done, he took a job somewhere else, leaving his buzzword-compliant half-baked system to rot.
ragebol 7y ago
Maybe you need less engineers trained in the new hotness but just experience with, uh, older hotness. Which may be easier to find.
- marcinzm 7y ago
  Engineers want what is best for them and not what is best for the company. That is the right choice for them. New hotness is one way to entice them. Far, far from the only one but it is a tool in the CTO's tool belt.
  - eeZah7Ux 7y ago

AznHisoka 7y ago

When I was younger, I was much more enticed with new technologies. But as I grow older, I've grown much more cynical. I just want to solve problems with ideally as little coding/thinking as possible.

deleted 7y ago
[deleted]
ttflee 7y ago
Old-man-yells-at-cloud.
- hkt 7y ago
  Damn cloud.
  - missosoup 7y ago
    Cloud get off my lawn.
    - TeMPOraL 7y ago
      In late-stage capitalism, the cloud gets you off "your" lawn, when your Lawn-as-a-Service provider gets acquihired by an adtech company.
      - dvtrn 7y ago
        "One of us...one of us..."
- raxxorrax 7y ago
  I am not that old, develop most apps besides embedded firmware for the cloud and have been yelling at it from the beginning to absolutely no effect. In my experience it is often C-level customers who embrace the cloud with most enthusiasm. Interestingly the older generation often had the most reservations against hosting data on external systems. They are generally very big on everything surveillance though.

rogerkirkness 7y ago

The lack of type checking must have cost way more than standing up Golang services when the time came that you actually had customers though right?

hkt 7y ago
People managed with dynamically typed languages for a long, long time in building pretty vast services. Static typing has its merits but it really depends on what you're building - if you're a bank, it offers excellent guarantees. If you're building something smaller scale or lower impact, you can cope if you're logical and have good tests.
- rogerkirkness 7y ago
  My understanding is that 1. Sooner or later you end up having to check all the types in your tests which means 2. You end up having the same number of lines of code in each, one where the type is declared in the model, which realistically is easier to document and to keep your data consistent, and the other where the only time you verify it is in the tests, or worse both tests and runtime, in which case by definition it's more lines of code. If lines of code is a proxy for bugs, and you build on something that doesn't have type safety, wouldn't that just increase your bugs across two very important immutable dimensions of computing? Stripe has a type checking team now...

TallGuyShort 7y ago

I've seen the pressure come from customers too, in the case of b2b between tech companies. My company embraced Kubernetes after years of customers asking us "what's your Kubernetes story?" and not liking the answer, "that doesn't solve any problems we have".

etxm 7y ago

I absolutely love Kubernetes, it makes my job so much easier... I can’t understand for the life of me why any start up uses it, it’s insane.

inkeddeveloper 7y ago

Sure, it is the job of the CTO to steer juniors in an appropriate direction. But, and I hate to say this, isn't it also the job of the CTO to help source funding and it seems VCs like to go after the buzzwords?

zeveb 7y ago

> I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time. If Kubernetes had only cost us a year and two hundred thousand dollars then we'd have been luckier than we actually are. It definitely has a place, but it is so not a good idea for a small team. You don't need K8s until you start to build a half-assed K8s.

merb 7y ago
> You don't need K8s until you start to build a half-assed K8s. which you start doing with more than 1 db server and more than one app server. until you realize you have a ansible script that is tied to your specific program. oh shit now you have two programs, and copied some stuff from ansible, but not everything is the same - damn. deployments occur a downtime (5 seconds), which some user notice - until you add like a thousand lines of ansible code. now you need monitoring and oh shit another ansible bloat, soon you outgrown k8s codebase with just ansible. (p.s. this does not account how you start your baremetal servers, etc. this would be another story)
- imtringued 7y ago
  What are you talking about? Just create two more virtual machines. Copy some files over and run `docker-compose up`. After that you only need to configure a load balancer.

lowercased 7y ago

> It is the job of the CTO to steer excitable juniors away from the new hotness Sometimes it's the CTO who is the excitable one pursuing new hotness...

rbanffy 7y ago
Then the company is doomed.

theptip 7y ago

Anecdata: series B startup. First year was a single VM with two docker containers for api and nginx. Deploy was a one shot “pull new container, stop old, start new” shell command. Year 2 onwards, k8s. No regrets, we only needed to make our first dedicated ops hire after 15 engineers, due in large part to the infra being so easy to program. I used GKE and I was also very familiar with k8s ahead of time. I would not recommend someone in my shoes to learn k8s from scratch at the stage I rolled it out, but if you know it already, it’s a solid choice for the first point that you want a second instance.

jiofih 7y ago
When people talk about companies using k8s too soon, they are talking about deploying it on your own, not using a hosted service like GCE. That’s a whole new ball game and takes 100x the effort.
- deleted 7y ago
  [deleted]
- theptip 7y ago
  I read TFA as mostly complaining about the conceptual and operational complexity of running your code on k8s, not so much about operating a cluster itself. Lots of ink spilled on irrelevant concepts that most users don't need to know or care about like EndpointSlices. And, arguing against microservices is a reasonable position -- but IF you have made that architectural choice, then Docker-for-Mac + the built-in kubernetes cluster is the most developer-friendly way of working on microservices that I am aware of. So a bit of a non-sequiteur there.
  - jiofih 7y ago
    I don’t see when you’d need to understand what an EndpointSlice is, unless running k8s itself. The concept does not leak through any of the pod management interfaces.

Kaze404 7y ago

I worked at a startup where the CTO was the impressionable junior and it caused all of those issues you described and more. I ended up leaving when I decided I shouldn't be having panic attacks in the middle of the night due to ridiculously tight deadlines that I couldn't meet because of the overhead introduced by k8s and microservices.

mirko22 7y ago

1000 users, almost 0 request per second / 10000 euro a month to deploy a very simple software on GKE in 3 continents and have some development instances. Why? Because Docker and "scalability" it offered looked much better on the investor slides... How? Instead of actually hiring someone that at least has experience with docker he decided it was very easy thing to learn so he did it him self and we ended up with thing like running two application in a same container, having database on containers that disappear (container restarts when ram is full is the best way to run a database) etc... And after all that started talking about mirco-services and how cool are they for code decoupling... Of all the things... I don't work there anymore... And if challenged on these reasons some people give (that supposedly have more experience) blanket statements like: "Docker is more safe by default thus we should use it..." Maybe when you go thought these situations you get to write articles like this. Of course containers, docker, k8 etc have their place, but in reality you can find all kinds of stunning non-sense.

simplecto 7y ago

Exactly this. For most startups Heroku, Dokku, or plain docker is enough. The stack will certainly evolve as growth (and success) comes. Building with 12-factor principles make that transition effortless when the time comes.

thu2111 7y ago
What about raw systemd? It can do resource control, service monitoring and restart and seems pretty lightweight compared to Docker. Fewer ways to footgun with data being erased, no separate shells or finding a tool you need isn't available inside the container.
- hedora 7y ago
  Or, you could use a distro with a sane init system, which eliminates all the systemd and reinvented wheel footguns. The last time I installed ubuntu 18.04, DNS queries took ~5 seconds. It’s a well known issue, with no diagnosed root cause. The solutions involved uninstalling the local dns stack, starting with systemd’s resolver. 2018 was well after DNS was reliable. How can stuff like that break in a long term support release?
  - montjoy 7y ago
    5 seconds is the amount of time Linux takes to fail over to the next dns resolver in resolv.conf. You likely had a bad/unreachable DNS server.
    - Spivak 7y ago
      Which is ironic because systemd got a lot of hate for doing the right thing and not just querying DNS servers in order but discovering the remembering the servers that are up/fastest. Turns out a lot of people just ignored the fact that all configured DNS servers are assumed to serve the same records and used DNS ordering to implement shitty split-horizon DNS.
- wdroz 7y ago