9 ms·

This past year I’ve finally been able to hit the $200/hour rate as a k8s consultant. I feel like I’m selling snake oil. I’m working on my 7th implementation an

This past year I’ve finally been able to hit the $200/hour rate as a k8s consultant. I feel like I’m selling snake oil. I’m working on my 7th implementation and none of my customers need kubernetes. Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. My largest implementation was a $3m/year, 9 year old company who could run their entire infrastructure on a couple dozen ec2 instances. However they are under the delusion of “hitting it big and needing to be able to scale 100x”. Paid off my student loans this year, and soon my house, thanks to superfluous k8s integrations!

droopyEyelids 6y ago
You are the real job creator, as there will be more work after your migrations than there was before
deagle50 6y ago
I've been on the fence about this. If you don't mind, how did you get started?
- waynesonfire 6y ago
  Oh, I can tell you because I work with these asshats. They come in, deploy the technology, usually get promoted and then leave. The people left behind have to maintain their garbage and because promotions are only given to new feature work we're the suckers holding the bag.
  - ic0n0cl4st 6y ago
    Actually This is exactly how I accidentally started consulting on k8s. Three of my customers had engineers who advocated for k8s, did 50-75% of an implementation then left. I have told all of my customers they don’t need this, but they are convinced this is the way of the future.
    - deagle50 6y ago
      "Right side of history" is a powerful motivator, for better or worse. How did you get the first customer?
      - ic0n0cl4st 6y ago
        Right side of history? I dunno about anything like that. It’s just business, show me the money. There will be a new trend next year or the year after, ad nauseum. I’ve been consulting on and off for over 20 years now, I dont even remember my first 10 customers anymore. I get a lot of referrals from customers and from a consulting company I sold. I was primarily focusing on terraform, which is the only technology I’ve actully ever enjoyed professionally, but there’s just so much stupid $$ in k8s.
ideamotor 6y ago
Better delete this ;)
- b0afc375b5 6y ago
  Is this one of those rare instances when honesty isn't the best policy?
  - statstutor 6y ago
    Doubtful. The consultant who is willing to say that straight (and with the experience to know it) is going to be worth a lot more than $200/h, to the right clients.
    - ideamotor 6y ago
      Worth, yes, but valued? Now if I were looking to hire someone, I might come right back to this thread.
jameshart 6y ago
I’m confused. Isn’t a kubernetes cluster just a way to deploy software onto a couple of dozen ec2 instances? Sure it has some accidental complexity, but so does having half a dozen different software systems each managing their own deployment pathway onto ec2. There seems to be some belief that kubernetes has to be some kind of crazy complex system to operate, when really it’s just an api for managing which docker processes get spun up.
- BlargMcLarg 6y ago
  Most tools are easier than people make them out to be. The average dev just isn't that great, and the average company overestimates the difficulty of these tools, while burying their devs with mindless drone work so they never get the time to study these things. The average dev isn't one which has studied CS/software dev and/or spends loads of free time still investigating tech off the clock, either.
- throwdbaaway 6y ago
  > when really it’s just an api for managing which docker processes get spun up On GCP, the goal can be achieved without kubernetes, by deploying containers on VMs and MIGs: https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c... Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate. The same can also be done either on GCP with non-COS images or on AWS, with a little bit of automation: https://news.ycombinator.com/item?id=26274033 https://news.ycombinator.com/item?id=26274033
  - jameshart 6y ago
    Right. I said k8s is a way. Not the way. There might be others. They might have their own trade offs.
  - thisiszilff 6y ago
    > Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate. I think the fair comparison here would be that to google's kubernetes engine. Getting a kubernetes cluster up and running in GCP is really easy.
kkoncevicius 6y ago
I don't think this kind of attitude should be encouraged. Imagine other professions celebrating things like that. Surgeon: Some of my patients request bladder surgeries while in reality their problems could be solved with a few simple diet supplements. But I just do the procedure and take the money. Just bought my 3rd Tesla, thanks to superfluous bladder operations! I know selling people shit they don't need is celebrated as an achievement nowadays, but feels like it shouldn't be.
- bombcar 6y ago
  Tons of work done in various industries is "shitwork" that the person doing the work wouldn't normally do except they're being paid to do it. At some point you give up arguing with the customer. The dieting is a perfect example - many health issues can be solved by losing weight but few people want to make the lifestyle changes necessary to do that.
  - foobarbecue 6y ago
    Is your "shitwork" related to "bullshit jobs"? https://en.wikipedia.org/wiki/Bullshit_Jobs https://en.wikipedia.org/wiki/Bullshit_Jobs
  - BLKNSLVR 6y ago
    > few people want to make the lifestyle changes necessary to do that. I didn't have the will to make lifestyle changes until I had a serious bout of sciatica. I needed that external motivator of torturous pain to be able to commit to changing my eating and fitness habits. Two and a bit years later I'm down 12Kgs and slim and healthy (but, ironically, still have back pain issues that requires minor surgery to alleviate). "Don't worry, it's only for the rest of your life". If you can come to terms with that, you're ready to start.
    - ic0n0cl4st 6y ago
      Dude, sciatica was my catalyst for lifestyle change as well. I had a bicycle accident 14 years ago and damaged my piraformis. It was the most humbling and painful experience of my life.
jrochkind1 6y ago
Thank you for this! What in your opinion is the optimal way to run "a couple dozen ec2 instances"? I'm kind of in that zone. I don't think "totally manually" works out very well once you are in even "one dozen EC2 instances", it's too easy to make mistakes, you spend too much time (with downtime) figuring out what mistake you made and what it was supposed to be instead. Generally too hard to orchestrate changes without downtime. But is there anything in between totally manual and kubernetes? I think that's one reason people are reaching for kubernetes, they actually are having challenges that need a solution, and can't figure out what the solution is other than kubernetes.
- throwawayzRUU6f 6y ago
  Obviously a personal opinion, but Ansible solves this problem really well. It does get sluggish once we're talking about more than 50 machines, though. To reliably end up with a working cluster: - spin up plain operating system VMs - spin up one more VM for Ansible instigator - collect the IPs, write them down to Ansible inventory - pull/upload your custom binaries - run Ansible playbooks to set the VMs up Sometimes it's beneficial to have a two-step process. One playbook to install rarely changing components. Then make a VM snapshot. The second playbook installs your custom binaries. Future installations can then start from the snapshot.
  - deckard1 6y ago
    There are lots of ways to skin the cat. Back in the day I worked on code to create custom Linux ISOs with network provisioning. Always wondered why more people didn't do this. You just need a kickstart script. With a simple provision script you can have a fully automated install. No messing about copying ssh keys or setting IP addresses, etc.
- ic0n0cl4st 6y ago
  It’s unpopular, but I embrace (AWS) lock-in for small and mid-sized infrastructure. AWS’ various services solve 99.9% of any infrastructure needs. High-bandwidh streaming and heavy gpu processing is one of the few places my customers find to be cost prohibitive. For compute I lean towards an immutable infrastructure. I was a Puppet / Chef engineer for 8 years, never again. It’s cleaner to configure apps via some mixture of pre-baked amis, docker files and some sort of parameter store. (Usually SSM or Vault). Fargate is a cleaner path but so much more expensive.
salmonlogs 6y ago
Firstly - congrats on making great income by providing valuable skills to a hot market. When you say "none of my customers need kubernetes" is that because they are not using any of its benefits, or because they could do it another way and kubernetes was one of the options rather than the only option? In my experience very few people are fully exploiting everything k8s has to offer, but many people are getting side benefits from greater standardisation of deployments, consistency for implementation of logging, tracing and telemetry, reproducibility of environments and to an extend talent retention and development. I'm very interested to understand where the line is for genuinely getting zero benefits of using k8s.
deleted 6y ago
[deleted]
spaetzleesser 6y ago
"Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. " That's what I am doing now. Knowing this stuff makes you much more employable and raises your salary. It's a stupid world but resume driven development is a rational thing to do and very encouraged by hiring practices. For a long time I thought the company would appreciate making things simple and cost efficient but when I look at the latest hires it's pretty clear that the company wouldn't even hire most of the people who keep the business running because they are "outdated". You really have to keep churning constantly or you fall off the bandwagon. On the other hand I am pretty new to "modern" server and web development and the complexity is just insane. Getting a simple API server and frontend going with comprehensive testing, CI/CD, containers, dependency injection and whatever is a huge effort. It barely leaves you time to think about writing efficient quality code because you are so busy with pull requests, writing test code and other stuff.
licebmi__at__ 6y ago
I'm in your position, but in my opinion, there are plenty of who are better in kubernetes even if they don't need it. There is complexity in the processes and even the scripting people use to glue things together, and there are enough places for the team to screw up.
ummonk 6y ago
I rarely see instances where kubernetes actually makes sense over just using regular AWS services (load balancing, autoscaling, AMIs, etc.) directly. It seems like k8s just adds a mostly superfluous extra layer of abstraction with a rather user-unfriendly interface.

droopyEyelids 6y ago

You are the real job creator, as there will be more work after your migrations than there was before

deagle50 6y ago

I've been on the fence about this. If you don't mind, how did you get started?

waynesonfire 6y ago
Oh, I can tell you because I work with these asshats. They come in, deploy the technology, usually get promoted and then leave. The people left behind have to maintain their garbage and because promotions are only given to new feature work we're the suckers holding the bag.
- ic0n0cl4st 6y ago
  Actually This is exactly how I accidentally started consulting on k8s. Three of my customers had engineers who advocated for k8s, did 50-75% of an implementation then left. I have told all of my customers they don’t need this, but they are convinced this is the way of the future.
  - deagle50 6y ago
    "Right side of history" is a powerful motivator, for better or worse. How did you get the first customer?
    - ic0n0cl4st 6y ago
      Right side of history? I dunno about anything like that. It’s just business, show me the money. There will be a new trend next year or the year after, ad nauseum. I’ve been consulting on and off for over 20 years now, I dont even remember my first 10 customers anymore. I get a lot of referrals from customers and from a consulting company I sold. I was primarily focusing on terraform, which is the only technology I’ve actully ever enjoyed professionally, but there’s just so much stupid $$ in k8s.

ideamotor 6y ago

Better delete this ;)

b0afc375b5 6y ago
Is this one of those rare instances when honesty isn't the best policy?
- statstutor 6y ago
  Doubtful. The consultant who is willing to say that straight (and with the experience to know it) is going to be worth a lot more than $200/h, to the right clients.
  - ideamotor 6y ago
    Worth, yes, but valued? Now if I were looking to hire someone, I might come right back to this thread.

jameshart 6y ago

I’m confused. Isn’t a kubernetes cluster just a way to deploy software onto a couple of dozen ec2 instances? Sure it has some accidental complexity, but so does having half a dozen different software systems each managing their own deployment pathway onto ec2. There seems to be some belief that kubernetes has to be some kind of crazy complex system to operate, when really it’s just an api for managing which docker processes get spun up.

BlargMcLarg 6y ago
Most tools are easier than people make them out to be. The average dev just isn't that great, and the average company overestimates the difficulty of these tools, while burying their devs with mindless drone work so they never get the time to study these things. The average dev isn't one which has studied CS/software dev and/or spends loads of free time still investigating tech off the clock, either.
throwdbaaway 6y ago
> when really it’s just an api for managing which docker processes get spun up On GCP, the goal can be achieved without kubernetes, by deploying containers on VMs and MIGs: https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c... Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate. The same can also be done either on GCP with non-COS images or on AWS, with a little bit of automation: https://news.ycombinator.com/item?id=26274033 https://news.ycombinator.com/item?id=26274033
- jameshart 6y ago
  Right. I said k8s is a way. Not the way. There might be others. They might have their own trade offs.
- thisiszilff 6y ago
  > Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate. I think the fair comparison here would be that to google's kubernetes engine. Getting a kubernetes cluster up and running in GCP is really easy.

kkoncevicius 6y ago

I don't think this kind of attitude should be encouraged. Imagine other professions celebrating things like that. Surgeon: Some of my patients request bladder surgeries while in reality their problems could be solved with a few simple diet supplements. But I just do the procedure and take the money. Just bought my 3rd Tesla, thanks to superfluous bladder operations! I know selling people shit they don't need is celebrated as an achievement nowadays, but feels like it shouldn't be.

bombcar 6y ago
Tons of work done in various industries is "shitwork" that the person doing the work wouldn't normally do except they're being paid to do it. At some point you give up arguing with the customer. The dieting is a perfect example - many health issues can be solved by losing weight but few people want to make the lifestyle changes necessary to do that.
- foobarbecue 6y ago
  Is your "shitwork" related to "bullshit jobs"? https://en.wikipedia.org/wiki/Bullshit_Jobs https://en.wikipedia.org/wiki/Bullshit_Jobs
- BLKNSLVR 6y ago
  > few people want to make the lifestyle changes necessary to do that. I didn't have the will to make lifestyle changes until I had a serious bout of sciatica. I needed that external motivator of torturous pain to be able to commit to changing my eating and fitness habits. Two and a bit years later I'm down 12Kgs and slim and healthy (but, ironically, still have back pain issues that requires minor surgery to alleviate). "Don't worry, it's only for the rest of your life". If you can come to terms with that, you're ready to start.
  - ic0n0cl4st 6y ago
    Dude, sciatica was my catalyst for lifestyle change as well. I had a bicycle accident 14 years ago and damaged my piraformis. It was the most humbling and painful experience of my life.

jrochkind1 6y ago

Thank you for this! What in your opinion is the optimal way to run "a couple dozen ec2 instances"? I'm kind of in that zone. I don't think "totally manually" works out very well once you are in even "one dozen EC2 instances", it's too easy to make mistakes, you spend too much time (with downtime) figuring out what mistake you made and what it was supposed to be instead. Generally too hard to orchestrate changes without downtime. But is there anything in between totally manual and kubernetes? I think that's one reason people are reaching for kubernetes, they actually are having challenges that need a solution, and can't figure out what the solution is other than kubernetes.

throwawayzRUU6f 6y ago
Obviously a personal opinion, but Ansible solves this problem really well. It does get sluggish once we're talking about more than 50 machines, though. To reliably end up with a working cluster: - spin up plain operating system VMs - spin up one more VM for Ansible instigator - collect the IPs, write them down to Ansible inventory - pull/upload your custom binaries - run Ansible playbooks to set the VMs up Sometimes it's beneficial to have a two-step process. One playbook to install rarely changing components. Then make a VM snapshot. The second playbook installs your custom binaries. Future installations can then start from the snapshot.
- deckard1 6y ago
  There are lots of ways to skin the cat. Back in the day I worked on code to create custom Linux ISOs with network provisioning. Always wondered why more people didn't do this. You just need a kickstart script. With a simple provision script you can have a fully automated install. No messing about copying ssh keys or setting IP addresses, etc.
ic0n0cl4st 6y ago
It’s unpopular, but I embrace (AWS) lock-in for small and mid-sized infrastructure. AWS’ various services solve 99.9% of any infrastructure needs. High-bandwidh streaming and heavy gpu processing is one of the few places my customers find to be cost prohibitive. For compute I lean towards an immutable infrastructure. I was a Puppet / Chef engineer for 8 years, never again. It’s cleaner to configure apps via some mixture of pre-baked amis, docker files and some sort of parameter store. (Usually SSM or Vault). Fargate is a cleaner path but so much more expensive.

salmonlogs 6y ago

Firstly - congrats on making great income by providing valuable skills to a hot market. When you say "none of my customers need kubernetes" is that because they are not using any of its benefits, or because they could do it another way and kubernetes was one of the options rather than the only option? In my experience very few people are fully exploiting everything k8s has to offer, but many people are getting side benefits from greater standardisation of deployments, consistency for implementation of logging, tracing and telemetry, reproducibility of environments and to an extend talent retention and development. I'm very interested to understand where the line is for genuinely getting zero benefits of using k8s.

deleted 6y ago

[deleted]

spaetzleesser 6y ago

"Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. " That's what I am doing now. Knowing this stuff makes you much more employable and raises your salary. It's a stupid world but resume driven development is a rational thing to do and very encouraged by hiring practices. For a long time I thought the company would appreciate making things simple and cost efficient but when I look at the latest hires it's pretty clear that the company wouldn't even hire most of the people who keep the business running because they are "outdated". You really have to keep churning constantly or you fall off the bandwagon. On the other hand I am pretty new to "modern" server and web development and the complexity is just insane. Getting a simple API server and frontend going with comprehensive testing, CI/CD, containers, dependency injection and whatever is a huge effort. It barely leaves you time to think about writing efficient quality code because you are so busy with pull requests, writing test code and other stuff.

licebmi__at__ 6y ago

I'm in your position, but in my opinion, there are plenty of who are better in kubernetes even if they don't need it. There is complexity in the processes and even the scripting people use to glue things together, and there are enough places for the team to screw up.

ummonk 6y ago

I rarely see instances where kubernetes actually makes sense over just using regular AWS services (load balancing, autoscaling, AMIs, etc.) directly. It seems like k8s just adds a mostly superfluous extra layer of abstraction with a rather user-unfriendly interface.