3 ms·

> I've never come across a devops person who'd say "Hey, that sofware is too simple to prematurely scale for sudden spikes of irrational amounts of traffic, so

by znpy 3y ago

> I've never come across a devops person who'd say "Hey, that sofware is too simple to prematurely scale for sudden spikes of irrational amounts of traffic, so why not just deploy it on a cheap vps?" Was a devops engineer at my previous job. We already had k8s clusters setup, pre-made CI templates and pre-tailored helm charts (along with monitoring and much more). All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place). Creating (and maintaining) a single vm for a pet project is way more work than using the pre-made and pre-customized and curated toolkit. This was at a 100+ developers organisation. If you think you could easily get off with a single vm then you've never seen devops done right, i'm fairly sure. EDIT: I probably fell for the bait, but the post i'm replying to really made me remember why we went on a killing spree in order to eradicate everything that was not k8s at my previous job and removing as much developer access to prod as possible. Some idiot developers think they know better, usually end up re-inventing a square wheel that breaks as soon as it's not running on their laptop anymore.

zer00eyz 3y ago
You're not wrong. K8s, containers in general, aren't solving the problem. Developers write apps, engineers write software. One is installable, it respects the system it lives on, It can be tuned to use the hardware it has or share. The other, the other is a bucket of shit in a bag. The bag protects you and everyone else from it contaminating everything it touches. Containers dont solve the problem they just enable it.
- worik 3y ago
  > Developers write apps, engineers write software. That is a classic distinction without a difference! Engineers have been to engineering school. If you have not been to engineering school, you are not an engineer!
  - zer00eyz 3y ago
    It's not a dead language, Im sorry the valley stole your title but that ship has sailed.
worik 3y ago
Yes you took the bait! > This was at a 100+ developers organisation. IMO the vast majority of software development happens in much smaller organisations than that. Dev Ops still matters there, and the requirements are different. I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here. It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data? That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things. Bitter. Experience.
- znpy 3y ago
  > I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here. not sure you're doing the right thing here. you might want to consider hiring some kind of linux guy that can can do some basic devops. or maybe hire some devops contractor that can work with you on a part-time basis and "curate" some specific aspects of your operations. I've seen this done in the past: so you've got this consultant on retainer, and you tell them something like: "i've got this issue, can we do something about it? our constraints would be x y z" and the consultant would make 1-3 proposals (different approaches, different pricing levels, different ETAs etc) and then you agree on what gets done. The key aspect here is that a good devops consultant can get stuff done very quickly. > It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data? Yes (to all questions). I ended up in working in heavily regulated environments. All the things you mentioned were not just niceties, but mandatory by legal requirements. > That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things. That's basic system administration. Most devops engineers are former sysadmins.
- jameshart 3y ago
  > IMO the vast majority of software development happens in much smaller organizations than that I guess it's okay to have an opinion about that, but this seems like something that should probably be a fact. Unfortunately, not sure I can find reliable stats on sizes of engineering organizations. The thing is, while there are obviously lots of small companies, there are also some really big software development organizations out there. A company like Netflix has 2,500 engineers. Microsoft employs over 100,000 engineers. Walmart employs over 15,000 software developers. You need a lot of little 10-50 engineer dev shops to add up to the combined size of the engineering orgs of the Fortune 500. According to https://www.statista.com/statistics/507530/united-states-distribution-it-industry-employment-by-company-size/ https://www.statista.com/statistics/507530/united-states-dis..., at least, 29.4+25.8 = 55.2% of the US "IT Industry" workforce are employed in companies with >100 employees. That's a Long way from telling us about sizes of engineering orgs though. But still... I'd be careful assuming that the vast majority of developers are in organizations of less than 100 people.
GauntletWizard 3y ago
Yeah, I'm an independent, selling these solutions. My minimum stack costs about $500/mo in AWS costs- and you could save 90% of that. But then you'd be paying me $20k more to set it up again when you expand to another dev team, while this way I can add your second dev team for $10k and practically zero additional AWS cost. Going straight to overkill is good business sense for any company that's going to make it to a medium-sized business, and it's not going to be the differentiating factor for a company that burns out.
ongy 3y ago
That's really the kicker for me. When the company is set up well on k8s, then choose k8s. If the company is set up well on VPS, then choose VPS. If it got neither, I'm unsure what's the better way to go on a greenfield. k8s has nice tooling, but part of that is required because it is massively complex. But with managed k8s providers (e.g. GKE with autopilot) you can just put in proper CPU and memory limits and essentially get yourself a VPS provisioned without having to worry about anything by k8s measures. If you have different pricing or location constraints, then it might be better to go with different models and some custom orchestration.
richardw 3y ago
But you’ve got 100 developers. That’s not “most web apps”, that’s firmly in the set of companies that need standardisation and potential scale, where the devops team makes the life of many devs far better. When it’s just a few devs and cash is limited, the business doesn’t need the complexity. It’s mostly for ego and branding.
ProblemExplorer 3y ago
> All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place). How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how?
- znpy 3y ago
  > How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how? All those questions are irrelevant to how software gets deployed. Do whatever you want on your laptop.
  - ProblemExplorer 3y ago
    Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems? The underlying orchestrator definitely affects how the software needs to behave and is definitely not irrelevant.
    - znpy 3y ago
      > Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems? nginx-ingress-controller does that. no custom stuff required. https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ https://kubernetes.github.io/ingress-nginx/user-guide/exposi...
    - ongy 3y ago
      Which LB doesn't support non-http flows?
    - jameshart 3y ago
      Follow the signposts laid out by https://12factor.net https://12factor.net. Think about how to apply those ideas to containerized applications. Consider that such applications can be run in the same way locally, on a server, or in an autoscaled cluster. Not sure why TCP/UDP load balancing complicates matters.

by znpy 3y ago

zer00eyz 3y ago

You're not wrong. K8s, containers in general, aren't solving the problem. Developers write apps, engineers write software. One is installable, it respects the system it lives on, It can be tuned to use the hardware it has or share. The other, the other is a bucket of shit in a bag. The bag protects you and everyone else from it contaminating everything it touches. Containers dont solve the problem they just enable it.

worik 3y ago
> Developers write apps, engineers write software. That is a classic distinction without a difference! Engineers have been to engineering school. If you have not been to engineering school, you are not an engineer!
- zer00eyz 3y ago
  It's not a dead language, Im sorry the valley stole your title but that ship has sailed.

worik 3y ago

Yes you took the bait! > This was at a 100+ developers organisation. IMO the vast majority of software development happens in much smaller organisations than that. Dev Ops still matters there, and the requirements are different. I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here. It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data? That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things. Bitter. Experience.

znpy 3y ago
> I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here. not sure you're doing the right thing here. you might want to consider hiring some kind of linux guy that can can do some basic devops. or maybe hire some devops contractor that can work with you on a part-time basis and "curate" some specific aspects of your operations. I've seen this done in the past: so you've got this consultant on retainer, and you tell them something like: "i've got this issue, can we do something about it? our constraints would be x y z" and the consultant would make 1-3 proposals (different approaches, different pricing levels, different ETAs etc) and then you agree on what gets done. The key aspect here is that a good devops consultant can get stuff done very quickly. > It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data? Yes (to all questions). I ended up in working in heavily regulated environments. All the things you mentioned were not just niceties, but mandatory by legal requirements. > That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things. That's basic system administration. Most devops engineers are former sysadmins.
jameshart 3y ago
> IMO the vast majority of software development happens in much smaller organizations than that I guess it's okay to have an opinion about that, but this seems like something that should probably be a fact. Unfortunately, not sure I can find reliable stats on sizes of engineering organizations. The thing is, while there are obviously lots of small companies, there are also some really big software development organizations out there. A company like Netflix has 2,500 engineers. Microsoft employs over 100,000 engineers. Walmart employs over 15,000 software developers. You need a lot of little 10-50 engineer dev shops to add up to the combined size of the engineering orgs of the Fortune 500. According to https://www.statista.com/statistics/507530/united-states-distribution-it-industry-employment-by-company-size/ https://www.statista.com/statistics/507530/united-states-dis..., at least, 29.4+25.8 = 55.2% of the US "IT Industry" workforce are employed in companies with >100 employees. That's a Long way from telling us about sizes of engineering orgs though. But still... I'd be careful assuming that the vast majority of developers are in organizations of less than 100 people.

GauntletWizard 3y ago

Yeah, I'm an independent, selling these solutions. My minimum stack costs about $500/mo in AWS costs- and you could save 90% of that. But then you'd be paying me $20k more to set it up again when you expand to another dev team, while this way I can add your second dev team for $10k and practically zero additional AWS cost. Going straight to overkill is good business sense for any company that's going to make it to a medium-sized business, and it's not going to be the differentiating factor for a company that burns out.

ongy 3y ago

That's really the kicker for me. When the company is set up well on k8s, then choose k8s. If the company is set up well on VPS, then choose VPS. If it got neither, I'm unsure what's the better way to go on a greenfield. k8s has nice tooling, but part of that is required because it is massively complex. But with managed k8s providers (e.g. GKE with autopilot) you can just put in proper CPU and memory limits and essentially get yourself a VPS provisioned without having to worry about anything by k8s measures. If you have different pricing or location constraints, then it might be better to go with different models and some custom orchestration.

richardw 3y ago

But you’ve got 100 developers. That’s not “most web apps”, that’s firmly in the set of companies that need standardisation and potential scale, where the devops team makes the life of many devs far better. When it’s just a few devs and cash is limited, the business doesn’t need the complexity. It’s mostly for ego and branding.

ProblemExplorer 3y ago

> All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place). How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how?

znpy 3y ago
> How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how? All those questions are irrelevant to how software gets deployed. Do whatever you want on your laptop.
- ProblemExplorer 3y ago
  Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems? The underlying orchestrator definitely affects how the software needs to behave and is definitely not irrelevant.
  - znpy 3y ago
    > Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems? nginx-ingress-controller does that. no custom stuff required. https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ https://kubernetes.github.io/ingress-nginx/user-guide/exposi...
  - ongy 3y ago
    Which LB doesn't support non-http flows?
  - jameshart 3y ago
    Follow the signposts laid out by https://12factor.net https://12factor.net. Think about how to apply those ideas to containerized applications. Consider that such applications can be run in the same way locally, on a server, or in an autoscaled cluster. Not sure why TCP/UDP load balancing complicates matters.