3 ms·

There's nothing wrong with feeling this way, but you're comparing a very small company (your own) to a much larger corporation. The appeal of AWS' "insane" bil

by dangus 3y ago

There's nothing wrong with feeling this way, but you're comparing a very small company (your own) to a much larger corporation. The appeal of AWS' "insane" bills are that they are not so "insane" when compared to the salaries required to maintain a home grown stack in-house, especially as complexity increases. It's not just application complexity, it's organizational complexity. Doing everything yourself is fine until that complexity increases and you start finding out that all your dozens or hundreds of expensive employees are spending a lot of time on home grown overhead. For example, when you use shell scripts for your deploys, you can't as easily prove things to auditors like you can with a CI/CD system that's tied in to role based access control. That sort of thing will be dealbreaker for customers who expect your company to maintain industry compliance standards. When you use a product like RDS you can tell your customers that Amazon is the one responsible for handling backups, security patching, etc. Also keep in mind that anyone with a significant amount of AWS infrastructure should be buying reservations and savings plans. The sticker price of AWS is not what corporations pay. As you grow your company, I encourage you to focus on the activities that make your business profitable and deliver value to customers. Every hour you spend upgrading a database or patching an operating system is an hour you could have been spending on developing a unique feature in your product that nobody else has. For now, I'm sure using this setup is much better than getting something too complex for your size, but that can change quickly if you're lucky enough to reach a larger scale.

shrubble 3y ago
In the mid-1990s a large ISP I am aware of, rented "pizzabox" and larger Sun SPARC systems (like SPARCstation 2s; but even the larger systems ran the same OS) to corporate clients. The team of 5 sysadmins setup, installed, secured and managed 2000 such systems, mostly running SunOS 4.1.x , which didn't have nearly the automation that we have now on newer Unix based OSes. Using only the bash shell and the many scripts they wrote for managing them. Guess the profitability of that kind of setup? 1 sysadmin per 400 systems... the kind of overhead people think exists, doesn't.
- theshrike79 3y ago
  "A large ISP" will have the money to hire a team of 5 experienced sysadmins to be 24/7 on-call to handle 2000 systems. A small to medium business won't have the 500k to 1M per year pay the sysadmins salaries. For them AWS or other managed services are the correct choice.
- qwytw 3y ago
  Wouldn't it still be 1 sysadmin per 5 systems instead of 400? It's not like you can hire 1.25% of a sysadmin directly, while on AWS you can you just pay a significant premium.
  - shrubble 3y ago
    Sysadmin consulting exists.
sgarland 3y ago
You make it sound as though there’s some obscene sprawl of interrelated tooling, barely held together with shell scripts, and only OP knows how it all works. Nothing described is that weird, nor should it have difficulty horizontally scaling when and if the time comes - add another node, and front it with a load balancer. The problem is that DevOps has shifted so heavily to Dev that “I’m running services natively on a Linux box” is somehow seen as Byzantine and arcane. > when you use shell scripts for deploys… can’t as easily prove things to auditors as you can with a CI/CD system What do you think a CI/CD system is running? Also, if /var/log/auth.log isn’t enough, there are other auditing systems available that could make this more granular. > industry compliance standards IME, these are a joke, and auditors routinely miss a dizzying amount of glaring problems, because they don’t look beyond records that humans generated. > every hour you spend upgrading a database or patching an OS… Patching the OS should be automated. If it isn’t, that’s on you. Ansible isn’t hard to learn. As to the database, if you can’t read docs (both MySQL and Postgres have excellent documentation on this procedure) and perform them as written, frankly you shouldn’t be dealing with RDS either. It’s not like AWS docs aren’t confusing, spread over a million pages, and sometimes contradictory. Pardon the seething undertone, but as an ops-heavy SRE/DBRE, I’m very tired of seeing people lambast ops as being somehow beneath modern practices, not worthy of their time, or worse, not cost-effective. A well-written app can absolutely run just fine on a tiny server, and does not need the miasma of shit that is cloud-native. Computers are blindingly fast. Stop demanding infinitely-scaling vCPUs because optimization is hard. Stop pretending that your time is more valuable than minutiae like “my ORM is producing garbage queries resulting in hideous latency because I don’t understand SQL or schema design.”
- deleted 3y ago
  [deleted]
- dangus 3y ago
  Try to see the forest for the trees here: ask yourself how does the business make money? If I throw my application on some PaaS thing and put the database on RDS, let’s say I can hire 5 developers who know almost nothing about infrastructure to develop and deploy the application. Each developer working on the application delivers valuable business logic. Let’s say each developer makes $200,000 total compensation and they bring in $500,000 in revenue. This setup makes me $1,500,000 in operating income per year. Now you’re saying I should find someone who knows enough about infrastructure architecture who database administration and all the other bare metal Linux fundamental topics to manage it in house rather than paying AWS all this wasteful money. So now I have 4 developers and 1 Ops/DBA person. My AWS bill was eliminated, saving $100,000 thanks to the infrastructure and database expert doing a wonderful job cost optimizing. But now my 4 developers have one less person to deliver features that motivate customers to sign deals with us, so now we’re making $1,300,000 operating income. (Losing $300k from the loss of a developer and gaining $100k back from cost savings. Obviously this is a made up scenario in my favor but that’s basically how it works. Businesses shopping for employees can’t get every skill they want in a job description. One of the most important parts of business strategy is making trade-offs. This scenario reminds me of a landlord that I had who fixed a simple problem with my dishwasher by just replacing it with a new one. They didn’t give a shit that a little bit of extra knowledge and a spare part would have fixed it for no exaggeration 100x lower cost, because resolving the problem quickly with no risk was worth more to them. In any event, I was giving that landlord many times the cost of that dishwasher every month. The compliance certifications are kind of like the real estate agent that has a big car payment on their Mercedes so that they can sell more expensive houses to wealthier clients. Yes, we all know that buying a Mercedes isn’t the most cost effective way to get from point A to B, but you can’t sell mansions if you show up in a Kia Rio.

by dangus 3y ago

shrubble 3y ago

In the mid-1990s a large ISP I am aware of, rented "pizzabox" and larger Sun SPARC systems (like SPARCstation 2s; but even the larger systems ran the same OS) to corporate clients. The team of 5 sysadmins setup, installed, secured and managed 2000 such systems, mostly running SunOS 4.1.x , which didn't have nearly the automation that we have now on newer Unix based OSes. Using only the bash shell and the many scripts they wrote for managing them. Guess the profitability of that kind of setup? 1 sysadmin per 400 systems... the kind of overhead people think exists, doesn't.

theshrike79 3y ago
"A large ISP" will have the money to hire a team of 5 experienced sysadmins to be 24/7 on-call to handle 2000 systems. A small to medium business won't have the 500k to 1M per year pay the sysadmins salaries. For them AWS or other managed services are the correct choice.
qwytw 3y ago
Wouldn't it still be 1 sysadmin per 5 systems instead of 400? It's not like you can hire 1.25% of a sysadmin directly, while on AWS you can you just pay a significant premium.
- shrubble 3y ago
  Sysadmin consulting exists.

sgarland 3y ago

You make it sound as though there’s some obscene sprawl of interrelated tooling, barely held together with shell scripts, and only OP knows how it all works. Nothing described is that weird, nor should it have difficulty horizontally scaling when and if the time comes - add another node, and front it with a load balancer. The problem is that DevOps has shifted so heavily to Dev that “I’m running services natively on a Linux box” is somehow seen as Byzantine and arcane. > when you use shell scripts for deploys… can’t as easily prove things to auditors as you can with a CI/CD system What do you think a CI/CD system is running? Also, if /var/log/auth.log isn’t enough, there are other auditing systems available that could make this more granular. > industry compliance standards IME, these are a joke, and auditors routinely miss a dizzying amount of glaring problems, because they don’t look beyond records that humans generated. > every hour you spend upgrading a database or patching an OS… Patching the OS should be automated. If it isn’t, that’s on you. Ansible isn’t hard to learn. As to the database, if you can’t read docs (both MySQL and Postgres have excellent documentation on this procedure) and perform them as written, frankly you shouldn’t be dealing with RDS either. It’s not like AWS docs aren’t confusing, spread over a million pages, and sometimes contradictory. Pardon the seething undertone, but as an ops-heavy SRE/DBRE, I’m very tired of seeing people lambast ops as being somehow beneath modern practices, not worthy of their time, or worse, not cost-effective. A well-written app can absolutely run just fine on a tiny server, and does not need the miasma of shit that is cloud-native. Computers are blindingly fast. Stop demanding infinitely-scaling vCPUs because optimization is hard. Stop pretending that your time is more valuable than minutiae like “my ORM is producing garbage queries resulting in hideous latency because I don’t understand SQL or schema design.”

deleted 3y ago
[deleted]
dangus 3y ago
Try to see the forest for the trees here: ask yourself how does the business make money? If I throw my application on some PaaS thing and put the database on RDS, let’s say I can hire 5 developers who know almost nothing about infrastructure to develop and deploy the application. Each developer working on the application delivers valuable business logic. Let’s say each developer makes $200,000 total compensation and they bring in $500,000 in revenue. This setup makes me $1,500,000 in operating income per year. Now you’re saying I should find someone who knows enough about infrastructure architecture who database administration and all the other bare metal Linux fundamental topics to manage it in house rather than paying AWS all this wasteful money. So now I have 4 developers and 1 Ops/DBA person. My AWS bill was eliminated, saving $100,000 thanks to the infrastructure and database expert doing a wonderful job cost optimizing. But now my 4 developers have one less person to deliver features that motivate customers to sign deals with us, so now we’re making $1,300,000 operating income. (Losing $300k from the loss of a developer and gaining $100k back from cost savings. Obviously this is a made up scenario in my favor but that’s basically how it works. Businesses shopping for employees can’t get every skill they want in a job description. One of the most important parts of business strategy is making trade-offs. This scenario reminds me of a landlord that I had who fixed a simple problem with my dishwasher by just replacing it with a new one. They didn’t give a shit that a little bit of extra knowledge and a spare part would have fixed it for no exaggeration 100x lower cost, because resolving the problem quickly with no risk was worth more to them. In any event, I was giving that landlord many times the cost of that dishwasher every month. The compliance certifications are kind of like the real estate agent that has a big car payment on their Mercedes so that they can sell more expensive houses to wealthier clients. Yes, we all know that buying a Mercedes isn’t the most cost effective way to get from point A to B, but you can’t sell mansions if you show up in a Kia Rio.