5 ms·

A lot of comments here seem to be along the lines of "you can hire one more engineer," but given the current economic situation remember that might be "keep one

by mike_d 3y ago

A lot of comments here seem to be along the lines of "you can hire one more engineer," but given the current economic situation remember that might be "keep one more engineer." Would you lay off someone on your team to keep AWS? Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is. I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running.

bee_rider 3y ago
Especially given the low unemployment rate, laying somebody off seems quite risky, if it doesn’t work out you’ll have trouble hiring some replacement I guess.
- cj 3y ago
  The current hiring market in tech is the easiest (for employers) than it has been in a really long time. It used to take 3-4 months to fill a role. In the current market it's more like 2-4 weeks.
  - sgarland 3y ago
    Not for SRE and DBRE lol. 4-6 months easy. I made the correct career choice. Downturn? What downturn?
unglaublich 3y ago
> I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running. Rightly so, because they're cloud native engineers, not system administrators. They're intimidated by the things they don't know. It'll be a very individual calculation whether it's worth it for your enterprise to organize and maintain hardware yourself, or isn't.
- moduspol 3y ago
  There's certainly no shortage of sysadmins comfortable with their on-prem skillsets that have their heads in the sand about the cloud. And there are plenty of us who've spent time managing hardware and physical networks, transitioned to cloud, and are very happy to not be looking back.
nwmcsween 3y ago
eh, to a degree, having to deal with failed hardware and worse buggy hardware is just a pain and really time consuming.
Jnr 3y ago
And in other places around the world those would be closer to 3 or 4 good engineers for the same money. And while each engineer costs some money, they probably bring in close to double of what they are being paid.
deepspace 3y ago
> Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is Keeping them humming along redundantly, with adequate power and cooling, and protection against cooling- and power failures is more of an undertaking, though. Now you are maintaining generators, UPSs and multiple HVAC systems in addition to your 'few racks of servers'. You also need to maintain full network redundancy (including ingress/egress) and all the cost that entails. All the above hardware needs maintenance and replacement when it becomes obsolete. Now you are good in one DC, but not protected against tornadoes, fire and flood like you would be if you used AWS with multiple availability zones. So, you have to build another DC far enough away, staff it, and buy tons of replication software, plus several FTEs to manage cross-site backups and deal with sync issues.
- lol768 3y ago
  Most of those requirements cease to exist if you decide to colo. It's not cloud or "run your own DC".
- 10000truths 3y ago
  You don't need to build your own datacenter unless your workload requires a datacenter's worth of hardware. Colocation is a feasible and popular option for handling all of the hands-on stuff you mention. Ship the racks to a colo center, they'll install them for you. Ship them replacement peripherals, and the operators will hot-swap them for you. If you need redundancy, that's just a matter of sending your hardware to multiple places instead of one. Slightly more involved, but it's hardly rocket science.
  - Sohcahtoa82 3y ago
    That all takes time. With cloud, you can have a system up and running in literal seconds, which is very nice when you find out that you severely underestimated how much traffic you web app will get. But yeah, in the long run, colocation becomes significantly cheaper than cloud. You use AWS, and you'll find yourself paying $200/month for hardware you could buy once for $2,000. Sometimes I think people forgot colocation is an option that exists.
bastawhiz 3y ago
The cost of one salary can probably be achieved by profiling and improving some software. Moving to graviton. Find and delete a few expensive queries. Add a database index. Switching from the cloud to on prem is a dramatic and expensive migration that changes a lot of constraints for your infra that you might not need to change in the first place.
I_Am_Nous 3y ago
>Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is. It isn't until hardware failures happen, and that requires different skills to deal with effectively. Like the time a core networking switch presented with a dead PSU and no backups, so you Frankenstein it back to temporarily working with another switch to pull the config off of it. With bare metal you may have to have more generalized, Jack of all trades staff to account for the unaccountable.

by mike_d 3y ago

bee_rider 3y ago

Especially given the low unemployment rate, laying somebody off seems quite risky, if it doesn’t work out you’ll have trouble hiring some replacement I guess.

cj 3y ago
The current hiring market in tech is the easiest (for employers) than it has been in a really long time. It used to take 3-4 months to fill a role. In the current market it's more like 2-4 weeks.
- sgarland 3y ago
  Not for SRE and DBRE lol. 4-6 months easy. I made the correct career choice. Downturn? What downturn?

unglaublich 3y ago

> I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running. Rightly so, because they're cloud native engineers, not system administrators. They're intimidated by the things they don't know. It'll be a very individual calculation whether it's worth it for your enterprise to organize and maintain hardware yourself, or isn't.

moduspol 3y ago
There's certainly no shortage of sysadmins comfortable with their on-prem skillsets that have their heads in the sand about the cloud. And there are plenty of us who've spent time managing hardware and physical networks, transitioned to cloud, and are very happy to not be looking back.

nwmcsween 3y ago

eh, to a degree, having to deal with failed hardware and worse buggy hardware is just a pain and really time consuming.

Jnr 3y ago

And in other places around the world those would be closer to 3 or 4 good engineers for the same money. And while each engineer costs some money, they probably bring in close to double of what they are being paid.

deepspace 3y ago

> Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is Keeping them humming along redundantly, with adequate power and cooling, and protection against cooling- and power failures is more of an undertaking, though. Now you are maintaining generators, UPSs and multiple HVAC systems in addition to your 'few racks of servers'. You also need to maintain full network redundancy (including ingress/egress) and all the cost that entails. All the above hardware needs maintenance and replacement when it becomes obsolete. Now you are good in one DC, but not protected against tornadoes, fire and flood like you would be if you used AWS with multiple availability zones. So, you have to build another DC far enough away, staff it, and buy tons of replication software, plus several FTEs to manage cross-site backups and deal with sync issues.

lol768 3y ago
Most of those requirements cease to exist if you decide to colo. It's not cloud or "run your own DC".
10000truths 3y ago
You don't need to build your own datacenter unless your workload requires a datacenter's worth of hardware. Colocation is a feasible and popular option for handling all of the hands-on stuff you mention. Ship the racks to a colo center, they'll install them for you. Ship them replacement peripherals, and the operators will hot-swap them for you. If you need redundancy, that's just a matter of sending your hardware to multiple places instead of one. Slightly more involved, but it's hardly rocket science.
- Sohcahtoa82 3y ago
  That all takes time. With cloud, you can have a system up and running in literal seconds, which is very nice when you find out that you severely underestimated how much traffic you web app will get. But yeah, in the long run, colocation becomes significantly cheaper than cloud. You use AWS, and you'll find yourself paying $200/month for hardware you could buy once for $2,000. Sometimes I think people forgot colocation is an option that exists.

bastawhiz 3y ago

The cost of one salary can probably be achieved by profiling and improving some software. Moving to graviton. Find and delete a few expensive queries. Add a database index. Switching from the cloud to on prem is a dramatic and expensive migration that changes a lot of constraints for your infra that you might not need to change in the first place.

I_Am_Nous 3y ago

>Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is. It isn't until hardware failures happen, and that requires different skills to deal with effectively. Like the time a core networking switch presented with a dead PSU and no backups, so you Frankenstein it back to temporarily working with another switch to pull the config off of it. With bare metal you may have to have more generalized, Jack of all trades staff to account for the unaccountable.