4 ms·

That's fine for the software they wrote. But there's a LOT we sysads do outside of that area. Have you seen a webapp go off the rails? There's millions of weir

That's fine for the software they wrote. But there's a LOT we sysads do outside of that area. Have you seen a webapp go off the rails? There's millions of weird fail-conditions. Can they learn what to monitor, at what frequency? Security... 'Principle of least privilege' is at odds with "make it work". I've seen devs assume the database server has commodity connection to the internet. They never asked if it did or didn't. And I've fought with them on the dev AD stack with an attempt in keeping the GPOs in line with production. But it's just 'too hard'. And I've seen devs go 'just spin up some more machines'. But unlike yesteryear where we provisioned hardware for cost for X years, aws/azure/gce "allows" devs to have a direct unlimited budget tied to the company. We sysads learned responsibilities, and how to budget, and how to requisition. Someone pressing a button on a dashboard, or writing a script doesn't. You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign.

jrockway 7y ago
> Can they learn what to monitor, at what frequency? Yes. That is part of the design of any system; observability is in everyone's mind when they are writing software. Software that fails mysteriously means less weekends and less time for drinking beer. Software that tells you exactly what it's doing means when something goes wrong, you fix it, and it never happens again. Or, it's so transparent you figure out the bug before you even check it in. > Security... 'Principle of least privilege' is at odds with "make it work". There is certainly value in having strong security principles, both in terms of policies and infrastructure. Infrastructure is key here; an ACL-checking proxy in front of all services, mTLS out of the box, the ability to write, deploy, and manage small services that are easy to audit... all help make security the rule instead of the exception. This is what all those "service meshes" and "policy frameworks" and "orchestration frameworks" aim to provide. > I've seen devs assume the database server has commodity connection to the internet. They never asked if it did or didn't. That's why the dev team should be operating that service. > And I've seen devs go 'just spin up some more machines'. [...] You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign. In my experience this has always been a utilization problem. One shared cluster for everyone is too complicated / expensive / dangerous / conflicts with empire-building, so every team has a bunch of t3.4xlarge instances that average 0% CPU load just because. Meanwhile, companies that care about their computers have seemingly-complicated orchestration frameworks so that they can extract value from every CPU core. The good news is, a lot of what used to be manual (wrangling machines, consolidating resources, mechanical aspects of security) has some serious open-source investment. People don't need to schedule programs to physica computers anymore. People don't need to write authentication and mTLS code into their application anymore. You don't need to fill out a form and create a purchase order to get a TLS certificate anymore. Times are changing. Your experience seems to have been shaped by supporting bad developers. We aren't all like that, though, and you'll find that there is less and less room for the incompetent these days. It's true of any field; someone builds you a house and the floor isn't level. "Don't build houses anymore," you say! No! Just don't hire the guy that did a crap job. Everything gets a lot easier when you set out to do a good job.
wwweston 7y ago
> You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign. The flip side of this is the C levels who have absolutely no experience monitoring cloud spend but have heard the marketing pitches or smelled the "best practices" in the air and are sure that moving to cloud services needs to happen yesterday whether or not demand patterns justify it.

jrockway 7y ago

> Can they learn what to monitor, at what frequency? Yes. That is part of the design of any system; observability is in everyone's mind when they are writing software. Software that fails mysteriously means less weekends and less time for drinking beer. Software that tells you exactly what it's doing means when something goes wrong, you fix it, and it never happens again. Or, it's so transparent you figure out the bug before you even check it in. > Security... 'Principle of least privilege' is at odds with "make it work". There is certainly value in having strong security principles, both in terms of policies and infrastructure. Infrastructure is key here; an ACL-checking proxy in front of all services, mTLS out of the box, the ability to write, deploy, and manage small services that are easy to audit... all help make security the rule instead of the exception. This is what all those "service meshes" and "policy frameworks" and "orchestration frameworks" aim to provide. > I've seen devs assume the database server has commodity connection to the internet. They never asked if it did or didn't. That's why the dev team should be operating that service. > And I've seen devs go 'just spin up some more machines'. [...] You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign. In my experience this has always been a utilization problem. One shared cluster for everyone is too complicated / expensive / dangerous / conflicts with empire-building, so every team has a bunch of t3.4xlarge instances that average 0% CPU load just because. Meanwhile, companies that care about their computers have seemingly-complicated orchestration frameworks so that they can extract value from every CPU core. The good news is, a lot of what used to be manual (wrangling machines, consolidating resources, mechanical aspects of security) has some serious open-source investment. People don't need to schedule programs to physica computers anymore. People don't need to write authentication and mTLS code into their application anymore. You don't need to fill out a form and create a purchase order to get a TLS certificate anymore. Times are changing. Your experience seems to have been shaped by supporting bad developers. We aren't all like that, though, and you'll find that there is less and less room for the incompetent these days. It's true of any field; someone builds you a house and the floor isn't level. "Don't build houses anymore," you say! No! Just don't hire the guy that did a crap job. Everything gets a lot easier when you set out to do a good job.

wwweston 7y ago

> You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign. The flip side of this is the C levels who have absolutely no experience monitoring cloud spend but have heard the marketing pitches or smelled the "best practices" in the air and are sure that moving to cloud services needs to happen yesterday whether or not demand patterns justify it.