4 ms·
I find your quote here and the follow up seriously unconvincing. You know what a miserable workflow is? Preparing all my build scripts according to the half bak
by mlthoughts2018 6y ago
I find your quote here and the follow up seriously unconvincing. You know what a miserable workflow is? Preparing all my build scripts according to the half baked build tool that our in-house Deployment team cooked up to use fabric to spray my deployment onto a bunch of nodes in our data center, all the tests and script linting pass, so then I go push it through our CI/CD pipeline (which can’t do the above tests for me because again this is all threadbare maintained by in-house tooling teams who could not be less aligned to end product deliverables), and lo and behold it breaks. Good thing I’m not paying so much for CloudWatch so I get to literally go ssh to the worker node of the CI/CD system where my task failed and cross my fingers that it wasn’t reclaimed for the next task and that the log files weren’t rotated off, grep my way to tracebacks that probably involve things no one on my team has ever heard of, spend 1/2 a day chasing them all down (if I’m lucky), and finally kicking the box to restart a build a few times and it goes. Then it goes to the in house homebrewed kubernetes deployment scripts where it promptly chokes on a secret that wasn’t configured right (thanks home made script linters), so then I have to go down a rabbit hole of eye melting Helm config of secrets for an hour but thank god I’m not paying for a simple cloud UI for this. So we get the secret cleared up and everything is great, until the kubernetes scheduler can’t schedule my pods because our data center is at capacity and now I literally can’t get work done until I chase down some probably incorrect and certainly unmaintained Grafana chart of who is using how much excess memory on which nodes, and wrangle someone with authority to force another team to stop eating lunch and downsize pods right now. After all this, 6-7 hours later and 20-30 labor hours later, my workload is finally deployed, but it will die overnight for an uncaught reason that doesn’t alert anyone, and even if it did, the team managing compute resources is in a permanent state of PTSD from never ending pages so they wouldn’t bother caring about why my workload failed and will just assume it’s application logic bugs or Docker bugs on my side for as long as they can just to avoid yet another un-debuggable failure landing on their plate, so that’ll be great for morale in the morning when I get to derail yet another day and spend time doing every type of ops work that is not my job and prevents us from delivering product goals.
This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.
Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.
I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.
- TheColorYellow 6y agoThis is the type of cultural accuracy I've been looking for. Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind. The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge. The ego and arrogance of people in tech really shines through here.
- papaf 6y agoPeople on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge. A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.
- mlthoughts2018 6y agoExactly, which points out how much better cloud tools are (given the level of seriousness most organizations are applying to them).
- nocman 6y ago> there is such a strong brand of "anything new must be snake oil" that runs through the HN community. I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well. I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions. Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it. This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.