4 ms·
Most of the suggestions in the article seem like sane things that most projects out there should have (not saying that the ones I have to help out with always d
by KronisLV 4y ago
Most of the suggestions in the article seem like sane things that most projects out there should have (not saying that the ones I have to help out with always do, but rather that I wish people cared enough about those), otherwise you're shipping what's essentially a black box, with relatively little insight into how well it works (or why it suddenly doesn't).
If you make the right technology choices, most of those should also be pretty doable (not necessarily easy, but not overwhelmingly hard):
- enable some basic healthchecks through configuration value in Spring or whatever you're using
- then do curl requests against that (or even the HEALTHCHECK command with containers)
- have something standardized take care of application lifecycle (like systemd services or once again containers)
- add some instrumentation along the way with something like Sentry or Apache Skywalking
- integrate with OpenAPI through whatever framework you use (so you don't have to write everything manually, but can use codegen)
- don't be lazy and write tests for your code, at least the parts that are easy to test (e.g. business logic, not the low level JSON serialization)
- hopefully have a few Markdown files that describe your service
Of course, those aren't strictly necessary for things to run (hence a lot might be called day 2 concerns), which many will use as an excuse not to care about it, as long as they can get something shipped and it seems to work now. I've seen that more often than I'd like, given that I'm sometimes the person who gets called in to fix the eventual issues.
An excellent group of suggestions for developing applications in ways that can limit headaches is the 12 Factor App site, which can help you avoid some issues ahead of time (it covers configuration, ports, logs and other concerns): https://12factor.net/ https://12factor.net/
Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working, with insights into potential problems ahead of time. And if your technology stacks are sufficiently boring, there's no reason why a lot of that knowledge couldn't be encapsulated into a few concise Wiki pages, Markdown files, code snippets or project templates - so anyone who needs to ship a new service in your org can grab one of those and get up and running quickly and properly.
- doublerabbit 4y ago> Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working However we don't live in such an ideal world. I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? I now have to waste my own time chasing managers, dev's and everybody else to setup what's required. My resources are finite, I don't have unlimited resources nor a budget to waste on virtual machines. We use multitude of OS versions ranging from CentOS, to Debian including FreeBSD and Solaris so what do you want? This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". I have to waste time debugging when turns out to be because the software isn't designed for what I thought was suitable for a project because someone decided to use some old version of a Ruby GEM that's been cross compiled and which I've had no clue. No one thinks about SysAdmins when they're running the actual show. If it wasn't for those, you wouldn't have a production, uptime and all the stuff we do. I can recite so many stories in many different jobs. Where this has been the case. But same-so, HN developer is bias and don't take in to account the flow that requires for their produce to reach production. /end vent. But your not allowed to do that on HN. Because you get thrown downvotes for expressing issues with how the teams integrates. 16 years of SysAdmin, 33 and I'm more than burnt out because of. The first thing I do every morning is decide the colour of the ethernet cable I wish to desire to hang myself with. I take pride in my work yet done dealing with the shit that's given, yet you still need to lick the plate because life. I've just had three months off, just exhausted all my savings and starting my new job on Monday. But hey-ho, maybe this company has stuff right.
- KronisLV 4y ago> I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? The cut and dry answer is: you aren't, because you cannot. What should happen is a polite response to the request along the lines of: "Insufficient information has been given for the release. Please see the attached Markdown template with the information that needs to be filled out. This request is on hold until this will be done. You can submit any suggestions to improve the template, or reach out with further questions to ..." > This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". So essentially you have all of the responsibility, without any of the power to actually do your job because of the circumstances that you're pigeonholed into? If you need the money, then I guess that's what you have to tolerate, but otherwise some lines should be drawn somewhere. In most cases, adding documentation/instrumentation and requiring it going forwards would be a good idea that any sane organization would get behind and support: especially if you can reference all of the past incidents that this would have helped guard against. Quite frankly, that sounds like a dysfunctional environment and absolutely nobody would fault you for quitting a year into it (or even not waiting that long), in search of something better. I suspect that a part of the problem is that in our market, we have mindsets that don't go beyond any of the following goals: - we got paid, regardless of what works or doesn't (can be seen in consulting) - we shipped something to meet a deadline and not get contractual penalties, regardless of quality - we shipped something that seems to work, though we don't care about much else (day 2 concerns) Sometimes it's because of ignorance and not knowing better, other times it's because there are cultural issues in the country as a whole, maybe a lot of people viewing development only as a step in the path to becoming a manager, instead of a craft that demands attention and care. So what you get globally can be companies that range anywhere from "Hey, we want you to be comfortable and not overworked: here are some learning materials or a budget for that, here's our knowledgebase and an overview of our procedures and architecture decision records (ADRs), here's a user group for this particular technology, feel free to reach out if you need anything." to "Hey, ship the software until monday. Why isn't it still done? I don't care about the details, get it done." Maybe I should write a blog post about that some day, just not sure what to title it: "OKRs/KPIs of caring about software" or something like that, probably. I suspect bad mindsets and a bad culture is one of the reasons for sites like this existing: https://devrant.com/ https://devrant.com/ or rather why articles like this ring true: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks At the end of the day, do what you can to take care of yourself!