3 ms·

Next in the trilogy: Things I want from SRE/DevOps as a SysAdmin. - What specs of a VM do you require? I'll assume that 16mb of RAM and 512mb of drive space r

Next in the trilogy: Things I want from SRE/DevOps as a SysAdmin. - What specs of a VM do you require? I'll assume that 16mb of RAM and 512mb of drive space running Slackware is suitable operating from 1.44mb floopy. - What do I do if it doesn't compile? It works in DevLand I assume I'll work anywhere. No, you cant growl at me, you asked for Linux and I gave you Linux. Documentation please.

KronisLV 4y ago
Most of the suggestions in the article seem like sane things that most projects out there should have (not saying that the ones I have to help out with always do, but rather that I wish people cared enough about those), otherwise you're shipping what's essentially a black box, with relatively little insight into how well it works (or why it suddenly doesn't). If you make the right technology choices, most of those should also be pretty doable (not necessarily easy, but not overwhelmingly hard): - enable some basic healthchecks through configuration value in Spring or whatever you're using - then do curl requests against that (or even the HEALTHCHECK command with containers) - have something standardized take care of application lifecycle (like systemd services or once again containers) - add some instrumentation along the way with something like Sentry or Apache Skywalking - integrate with OpenAPI through whatever framework you use (so you don't have to write everything manually, but can use codegen) - don't be lazy and write tests for your code, at least the parts that are easy to test (e.g. business logic, not the low level JSON serialization) - hopefully have a few Markdown files that describe your service Of course, those aren't strictly necessary for things to run (hence a lot might be called day 2 concerns), which many will use as an excuse not to care about it, as long as they can get something shipped and it seems to work now. I've seen that more often than I'd like, given that I'm sometimes the person who gets called in to fix the eventual issues. An excellent group of suggestions for developing applications in ways that can limit headaches is the 12 Factor App site, which can help you avoid some issues ahead of time (it covers configuration, ports, logs and other concerns): https://12factor.net/ https://12factor.net/ Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working, with insights into potential problems ahead of time. And if your technology stacks are sufficiently boring, there's no reason why a lot of that knowledge couldn't be encapsulated into a few concise Wiki pages, Markdown files, code snippets or project templates - so anyone who needs to ship a new service in your org can grab one of those and get up and running quickly and properly.
- doublerabbit 4y ago
  > Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working However we don't live in such an ideal world. I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? I now have to waste my own time chasing managers, dev's and everybody else to setup what's required. My resources are finite, I don't have unlimited resources nor a budget to waste on virtual machines. We use multitude of OS versions ranging from CentOS, to Debian including FreeBSD and Solaris so what do you want? This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". I have to waste time debugging when turns out to be because the software isn't designed for what I thought was suitable for a project because someone decided to use some old version of a Ruby GEM that's been cross compiled and which I've had no clue. No one thinks about SysAdmins when they're running the actual show. If it wasn't for those, you wouldn't have a production, uptime and all the stuff we do. I can recite so many stories in many different jobs. Where this has been the case. But same-so, HN developer is bias and don't take in to account the flow that requires for their produce to reach production. /end vent. But your not allowed to do that on HN. Because you get thrown downvotes for expressing issues with how the teams integrates. 16 years of SysAdmin, 33 and I'm more than burnt out because of. The first thing I do every morning is decide the colour of the ethernet cable I wish to desire to hang myself with. I take pride in my work yet done dealing with the shit that's given, yet you still need to lick the plate because life. I've just had three months off, just exhausted all my savings and starting my new job on Monday. But hey-ho, maybe this company has stuff right.
  - KronisLV 4y ago
    > I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? The cut and dry answer is: you aren't, because you cannot. What should happen is a polite response to the request along the lines of: "Insufficient information has been given for the release. Please see the attached Markdown template with the information that needs to be filled out. This request is on hold until this will be done. You can submit any suggestions to improve the template, or reach out with further questions to ..." > This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". So essentially you have all of the responsibility, without any of the power to actually do your job because of the circumstances that you're pigeonholed into? If you need the money, then I guess that's what you have to tolerate, but otherwise some lines should be drawn somewhere. In most cases, adding documentation/instrumentation and requiring it going forwards would be a good idea that any sane organization would get behind and support: especially if you can reference all of the past incidents that this would have helped guard against. Quite frankly, that sounds like a dysfunctional environment and absolutely nobody would fault you for quitting a year into it (or even not waiting that long), in search of something better. I suspect that a part of the problem is that in our market, we have mindsets that don't go beyond any of the following goals: - we got paid, regardless of what works or doesn't (can be seen in consulting) - we shipped something to meet a deadline and not get contractual penalties, regardless of quality - we shipped something that seems to work, though we don't care about much else (day 2 concerns) Sometimes it's because of ignorance and not knowing better, other times it's because there are cultural issues in the country as a whole, maybe a lot of people viewing development only as a step in the path to becoming a manager, instead of a craft that demands attention and care. So what you get globally can be companies that range anywhere from "Hey, we want you to be comfortable and not overworked: here are some learning materials or a budget for that, here's our knowledgebase and an overview of our procedures and architecture decision records (ADRs), here's a user group for this particular technology, feel free to reach out if you need anything." to "Hey, ship the software until monday. Why isn't it still done? I don't care about the details, get it done." Maybe I should write a blog post about that some day, just not sure what to title it: "OKRs/KPIs of caring about software" or something like that, probably. I suspect bad mindsets and a bad culture is one of the reasons for sites like this existing: https://devrant.com/ https://devrant.com/ or rather why articles like this ring true: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks At the end of the day, do what you can to take care of yourself!
Macha 4y ago
> - What specs of a VM do you require? What are my options? How much do those cost the company? I can run N requests on my laptop with specs of X. CPU usage hits 100% and memory usage hits 50%. This is 10% of our projected load. Therefore I project that I will need 10X CPU and 5X memory resources for this system for our entire userbase. My laptop is pretty powerful, but there are systems with 320GB of RAM and 80 cores. Do we get 2 of them? Oh, what you have is a bunch of 3 generation old 28 core Xeons with 128gb of RAM. Can I get access to one of them to test thoroughput? One Xeon core is not as good as one macbook core, especially not that old, so I really can't make any promises of perf without testing on representative hardware. No? Fine, let's add 30% buffer for poor IPC on old hardware. Oh, you've got a custom wrapper for Java that launches it with a bunch of custom JVM options that the SRE org mandates everyone uses? Got any documentation on what those are? What's our lead time for getting more? We need to know that for an idea of how much buffer we need. So apparently lead time is six months? Ok, what's our load going to be in 6 months? I'll ask bizdev how many customers they think we'll have in six months. Oh, they answered with a vague "We want to have lots of users". Fine, we'll add an extra 100% buffer. What, the operational efficiency team are pissed our VMs are at 25% usage? While there are certainly dev teams that throw capacity management over the wall to SRE, the inverse is also nonsense as it's the SRE team who usually get informed of hardware options, deployment standards (especially in bigco) and company operational standards.
- doublerabbit 4y ago
  > What are my options? How much do those cost the company? Do we get 2 of them? What options do you want? You come to me with the specifications, and I'll get back to you if you can. I'll work with you to get what you want. If I am the one with the power to create, the gatekeeper to the virtualization cluster and with a ballpark figure this makes my life so much easier. It allows me to make my justification easier too. "Hi Manager, X needs this. I think it needs this. I'm going to setup this and evaluate, no manager I don't think your correct." "Here's my tech specification for what I'm setting up. here's the documentation for configuration and setup and these are the results" Lets ride. Let me give you 8x core and 64GB see how the performance spikes and go from there. I'm not stingy and always happy to give more to test performance and than decrease if it's overkill. But don't dispute if I start to take away because of. > what you have is a bunch of 3 generation old 28 core Xeons with 128gb of RAM. Can I get access to one of them to test thoroughput? Sure. Yes you can. Why do you think you can't? > Lead time As fast as you require me to setup the VM. If I have all the docs, I can fly-by and have this thing setup under a day at minimum. Heck, I'll even work no-paid overtime to get this for you. I can escalate this on the fly. I'm in the good books with the NetOps, I even know the backup-ops. Tough crowd every time to please, but I manage so. Production lead time? Sure, probably six months to get stake-holder approval and the rest, but I'll try to get it sooner. Prototype lead: a week. Just give me figures and I'll do the rest. Enterprise or not. I'll push for what you want but you have to work with me. However I need the figures and documentation before I can. I can't be seen creating the documentation on my own time based on made-up figures when I have an estate of 100 VM's needing security patches. That makes me look bad and if it fails the tests, I'll get the blame.

KronisLV 4y ago

Most of the suggestions in the article seem like sane things that most projects out there should have (not saying that the ones I have to help out with always do, but rather that I wish people cared enough about those), otherwise you're shipping what's essentially a black box, with relatively little insight into how well it works (or why it suddenly doesn't). If you make the right technology choices, most of those should also be pretty doable (not necessarily easy, but not overwhelmingly hard): - enable some basic healthchecks through configuration value in Spring or whatever you're using - then do curl requests against that (or even the HEALTHCHECK command with containers) - have something standardized take care of application lifecycle (like systemd services or once again containers) - add some instrumentation along the way with something like Sentry or Apache Skywalking - integrate with OpenAPI through whatever framework you use (so you don't have to write everything manually, but can use codegen) - don't be lazy and write tests for your code, at least the parts that are easy to test (e.g. business logic, not the low level JSON serialization) - hopefully have a few Markdown files that describe your service Of course, those aren't strictly necessary for things to run (hence a lot might be called day 2 concerns), which many will use as an excuse not to care about it, as long as they can get something shipped and it seems to work now. I've seen that more often than I'd like, given that I'm sometimes the person who gets called in to fix the eventual issues. An excellent group of suggestions for developing applications in ways that can limit headaches is the 12 Factor App site, which can help you avoid some issues ahead of time (it covers configuration, ports, logs and other concerns): https://12factor.net/ https://12factor.net/ Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working, with insights into potential problems ahead of time. And if your technology stacks are sufficiently boring, there's no reason why a lot of that knowledge couldn't be encapsulated into a few concise Wiki pages, Markdown files, code snippets or project templates - so anyone who needs to ship a new service in your org can grab one of those and get up and running quickly and properly.

doublerabbit 4y ago
> Ideally, everyone who has the applicable skills or knowledge would collaborate and work together towards having their software both work now and also keep working However we don't live in such an ideal world. I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? I now have to waste my own time chasing managers, dev's and everybody else to setup what's required. My resources are finite, I don't have unlimited resources nor a budget to waste on virtual machines. We use multitude of OS versions ranging from CentOS, to Debian including FreeBSD and Solaris so what do you want? This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". I have to waste time debugging when turns out to be because the software isn't designed for what I thought was suitable for a project because someone decided to use some old version of a Ruby GEM that's been cross compiled and which I've had no clue. No one thinks about SysAdmins when they're running the actual show. If it wasn't for those, you wouldn't have a production, uptime and all the stuff we do. I can recite so many stories in many different jobs. Where this has been the case. But same-so, HN developer is bias and don't take in to account the flow that requires for their produce to reach production. /end vent. But your not allowed to do that on HN. Because you get thrown downvotes for expressing issues with how the teams integrates. 16 years of SysAdmin, 33 and I'm more than burnt out because of. The first thing I do every morning is decide the colour of the ethernet cable I wish to desire to hang myself with. I take pride in my work yet done dealing with the shit that's given, yet you still need to lick the plate because life. I've just had three months off, just exhausted all my savings and starting my new job on Monday. But hey-ho, maybe this company has stuff right.
- KronisLV 4y ago
  > I'm given tasks such as "set this up for devs for release of X" with nothing specified. How am I suppose complete the request? The cut and dry answer is: you aren't, because you cannot. What should happen is a polite response to the request along the lines of: "Insufficient information has been given for the release. Please see the attached Markdown template with the information that needs to be filled out. This request is on hold until this will be done. You can submit any suggestions to improve the template, or reach out with further questions to ..." > This is SRE playing lax on the case of not defining and just expecting it. I don't have any access to the DevKit side of things, I don't have the ability find out what is actually required. You want it to run in production, fine. But it makes my job insane when no documentation is provided and a JIRA ticket consists of some attachment with "this please" is handed to me. Something breaks and I am the first to get the grunt because it's "my fault". So essentially you have all of the responsibility, without any of the power to actually do your job because of the circumstances that you're pigeonholed into? If you need the money, then I guess that's what you have to tolerate, but otherwise some lines should be drawn somewhere. In most cases, adding documentation/instrumentation and requiring it going forwards would be a good idea that any sane organization would get behind and support: especially if you can reference all of the past incidents that this would have helped guard against. Quite frankly, that sounds like a dysfunctional environment and absolutely nobody would fault you for quitting a year into it (or even not waiting that long), in search of something better. I suspect that a part of the problem is that in our market, we have mindsets that don't go beyond any of the following goals: - we got paid, regardless of what works or doesn't (can be seen in consulting) - we shipped something to meet a deadline and not get contractual penalties, regardless of quality - we shipped something that seems to work, though we don't care about much else (day 2 concerns) Sometimes it's because of ignorance and not knowing better, other times it's because there are cultural issues in the country as a whole, maybe a lot of people viewing development only as a step in the path to becoming a manager, instead of a craft that demands attention and care. So what you get globally can be companies that range anywhere from "Hey, we want you to be comfortable and not overworked: here are some learning materials or a budget for that, here's our knowledgebase and an overview of our procedures and architecture decision records (ADRs), here's a user group for this particular technology, feel free to reach out if you need anything." to "Hey, ship the software until monday. Why isn't it still done? I don't care about the details, get it done." Maybe I should write a blog post about that some day, just not sure what to title it: "OKRs/KPIs of caring about software" or something like that, probably. I suspect bad mindsets and a bad culture is one of the reasons for sites like this existing: https://devrant.com/ https://devrant.com/ or rather why articles like this ring true: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks At the end of the day, do what you can to take care of yourself!

Macha 4y ago

> - What specs of a VM do you require? What are my options? How much do those cost the company? I can run N requests on my laptop with specs of X. CPU usage hits 100% and memory usage hits 50%. This is 10% of our projected load. Therefore I project that I will need 10X CPU and 5X memory resources for this system for our entire userbase. My laptop is pretty powerful, but there are systems with 320GB of RAM and 80 cores. Do we get 2 of them? Oh, what you have is a bunch of 3 generation old 28 core Xeons with 128gb of RAM. Can I get access to one of them to test thoroughput? One Xeon core is not as good as one macbook core, especially not that old, so I really can't make any promises of perf without testing on representative hardware. No? Fine, let's add 30% buffer for poor IPC on old hardware. Oh, you've got a custom wrapper for Java that launches it with a bunch of custom JVM options that the SRE org mandates everyone uses? Got any documentation on what those are? What's our lead time for getting more? We need to know that for an idea of how much buffer we need. So apparently lead time is six months? Ok, what's our load going to be in 6 months? I'll ask bizdev how many customers they think we'll have in six months. Oh, they answered with a vague "We want to have lots of users". Fine, we'll add an extra 100% buffer. What, the operational efficiency team are pissed our VMs are at 25% usage? While there are certainly dev teams that throw capacity management over the wall to SRE, the inverse is also nonsense as it's the SRE team who usually get informed of hardware options, deployment standards (especially in bigco) and company operational standards.

doublerabbit 4y ago
> What are my options? How much do those cost the company? Do we get 2 of them? What options do you want? You come to me with the specifications, and I'll get back to you if you can. I'll work with you to get what you want. If I am the one with the power to create, the gatekeeper to the virtualization cluster and with a ballpark figure this makes my life so much easier. It allows me to make my justification easier too. "Hi Manager, X needs this. I think it needs this. I'm going to setup this and evaluate, no manager I don't think your correct." "Here's my tech specification for what I'm setting up. here's the documentation for configuration and setup and these are the results" Lets ride. Let me give you 8x core and 64GB see how the performance spikes and go from there. I'm not stingy and always happy to give more to test performance and than decrease if it's overkill. But don't dispute if I start to take away because of. > what you have is a bunch of 3 generation old 28 core Xeons with 128gb of RAM. Can I get access to one of them to test thoroughput? Sure. Yes you can. Why do you think you can't? > Lead time As fast as you require me to setup the VM. If I have all the docs, I can fly-by and have this thing setup under a day at minimum. Heck, I'll even work no-paid overtime to get this for you. I can escalate this on the fly. I'm in the good books with the NetOps, I even know the backup-ops. Tough crowd every time to please, but I manage so. Production lead time? Sure, probably six months to get stake-holder approval and the rest, but I'll try to get it sooner. Prototype lead: a week. Just give me figures and I'll do the rest. Enterprise or not. I'll push for what you want but you have to work with me. However I need the figures and documentation before I can. I can't be seen creating the documentation on my own time based on made-up figures when I have an estate of 100 VM's needing security patches. That makes me look bad and if it fails the tests, I'll get the blame.