5 ms·

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so

by tasuki 4y ago

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...

amtamt 4y ago
In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.
- tasuki 4y ago
  You're of course right. But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?
  - amtamt 4y ago
    How are gc/ syslog/ kernel logs are special and not sent to central logging? Ad-hoc logs (such as thread dumps etc.) can also be triggered via safe application end points from restricted locations. > "Is it never necessary to run ad hoc commands on your production systems?" Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team. Even the most competent people, sysadmin or developer, can make a mistake. I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.
    - tasuki 4y ago
      Well done you, I guess? Nothing to disagree with. I'm just amazed this can be actual reality :)
      - glotzerhotze 4y ago
        It takes a while to get there - and a lot of discipline of those involved.
  - glotzerhotze 4y ago
    In an ideal world, you wouldn‘t need that. But most systems never matured to be considered being in an ideal state / world.
- sgarland 4y ago
  As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] that fixes this, but it also requires at the very least some Linux capabilities that might raise eyebrows with security, and as written it requires privileged containers. Thankfully this particular example will hopefully be fixed with cgroupsv2, but still. SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host. "Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it. [0] https://github.com/kubernetes/kubernetes/issues/50632 https://github.com/kubernetes/kubernetes/issues/50632 [1] https://github.com/transferwise/oomie https://github.com/transferwise/oomie
  - amtamt 4y ago
    Seems "not well deployed system" or "system not well understood by developer", what stops one reducing retention to few hours for node kernel logs or having read only access to certain parts, if at all there are so many kernel/ sys logs. I have never come across a system where application logs were not multiple orders of magnitude more in volume than syslogs. Not sure what RDS vs in-house DB example was meant to convey. Ironically, these examples are only proving the point made in first comment.
bsagdiyev 4y ago
Anecdotally I've had the opposite experience. Majority of developers I've interacted with didn't understand how anything systems admin side worked. This isn't to say all developers are like that, but that you may be more of an outlier in recent times. Also your admin team sounds awful to work with.
arcbyte 4y ago
A metaphor that just occurred to me. When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done. SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.
throwaway892238 4y ago
Admittedly, that experience sucks. But there's a couple reasons they did that: 1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process. 2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time. 3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens. 4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.
- tasuki 4y ago
  1. I don't want root on prod servers, sounds like a nightmare! 2. Yes, I'm sure this was the case. 3. Indeed! 4. I haven't! My original comment just took some situation out of context. I know nothing about what else the sysadmins were doing, they probably had other things on their plate about which I knew nothing...

by tasuki 4y ago

amtamt 4y ago

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.

tasuki 4y ago
You're of course right. But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?
- amtamt 4y ago
  How are gc/ syslog/ kernel logs are special and not sent to central logging? Ad-hoc logs (such as thread dumps etc.) can also be triggered via safe application end points from restricted locations. > "Is it never necessary to run ad hoc commands on your production systems?" Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team. Even the most competent people, sysadmin or developer, can make a mistake. I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.
  - tasuki 4y ago
    Well done you, I guess? Nothing to disagree with. I'm just amazed this can be actual reality :)
    - glotzerhotze 4y ago
      It takes a while to get there - and a lot of discipline of those involved.
- glotzerhotze 4y ago
  In an ideal world, you wouldn‘t need that. But most systems never matured to be considered being in an ideal state / world.
sgarland 4y ago
As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] that fixes this, but it also requires at the very least some Linux capabilities that might raise eyebrows with security, and as written it requires privileged containers. Thankfully this particular example will hopefully be fixed with cgroupsv2, but still. SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host. "Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it. [0] https://github.com/kubernetes/kubernetes/issues/50632 https://github.com/kubernetes/kubernetes/issues/50632 [1] https://github.com/transferwise/oomie https://github.com/transferwise/oomie
- amtamt 4y ago
  Seems "not well deployed system" or "system not well understood by developer", what stops one reducing retention to few hours for node kernel logs or having read only access to certain parts, if at all there are so many kernel/ sys logs. I have never come across a system where application logs were not multiple orders of magnitude more in volume than syslogs. Not sure what RDS vs in-house DB example was meant to convey. Ironically, these examples are only proving the point made in first comment.

bsagdiyev 4y ago

Anecdotally I've had the opposite experience. Majority of developers I've interacted with didn't understand how anything systems admin side worked. This isn't to say all developers are like that, but that you may be more of an outlier in recent times. Also your admin team sounds awful to work with.

arcbyte 4y ago

A metaphor that just occurred to me. When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done. SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.

throwaway892238 4y ago

Admittedly, that experience sucks. But there's a couple reasons they did that: 1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process. 2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time. 3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens. 4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.

tasuki 4y ago
1. I don't want root on prod servers, sounds like a nightmare! 2. Yes, I'm sure this was the case. 3. Indeed! 4. I haven't! My original comment just took some situation out of context. I know nothing about what else the sysadmins were doing, they probably had other things on their plate about which I knew nothing...