4 ms·

> (Most likely the answer is NO, so you wouldn't know about brute-force attacks, invalid requests, corrupted data, or other anomalies the developers should have

> (Most likely the answer is NO, so you wouldn't know about brute-force attacks, invalid requests, corrupted data, or other anomalies the developers should have their eyes on). I would be much more worried about security if the developers had access to the production environment than if they didn't.

alasdair_ 6y ago
I once worked for a fortune 500 that blocked all but a few unix sysadmins from executing shell commands on prod servers. So far, so good. The issue was that the “sysadmins” knew almost nothing about how anything worked, so the procedure was that the devs would give them commands to type and they would just type them in to a terminal. Of course, if anything went wrong, or was ambiguous, the dev would need to check on it, and would end up standing next to the “sysadmin”’s desk and just telling them which characters to type. I once had to explain what a pipe character was... Anyway, the end result was that all of the devs had prod access, they just had a very slow interface to it.
- pvtmert 6y ago
  I'm system engineer in a decent company. I wish devs could access to prod systems so they have to get to cleanup their mess / get responsible for actions they do... They already have access to staging and preprod what they do is basically coming to me and saying 'it did run well on their macos machine' There are outliers, some few good people and some random people having no idea what they are doing. IMHO there should be random quiz/task/test each day you login. Something obvious but not trivial at the same time related with domain of the system. So you would get 24h access if you pass, get denied for 3 hours if fail. 3 questions in each session... At least people might learn something out of requirement...
- henryfjordan 6y ago
  > Anyway, the end result was that all of the devs had prod access, they just had a very slow interface to it. They also had a person who could see if the dev was stalking an ex-girlfriend or pilfering bank account info. Sounds perfect.
  - alasdair_ 6y ago
    Normally, yes, but the people employed as unix admins literally didn't understand what most basic shell commands did, to the point that copying prod data to a remote server would have been completely trivial.
- caiobegotti 6y ago
  HSBC production support used to be pretty much exactly like that during rollouts (no idea if that has changed in the last 5 years).
aerostable_slug 6y ago
Me consulting at a major media company, paraphrasing:"It's, um, interesting that most of the development staff has ssh to essentially every production system as well as sudo privileges. And logging isn't super great." PHB: "They need access to all the systems." Me: "Well, um, what if one of them got disgruntled and just tossed some malicious JS in one of your many, many front end servers? Given the procedures here, it would be unlikely anyone would stumble across it, but they'd get a good amount of hits before the browsers block the entire site and traffic precipitously drops." PHB: "Nobody would ever actually do that. Besides: we have to give them access. (looks at me as if he's explaining something to a five year old) We're a devops shop..."
- malka 6y ago
  With GDPR, it's not really kosher to give devs access to production database if it contains data about people.
  - fendy3002 6y ago
    Honest question, who can keep the access to production in the eye of GDPR?
    - malka 6y ago
      the production team can. it has made very hard to debug. Now the production team has to do most of the debug work, they have to give us anonymised data (and you cant turn personal data in anomymous data. it would be pseudonymous at best) that trigger the bug. It can be pretty hard if your organization was not organized with this in mind in the first place.
    - saratogacx 6y ago
      Operations staff gets access to production machines with Operations being explicitly forbidden from producing code that runs on the systems. There are still vectors for bad actors of course, but the idea is to firewall those who write the code from those who run it.
    - 6y ago
ric2b 6y ago
Why is that?

alasdair_ 6y ago

I once worked for a fortune 500 that blocked all but a few unix sysadmins from executing shell commands on prod servers. So far, so good. The issue was that the “sysadmins” knew almost nothing about how anything worked, so the procedure was that the devs would give them commands to type and they would just type them in to a terminal. Of course, if anything went wrong, or was ambiguous, the dev would need to check on it, and would end up standing next to the “sysadmin”’s desk and just telling them which characters to type. I once had to explain what a pipe character was... Anyway, the end result was that all of the devs had prod access, they just had a very slow interface to it.

pvtmert 6y ago
I'm system engineer in a decent company. I wish devs could access to prod systems so they have to get to cleanup their mess / get responsible for actions they do... They already have access to staging and preprod what they do is basically coming to me and saying 'it did run well on their macos machine' There are outliers, some few good people and some random people having no idea what they are doing. IMHO there should be random quiz/task/test each day you login. Something obvious but not trivial at the same time related with domain of the system. So you would get 24h access if you pass, get denied for 3 hours if fail. 3 questions in each session... At least people might learn something out of requirement...
henryfjordan 6y ago
> Anyway, the end result was that all of the devs had prod access, they just had a very slow interface to it. They also had a person who could see if the dev was stalking an ex-girlfriend or pilfering bank account info. Sounds perfect.
- alasdair_ 6y ago
  Normally, yes, but the people employed as unix admins literally didn't understand what most basic shell commands did, to the point that copying prod data to a remote server would have been completely trivial.
caiobegotti 6y ago
HSBC production support used to be pretty much exactly like that during rollouts (no idea if that has changed in the last 5 years).

aerostable_slug 6y ago

Me consulting at a major media company, paraphrasing:"It's, um, interesting that most of the development staff has ssh to essentially every production system as well as sudo privileges. And logging isn't super great." PHB: "They need access to all the systems." Me: "Well, um, what if one of them got disgruntled and just tossed some malicious JS in one of your many, many front end servers? Given the procedures here, it would be unlikely anyone would stumble across it, but they'd get a good amount of hits before the browsers block the entire site and traffic precipitously drops." PHB: "Nobody would ever actually do that. Besides: we have to give them access. (looks at me as if he's explaining something to a five year old) We're a devops shop..."

malka 6y ago
With GDPR, it's not really kosher to give devs access to production database if it contains data about people.
- fendy3002 6y ago
  Honest question, who can keep the access to production in the eye of GDPR?
  - malka 6y ago
    the production team can. it has made very hard to debug. Now the production team has to do most of the debug work, they have to give us anonymised data (and you cant turn personal data in anomymous data. it would be pseudonymous at best) that trigger the bug. It can be pretty hard if your organization was not organized with this in mind in the first place.
  - saratogacx 6y ago
    Operations staff gets access to production machines with Operations being explicitly forbidden from producing code that runs on the systems. There are still vectors for bad actors of course, but the idea is to firewall those who write the code from those who run it.
  - 6y ago

ric2b 6y ago

Why is that?