7 ms·

You know what developers do with root? They install bitcoin miners. They install malware. They install all sorts of shit. It's unfortunate but it's true, I'

You know what developers do with root? They install bitcoin miners. They install malware. They install all sorts of shit. It's unfortunate but it's true, I've seen it 100x. We let our devs have local admin at my company but please don't misunderstand the serious risks involved in that decision. We take great measures to ensure that we can do that while still keeping our customers safe. We would never allow admin AWS access, that's absurd. An attacker on your box would be able to own prod. Sorry that it's an inconvenience! When you manage to solve infosec, hit your ops/sec team up, they'll be happy to hand you the keys after that point.

scarface74 8y ago
Trust but verify. AWS has plenty of logging capabilities. I'm not saying that all developers should have unfettered access. But someone has to be trusted.
- staticassertion 8y ago
  Of course, some people need admin capabilities. But a senior developer? Nah. Engineers whose specific job it is to admin infra, yes. Logging is great, but it's a 'we are owned, how do we get them out' measure - we want to avoid getting to that point. edit: Of course, this all depends on your threat model, posture, maturity, etc. I'm just saying - we have lots of very good reasons to lock devs out.
  - scarface74 8y ago
    You're assuming that a small company has a separate person solely dedicated to infrastructure. Yes I have an AWS certification and on paper I am qualified to be an "AWS Architect". But I would be twiddling my thumbs all day with not enough work to do and would die a thousands deaths if I didn't do hands on coding.
    - staticassertion 8y ago
      As I said in my edit, this is all subject to the details of a company's posture.
    - deleted 8y ago
      [deleted]
  - AlphaSite 8y ago
    This sounds like Amin access on local machines, not admin access at all?
    - staticassertion 8y ago
      Actually, I realized we don't even give local admin. I forgot. Unfortunately, no editing.
- msost 8y ago
  "I'm not saying _all_ developers should have unfettered access", but I'm an exception. And then add, "otherwise I'll leave for a company that gives me that access". See how that sounds?
  - mickronome 8y ago
    It sounds like someone who knows what they want and who like to get their hands dirty in a job market where they habe that choice. Not all developers are irresponsible, and not all ops people are impossible to work with. But both certainly exist.
    - scarface74 8y ago
      That's what made things more infuriating at the company I left. I came in as the lead developer knowing that if I wanted to get things done, I would have to ingratiate myself to the net ops people. I could fire off a Skype, ask for what I needed on prem (VMs and hard drive space mostly) and by the time I sent the ticket request as a formality, it was already done. But then they decided to "go the cloud" and instead of training their internal network ops people and having them work with the vendor who was creating the AWS infrastructure, the vendor took everything over and even our internal folks couldn't get anything done without layers of approvals. So I ended up setting up my own AWS VPC at home, doing proof of concepts just so I could learn how to talk the talk, studied for the system administrator cert (even though I was a developer) and then got so frustrated it was easier to change my environment than to try to change my environment. So now they are spending more money on AWS than they would have in their colo because no developer wants to go through the hassle of going through the red tape of trying to get AWS services and are just throwing things on EC2 instances. In today's world, an EC2 instance for custom developed code is almost always sub optimal when you have things like AWS Lambda for serverless functions, Fargate for serverless Docker containers and dozens of other services that allows you to use AWS to do the "undifferentiated heavy lifting".
  - scarface74 8y ago
    Yes that sounds like someone who doesn't want to have to wait two weeks to get approvals to create resources in a Dev environment. But as the team lead, I already had the final say into what code went into production and could do all kind of nefarious acts if I desired. Yes we had a CI/CD process in place with sign offs. But there was nothing stopping me from only doing certain actions based on which environment the program was running in. I've seen what happens to people who are "just developers" that spend all their life working in large companies where they never learn anything about database administration, Dev ops, Net ops, or in the modern era - cloud administration. They aren't as highly valued as someone who really is full stack - from the browser all the way down to architecting the infrastructure. Why wouldn't I choose a company if given that option that lets me increase my marketability, and gives me hands on experience in an enterprise environment instead of just being a "paper tiger" who has certifications but no experience at scale?
- ntnn 8y ago
  Logging doesn't help you when you business has to shut down because someone took over you account and deleted everything. Separation of access is important and _required_. Developers don't need access to prod, admins maintaining the infrastructure don't need access to the directory, IDM doesn't need access to either QA or prod. Developers do need full access in an environment to properly test - but that environment should be basically hermetically sealed from the rest of the companies infrastructure. So even if they do screw up the whole business won't be affected.
  - andrewstuart2 8y ago
    If someone took over your account and deleted everything, and you couldn't get any of it back, you weren't taking care of the "availability" third of security. I agree that developers don't need access to everything, but I completely disagree that they should have no access to prod. The games of phone tag and "try typing this" that happens during prod issues is a waste of everybody's time, and I fully believe that the people who write the code should be the ones with both the responsibility of the pagers and the ability to fix the code they've deployed. Everybody is happier, and the job gets done more quickly, when the job gets done by the people most qualified to do it (because they wrote it), and when they bear the consequences of writing bad code. The environment needs to be set up to be forgiving of mistakes, yes, but that's easily done these days and should never result in loss of data if the infrastructure is properly automated. If giving production access means your developers can screw something up, then your admins can just as easily screw something up. Create environments that forgive these failures because they'll happen one way or another.
    - georgebarnett 8y ago
      There are already examples companies which have folded overnight after losing creds and having everything deleted. Removing root is not a trust issue - it’s a security surface area issue. You increase the number of audit points and attack options by at least an order of magnitude (1 admin : 10 devs). In a small shop this might be acceptable, however in a large org it’s plain old insane. If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.
jacques_chester 8y ago
> They install bitcoin miners. I think of bitcoin mining attacks as pentesting with a contingent fee. Probably the cheapest and least destructive way to learn you have security weaknesses.
peterwwillis 8y ago
I think the GP is talking about things like provisioning AWS resources, not having root on the production box. Anyone should be able to provision new resources without harming production. Most of the time you don't see this, because management has still not gotten it through their thick heads that in order for the business to move fast, devs need the agency to implement solutions fast. Cost-cutting of AWS leads to one (or several!) groups that are the gatekeepers to AWS, purportedly in order to make it more "efficient" by everyone doing everything one way, or having one group of experts handle it. But that's just one group of plumbers handling repairs for an entire city. Leaks spring up and nobody is there to fix them, and meanwhile, the developer is sitting there staring at the leak, going "will you please just let me duct tape this leak so I can get back to work!?"
- scarface74 8y ago
  Exactly. When the last company I worked for decided to "go to the cloud", it took weeks to get simple EC2 instances setup because we had to go through my manager, say why we needed it, then wait on an outside vendor. We had layers of consultants getting in the way. I still ended up doing stuff suboptimally like using all of the stuff I would use on prem like Consul (instead of Parameter store),Fabio (instead of load balancers), Nomad (instead of ECS), VSTS (CI/CD - when I should have use the Codepipeline, Code Build, code deploy ) and even a local copy of Memcached and RabbitMQ just so I wouldn't have to go through all of the red tape just to provision resources so we could get our work done. Our projects ended up costing more because it took so long to get anything approved, I would overprovision "just in case". Anytime that I wanted to stand up VMs for testing, I would send a Skype message to our local net ops person for an on prem VM and have it within an hour. I was the "architect" for the company - I couldn't afford to make excuses and it's never helpful at my level of responsibility to blame management when things don't go right. I had to get it done. So, I learned what I could, studied AWS services on a high level inside and out, got a certification and left for greener pastures. And people wonder why I never do large organizations.
voidr 8y ago
> They install bitcoin miners. You can do that without root. > They install malware. What developers are you talking about? I want to know what developer would risk their career and prison time. And if a developer has no problem with going to prison, surely they have no problem finding some 0day privilege escalation exploit. > It's unfortunate but it's true, I've seen it 100x. That's not really an argument, how many developers did you have and how many of them risked prison time to install malware on dev machines? > We take great measures to ensure that we can do that while still keeping our customers safe. > We would never allow admin AWS access, that's absurd. An attacker on your box would be able to own prod. You aren't doing a great job then, because your production stuff should be on a separate AWS account altogether.
- scarface74 8y ago
  Even if your production stuff is in a separate account, that just helps prevent someone from accidentally screwing up production. To think that not giving your developers - the people who are creating code that you are putting on your servers and know the infrastructure as well as anyone - will prevent them from being malicious is just security theatre. It may help you check the box about being compliant with some type of standard but it really doesn't help you. If the developers program has access to production resources, they can gain access to those resources.
scarface74 8y ago
A developer wouldn’t have to install malware. A developer could create malware. Even if you have all of your deployments automated, any developer worth anything could sneak malicious code into the process.
- staticassertion 8y ago
  I don't see how that's really relevant.
  - scarface74 8y ago
    A developer doesn't need admin access to the AWS console to install malware, bitcoin miners, etc. He just needs to have his code installed. The person who is deploying the developer's code installed is rarely going to code review the code before its installed. If my code has access to production when you deploy it, I can make it do anything I want and you would never know.

scarface74 8y ago

Trust but verify. AWS has plenty of logging capabilities. I'm not saying that all developers should have unfettered access. But someone has to be trusted.

staticassertion 8y ago
Of course, some people need admin capabilities. But a senior developer? Nah. Engineers whose specific job it is to admin infra, yes. Logging is great, but it's a 'we are owned, how do we get them out' measure - we want to avoid getting to that point. edit: Of course, this all depends on your threat model, posture, maturity, etc. I'm just saying - we have lots of very good reasons to lock devs out.
- scarface74 8y ago
  You're assuming that a small company has a separate person solely dedicated to infrastructure. Yes I have an AWS certification and on paper I am qualified to be an "AWS Architect". But I would be twiddling my thumbs all day with not enough work to do and would die a thousands deaths if I didn't do hands on coding.
  - staticassertion 8y ago
    As I said in my edit, this is all subject to the details of a company's posture.
  - deleted 8y ago
    [deleted]
- AlphaSite 8y ago
  This sounds like Amin access on local machines, not admin access at all?
  - staticassertion 8y ago
    Actually, I realized we don't even give local admin. I forgot. Unfortunately, no editing.
msost 8y ago
"I'm not saying _all_ developers should have unfettered access", but I'm an exception. And then add, "otherwise I'll leave for a company that gives me that access". See how that sounds?
- mickronome 8y ago
  It sounds like someone who knows what they want and who like to get their hands dirty in a job market where they habe that choice. Not all developers are irresponsible, and not all ops people are impossible to work with. But both certainly exist.
  - scarface74 8y ago
    That's what made things more infuriating at the company I left. I came in as the lead developer knowing that if I wanted to get things done, I would have to ingratiate myself to the net ops people. I could fire off a Skype, ask for what I needed on prem (VMs and hard drive space mostly) and by the time I sent the ticket request as a formality, it was already done. But then they decided to "go the cloud" and instead of training their internal network ops people and having them work with the vendor who was creating the AWS infrastructure, the vendor took everything over and even our internal folks couldn't get anything done without layers of approvals. So I ended up setting up my own AWS VPC at home, doing proof of concepts just so I could learn how to talk the talk, studied for the system administrator cert (even though I was a developer) and then got so frustrated it was easier to change my environment than to try to change my environment. So now they are spending more money on AWS than they would have in their colo because no developer wants to go through the hassle of going through the red tape of trying to get AWS services and are just throwing things on EC2 instances. In today's world, an EC2 instance for custom developed code is almost always sub optimal when you have things like AWS Lambda for serverless functions, Fargate for serverless Docker containers and dozens of other services that allows you to use AWS to do the "undifferentiated heavy lifting".
- scarface74 8y ago
  Yes that sounds like someone who doesn't want to have to wait two weeks to get approvals to create resources in a Dev environment. But as the team lead, I already had the final say into what code went into production and could do all kind of nefarious acts if I desired. Yes we had a CI/CD process in place with sign offs. But there was nothing stopping me from only doing certain actions based on which environment the program was running in. I've seen what happens to people who are "just developers" that spend all their life working in large companies where they never learn anything about database administration, Dev ops, Net ops, or in the modern era - cloud administration. They aren't as highly valued as someone who really is full stack - from the browser all the way down to architecting the infrastructure. Why wouldn't I choose a company if given that option that lets me increase my marketability, and gives me hands on experience in an enterprise environment instead of just being a "paper tiger" who has certifications but no experience at scale?
ntnn 8y ago
Logging doesn't help you when you business has to shut down because someone took over you account and deleted everything. Separation of access is important and _required_. Developers don't need access to prod, admins maintaining the infrastructure don't need access to the directory, IDM doesn't need access to either QA or prod. Developers do need full access in an environment to properly test - but that environment should be basically hermetically sealed from the rest of the companies infrastructure. So even if they do screw up the whole business won't be affected.
- andrewstuart2 8y ago
  If someone took over your account and deleted everything, and you couldn't get any of it back, you weren't taking care of the "availability" third of security. I agree that developers don't need access to everything, but I completely disagree that they should have no access to prod. The games of phone tag and "try typing this" that happens during prod issues is a waste of everybody's time, and I fully believe that the people who write the code should be the ones with both the responsibility of the pagers and the ability to fix the code they've deployed. Everybody is happier, and the job gets done more quickly, when the job gets done by the people most qualified to do it (because they wrote it), and when they bear the consequences of writing bad code. The environment needs to be set up to be forgiving of mistakes, yes, but that's easily done these days and should never result in loss of data if the infrastructure is properly automated. If giving production access means your developers can screw something up, then your admins can just as easily screw something up. Create environments that forgive these failures because they'll happen one way or another.
  - georgebarnett 8y ago
    There are already examples companies which have folded overnight after losing creds and having everything deleted. Removing root is not a trust issue - it’s a security surface area issue. You increase the number of audit points and attack options by at least an order of magnitude (1 admin : 10 devs). In a small shop this might be acceptable, however in a large org it’s plain old insane. If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.

jacques_chester 8y ago

> They install bitcoin miners. I think of bitcoin mining attacks as pentesting with a contingent fee. Probably the cheapest and least destructive way to learn you have security weaknesses.

peterwwillis 8y ago

I think the GP is talking about things like provisioning AWS resources, not having root on the production box. Anyone should be able to provision new resources without harming production. Most of the time you don't see this, because management has still not gotten it through their thick heads that in order for the business to move fast, devs need the agency to implement solutions fast. Cost-cutting of AWS leads to one (or several!) groups that are the gatekeepers to AWS, purportedly in order to make it more "efficient" by everyone doing everything one way, or having one group of experts handle it. But that's just one group of plumbers handling repairs for an entire city. Leaks spring up and nobody is there to fix them, and meanwhile, the developer is sitting there staring at the leak, going "will you please just let me duct tape this leak so I can get back to work!?"

scarface74 8y ago
Exactly. When the last company I worked for decided to "go to the cloud", it took weeks to get simple EC2 instances setup because we had to go through my manager, say why we needed it, then wait on an outside vendor. We had layers of consultants getting in the way. I still ended up doing stuff suboptimally like using all of the stuff I would use on prem like Consul (instead of Parameter store),Fabio (instead of load balancers), Nomad (instead of ECS), VSTS (CI/CD - when I should have use the Codepipeline, Code Build, code deploy ) and even a local copy of Memcached and RabbitMQ just so I wouldn't have to go through all of the red tape just to provision resources so we could get our work done. Our projects ended up costing more because it took so long to get anything approved, I would overprovision "just in case". Anytime that I wanted to stand up VMs for testing, I would send a Skype message to our local net ops person for an on prem VM and have it within an hour. I was the "architect" for the company - I couldn't afford to make excuses and it's never helpful at my level of responsibility to blame management when things don't go right. I had to get it done. So, I learned what I could, studied AWS services on a high level inside and out, got a certification and left for greener pastures. And people wonder why I never do large organizations.

voidr 8y ago

> They install bitcoin miners. You can do that without root. > They install malware. What developers are you talking about? I want to know what developer would risk their career and prison time. And if a developer has no problem with going to prison, surely they have no problem finding some 0day privilege escalation exploit. > It's unfortunate but it's true, I've seen it 100x. That's not really an argument, how many developers did you have and how many of them risked prison time to install malware on dev machines? > We take great measures to ensure that we can do that while still keeping our customers safe. > We would never allow admin AWS access, that's absurd. An attacker on your box would be able to own prod. You aren't doing a great job then, because your production stuff should be on a separate AWS account altogether.

scarface74 8y ago
Even if your production stuff is in a separate account, that just helps prevent someone from accidentally screwing up production. To think that not giving your developers - the people who are creating code that you are putting on your servers and know the infrastructure as well as anyone - will prevent them from being malicious is just security theatre. It may help you check the box about being compliant with some type of standard but it really doesn't help you. If the developers program has access to production resources, they can gain access to those resources.

scarface74 8y ago

A developer wouldn’t have to install malware. A developer could create malware. Even if you have all of your deployments automated, any developer worth anything could sneak malicious code into the process.

staticassertion 8y ago
I don't see how that's really relevant.
- scarface74 8y ago
  A developer doesn't need admin access to the AWS console to install malware, bitcoin miners, etc. He just needs to have his code installed. The person who is deploying the developer's code installed is rarely going to code review the code before its installed. If my code has access to production when you deploy it, I can make it do anything I want and you would never know.