7 ms·

If you can't trust engineers to work in production, then you can't trust their code. Extensive logging, auditing, and granular access control are also critical

by flyt 16y ago

If you can't trust engineers to work in production, then you can't trust their code. Extensive logging, auditing, and granular access control are also critical to making this work in a large engineering org.

ianhawes 16y ago
I don't think it's a matter of trust but rather mitigating your exposure to hackers.
- strlen 16y ago
  How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).
  - kragen 16y ago
    You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
    - strlen 16y ago
      Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).
      - kragen 16y ago
        Sorry, there were some shorthands in my post. Let me expand. If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't. If you can run strace on a process, you can inject malicious code into it. While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero! I wasn't suggesting that developers themselves would be putting malicious code into production.

by flyt 16y ago

ianhawes 16y ago

I don't think it's a matter of trust but rather mitigating your exposure to hackers.

strlen 16y ago
How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).
- kragen 16y ago
  You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
  - strlen 16y ago
    Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).
    - kragen 16y ago
      Sorry, there were some shorthands in my post. Let me expand. If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't. If you can run strace on a process, you can inject malicious code into it. While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero! I wasn't suggesting that developers themselves would be putting malicious code into production.