4 ms·

> Contrast this with Facebook, Google et al-- where every process (from corporate IT to production operations to HR and recruiting) is designed with needs of t

> Contrast this with Facebook, Google et al-- where every process (from corporate IT to production operations to HR and recruiting) is designed with needs of the engineering organization in mind: Completely agree. > Want ssh access to production? No problem. This makes me a little uneasy, I'm not sure everyone should have ssh access to the production server.

strlen 16y ago
> This makes me a little uneasy, I'm not sure everyone should have ssh access to the production server. Every _developer_ should. No question about it. Sudo should given on an is-needed basis, but ultimately, as a developer I can screw up a lot more by simply writing bad code. Simple philosophy: you build the software, you should be involved in running it (including carrying a pager). Amazon's CTO agrees: http://twitter.com/#!/Werner/status/50957550908223490 http://twitter.com/#!/Werner/status/50957550908223490
- chollida1 16y ago
  > Every _developer_ should. No question about it. Sudo should given on an is-needed basis, but ultimately, as a developer I can screw up a lot more by simply writing bad code. Fair enough. This, more nuanced point, I agree with.
flyt 16y ago
If you can't trust engineers to work in production, then you can't trust their code. Extensive logging, auditing, and granular access control are also critical to making this work in a large engineering org.
- ianhawes 16y ago
  I don't think it's a matter of trust but rather mitigating your exposure to hackers.
  - strlen 16y ago
    How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).
    - kragen 16y ago
      You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
      - strlen 16y ago
        Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).

strlen 16y ago

> This makes me a little uneasy, I'm not sure everyone should have ssh access to the production server. Every _developer_ should. No question about it. Sudo should given on an is-needed basis, but ultimately, as a developer I can screw up a lot more by simply writing bad code. Simple philosophy: you build the software, you should be involved in running it (including carrying a pager). Amazon's CTO agrees: http://twitter.com/#!/Werner/status/50957550908223490 http://twitter.com/#!/Werner/status/50957550908223490

chollida1 16y ago
> Every _developer_ should. No question about it. Sudo should given on an is-needed basis, but ultimately, as a developer I can screw up a lot more by simply writing bad code. Fair enough. This, more nuanced point, I agree with.

flyt 16y ago

If you can't trust engineers to work in production, then you can't trust their code. Extensive logging, auditing, and granular access control are also critical to making this work in a large engineering org.

ianhawes 16y ago
I don't think it's a matter of trust but rather mitigating your exposure to hackers.
- strlen 16y ago
  How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).
  - kragen 16y ago
    You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
    - strlen 16y ago
      Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).