2 ms·

How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords a

by strlen 16y ago

How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).

kragen 16y ago
You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
- strlen 16y ago
  Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).
  - kragen 16y ago
    Sorry, there were some shorthands in my post. Let me expand. If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't. If you can run strace on a process, you can inject malicious code into it. While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero! I wasn't suggesting that developers themselves would be putting malicious code into production.
    - strlen 16y ago
      How is getting onto a developer's workstation more difficult than on an operation engineer's workstation? If you don't allow developers to do some operational duties, this also means having to have more operations staff (which typically will have higher privileges than developers, anyway). You are also forgetting that there is usually a step between a developer workstation and production, and at that gateway you'll typically have additional security measures (so that simply getting to the gateway doesn't get you to production). I don't, however, disagree with your overall idea: yes, technically, developers having ssh access to production might (to a very small degree) reduce security, all else being equal. However, there countless benefits to giving developers ssh access that result in greater security. Nor do you have to use the same policy for all machines: SOX, for example, mandates that developers that write the code that handles financial transactions shouldn't have access to machines that run this code (to prevent fraud). There are other types of machines I'd include in this case (databases holding sensitive user data, machines holding sensitive configuration, etc...). However, for a vanilla machine running an application server, or a database server holding strictly non-sensitive/non-revenue data, that's not the case. There are also far worse mistakes one can make (e.g., don't use version control, don't put proper review procedures in place, hire/don't fire incompetent developers) which will impact security.

by strlen 16y ago

kragen 16y ago

You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.

strlen 16y ago
Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't. For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications. My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).
- kragen 16y ago
  Sorry, there were some shorthands in my post. Let me expand. If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't. If you can run strace on a process, you can inject malicious code into it. While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero! I wasn't suggesting that developers themselves would be putting malicious code into production.
  - strlen 16y ago
    How is getting onto a developer's workstation more difficult than on an operation engineer's workstation? If you don't allow developers to do some operational duties, this also means having to have more operations staff (which typically will have higher privileges than developers, anyway). You are also forgetting that there is usually a step between a developer workstation and production, and at that gateway you'll typically have additional security measures (so that simply getting to the gateway doesn't get you to production). I don't, however, disagree with your overall idea: yes, technically, developers having ssh access to production might (to a very small degree) reduce security, all else being equal. However, there countless benefits to giving developers ssh access that result in greater security. Nor do you have to use the same policy for all machines: SOX, for example, mandates that developers that write the code that handles financial transactions shouldn't have access to machines that run this code (to prevent fraud). There are other types of machines I'd include in this case (databases holding sensitive user data, machines holding sensitive configuration, etc...). However, for a vanilla machine running an application server, or a database server holding strictly non-sensitive/non-revenue data, that's not the case. There are also far worse mistakes one can make (e.g., don't use version control, don't put proper review procedures in place, hire/don't fire incompetent developers) which will impact security.