8 ms·

is a bad idea

71 points by gehaxelt 7y ago

closeparen 7y ago
You do not need to put keys on the web sever’s disk to SSH onward from it. Use agent forwarding instead.
- egwynn 7y ago
  Or even use `ProxyJump`, which doesn’t put as much trust on the jump host!
  - gehaxelt 7y ago
    Good points! I've updated the post. Thanks!
bfgpereira 7y ago
This is what I have learned in many years of work: people who know systems should be let to handle those systems. This is what happens when a developer is left to do the work that a system administrator should be trained to do - not all are -. For a developer, in most cases, "just works" is the end goal, when referring to systems. Not "how it works", and what are the implications of making it work like this. This really makes me sad.
- busterarm 7y ago
  I'm glad I can do both, but being the pivot point is less than stellar sometimes and can stick you with odd tasks (or way more than you want). For too many developers "i shouldn't be blocked" is the highest virtue and that's when things like this happen.
  - bfgpereira 7y ago
    You describe precisely one of my points there. Its incredible how comfortable people are touching components that they don't fully understand. :D
- scarejunba 7y ago
  Sys admins are dead. I have GKE now via a reliable Terraform module. None of my production instances can be logged onto.
  - haydn3 7y ago
    Are you sure about that?
    - scarejunba 7y ago
      Only time will tell if I am in fact right. I'm counting on being more right than the guys who have dedicated staff who routinely shell into their servers. I suppose we'll see if companies with sysadmins have more breaches than the guys who run their own ops using container orchestration etc. I think I'd go even odds $1k that over the next five years, most large scale data breaches will be at organizations where sys admins run the majority of ops. There's a whole new category of errors you can make (making your bucket open, etc.) with cloud providers but the tooling has better defaults.
      - bfgpereira 7y ago
        > Only time will tell if I am in fact right. I guess you prove a point here: you trust that the deployment of your GKE module allows you to be safe, so your investment vs risk trade seems to satisfy you. But you, yourself, cannot even predict how insecure you are at the moment due to the complexity of the software solutions you are using. > I suppose we'll see if companies with sysadmins have more breaches than the guys who run their own ops using container orchestration etc. Oh, I do agree, at least with the current state of sysadmins out there. But the problem is that you loose control of the integrity of your system once you reach a point where software complexity becomes your security entry point. If you are sure the containers you will be using are secure, have sane defaults, are up to date, etc, then fine, good job! I just don't trust that most people will be able to reassure me that. And please keep in mind that in your case, the target is not your container platform, the target would be the containers in that platform, and the services they run.
- swiley 7y ago
  I would disagree: The problem is developers (and users in general, but their lack of formal training is an excuse) being comfortable using interfaces and abstractions they don't fully understand. Note that the result of this might sound like it makes the idea of a professional system administrator invalid but that's not true: I think the better SAs of the past had a thorough understanding of what their tools did and many of probably even modified them, this contrasts the current situation where people are poking things in PAS GUIs and accidentally running up huge bills.
  - akira2501 7y ago
    > being comfortable using interfaces and abstractions they don't fully understand. I don't think it was an interface or abstraction that got the user in trouble here, they were using a pair of systems in ways that were fine on their own, but combined led to an emergent vulnerability that they didn't even know to consider. It may be sheer pedantry but I really do see this as a unique "systems" issue, and this type of 'emergent' property between separate self-contained programs is fully within that domain.
    - wwright 7y ago
      I think it is an abstraction: the one underlying both of those components that combined to create the vulnerability. The abstractions provided by the OS compose in very surprising and hard to predict ways for humans. This is why newer systems don’t use the same abstractions (JavaScript and browser APIs), or else sandbox them much more thoroughly (iOS). Those newer tools have newer problems, of course, but I think a lot of the churn and reinvention of tech that we complain about is really about trying to find abstractions that combine in more predictable and useful ways.
      - pdkl95 7y ago
        > The abstractions ... [are] very surprising From this[1] insightful video essay by Kyle Kallgren: >> Metaphor Shear -- That feeling all users experience when you realize the metaphor you are working in is bogus. When the computer fails you and you remember that there are a hundred translations between input and output. Codes and translations we don't have the time or patience to do ourselves. Intellectual labor that we've surrendered to a device. >> The joke at the center of Douglas Adams Hitchhiker's Guide To The Galaxy is about metaphor shear. The answer to an important question lost on its long journey from input to output. A computer glitch so huge, so strange and so embarrassing that its programmers have to make a computer the size of a planet to file a bug report. [1] https://www.youtube.com/watch?v=hr9_DcO6G3A https://www.youtube.com/watch?v=hr9_DcO6G3A
- sombremesa 7y ago
  I've encountered a system administrator who left the admin LDAP password (for the entire organization) in plaintext in a world accessible script. I'd tell you the name but I don't want to drag the institution through the mud unnecessarily.
  - mschuster91 7y ago
    Oh holy hell this is a common mistake. Especially if your developers have local admin rights. Don't expect /etc/skel (OS X) to be unreadable.
- thosakwe 7y ago
  Honestly, I wish more developers were familiar with the Linux command line. It definitely has a learning curve, so it makes sense that people avoid it if it's not necessary to launch their app. That being said, there's a lot of value in knowing how to set up a server yourself, and secure it.
- lubujackson 7y ago
  Isn't this exactly the reason people say PHP is awful, though? Hidden pitfalls if you just do what works and aren't security conscious, and too much flexibility instead of one clean standard. And at the end of the day, putting blame on rookie users who keep wandering onto the busy street and get run over by a bus. At this point there is no reason sane network security shouldn't be baked into popular Linux OSes (except that the cloud is busy abstracting the problem away). Sure, real sysadmins can have the keys to the gun safe, but these are structural problems that could and should be mitigated through modernizing OS design.
  - arpa 7y ago
    Literally nothing you said makes sense.
Scarbutt 7y ago
You really have to go out of your way to make the www-data user available through ssh, weird some do this.
- linuxdude314 7y ago
  My thoughts exactly. For someone responsible for a system to do this is such an act of gross negligence, and demonstrates such a lack of competency I would be remiss to keep the person at the company were they my employee.
- kjs3 7y ago
  I can tell you exactly how it happened: someone thought that this was the most expedient way to do something they wanted to do, they unfortunately had rights to do it, and they just did it without considering for a minute the consequences. Happens all the time.
jrochkind1 7y ago
Exposing a private key to the world is a security problem? No kidding?
- deleted 7y ago
  [deleted]
- the_alchemist 7y ago
  As long as you keep the public part private, there is no issue ;)
  - MaxBarraclough 7y ago
    From what I see on StackExchange [0] that's probably untrue. [0] https://security.stackexchange.com/q/172274/ https://security.stackexchange.com/q/172274/
AdmiralAsshat 7y ago
The files in ~/.ssh are usually initialized with restrictive permissions, so how do they end up getting exposed? The only way I can think off-the-bat is that someone absent-mindedly commits them to their git dotfiles and ends up copying them over to another machine when they do a `git clone` command.
- znpy 7y ago
  > so how do they end up getting exposed? the author cites "allowing developers to connect to the host with the www-data user", and this is a very specific form of incompetence. www-data is the name commonly used by debian and debian-based distros to run apache and other http servers. it's literally, just designed to run the executable, not to upload new version of webpages or anything. there are countless ways to avoid this pitfall, the simplest that comes to my mind is creating another user for uploading stuff and adding such user to the www-data group. at the end of the day... meh. people might start a campaign about how not to use the www-data or something else, but not-very-techy people will find another way to misuse a webserver.
danielparks 7y ago
The www-data user (or whatever the web server is running as) should not own any files that are served by the web server. The user should not be able to log in either (its shell should be /bin/false or something similar). Use an entirely different user for file ownership.
DaniloDias 7y ago
TL;DR: Antipattern: pointing web server config to any files based in /home.
- asveikau 7y ago
  Not just that. Even if you don't make that mistake, having servers ssh into other hosts and leaving keys on them for this purpose means if one machine is compromised, others can be too. And they can use known_hosts to discover which ones.
  - arpa 7y ago
    ssh -A is a thing. A risky thing, but so much better than keeping private keys on server.
coldtea 7y ago
Next up "Leaving your Ferrari parked with its doors open and loaded with new iPhones in Downtown L.A. can incur a theft problem"...
terlisimo 7y ago
Sane nginx default for 99.999% sites out there: # prevent access to any file/dir beginning with a dot location ~ /\. { return 404; }
pdkl95 7y ago
> For some of them, the www-data's home directory is the DocumentRoot “Well, there’s your problem.” Don't expose any ${HOME} to the world! SSH keys are not the only exploitable file in a typical homedir (or even auto-generated from /etc/skel/)! A few files that come to mind: # other keys ~/.gnupg/ # probably lots of app-specific risks # (e.g. saved login info) ~/.cache/ ~/.config/ ~/.local/share/ # if it's also a desktop system running X ~/.Xauthority ~/.mozilla/ Yes, protecting ~/.id* with a passphrase is important and leaking ~/.ssh/known_hosts can have consequences, but this type of exploit shouldn't even be possible. Don't share your homedir - which contains most user-level config files on UNIX systems - with the world. DocumentRoot needs to be contained in a subdir. (edit: or even better, contained somewhere outside of /home where it won't overlap with common file paths)
- jmole 7y ago
  why would www-data need a private key though?
  - sowbug 7y ago
    It doesn't, but if someone is shelling around on the server, they might throw a key in there for convenience, maybe in order to scp something from another machine, and then forget to remove it when they're done. One can debate whether the root cause is forgetfulness, or rather that people shouldn't be sshing into prod servers to begin with.
    - deleted 7y ago
      [deleted]
    - im3w1l 7y ago
      Ever putting private data in a public place, is an unacceptable risk. Even if you remember to remove it there is a window of vulnerability. And there are people out there constantly probing for weaknesses.
  - deleted 7y ago
    [deleted]
air7 7y ago
Off topic but I personally find the use of the female pronouns in technical articles distracting. Referring to a hypothetical "Attacker" as "he" or "they" is what my brain expects which means I can parse the text faster and get to the point quicker. When the "attacker" is a "she" I'm surprised and the flow is interrupted. I'm forced to think about the fact that the author is signaling to me that they believe that women can be anything they want and shouldn't be boxed in into predetermined gender special professions etc. Perhaps one might argue that this interruption is precisely the goal, and therein lies my point: I see this choice of words as a form of guerrilla advertisement by the author to showcase their agenda about a subject which has no relevance to the topic of the essay. Basically every "she" is a little ad for "Women's Rights" (for lack of a better term), and ads are annoying. Note that this has nothing to do with my personal opinion on the subject. It's just like mangling in an opinion about climate change, veganism etc. While legitimate and worthwhile, it shouldn't be purported as part of a technical article. I'm obviously "zooming-in" a little on a minute matter, but nevertheless it's a minor pet peeve of mine. I wonder how others see this.
OrgNet 7y ago
lol... in other news: Pwning your passwordless web server the easy way
eqqn 7y ago
You can sometimes grab these ssh keys if the server has a severe directory traversal, local-file-inclusion vulnerability(run server/DB as root) , or to use for pivoting to different user/machine... But never heard of these being exposed outright.
maury91 7y ago
Or "how leaving the keys in the lock when you are not a home is a bad idea"

71 points by gehaxelt 7y ago

closeparen 7y ago

You do not need to put keys on the web sever’s disk to SSH onward from it. Use agent forwarding instead.

egwynn 7y ago
Or even use `ProxyJump`, which doesn’t put as much trust on the jump host!
- gehaxelt 7y ago
  Good points! I've updated the post. Thanks!

bfgpereira 7y ago

This is what I have learned in many years of work: people who know systems should be let to handle those systems. This is what happens when a developer is left to do the work that a system administrator should be trained to do - not all are -. For a developer, in most cases, "just works" is the end goal, when referring to systems. Not "how it works", and what are the implications of making it work like this. This really makes me sad.

busterarm 7y ago
I'm glad I can do both, but being the pivot point is less than stellar sometimes and can stick you with odd tasks (or way more than you want). For too many developers "i shouldn't be blocked" is the highest virtue and that's when things like this happen.
- bfgpereira 7y ago
  You describe precisely one of my points there. Its incredible how comfortable people are touching components that they don't fully understand. :D
scarejunba 7y ago
Sys admins are dead. I have GKE now via a reliable Terraform module. None of my production instances can be logged onto.
- haydn3 7y ago
  Are you sure about that?
  - scarejunba 7y ago
    Only time will tell if I am in fact right. I'm counting on being more right than the guys who have dedicated staff who routinely shell into their servers. I suppose we'll see if companies with sysadmins have more breaches than the guys who run their own ops using container orchestration etc. I think I'd go even odds $1k that over the next five years, most large scale data breaches will be at organizations where sys admins run the majority of ops. There's a whole new category of errors you can make (making your bucket open, etc.) with cloud providers but the tooling has better defaults.
    - bfgpereira 7y ago
      > Only time will tell if I am in fact right. I guess you prove a point here: you trust that the deployment of your GKE module allows you to be safe, so your investment vs risk trade seems to satisfy you. But you, yourself, cannot even predict how insecure you are at the moment due to the complexity of the software solutions you are using. > I suppose we'll see if companies with sysadmins have more breaches than the guys who run their own ops using container orchestration etc. Oh, I do agree, at least with the current state of sysadmins out there. But the problem is that you loose control of the integrity of your system once you reach a point where software complexity becomes your security entry point. If you are sure the containers you will be using are secure, have sane defaults, are up to date, etc, then fine, good job! I just don't trust that most people will be able to reassure me that. And please keep in mind that in your case, the target is not your container platform, the target would be the containers in that platform, and the services they run.
swiley 7y ago
I would disagree: The problem is developers (and users in general, but their lack of formal training is an excuse) being comfortable using interfaces and abstractions they don't fully understand. Note that the result of this might sound like it makes the idea of a professional system administrator invalid but that's not true: I think the better SAs of the past had a thorough understanding of what their tools did and many of probably even modified them, this contrasts the current situation where people are poking things in PAS GUIs and accidentally running up huge bills.
- akira2501 7y ago
  > being comfortable using interfaces and abstractions they don't fully understand. I don't think it was an interface or abstraction that got the user in trouble here, they were using a pair of systems in ways that were fine on their own, but combined led to an emergent vulnerability that they didn't even know to consider. It may be sheer pedantry but I really do see this as a unique "systems" issue, and this type of 'emergent' property between separate self-contained programs is fully within that domain.
  - wwright 7y ago
    I think it is an abstraction: the one underlying both of those components that combined to create the vulnerability. The abstractions provided by the OS compose in very surprising and hard to predict ways for humans. This is why newer systems don’t use the same abstractions (JavaScript and browser APIs), or else sandbox them much more thoroughly (iOS). Those newer tools have newer problems, of course, but I think a lot of the churn and reinvention of tech that we complain about is really about trying to find abstractions that combine in more predictable and useful ways.
    - pdkl95 7y ago
      > The abstractions ... [are] very surprising From this[1] insightful video essay by Kyle Kallgren: >> Metaphor Shear -- That feeling all users experience when you realize the metaphor you are working in is bogus. When the computer fails you and you remember that there are a hundred translations between input and output. Codes and translations we don't have the time or patience to do ourselves. Intellectual labor that we've surrendered to a device. >> The joke at the center of Douglas Adams Hitchhiker's Guide To The Galaxy is about metaphor shear. The answer to an important question lost on its long journey from input to output. A computer glitch so huge, so strange and so embarrassing that its programmers have to make a computer the size of a planet to file a bug report. [1] https://www.youtube.com/watch?v=hr9_DcO6G3A https://www.youtube.com/watch?v=hr9_DcO6G3A
sombremesa 7y ago
I've encountered a system administrator who left the admin LDAP password (for the entire organization) in plaintext in a world accessible script. I'd tell you the name but I don't want to drag the institution through the mud unnecessarily.
- mschuster91 7y ago
  Oh holy hell this is a common mistake. Especially if your developers have local admin rights. Don't expect /etc/skel (OS X) to be unreadable.
thosakwe 7y ago
Honestly, I wish more developers were familiar with the Linux command line. It definitely has a learning curve, so it makes sense that people avoid it if it's not necessary to launch their app. That being said, there's a lot of value in knowing how to set up a server yourself, and secure it.
lubujackson 7y ago
Isn't this exactly the reason people say PHP is awful, though? Hidden pitfalls if you just do what works and aren't security conscious, and too much flexibility instead of one clean standard. And at the end of the day, putting blame on rookie users who keep wandering onto the busy street and get run over by a bus. At this point there is no reason sane network security shouldn't be baked into popular Linux OSes (except that the cloud is busy abstracting the problem away). Sure, real sysadmins can have the keys to the gun safe, but these are structural problems that could and should be mitigated through modernizing OS design.
- arpa 7y ago
  Literally nothing you said makes sense.

Scarbutt 7y ago

You really have to go out of your way to make the www-data user available through ssh, weird some do this.

linuxdude314 7y ago
My thoughts exactly. For someone responsible for a system to do this is such an act of gross negligence, and demonstrates such a lack of competency I would be remiss to keep the person at the company were they my employee.
kjs3 7y ago
I can tell you exactly how it happened: someone thought that this was the most expedient way to do something they wanted to do, they unfortunately had rights to do it, and they just did it without considering for a minute the consequences. Happens all the time.

jrochkind1 7y ago

Exposing a private key to the world is a security problem? No kidding?

deleted 7y ago
[deleted]
the_alchemist 7y ago
As long as you keep the public part private, there is no issue ;)
- MaxBarraclough 7y ago
  From what I see on StackExchange [0] that's probably untrue. [0] https://security.stackexchange.com/q/172274/ https://security.stackexchange.com/q/172274/

AdmiralAsshat 7y ago

The files in ~/.ssh are usually initialized with restrictive permissions, so how do they end up getting exposed? The only way I can think off-the-bat is that someone absent-mindedly commits them to their git dotfiles and ends up copying them over to another machine when they do a `git clone` command.

znpy 7y ago
> so how do they end up getting exposed? the author cites "allowing developers to connect to the host with the www-data user", and this is a very specific form of incompetence. www-data is the name commonly used by debian and debian-based distros to run apache and other http servers. it's literally, just designed to run the executable, not to upload new version of webpages or anything. there are countless ways to avoid this pitfall, the simplest that comes to my mind is creating another user for uploading stuff and adding such user to the www-data group. at the end of the day... meh. people might start a campaign about how not to use the www-data or something else, but not-very-techy people will find another way to misuse a webserver.

danielparks 7y ago

The www-data user (or whatever the web server is running as) should not own any files that are served by the web server. The user should not be able to log in either (its shell should be /bin/false or something similar). Use an entirely different user for file ownership.

DaniloDias 7y ago

TL;DR: Antipattern: pointing web server config to any files based in /home.

asveikau 7y ago
Not just that. Even if you don't make that mistake, having servers ssh into other hosts and leaving keys on them for this purpose means if one machine is compromised, others can be too. And they can use known_hosts to discover which ones.
- arpa 7y ago
  ssh -A is a thing. A risky thing, but so much better than keeping private keys on server.

coldtea 7y ago

Next up "Leaving your Ferrari parked with its doors open and loaded with new iPhones in Downtown L.A. can incur a theft problem"...

terlisimo 7y ago

Sane nginx default for 99.999% sites out there: # prevent access to any file/dir beginning with a dot location ~ /\. { return 404; }

pdkl95 7y ago

> For some of them, the www-data's home directory is the DocumentRoot “Well, there’s your problem.” Don't expose any ${HOME} to the world! SSH keys are not the only exploitable file in a typical homedir (or even auto-generated from /etc/skel/)! A few files that come to mind: # other keys ~/.gnupg/ # probably lots of app-specific risks # (e.g. saved login info) ~/.cache/ ~/.config/ ~/.local/share/ # if it's also a desktop system running X ~/.Xauthority ~/.mozilla/ Yes, protecting ~/.id* with a passphrase is important and leaking ~/.ssh/known_hosts can have consequences, but this type of exploit shouldn't even be possible. Don't share your homedir - which contains most user-level config files on UNIX systems - with the world. DocumentRoot needs to be contained in a subdir. (edit: or even better, contained somewhere outside of /home where it won't overlap with common file paths)

jmole 7y ago
why would www-data need a private key though?
- sowbug 7y ago
  It doesn't, but if someone is shelling around on the server, they might throw a key in there for convenience, maybe in order to scp something from another machine, and then forget to remove it when they're done. One can debate whether the root cause is forgetfulness, or rather that people shouldn't be sshing into prod servers to begin with.
  - deleted 7y ago
    [deleted]
  - im3w1l 7y ago
    Ever putting private data in a public place, is an unacceptable risk. Even if you remember to remove it there is a window of vulnerability. And there are people out there constantly probing for weaknesses.
- deleted 7y ago
  [deleted]

air7 7y ago

Off topic but I personally find the use of the female pronouns in technical articles distracting. Referring to a hypothetical "Attacker" as "he" or "they" is what my brain expects which means I can parse the text faster and get to the point quicker. When the "attacker" is a "she" I'm surprised and the flow is interrupted. I'm forced to think about the fact that the author is signaling to me that they believe that women can be anything they want and shouldn't be boxed in into predetermined gender special professions etc. Perhaps one might argue that this interruption is precisely the goal, and therein lies my point: I see this choice of words as a form of guerrilla advertisement by the author to showcase their agenda about a subject which has no relevance to the topic of the essay. Basically every "she" is a little ad for "Women's Rights" (for lack of a better term), and ads are annoying. Note that this has nothing to do with my personal opinion on the subject. It's just like mangling in an opinion about climate change, veganism etc. While legitimate and worthwhile, it shouldn't be purported as part of a technical article. I'm obviously "zooming-in" a little on a minute matter, but nevertheless it's a minor pet peeve of mine. I wonder how others see this.

OrgNet 7y ago

lol... in other news: Pwning your passwordless web server the easy way

eqqn 7y ago

You can sometimes grab these ssh keys if the server has a severe directory traversal, local-file-inclusion vulnerability(run server/DB as root) , or to use for pivoting to different user/machine... But never heard of these being exposed outright.

maury91 7y ago

Or "how leaving the keys in the lock when you are not a home is a bad idea"