5 ms·

Cache on Delivery (Mining random people's memcached)

85 points by jackowayed 16y ago

tptacek 16y ago
I like the Sensepost people, but can sum up this talk very succinctly: * You need to make sure you've firewalled (and/or on single-host configurations bound it exclusively to lo0) memcached so people can't talk to it from the Internet. * If you survey the Internet, you'll find a lot of people who have not obeyed this rule. * If you don't obey this rule attackers can get and possibly alter most of your data; an exposed memcached is probably game-over for your site. The only things you should be able to talk to on anything in your data center are ports 443 and 80, plus 25 on your single mail relay if you have one, plus 22 (heavily filtered) on a single relay SSH server. If you can talk to more than that, you've probably done something very wrong.
- tomjen3 16y ago
  Personally I am so paranoid with SSH that I don't run it on the default port. That alone allows you to avoid most attackers trying to guess valid server ips. Now to put a honey pot on port 22.
  - KTamas 16y ago
    Oh yes, the first thing I do is change the SSH port, eleminates 99,99% of the auth attacks.
    - Vivtek 16y ago
      Damn. Why didn't I ever think of that? I'm doing that tomorrow.
      - dugmartin 16y ago
        Remove root login over ssh and disable password logins and you are good to go. If you want to get fancy Google for ssh port knocking.
        mrduncan 16y ago
        disable password logins I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.
        InclinedPlane 16y ago
        If you host your systems on a VPS service like linode or slicehost then you have the backup of a web based console in case you screw up royally and have a HD crash on the one machine your SSH key was on (for example).
        KTamas 16y ago
        Heh, port knocking looks cool, might do that just for the heck of it on a server that only I use. Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.
- bretthoerner 16y ago
  What happens when your 1 relay host goes down?
  - jrockway 16y ago
    You restart it.
- seiji 16y ago
  You missed the point where they found major sites wide open and extracted credentials for gowalla, bitly, and poked around PBS. Bad gowalla, bitly, and PBS operations departments. Shame on you.
- mls 16y ago
  Absolutely, there's no doubt that it's an operations issue rather than a memcached issue and, when found, indicates poor ops standards (remote enumeration of a cache is a debug feature apparently). With services moving out of private data centers and into the cloud, security tools are different from the arsenal used for internal apps; if you're an app guy deploying on EC2, you don't have a network security guy who's responsible for protecting your services. It's now your job to configure EC2 to protect your services, which is a small but significant difference. In terms of scanning for open ports, I think the main takeaway was that in distributed apps running off shared infrastructure, the basics often aren't followed by under pressure admins and devs. Memcached density was about 1 per 274 scanned addresses, which was higher than I'd have thought; it's apparently a mistake that's not uncommon. Unfortunately it's often not good enough to point out open ports, and one needs to demonstrate exploitability. The main thrust of the talk was so say "if you come across memcacheds, don't skip them, there's coolness there". e.g. an open memcached used by Django directly equates to remote code exec due to Python's pickle. You're spot on with the final point; firewall firewall firewall. [fd: that's my name on the preso]
davidfischer 16y ago
This was one of my favorite briefings at Blackhat.
lsb 16y ago
Memcached by default only connects to localhost. Were these people using it like that and/or does that matter?
- yanowitz 16y ago
  In most deployed configurations, you use memcache in a distributed fashion (so N machines can connect to a pool of caches), so you usually have them bind to a public interface. One that you stick behind at least one firewall. Exposing memcached (or riak or whatever) is operational malpractice.
  - _delirium 16y ago
    It's definitely dumb, but people might be lulled into thinking it isn't necessary because the trend among other servers is to essentially include their own firewalling setup, with a default of blacklist-everything that you have to explicitly override by whitelisting IPs, or by having clients authenticate. For example, you used to rely on firewalls to make sure nobody was sending spam through your internal mail relays, but these days most MTAs handle that filtering themselves, taking whitelists of "relay from here is ok" IPs in their config files, even though in theory iptables/ipfw/etc. could handle that job just fine, and more efficiently. More relevantly, most RDBMSs have a similar setup--- they won't just let any client talk to them, even when bound to a public IP. For efficiency, though, memcached explicitly decided not to handle that kind of filtering itself, e.g. by having the memcached config file take a list of IPs of other memcached servers to talk to (defaulting to none), or including some kind of auth mechanism, and instead relies on the system firewall for that. That's a reasonable design decision, but increasingly unusual. Most servers these days see the system firewall as a backup level of defense, not the primary one, and aim to be secure even when unfirewalled.
    - macca321 16y ago
      I lulled
- sensepost 16y ago
  AFAIK one of the points in the slides was that, by default, memcached binds to all interfaces.
werrett 16y ago
I've come across unprotected Memcached deployments in a couple of pen-tests. On one engagement in particular someone poking Memcached would be able to (temporarily) increase their account balance and even access CC numbers from recent transactions. :( As far as I see it, this is one of the unintentional side effects of "hosting in the cloud". If you had co-located servers you'd whack up a firewall and only allow your internal IPs to access non-HTTP ports. Alas everyone now just spins up an S3 image and palms it off to Amazon.
- bdr 16y ago
  With EC2, you have to opt-in any ports open to the public.
  - werrett 16y ago
    There you go, I stand corrected. Are you able to make requests between instances on non-public ports? As someone else pointed out Memcached infrastructure typically won't sit on your local webserver.
    - fleitz 16y ago
      Yes, it's pretty easy to set this up with security groups, you can restrict ports to only open to machines within another security group. So lets say you've got memcached, mysql and a bunch of webservers. On the memcached security group you open 11211 to the webservers group On the db security group you open 3306 to webservers On the webserver group you open 80 and 443 to everyone.
mikeklaas 16y ago
How can people not run a restrictive firewall on their externally-accessible boxes? EC2 even has this built in as a network-wide feature. Our production servers run with the following setup: * block everything from outside internal network * open port 80 on web server * open port 22 on all boxes, but only allow key-based authentication. Oh, and only allow connections from an ip whitelist. The restrictions on port 22 are probably a little overkill, I admit.
- maukdaddy 16y ago
  Restrictions on port 22 are NEVER overkill. In fact, you should run another port entirely.
  - _delirium 16y ago
    If you're already using key authentication, that doesn't add much security. A port is just a very short password shared by all users--- instead of logging in with "I'd like to login, please initiate public-key authentication", you now have to login with "I'd like to login, the magic number is 2222, and please initiate public-key authentication", which isn't really any more secure.
    - peripitea 16y ago
      Call me an engineer, but I don't understand why anyone would dismiss security through obscurity just because it's theoretically useless. If you can reduce the lifetime expected value of successful attacks on your system, it should be considered a win regardless of how you do it. (Of course, relying solely on security through obscurity is a sure-fire way to greatly increase said EV.)
    - billpg 16y ago
      It will deflect a lot of network-capacity-wasting opportunistic attempts if the masses don't even realise there's an SSH port there.
sensepost 16y ago
The blog entry on the topic is available at http://www.sensepost.com/blog/4873.html http://www.sensepost.com/blog/4873.html Also, GoWalla & Bit.Ly were informed pre-publication, and fixed their stuff.

85 points by jackowayed 16y ago

I like the Sensepost people, but can sum up this talk very succinctly: * You need to make sure you've firewalled (and/or on single-host configurations bound it exclusively to lo0) memcached so people can't talk to it from the Internet. * If you survey the Internet, you'll find a lot of people who have not obeyed this rule. * If you don't obey this rule attackers can get and possibly alter most of your data; an exposed memcached is probably game-over for your site. The only things you should be able to talk to on anything in your data center are ports 443 and 80, plus 25 on your single mail relay if you have one, plus 22 (heavily filtered) on a single relay SSH server. If you can talk to more than that, you've probably done something very wrong.

tomjen3 16y ago
Personally I am so paranoid with SSH that I don't run it on the default port. That alone allows you to avoid most attackers trying to guess valid server ips. Now to put a honey pot on port 22.
- KTamas 16y ago
  Oh yes, the first thing I do is change the SSH port, eleminates 99,99% of the auth attacks.
  - Vivtek 16y ago
    Damn. Why didn't I ever think of that? I'm doing that tomorrow.
    - dugmartin 16y ago
      Remove root login over ssh and disable password logins and you are good to go. If you want to get fancy Google for ssh port knocking.
      - mrduncan 16y ago
        disable password logins I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.
        InclinedPlane 16y ago
        If you host your systems on a VPS service like linode or slicehost then you have the backup of a web based console in case you screw up royally and have a HD crash on the one machine your SSH key was on (for example).
      - KTamas 16y ago
        Heh, port knocking looks cool, might do that just for the heck of it on a server that only I use. Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.
bretthoerner 16y ago
What happens when your 1 relay host goes down?
- jrockway 16y ago
  You restart it.
seiji 16y ago
You missed the point where they found major sites wide open and extracted credentials for gowalla, bitly, and poked around PBS. Bad gowalla, bitly, and PBS operations departments. Shame on you.
mls 16y ago
Absolutely, there's no doubt that it's an operations issue rather than a memcached issue and, when found, indicates poor ops standards (remote enumeration of a cache is a debug feature apparently). With services moving out of private data centers and into the cloud, security tools are different from the arsenal used for internal apps; if you're an app guy deploying on EC2, you don't have a network security guy who's responsible for protecting your services. It's now your job to configure EC2 to protect your services, which is a small but significant difference. In terms of scanning for open ports, I think the main takeaway was that in distributed apps running off shared infrastructure, the basics often aren't followed by under pressure admins and devs. Memcached density was about 1 per 274 scanned addresses, which was higher than I'd have thought; it's apparently a mistake that's not uncommon. Unfortunately it's often not good enough to point out open ports, and one needs to demonstrate exploitability. The main thrust of the talk was so say "if you come across memcacheds, don't skip them, there's coolness there". e.g. an open memcached used by Django directly equates to remote code exec due to Python's pickle. You're spot on with the final point; firewall firewall firewall. [fd: that's my name on the preso]

davidfischer 16y ago

This was one of my favorite briefings at Blackhat.

lsb 16y ago

Memcached by default only connects to localhost. Were these people using it like that and/or does that matter?

yanowitz 16y ago
In most deployed configurations, you use memcache in a distributed fashion (so N machines can connect to a pool of caches), so you usually have them bind to a public interface. One that you stick behind at least one firewall. Exposing memcached (or riak or whatever) is operational malpractice.
- _delirium 16y ago
  It's definitely dumb, but people might be lulled into thinking it isn't necessary because the trend among other servers is to essentially include their own firewalling setup, with a default of blacklist-everything that you have to explicitly override by whitelisting IPs, or by having clients authenticate. For example, you used to rely on firewalls to make sure nobody was sending spam through your internal mail relays, but these days most MTAs handle that filtering themselves, taking whitelists of "relay from here is ok" IPs in their config files, even though in theory iptables/ipfw/etc. could handle that job just fine, and more efficiently. More relevantly, most RDBMSs have a similar setup--- they won't just let any client talk to them, even when bound to a public IP. For efficiency, though, memcached explicitly decided not to handle that kind of filtering itself, e.g. by having the memcached config file take a list of IPs of other memcached servers to talk to (defaulting to none), or including some kind of auth mechanism, and instead relies on the system firewall for that. That's a reasonable design decision, but increasingly unusual. Most servers these days see the system firewall as a backup level of defense, not the primary one, and aim to be secure even when unfirewalled.
  - macca321 16y ago
    I lulled
sensepost 16y ago
AFAIK one of the points in the slides was that, by default, memcached binds to all interfaces.

werrett 16y ago

I've come across unprotected Memcached deployments in a couple of pen-tests. On one engagement in particular someone poking Memcached would be able to (temporarily) increase their account balance and even access CC numbers from recent transactions. :( As far as I see it, this is one of the unintentional side effects of "hosting in the cloud". If you had co-located servers you'd whack up a firewall and only allow your internal IPs to access non-HTTP ports. Alas everyone now just spins up an S3 image and palms it off to Amazon.

bdr 16y ago
With EC2, you have to opt-in any ports open to the public.
- werrett 16y ago
  There you go, I stand corrected. Are you able to make requests between instances on non-public ports? As someone else pointed out Memcached infrastructure typically won't sit on your local webserver.
  - fleitz 16y ago
    Yes, it's pretty easy to set this up with security groups, you can restrict ports to only open to machines within another security group. So lets say you've got memcached, mysql and a bunch of webservers. On the memcached security group you open 11211 to the webservers group On the db security group you open 3306 to webservers On the webserver group you open 80 and 443 to everyone.

mikeklaas 16y ago

How can people not run a restrictive firewall on their externally-accessible boxes? EC2 even has this built in as a network-wide feature. Our production servers run with the following setup: * block everything from outside internal network * open port 80 on web server * open port 22 on all boxes, but only allow key-based authentication. Oh, and only allow connections from an ip whitelist. The restrictions on port 22 are probably a little overkill, I admit.

maukdaddy 16y ago
Restrictions on port 22 are NEVER overkill. In fact, you should run another port entirely.
- _delirium 16y ago
  If you're already using key authentication, that doesn't add much security. A port is just a very short password shared by all users--- instead of logging in with "I'd like to login, please initiate public-key authentication", you now have to login with "I'd like to login, the magic number is 2222, and please initiate public-key authentication", which isn't really any more secure.
  - peripitea 16y ago
    Call me an engineer, but I don't understand why anyone would dismiss security through obscurity just because it's theoretically useless. If you can reduce the lifetime expected value of successful attacks on your system, it should be considered a win regardless of how you do it. (Of course, relying solely on security through obscurity is a sure-fire way to greatly increase said EV.)
  - billpg 16y ago
    It will deflect a lot of network-capacity-wasting opportunistic attempts if the masses don't even realise there's an SSH port there.

sensepost 16y ago

The blog entry on the topic is available at http://www.sensepost.com/blog/4873.html http://www.sensepost.com/blog/4873.html Also, GoWalla & Bit.Ly were informed pre-publication, and fixed their stuff.