4 ms·

This actually got me a while ago but with redis and some script kiddy turning my dev server into a bitcoin miner. Anyone else running docker and using iptables

This actually got me a while ago but with redis and some script kiddy turning my dev server into a bitcoin miner. Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/

squeaky-clean 5y ago
Almost exactly the same thing has happened to me except Selenium and they were trying to log into Playstation Network accounts.
- deleted 5y ago
  [deleted]
qbasic_forever 5y ago
The insidious thing is that there's no indication, failure, log or anything to tell you something is out of the ordinary either. It would be one thing if it just exploded and failed to run, but it's even worse that it silently interacts with iptables & ufw to allow all the traffic through. The exact opposite of what you want or intended.
- dawnerd 5y ago
  Only reason I noticed was a command was taking slightly longer than normal and I checked htop and saw redis using 100% cpu. Absolutely insane this is the Docker default.
frakkingcylons 5y ago
Same exact thing happened to my test redis instance two years ago.
bushbaba 5y ago
I’m kind of shocked this is even deemed acceptable architecture. You’d think docker wouldn’t even touch iptables unless explicitly told to.
- qbasic_forever 5y ago
  It's how they can make containers feel like isolated little subnets without resorting to vxlan or other kernel-level stuff. It's a great development experience and I'd be sad to see it go. But.... it really needs to proactively detect and warn users. The issue has been known for many years. A quick little check and error out on startup if you're running on Ubuntu or have ufw enabled would probably save 99% of the pain people have had with it over the years.
  - 1337shadow 5y ago
    It works fine when users create docker networks for their containers to communicate, which docker-compose does by default, instead of publishing ports on 0.0.0.0 like yolo
- AtNightWeCode 5y ago
  I would say that this is the expected behavior. Also, I don't think one should rely on firewalls in this way.
  - Shank 5y ago
    Relying on firewalls to do what firewalls do and have done and continue to do seems perfectly acceptable. Yes, your database should have authentication enabled too, but expecting ports to not be unexpectedly open is the entire point of firewalls.
    - AtNightWeCode 5y ago
      Kind of what I said. Firewalls are for blocking unwanted traffic. It should not be used as a replacement for other security measures. "unexpectedly open", well, there I simply disagree.
      - Aeolun 5y ago
        So if you have a firewall set to block everything, and you run a docker container that listens to your global IP, you expect it to magic your firewall for you?
js4ever 5y ago
Redis can be exploited to run executables ???
- qbasic_forever 5y ago
  It can execute Lua, I'm sure there's plenty of fun for hackers to have with this: https://redis.io/commands/eval https://redis.io/commands/eval
- dawnerd 5y ago
  Yeah - kinda crazy theres no auth by default AND eval is allowed. Pretty trivial for someone to have it download a script and run it pretty much with free reign.
  - slavak 5y ago
    Redis doesn't accept unauthenticated external connections by default for a while now, specifically to try and eliminate this footgun. https://github.com/redis/redis/commit/edd4d555df57dc84265fdfb4ef59a4678832f6da https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
    - dawnerd 5y ago
      Interesting, this definitely happened with a more recent version... Wonder if theres some other exploit at play too (could also be the containerized version?)
    - SilverRed 5y ago
      I had an issue where I used the redis docker image and didn't understand docker networking properly so I set the network mode as host so my other container could connect. Not knowing this had exposed redis to the world unauthenticated (in about 2018). Eventually a kind script set a password on redis which caused me to notice and fix this issue.
- bigiain 5y ago
  From antirez the guy who wrote redis's blog: 'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96 http://antirez.com/news/96 That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...) Note: that's about 5 years old now...
tpetry 5y ago
I guess most people have faced this. I got years ago too but after a few seconds some internal monitoring system alerted me that new ports have been exposed. I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.
Aeolun 5y ago
What is this black magic? Why is docker concerned with iptables at all? And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”

squeaky-clean 5y ago

Almost exactly the same thing has happened to me except Selenium and they were trying to log into Playstation Network accounts.

deleted 5y ago
[deleted]

qbasic_forever 5y ago

The insidious thing is that there's no indication, failure, log or anything to tell you something is out of the ordinary either. It would be one thing if it just exploded and failed to run, but it's even worse that it silently interacts with iptables & ufw to allow all the traffic through. The exact opposite of what you want or intended.

dawnerd 5y ago
Only reason I noticed was a command was taking slightly longer than normal and I checked htop and saw redis using 100% cpu. Absolutely insane this is the Docker default.

frakkingcylons 5y ago

Same exact thing happened to my test redis instance two years ago.

bushbaba 5y ago

I’m kind of shocked this is even deemed acceptable architecture. You’d think docker wouldn’t even touch iptables unless explicitly told to.

qbasic_forever 5y ago
It's how they can make containers feel like isolated little subnets without resorting to vxlan or other kernel-level stuff. It's a great development experience and I'd be sad to see it go. But.... it really needs to proactively detect and warn users. The issue has been known for many years. A quick little check and error out on startup if you're running on Ubuntu or have ufw enabled would probably save 99% of the pain people have had with it over the years.
- 1337shadow 5y ago
  It works fine when users create docker networks for their containers to communicate, which docker-compose does by default, instead of publishing ports on 0.0.0.0 like yolo
AtNightWeCode 5y ago
I would say that this is the expected behavior. Also, I don't think one should rely on firewalls in this way.
- Shank 5y ago
  Relying on firewalls to do what firewalls do and have done and continue to do seems perfectly acceptable. Yes, your database should have authentication enabled too, but expecting ports to not be unexpectedly open is the entire point of firewalls.
  - AtNightWeCode 5y ago
    Kind of what I said. Firewalls are for blocking unwanted traffic. It should not be used as a replacement for other security measures. "unexpectedly open", well, there I simply disagree.
    - Aeolun 5y ago
      So if you have a firewall set to block everything, and you run a docker container that listens to your global IP, you expect it to magic your firewall for you?

js4ever 5y ago

Redis can be exploited to run executables ???

qbasic_forever 5y ago
It can execute Lua, I'm sure there's plenty of fun for hackers to have with this: https://redis.io/commands/eval https://redis.io/commands/eval
dawnerd 5y ago
Yeah - kinda crazy theres no auth by default AND eval is allowed. Pretty trivial for someone to have it download a script and run it pretty much with free reign.
- slavak 5y ago
  Redis doesn't accept unauthenticated external connections by default for a while now, specifically to try and eliminate this footgun. https://github.com/redis/redis/commit/edd4d555df57dc84265fdfb4ef59a4678832f6da https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
  - dawnerd 5y ago
    Interesting, this definitely happened with a more recent version... Wonder if theres some other exploit at play too (could also be the containerized version?)
  - SilverRed 5y ago
    I had an issue where I used the redis docker image and didn't understand docker networking properly so I set the network mode as host so my other container could connect. Not knowing this had exposed redis to the world unauthenticated (in about 2018). Eventually a kind script set a password on redis which caused me to notice and fix this issue.
bigiain 5y ago
From antirez the guy who wrote redis's blog: 'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96 http://antirez.com/news/96 That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...) Note: that's about 5 years old now...

tpetry 5y ago

I guess most people have faced this. I got years ago too but after a few seconds some internal monitoring system alerted me that new ports have been exposed. I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.

Aeolun 5y ago

What is this black magic? Why is docker concerned with iptables at all? And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”