4 ms·

Author here. TPROXY module is pretty special, it really would have been hard to handle any inbound port without it. I guess it shows that there are benefits in

by majke 9y ago

Author here. TPROXY module is pretty special, it really would have been hard to handle any inbound port without it. I guess it shows that there are benefits in keeping firewall and network stack code tied close.

cjcampbell 9y ago
Excellent article all the way around. I may assign this to my networking students further into the quarter. It provides a nice alternative use case that should help them think more critically about the relationship between applications, the socket API, and the rest of the network stack. Thanks!
c22 9y ago
I have a vpn for myself and some friends that accepts connections on all ports. I set it up over a decade ago on OpenBSD with a simple pf redirect. I have never had any problems with it, but it obviously doesn't see nearly as much traffic as Cloudflare. Does OpenBSD handle this differently than Linux, or am I doing this wrong?
- ryan-c 9y ago
  The trick is that they want the application to be able to see what the original destination IP and port were. I'm not sure if a pf redirect preserves that information.
  - jlgaddis 9y ago
    It does -- or appears to, at least, for my instances of (OpenBSD's) spamd.
zng00 9y ago
This is great, thanks for sharing. I'm curious about the downstream proxy process (i.e. ::1234) and how you scale it and balance load across multiple instances of the process. You can't really use iptables to load balance your processes as either the DNAT or REDIRECT mechanism will modify the destination address, right? Ex. # TPROXY directs all traffic to :1234, and these rules load balance to 4 different processes iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination 127.0.0.1:8080 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 1 -j DNAT --to-destination 127.0.0.1:8081 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 2 -j DNAT --to-destination 127.0.0.1:8082 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 3 -j DNAT --to-destination 127.0.0.1:8083
- majke 9y ago
  We have a single Accept queue for all the ports. For TCP it doesn't create any problems - the new connection rate is rarely significant. For the accept-queue load balancing see these blog posts: https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/ https://blog.cloudflare.com/the-sad-state-of-linux-socket-ba... https://blog.cloudflare.com/syn-packet-handling-in-the-wild/ https://blog.cloudflare.com/syn-packet-handling-in-the-wild/
  - zng00 9y ago
    Wow, these are some great resources. Thanks for sharing! I have a call with one of your colleagues in 5 minutes ;)

by majke 9y ago

cjcampbell 9y ago

Excellent article all the way around. I may assign this to my networking students further into the quarter. It provides a nice alternative use case that should help them think more critically about the relationship between applications, the socket API, and the rest of the network stack. Thanks!

c22 9y ago

I have a vpn for myself and some friends that accepts connections on all ports. I set it up over a decade ago on OpenBSD with a simple pf redirect. I have never had any problems with it, but it obviously doesn't see nearly as much traffic as Cloudflare. Does OpenBSD handle this differently than Linux, or am I doing this wrong?

ryan-c 9y ago
The trick is that they want the application to be able to see what the original destination IP and port were. I'm not sure if a pf redirect preserves that information.
- jlgaddis 9y ago
  It does -- or appears to, at least, for my instances of (OpenBSD's) spamd.

zng00 9y ago

This is great, thanks for sharing. I'm curious about the downstream proxy process (i.e. ::1234) and how you scale it and balance load across multiple instances of the process. You can't really use iptables to load balance your processes as either the DNAT or REDIRECT mechanism will modify the destination address, right? Ex. # TPROXY directs all traffic to :1234, and these rules load balance to 4 different processes iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination 127.0.0.1:8080 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 1 -j DNAT --to-destination 127.0.0.1:8081 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 2 -j DNAT --to-destination 127.0.0.1:8082 iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 3 -j DNAT --to-destination 127.0.0.1:8083

majke 9y ago
We have a single Accept queue for all the ports. For TCP it doesn't create any problems - the new connection rate is rarely significant. For the accept-queue load balancing see these blog posts: https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/ https://blog.cloudflare.com/the-sad-state-of-linux-socket-ba... https://blog.cloudflare.com/syn-packet-handling-in-the-wild/ https://blog.cloudflare.com/syn-packet-handling-in-the-wild/
- zng00 9y ago
  Wow, these are some great resources. Thanks for sharing! I have a call with one of your colleagues in 5 minutes ;)