8 ms·

TCP over TCP is a bad idea (2000)

86 points by Deeg9rie9usi 2y ago

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
dang 2y ago
Related: Why TCP over TCP is a bad idea (2001) - https://news.ycombinator.com/item?id=25080693 https://news.ycombinator.com/item?id=25080693 - Nov 2020 (68 comments) Why TCP Over TCP Is a Bad Idea (2001) - https://news.ycombinator.com/item?id=9281954 https://news.ycombinator.com/item?id=9281954 - March 2015 (43 comments) Why TCP Over TCP Is A Bad Idea - https://news.ycombinator.com/item?id=2409090 https://news.ycombinator.com/item?id=2409090 - April 2011 (26 comments)
Svip 2y ago
I notice that the earliest version of this post[0] is dated 1999, whilst the latest version is modified in 2001 (see the main link). Which year would be appropriate to mark it on HN? 1999? 2001? [0] https://web.archive.org/web/20000310230940/http://sites.inka.de/~W1011/devel/tcp-tcp.html https://web.archive.org/web/20000310230940/http://sites.inka...
- dmitrygr 2y ago
  sqrt(1999 * 2001)
  - schmidtleonard 2y ago
    (1999^-1 + 2001^-1)^-1
    - exe34 2y ago
      2000ish
  - pif 2y ago
    (|1999> + |2001>)*Sqrt(1/2)
- Shared404 2y ago
  IMO the latest update is the best option. Or a syntax like (<original date>, updated <update date>) if the update was not super substantial.
01HNNWZ0MV43FF 2y ago
Port forwarding doesn't count, right?
- duskwuff 2y ago
  Correct, because that doesn't transform the TCP stream. "TCP over TCP" specifically means a TCP stream whose payload represents a sequence of TCP packets.
  - AStonesThrow 2y ago
    TCP doesn't use "packets". They're called "segments".
    - lxgr 2y ago
      A TCP packet is a pretty common term for an IP packet containing a TCP segment. VPNs usually forward IP packets, so the usage seems correct to me here.
      - AStonesThrow 2y ago
        It's wrong. "TCP packet" is a pretty common colloquial term which is also factually wrong. There is no such thing as "packet" at the TCP protocol layer. This paper is discussing the tunnelling of payloads with PPP over SSH. You can see in the stacked diagram that "TCP" appears in two layers of the protocol stack, as does IP. Essentially, they're encapsulating TCP segments inside other TCP segments, although there are also trappings of IP, PPP, and SSH protocols in-between those. This isn't a VPN use case at all. And if a VPN is forwarding IP packets, then it's forwarding IP packets.
        adrian_b 2y ago
        "TCP packet" is an absolutely correct term. As the previous poster said, a "TCP packet" is not a "TCP segment". A "TCP packet" is an IP packet that carries some part of the TCP byte stream, i.e. normally some part of one TCP segment. Each "TCP packet" carries its own TCP header, so the division of the TCP segments into TCP packets is visible and significant when you look at a TCP data flow from outside, even if it does not matter internally for the processes that communicate through TCP. There are probably much more people who care about TCP packets than about TCP segments. When you are configuring firewall rules or monitoring network traffic, you are concerned about TCP packets, almost never about TCP segments.
aarmenaa 2y ago
I've just spent the last month learning exactly why I definitely do want a TCP over TCP VPN. The short answer is almost every cloud vendor assumes you're doing TCP, and they've taken the "unreliable" part of UDP to heart. It is practically impossible run any modern VPN on most cloud providers anymore. Over the last month, I've been attempting to set up a fast Wireguard VPN tunnel between AWS and OVH. AWS killed all internet access on the instance with zero warning and sent us an email indicating that they suspected the instance was compromised and being used as part of a DDOS attack. OVH randomly performs "DDOS mitigation" anytime the tunnel is under any load. In both cases we were able to talk to someone and have the issue addressed, but I wanna stress: this is one stream between two IPs -- there's nothing that makes this anything close to looking like a DDOS. Even after getting everything properly blessed, OVH drops all UDP traffic over 1 Gbps. It took them a month of back-and-forth troubleshooting to tell us this. The really terrible part is "TCP over TCP is bad" is now so prevalent there's basically no good VPN options for it if you need it. Wireguard won't do it directly, but there's hacks involving udp2raw. I tried it, and wasn't able to achieve more than 100 Mbps. OpenVPN can do it, but is single-threaded and won't reasonably do more than 1 Gbps without hardware acceleration, which didn't appear to work on EC2 instances. strongSwan cannot be configured to do unencapsulated ESP anymore -- they removed the option -- so it's UDP encapsulated only. Their reasoning is UDP is necessary for NAT traversal, and of course everybody needs that. It's also thread-per-SA so also not fast. The only solution I've found than can do something not UDP is Libreswan, which can still do unencapsulated ESP (IP Protocol 50) if you ask nicely. It's also thread-per-SA, but I've managed to wring 2 - 3 Gbps out of a single core after tinkering with the configuration. For the love of all that's good in the world, just add performant TCP support to Wireguard. I do not care about what happens in non-optimal conditions. /rant
- throwing_away 2y ago
  Did you go down the shadowsocks path at all?
  - aarmenaa 2y ago
    I did not. I'm not terribly familiar with it, but it doesn't look like I can do general routing with it, right? My end goal is to route between two subnets.
some_furry 2y ago
Yes, but what about IPv6 over Amazon S3? https://xeiaso.net/blog/anything-message-queue/ https://xeiaso.net/blog/anything-message-queue/
- bcrl 2y ago
  Time to pull RFC 1149: A Standard for the Transmission of IP Datagrams on Avian Carriers. https://www.rfc-editor.org/rfc/rfc1149 https://www.rfc-editor.org/rfc/rfc1149
cma 2y ago
If you are in a situation where you have to anyway, you can use multiple TCP sockets and round robin them (with Nagle off) such that you are always sending just one packet over each. You'll get overhead and some unneeded acks, but no front of line blocking of the second layer of TCP mechanics going on.

86 points by Deeg9rie9usi 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

dang 2y ago

Related: Why TCP over TCP is a bad idea (2001) - https://news.ycombinator.com/item?id=25080693 https://news.ycombinator.com/item?id=25080693 - Nov 2020 (68 comments) Why TCP Over TCP Is a Bad Idea (2001) - https://news.ycombinator.com/item?id=9281954 https://news.ycombinator.com/item?id=9281954 - March 2015 (43 comments) Why TCP Over TCP Is A Bad Idea - https://news.ycombinator.com/item?id=2409090 https://news.ycombinator.com/item?id=2409090 - April 2011 (26 comments)

Svip 2y ago

I notice that the earliest version of this post[0] is dated 1999, whilst the latest version is modified in 2001 (see the main link). Which year would be appropriate to mark it on HN? 1999? 2001? [0] https://web.archive.org/web/20000310230940/http://sites.inka.de/~W1011/devel/tcp-tcp.html https://web.archive.org/web/20000310230940/http://sites.inka...

dmitrygr 2y ago
sqrt(1999 * 2001)
- schmidtleonard 2y ago
  (1999^-1 + 2001^-1)^-1
  - exe34 2y ago
    2000ish
- pif 2y ago
  (|1999> + |2001>)*Sqrt(1/2)
Shared404 2y ago
IMO the latest update is the best option. Or a syntax like (<original date>, updated <update date>) if the update was not super substantial.

01HNNWZ0MV43FF 2y ago

Port forwarding doesn't count, right?

duskwuff 2y ago
Correct, because that doesn't transform the TCP stream. "TCP over TCP" specifically means a TCP stream whose payload represents a sequence of TCP packets.
- AStonesThrow 2y ago
  TCP doesn't use "packets". They're called "segments".
  - lxgr 2y ago
    A TCP packet is a pretty common term for an IP packet containing a TCP segment. VPNs usually forward IP packets, so the usage seems correct to me here.
    - AStonesThrow 2y ago
      It's wrong. "TCP packet" is a pretty common colloquial term which is also factually wrong. There is no such thing as "packet" at the TCP protocol layer. This paper is discussing the tunnelling of payloads with PPP over SSH. You can see in the stacked diagram that "TCP" appears in two layers of the protocol stack, as does IP. Essentially, they're encapsulating TCP segments inside other TCP segments, although there are also trappings of IP, PPP, and SSH protocols in-between those. This isn't a VPN use case at all. And if a VPN is forwarding IP packets, then it's forwarding IP packets.
      - adrian_b 2y ago
        "TCP packet" is an absolutely correct term. As the previous poster said, a "TCP packet" is not a "TCP segment". A "TCP packet" is an IP packet that carries some part of the TCP byte stream, i.e. normally some part of one TCP segment. Each "TCP packet" carries its own TCP header, so the division of the TCP segments into TCP packets is visible and significant when you look at a TCP data flow from outside, even if it does not matter internally for the processes that communicate through TCP. There are probably much more people who care about TCP packets than about TCP segments. When you are configuring firewall rules or monitoring network traffic, you are concerned about TCP packets, almost never about TCP segments.

aarmenaa 2y ago

I've just spent the last month learning exactly why I definitely do want a TCP over TCP VPN. The short answer is almost every cloud vendor assumes you're doing TCP, and they've taken the "unreliable" part of UDP to heart. It is practically impossible run any modern VPN on most cloud providers anymore. Over the last month, I've been attempting to set up a fast Wireguard VPN tunnel between AWS and OVH. AWS killed all internet access on the instance with zero warning and sent us an email indicating that they suspected the instance was compromised and being used as part of a DDOS attack. OVH randomly performs "DDOS mitigation" anytime the tunnel is under any load. In both cases we were able to talk to someone and have the issue addressed, but I wanna stress: this is one stream between two IPs -- there's nothing that makes this anything close to looking like a DDOS. Even after getting everything properly blessed, OVH drops all UDP traffic over 1 Gbps. It took them a month of back-and-forth troubleshooting to tell us this. The really terrible part is "TCP over TCP is bad" is now so prevalent there's basically no good VPN options for it if you need it. Wireguard won't do it directly, but there's hacks involving udp2raw. I tried it, and wasn't able to achieve more than 100 Mbps. OpenVPN can do it, but is single-threaded and won't reasonably do more than 1 Gbps without hardware acceleration, which didn't appear to work on EC2 instances. strongSwan cannot be configured to do unencapsulated ESP anymore -- they removed the option -- so it's UDP encapsulated only. Their reasoning is UDP is necessary for NAT traversal, and of course everybody needs that. It's also thread-per-SA so also not fast. The only solution I've found than can do something not UDP is Libreswan, which can still do unencapsulated ESP (IP Protocol 50) if you ask nicely. It's also thread-per-SA, but I've managed to wring 2 - 3 Gbps out of a single core after tinkering with the configuration. For the love of all that's good in the world, just add performant TCP support to Wireguard. I do not care about what happens in non-optimal conditions. /rant

throwing_away 2y ago
Did you go down the shadowsocks path at all?
- aarmenaa 2y ago
  I did not. I'm not terribly familiar with it, but it doesn't look like I can do general routing with it, right? My end goal is to route between two subnets.

some_furry 2y ago

Yes, but what about IPv6 over Amazon S3? https://xeiaso.net/blog/anything-message-queue/ https://xeiaso.net/blog/anything-message-queue/

bcrl 2y ago
Time to pull RFC 1149: A Standard for the Transmission of IP Datagrams on Avian Carriers. https://www.rfc-editor.org/rfc/rfc1149 https://www.rfc-editor.org/rfc/rfc1149

cma 2y ago

If you are in a situation where you have to anyway, you can use multiple TCP sockets and round robin them (with Nagle off) such that you are always sending just one packet over each. You'll get overhead and some unneeded acks, but no front of line blocking of the second layer of TCP mechanics going on.