5 ms·

Why TCP Over TCP Is A Bad Idea

100 points by kqr2 16y ago

ivank 16y ago
If someone really wanted to do TCP over TCP (say, because UDP packets are blocked), couldn't they build something just like TCP but without retransmits? Does this already exist in some pile of code somewhere?
- xtacy 16y ago
  TCP does congestion control + reliable delivery. Retransmits help reliable delivery. So, TCP without retransmits is basically a datagram protocol that does only congestion control. DCCP does precisely this and its congestion control is TCP friendly. It's available in Linux afaik. http://en.wikipedia.org/wiki/Datagram_Congestion_Control_Protocol http://read.cs.ucla.edu/dccp/
  - bostonvaulter2 16y ago
    See also: SCTP http://en.wikipedia.org/wiki/SCTP http://en.wikipedia.org/wiki/SCTP
- dfox 16y ago
  Yes you certainly can, see sshuttle (on the frontpage just now) for example. But you will not get transparent end-to-end connection (which hardly matters). Idea is to reconstruct something similar to kernel socket calls from actual TCP packets and send those calls to the other side (with retransmission handled only by outer reliable transport). It's similar idea as "User"/"NAT-only"/"Outgoing only" networking option offered by various virtualization systems and emulators. Probably original historical implementation of this hack is slirp, which is unix tool that works as NATing SLIP endpoint without needing root access (or anything OS-specific), that was often used in early '90s to get "full" IP connectivity from UNIX account on internet-connected modem-accessible system (usually without official approval to do so).
- ay 16y ago
  See my comment above about the TCP minions.
bigiain 16y ago
Unfortunately, I think this is fundamentally how 3G works, isn't it? Wich is why carriers around the world are all melting down as soon as their customers start using the 3G service they're paying for. And somehow the "network experts" at AT&T (or in my case, Vodafone) never learned from the exact same issues when the whole world was tunneling tcp over ppp/slip back in the late '80s...
- m0nastic 16y ago
  At least as relating to WCDMA/UMTS, I don't think it's technically TCP over TCP. TCP is encapsulated in PDCP for the wireless portion of the transmission, and the header's are compressed to hell (using RFC 2507). On the CDMA and CDMA2000 side of the house (I have no idea about LTE), it's using a similar encapsulation, but our good friend PPP makes an appearance. I'm pretty sure that the wireless data networks melting down is from a combination of too much density per tower (in cities) and insufficient upstream connectivity.
  - bigiain 16y ago
    OK, I'm probably misunderstanding the cellular details... I do however expect the issues alluded to here: http://www.computer.org/portal/web/csdl/doi/10.1109/FGCNS.2008.159 http://www.computer.org/portal/web/csdl/doi/10.1109/FGCNS.20... http://portal.acm.org/citation.cfm?id=1095972 http://portal.acm.org/citation.cfm?id=1095972 http://doc.utwente.nl/62891/ http://doc.utwente.nl/62891/ To have some strong bearing on me seeing crap performance like this: http://www.flickr.com/photos/bigiain/5222203911/ http://www.flickr.com/photos/bigiain/5222203911/
    - m0nastic 16y ago
      No, I think you're right, it seems like there's a discrepancy between the protocol specification and the implementation (I suppose it's not the first time that's happened). Some quick Googling turns up a lot of people who complain about latency issues with 3G (which bizarrely enough, I've never noticed. Must be the one situation where I'm not being screwed over by ATT), but I don't actually see a consensus around what's responsible for it. It looks like the SGSN is the likely culprit, as I see some proposals to bypass it completely (or install a Gateway when PDP is detected). Those ping times are rough, I'd be pissed too if I saw that.
mike463 16y ago
I thought this was going to be something like when I ssh into a remote system and run 'tcpdump' without filtering out port 22. :)
donw 16y ago
Learned this the hard way while setting up OpenVPN a few years ago. In the configuration file, there's an option to use TCP as the transport protocol, and being young and naïve, I figured that it was best to have a reliable protocol underpinning my VPN. What I didn't stop to consider, was that all of the traffic going over the VPN was also TCP, and as such was reliable enough on its own. Took me a week to figure out why connections would drop randomly, transfers would magically slow down and speed up, and all other forms of network-based mayhem.
- simias 16y ago
  Unfortunately it's not always feasible to route UDP properly everywhere. I'm currently using OpenVPN over TCP because the NAT here won't let me punch a hole for UDP, so I can assert that, while not ideal, it works well enough for web browsing assuming you have a good enough internet connection to back it up.
  - ay 16y ago
    TCP minions is there to solve precisely this problem: http://dedis.cs.yale.edu/2009/tng/papers/pfldnet10-slides.pdf http://dedis.cs.yale.edu/2009/tng/papers/pfldnet10-slides.pd... TCP over TCP works well enough as soon as your packet loss is small enough to be within the fast retransmit. As soon as you have to timeout, all bets are off.
    - simias 16y ago
      Thanks for this link, it's an interesting read. Where can I find out more about this TCP minion? The slides don't mention any external reference and Google only brings me back to this PDF.
      - ay 16y ago
        Catch up with Jana on twitter @janaiyengar to check if he has more details to share. I think having minions is a great idea to get the networking out of (TCP|UDP) ghetto.
neilc 16y ago
See also: http://en.wikipedia.org/wiki/End-to-end_principle http://en.wikipedia.org/wiki/End-to-end_principle
pyre 16y ago
Interestingly enough, in the opposite direction. ICQ chose to do their protocol over UDP, but basically re-implemented TCP into their protocol.
- muppetman 16y ago
  In the old days (I'm thinking using the Miranda client especially) you could disable this confirmation though. Made sending messages very fast, but you never knew _for sure_ that your recipient got them. Alas poor ICQ, it was so fantastic (it's how I met my wife)
- mukyu 16y ago
  That isn't surprising or at all uncommon.
dedward 16y ago
Similar to why TCP over any medium subject to packet loss for reasons other than contention is a bad idea... say random radio noise on wireless. You'll end up with TCP slowing down to a crawl trying to correct for what it was designed for - contention, when it's really just random loss - so the best move would be to just keep going.
- JoachimSchipper 16y ago
  Doesn't wifi include retransmission (at the media layer!) for this very reason? (Yes, it's ugly.)
  - dfox 16y ago
    Contrast that with wired ethernet that includes retransmission in media layer as a solution to congestion (the retransmission algorithm works only for congestion and other causes of packet loss on wired ethernet ar insignificant).
    - JoachimSchipper 16y ago
      Good point.
albertzeyer 16y ago
In the upper layer, wouldn't it be possible to just drop those retransmitted TCP packets again by the tunneling application?
jeza 16y ago
Scary thing is, I'm sure I read this article nearly ten years ago. I remember creating a few of these PPP over SSH VPNs as well, though mainly for testing so I can't remember if we actually encountered this issue. Fortunately CIPE came along before I really needed a VPN.

100 points by kqr2 16y ago

ivank 16y ago

If someone really wanted to do TCP over TCP (say, because UDP packets are blocked), couldn't they build something just like TCP but without retransmits? Does this already exist in some pile of code somewhere?

xtacy 16y ago
TCP does congestion control + reliable delivery. Retransmits help reliable delivery. So, TCP without retransmits is basically a datagram protocol that does only congestion control. DCCP does precisely this and its congestion control is TCP friendly. It's available in Linux afaik. http://en.wikipedia.org/wiki/Datagram_Congestion_Control_Protocol http://read.cs.ucla.edu/dccp/
- bostonvaulter2 16y ago
  See also: SCTP http://en.wikipedia.org/wiki/SCTP http://en.wikipedia.org/wiki/SCTP
dfox 16y ago
Yes you certainly can, see sshuttle (on the frontpage just now) for example. But you will not get transparent end-to-end connection (which hardly matters). Idea is to reconstruct something similar to kernel socket calls from actual TCP packets and send those calls to the other side (with retransmission handled only by outer reliable transport). It's similar idea as "User"/"NAT-only"/"Outgoing only" networking option offered by various virtualization systems and emulators. Probably original historical implementation of this hack is slirp, which is unix tool that works as NATing SLIP endpoint without needing root access (or anything OS-specific), that was often used in early '90s to get "full" IP connectivity from UNIX account on internet-connected modem-accessible system (usually without official approval to do so).
ay 16y ago
See my comment above about the TCP minions.

bigiain 16y ago

Unfortunately, I think this is fundamentally how 3G works, isn't it? Wich is why carriers around the world are all melting down as soon as their customers start using the 3G service they're paying for. And somehow the "network experts" at AT&T (or in my case, Vodafone) never learned from the exact same issues when the whole world was tunneling tcp over ppp/slip back in the late '80s...

m0nastic 16y ago
At least as relating to WCDMA/UMTS, I don't think it's technically TCP over TCP. TCP is encapsulated in PDCP for the wireless portion of the transmission, and the header's are compressed to hell (using RFC 2507). On the CDMA and CDMA2000 side of the house (I have no idea about LTE), it's using a similar encapsulation, but our good friend PPP makes an appearance. I'm pretty sure that the wireless data networks melting down is from a combination of too much density per tower (in cities) and insufficient upstream connectivity.
- bigiain 16y ago
  OK, I'm probably misunderstanding the cellular details... I do however expect the issues alluded to here: http://www.computer.org/portal/web/csdl/doi/10.1109/FGCNS.2008.159 http://www.computer.org/portal/web/csdl/doi/10.1109/FGCNS.20... http://portal.acm.org/citation.cfm?id=1095972 http://portal.acm.org/citation.cfm?id=1095972 http://doc.utwente.nl/62891/ http://doc.utwente.nl/62891/ To have some strong bearing on me seeing crap performance like this: http://www.flickr.com/photos/bigiain/5222203911/ http://www.flickr.com/photos/bigiain/5222203911/
  - m0nastic 16y ago
    No, I think you're right, it seems like there's a discrepancy between the protocol specification and the implementation (I suppose it's not the first time that's happened). Some quick Googling turns up a lot of people who complain about latency issues with 3G (which bizarrely enough, I've never noticed. Must be the one situation where I'm not being screwed over by ATT), but I don't actually see a consensus around what's responsible for it. It looks like the SGSN is the likely culprit, as I see some proposals to bypass it completely (or install a Gateway when PDP is detected). Those ping times are rough, I'd be pissed too if I saw that.

mike463 16y ago

I thought this was going to be something like when I ssh into a remote system and run 'tcpdump' without filtering out port 22. :)

donw 16y ago

Learned this the hard way while setting up OpenVPN a few years ago. In the configuration file, there's an option to use TCP as the transport protocol, and being young and naïve, I figured that it was best to have a reliable protocol underpinning my VPN. What I didn't stop to consider, was that all of the traffic going over the VPN was also TCP, and as such was reliable enough on its own. Took me a week to figure out why connections would drop randomly, transfers would magically slow down and speed up, and all other forms of network-based mayhem.

simias 16y ago
Unfortunately it's not always feasible to route UDP properly everywhere. I'm currently using OpenVPN over TCP because the NAT here won't let me punch a hole for UDP, so I can assert that, while not ideal, it works well enough for web browsing assuming you have a good enough internet connection to back it up.
- ay 16y ago
  TCP minions is there to solve precisely this problem: http://dedis.cs.yale.edu/2009/tng/papers/pfldnet10-slides.pdf http://dedis.cs.yale.edu/2009/tng/papers/pfldnet10-slides.pd... TCP over TCP works well enough as soon as your packet loss is small enough to be within the fast retransmit. As soon as you have to timeout, all bets are off.
  - simias 16y ago
    Thanks for this link, it's an interesting read. Where can I find out more about this TCP minion? The slides don't mention any external reference and Google only brings me back to this PDF.
    - ay 16y ago
      Catch up with Jana on twitter @janaiyengar to check if he has more details to share. I think having minions is a great idea to get the networking out of (TCP|UDP) ghetto.

neilc 16y ago

See also: http://en.wikipedia.org/wiki/End-to-end_principle http://en.wikipedia.org/wiki/End-to-end_principle

pyre 16y ago

Interestingly enough, in the opposite direction. ICQ chose to do their protocol over UDP, but basically re-implemented TCP into their protocol.

muppetman 16y ago
In the old days (I'm thinking using the Miranda client especially) you could disable this confirmation though. Made sending messages very fast, but you never knew _for sure_ that your recipient got them. Alas poor ICQ, it was so fantastic (it's how I met my wife)
mukyu 16y ago
That isn't surprising or at all uncommon.

dedward 16y ago

Similar to why TCP over any medium subject to packet loss for reasons other than contention is a bad idea... say random radio noise on wireless. You'll end up with TCP slowing down to a crawl trying to correct for what it was designed for - contention, when it's really just random loss - so the best move would be to just keep going.

JoachimSchipper 16y ago
Doesn't wifi include retransmission (at the media layer!) for this very reason? (Yes, it's ugly.)
- dfox 16y ago
  Contrast that with wired ethernet that includes retransmission in media layer as a solution to congestion (the retransmission algorithm works only for congestion and other causes of packet loss on wired ethernet ar insignificant).
  - JoachimSchipper 16y ago
    Good point.

albertzeyer 16y ago

In the upper layer, wouldn't it be possible to just drop those retransmitted TCP packets again by the tunneling application?

jeza 16y ago

Scary thing is, I'm sure I read this article nearly ten years ago. I remember creating a few of these PPP over SSH VPNs as well, though mainly for testing so I can't remember if we actually encountered this issue. Fortunately CIPE came along before I really needed a VPN.