8 ms·
Upstreaming multipath TCP
- Animats 7y agoWe had that in the early days of TCP. Then Berkeley broke it. Originally, an IP address identified a destination host, not an interface card. Didn't matter over what device a packet arrived. The BSD people, though, tied IP addresses to the network interface, so it mattered over which device a packet arrived. This simplified outbound routing in BSD, but broke multipath TCP.
- jerkstate 7y agothat's neat, but isn't really the same thing. a mptcp socket has multiple subcarrier sockets, potentially routed over different paths, which each have their own loss statistics, so mptcp will dynamically balance over the links that work the best.
- Animats 7y agoRight. Early thinking was to do that in the routing algorithm, before the Internet became so hub and spoke.
- keithwinstein 7y agoHmm, I don't quite follow (but I know you were there!). I know there was disagreement about the right way to do multihomed hosts on the Internet; RFC 1122 gets at this with its discussion of the "strong ES model" vs. "weak ES model," and RFC 6418/6419 has a survey of current-ish behavior. By now, most implementations seem to use the "weak ES model" by default or have it configurable. This allows all interfaces to have the same IP address if you want. But that doesn't solve the problems that MPTCP solves, i.e.: (a) break-before-make failover of a TCP connection across different network paths between two hosts, and (b) combining the resources of multiple end-to-end network paths in one connection. Because even if a host uses the same IP address for different interfaces (and therefore a TCP connection can survive failure of one interface), it's not like that IP address is going to be individually globally routable. There's no way for some router in the middle of the Internet to know that you just walked out of range of the coffeeshop Wi-Fi and are now only reachable via a commercial LTE ISP, and would like to have incoming datagrams start arriving via the LTE interface (and ISP) instead. They won't let everybody's laptop be its own one-host AS and do a BGP announcement every time it loses a Wi-Fi interface, and even if they did, it would take too long to propagate to be useful. MPTCP (and the mobility in QUIC and Mosh) solves the problem by keeping the network ignorant of the roaming and letting the connection failover to a different network path by having the end hosts address each other at different IP addresses. Similar story, I think, for aggregating network paths between two individually multihomed hosts.
- namibj 7y agoSo QUIC provides Mosh-like semi-arbitrary data transfers? This would be great for stuff like an IRC bouncer, because while Mosh exhibits excellent resilience, it prevents efficient rich-client functionality like instant auto-complete (Mosh triggers a prediction delay in that case) and window-switching.
- yrro 7y agoThis is how Linux still works by default, right? Behaviour that is quite surprising, at least it was to me when I discovered that it, in conjunction with many tools (e.g., docker) conveniently enabling up forwarding on all interfaces, unwittingly transforms multi-homed hosts into routers...
- bboreham 7y agoI believe Docker stopped doing that a couple of years back. Or it sets the default on the forward chain to drop.
- yrro 7y agoYou're right, something is setting the FORWARD chain policy to DROP and I guess it's docker.
- bogomipz 7y agoI don't think this is correct, it should be disabled by default(net.ipv4.ip_forward=0). At least it is in the major distros I'm familiar with. What distro are you using?
- yrro 7y agoDistros disable it by default but it gets silently enabled when you install things like docker (and maybe livbirt) and so on. At least Docker (as it was pointed out by a sibling reply to my comment) also sets the FORWARD chain policy to DROP.
- rocqua 7y agoThere are good reasons for layer 3 addresses to be network interface specific. In fact, that is how layer 3 is supposed to work. After all, each interface is potentially connected to acdifferent network. Hence, routing to those interfaces is different, and layer 3 is about routing. The mistake instead is that TCP is not fully layer 4. They are entangled with layer 3. Specifically, a TCP socket is defined by an (IP, port) pair. Where the IP is layer 3. As such, it is fundamentally impossible to persist a TCP connection accross multiple IPs. There is no reason for this. If instead a socket were identified as (uuid, port) then after I change my ip address, I can continue receiving packets sent to (uuid, port). And the other side will still recognize packets from me, because only my IP changed, not the connection uuid. You'd maybe need to add some spoofing defenses, but we need those in current TCP too.
- rocqua 7y agoP.s. this rant/idea was taken from this excellent blog post: https://apenwarr.ca/log/20170810 https://apenwarr.ca/log/20170810
- throw0101a 7y agoSCTP, another Layer 4 protocol, was designed with multi-homing in mind: * https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol#Multi_homing https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr... I think a lot of the effort put into QUIC, HTTP/2,3,4,whatever, etc, could have been avoided if SCTP was more broadly adopted.
- imglorp 7y agoIt is a shame. Do note that SCTP is heavily used in telecom. Every network element in that world needs to be redundant, along with the multiple routes to get to each of those redundant elements. SCTP helps hide some of that from the application layers. https://en.wikipedia.org/wiki/SIGTRAN https://en.wikipedia.org/wiki/SIGTRAN
- throw0101a 7y agoYes, I think that SS7 architecture was used as some design criteria when it came SCTP. Just about everything in the telco world has (at least) an A- and a B-side for HA.
- despair3435 7y agoFully agree. This is one of the biggest mistakes that was made in the early days of TCP/IP, with far reaching consequences (such as breaking multihoming, but also for instance increasing routing table sizes). In Ouroboros (our recursive network implementation: https://ouroboros.rocks/ https://ouroboros.rocks/), we only identify the interface once (MAC address). Addresses are contained within each recursive network layer, and identify a specific process at each layer (similar to how an IP address should actually identify the host).
- isostatic 7y agoI can add an IP as a loopback just fine - my routers have that all the time. The problem is then how does the network know where my IP address is. When a packet aims at my loopback, how does it work its way to me? The answer is the same as interface addresses - we use routing protocols for that, mainly today that's OSPF (in local networks) and BGP (globally). Convergence takes a relatively long time for BGP, and BGP (at least at a global scale) is limited to /24s. ISPs won't accept your /32 advert. Even if they did, traffic still only goes in one direction from a given host. On top of that, theoretically if every BGP peer rand BFD you might see failovers in a few seconds, but that's not the real world.
- smz2 7y agoAnd we still need to assign at least one /32 to every mobile device.
- isostatic 7y agoIPV6 and give a /64 to each device. That's fine, however you're still with a routing table of upto 2^64 entries. Currently BGP routing table is upto 2^24 entries, but it's really less than 1 million entries. BGP is not the answer to multi path devices for many many reasons. Tackling it at higher levels (OSI 4-8) is the solution.
- throw0101a 7y ago> The problem is then how does the network know where my IP address is. When a packet aims at my loopback, how does it work its way to me? Mobile IP: > The basic idea behind the separation is that the Internet architecture combines two functions, routing locators (where a client is attached to the network) and identifiers (who the client is) in one number space: the IP address. LISP supports the separation of the IPv4 and IPv6 address space following a network-based map-and-encapsulate scheme (RFC 1955). In LISP, both identifiers and locators can be IP addresses or arbitrary elements like a set of GPS coordinates or a MAC address.[2] * https://en.wikipedia.org/wiki/Locator/Identifier_Separation_Protocol https://en.wikipedia.org/wiki/Locator/Identifier_Separation_...
- ay 7y agoNo we didn’t in practice - because you also need the return routability. Modern day example: If you are connected over WiFi and 3G over two different ISPs, the packet for your (probably RFC1918, statefully NATted by your home gateway) address of the WiFi interface has absolutely zero chance of arriving over 3G (which probably has a different RFC1918 address, statefully NATted in the CGN in the mobile infra). And vice versa. So strong host model vs weak host model is irrelevant in this context. MPTCP works absolutely fine in this scenario. MIPv6 reportedly works to roam (assuming no NAT66 in the path), but can not use two paths at once.
- cat199 7y agosource? or further details? e.g. either historical discussion or example of other IPv4 system which implemented things as you describe Pre-existence of IMP's doesn't apply I don't think since we are talking IPv4
- despair3435 7y agoCyclades [0] for instance also named the host, as a historical example. [0] https://en.wikipedia.org/wiki/CYCLADES https://en.wikipedia.org/wiki/CYCLADES
- dependenttypes 7y agoBased on what I have read so far Berkeley is responsible for everything that is bad with the world, such as signals for example.
- bodyfour 7y agosignals might have gotten more complicated in BSD, but the concept certainly existed earlier: https://github.com/dspinellis/unix-history-repo/blob/Research-V4-Snapshot-Development/man/man2/signal.2 https://github.com/dspinellis/unix-history-repo/blob/Researc...
- bogomipz 7y ago>"Originally, an IP address identified a destination host, not an interface card. Didn't matter over what device a packet arrived." How would an upstream L2/L3 device be aware and be able to handle this. We have things like LACP now but that requires more physical ports and chipsets that understand the protocol. In the early days of TCP a router was just another UNIX box. But even something like LACP requires multiple links to the same upstream device how would this have worked if the box was connected to say two different upstream switches or routers? Today we have things like Juniper's MLAG and Cisco vCP but those are proprietary and very expensive solutions.
- dooglius 7y agoA shame this has to be done within the kernel. There was never much reason to implement TCP handling in ring zero.
- etaioinshrdlu 7y agoYes, I have not heard any good arguments for why TCP should be in a kernel, other than, it's convenient, and it's always been done that way, and your apps get to share it. Like using the kernel as a shared library... You could put BitTorrent in the kernel. It makes about as much sense architecturally, but isn't too widely used.
- phicoh 7y agoPOSIX supports sharing filedescriptors between processes. For example, you can have a process that reads a few bytes from a TCP socket and then passes the socket to another process. Unix tries to model all I/O, including networking, as operations on files. Realistically it is only possible to get this right if the kernel is involved. Of course it is quite possible to come up with different models. But Unix seems to be uniquely powerful in its ability to create complex systems from lots of small processes. The kernel being involved does not imply that all code has to be in the kernel. The FUSE file system interface is a well known way to run filesystem code as user processes. Likewise, there are ways to run device drivers in user space. The disavantage is that the extra context switches cause performance loss. So this approach is use for protocols that are rarely used and do not warrent full kernel implementation.
- cat199 7y agotaken to the logical extreme, why do anything in the kernel for that matter?
- etaioinshrdlu 7y agoI think a good rule of thumb should be that a kernel should be responsible making hardware devices safe to use among multiple processes, and little else.
- beagle3 7y agoDJB's CurveCP / MinimalT framework is probably a much better solution to the problems MPTCP is trying to solve - and also adds security, increases address space as well. But .. MPTCP it is; at least something is gaining wide use.
- est 7y ago> Baerts said, so there will be no user-space access to subflows for now. So one has to tcpdump to observe the sub flows? > But, naturally, there are users who want their unmodified binary programs to start using MPTCP once it's available. There is a working, if inelegant, solution to this problem. A new control-group hook allows the installation of a BPF program that runs when a program calls socket(); it can change the requested protocol to IPPROTO_MPTCP and the calling application will be none the wiser. Uhhhhh!
- ksec 7y agoDoes anyone know the state of MPTCP on iOS? I heard they were testing it long ago on iOS. And then we haven't heard anything about it.
- flounder3 7y agoWe implemented it for Siri in iOS 7. Exposed APIs for third party developers in iOS 11.
- danceparty 7y agoAPIs are available for developers, and MPTCP is also now used for Apple Music