9 ms·

Sequenced Packets Over Ordinary TCP

43 points by colinprince 8y ago

convolvatron 8y ago
sctp (https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...) has both reliable delivery and transport level message framing. multistream, multipath, and linux and bsd support. not much uptake though. maybe if the quic people had built on it instead of udp
- kev009 8y ago
  It's used in SIGTRAN/SS7 so just about any phone call, cell session etc is sctp. As an aside rrs, the primary author of spec, book, and reference implementation, still occasionally works on the FreeBSD sctp and has recently rewritten the FreeBSD TCP stack over the past few years. It's called RACK, and there's another extension in the pipe called BBR with incorporates ideas from Google's work. A very amazing person! * https://penturalabs.wordpress.com/2013/11/24/what-is-sigtran-ss7-sctp/ https://penturalabs.wordpress.com/2013/11/24/what-is-sigtran... * https://github.com/freebsd/freebsd/blob/master/sys/netinet/tcp_stacks/rack.c https://github.com/freebsd/freebsd/blob/master/sys/netinet/t...
- lallysingh 8y ago
  It's used in WebRTC data channels, atop of UDP and DTLS. It's not used above raw IP because few intermediate routers support it.
- pilif 8y ago
  > maybe if the quic people had built on it instead of udp Then QUIC would be as dead in the water as sctp is: between all the middleboxes currently running the internet it’s impossible to pass any low level protocol that wasn’t in use in the late 90ies when firewalls started to happen. As a matter of fact, quic is an attempt to get sctp like semantics going over a protocol that has a chance of being transmittable over the internet. In order for sctp to gain uptake, we need to collectively update all routers, firewalls, OSes and applications. In order for quic to gain uptake, we need to update applications.
  - kev009 8y ago
    QUIC is kind of like a relaxed TCP over UDP. You can do SCTP over UDP (aka WebRTC), and TCP over UDP to gain those application-managed benefits with varying upper level trade offs. SCTP has pretty wide support, but it's kind of like an IPv6-lite in terms of re-education and that is the barrier to move forward if to supplant or scalably supplement TCP for the web.
- peterwwillis 8y ago
  > maybe if the quic people had built on it instead of udp I've noticed that all modern protocols are designed by lower-tier engineers in teams in departments in corporations that want new functionality in some product they make, but can't wait to propose a modification to an internet or web standard that will take 5 years to get adopted by all the vendors and service providers they want to use it, and they refuse to make a native app to do it. This gives you protocols like WebSockets, WebRTC, QUIC. On the application side you get things like WebAssembly. So a few companies are slowly cobbling together their distributed operating system for web applications called a "web browser". I think the unspoken plan is to subvert all native computing platforms by evolving web browsers into a reinvented Java-esque application platform. But ironically I don't think this was intentional. I think everyone's just lazy and reinventing every wheel to fit their niche, which just happens to be web apps. In 10+ years there may be a new platform and paradigm, and all this stuff will be obsolete, even though it'll do basically the same things.
  - dagenix 8y ago
    I don't see how any of this is true. The problem with SCTP is that middleboxes don't understand it and tend to drop it. This create an unfortunate situation where middleboxes refuse to support it because no-one uses it and no-one uses it because middleboxes refuse to support it. You can't just create a standard that says everyone should support SCTP and make everyone do it - standards already say that middleboxes should support it and they don't. Its bizarre that you picked WebSockets, WebRTC, and QUIC as examples of how no one supports the standardization process. WebSockets were standardized via RFC 6455 in 2011, WebRTC via W3C, work with starting in ~2011, and QUIC implemented in 2012, with IETF standardization currently underway. These are all 6+ year investments by their sponsors.
    - wtallis 8y ago
      The sane development path would have been to build QUIC as a thin layer on top of SCTP, with a fallback to UDP implemented only if it proved necessary. I don't think the middlebox problem is a showstopper for a limited-audience experimental protocol like QUIC, and a successful QUIC-over-SCTP experiment could have gotten rid of the middlebox worries (by either motivating people to get rid of the broken middleboxes, or by proving they weren't common enough to care about), in much the same way that Apple's limited deployment of ECN has eliminated the worries about middleboxes that kill ECN flows. As for WebSockets, WebRTC, and WebAssembly, the fact that they've been standardized within their insular niche says nothing about the extent to which they duplicate functionality that has long existed outside the browser as part of the host OS or as libraries that use the low-level networking constructs that any native application has access to (but JavaScript didn't).
- MertsA 8y ago
  I really wish QUIC, WebRTC, etc would have standardized on SCTP over UDP for IPv4 and raw SCTP for IPv6. We don't need to reinvent the wheel, and we had (still have?) a bonafide opportunity to get out in front of the situation with IPv6. If we got actual adoption of SCTP before ye olde middlebox vendors started mucking things up on IPv6 like on IPv4 we wouldn't have to tunnel a protocol that was standardized over 18 years ago just to be able to use it on the internet. Things like multihoming make even more sense with IPv6 and the explosion of mobile devices. It just kills me inside that we already solved this problem in a way that's just about transparent to most all applications but it looks like we're never going to get widespread support because some middleboxes and NAT implementations couldn't be bothered to support it.
est 8y ago
How am I supposed to work with it? s=socket.socket(socket.AF_INET, socket.SOCK_SEQPACKET) [Errno 43] Protocol not supported
- lallysingh 8y ago
  You don't. Only the popular parts of TCP/IP are reliably implemented and relatively bug free. Use those, ignore the stuff in this article.
- saurik 8y ago
  This is a proposal, not documentation.
gwbas1c 8y ago
Just send either a 16-bit or 32-bit packet length before sending a packet. Much easier than abusing the protocol, and the reader knows how large of a buffer to allocate.
tonyg 8y ago
The urgent pointer mechanism is not reliable or well-implemented enough for widespread use. Excerpts from RFC 6093 (from 2011, https://tools.ietf.org/html/rfc6093 https://tools.ietf.org/html/rfc6093), "On the Implementation of the TCP Urgent Mechanism", which examines current implementation practices in detail, and ultimately "recommends against the use of urgent indications": Unfortunately, virtually all TCP implementations process TCP urgent indications differently. (sec. 3.1) All the popular implementations [...] interpret the semantics of the TCP Urgent Pointer as specified in Section 3.1 of RFC 793. This means that even when RFC 1122 formally updated RFC 793 to clarify the ambiguity [...] this clarification was never reflected in actual implementations (sec 3.2) Because implementations just never bothered to implement RFCs 1011 and 1122's update, later in the document they retract the advice given by RFCs 1011 and 1122, overriding the earlier overriding of RFC 793, effectively returning to RFC 793's definition for the urgent pointer! (!) There's more: [...] an application might be unable to interoperate with itself if both the TCP sender and the TCP receiver are running on the same host. (sec 3.2) [S]ome middleboxes clear the urgent indications by clearing the URG flag and setting the Urgent Pointer to zero. (sec 3.4) Ultimately: don't do it. You will not have a fun time. As a result of the issues discussed in Section 3.2 and Section 3.4, new applications SHOULD NOT employ the TCP urgent mechanism. (sec 5) Additionally, applications that still decide to use the urgent mechanism need to be designed for correct operation even when the URG flag is cleared by middleboxes. (sec 6)
- tonyg 8y ago
  It's a bit sad that, as a result of the urgent pointer being useless, there are 16 zero bits sent in every TCP header. They can't even be repurposed - the specs say they're meaningful only when the URG bit is set, and leave their condition when URG is clear unspecified, but various middleboxes take a clear URG bit as licence to explicitly zero the urgent pointer on the way through. (And of course some clear it no matter what.) This is one of those areas where IPv6 could have been something quite different and amazing. Separating TCP and IP was, in hindsight, a mistake; some alternate universe out there had the IPv6 process focus on finding a better way of carving up the problem.
- AndyMcConachie 8y ago
jrockway 8y ago
I feel like when you start using protocol features you heard about from an RFC, you just get disappointed, because they never work as well as the author of the RFC envisioned. Certainly if you have two nodes both using software that you control, everything is going to be great. But that never happens. Someday, someone is going to want a load balancer in the middle. Does your favorite load balancer read the urgent flag in the TCP packet? Probably not. So now you are shaving that yak, or rewriting your app when you don't actually control the code (think cloud providers), wasting time and money because you read something you thought was "neat". If you want records in your TCP connection, define the first n octets of your stream to be a record length. Read the length, then read that many octets off the stream. There's your record. The next n octets of the stream are the length of the next record... and so on. This will work with every TCP stack everywhere, forever. Done. As for RFCs, I've recently been working on an aggregator for syslog messages. I read the RFC, and the protocol sounds quite nice, well-defined positions in the stream for all sorts of relevant data like the time, hostname, and even arbitrary structured data. Back in the real world, though, nobody read the RFC and a syslog message looks like "'<' <some numbers that are supposed to be the priority, but nobody can decide what the priority means so it's just some digits that you throw away> '>' <arbitrary bytes that have no standard meaning, sometimes there's something that looks like a timestamp or a hostname, but really trusting a device to know its own name is way too much to ask, and timestamps... don't get me started>". Sometimes the message ends with a \n, but I wouldn't count on it. TL;DR: it's nice that you read the RFC. Too bad nobody else did.

43 points by colinprince 8y ago

sctp (https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...) has both reliable delivery and transport level message framing. multistream, multipath, and linux and bsd support. not much uptake though. maybe if the quic people had built on it instead of udp

kev009 8y ago
It's used in SIGTRAN/SS7 so just about any phone call, cell session etc is sctp. As an aside rrs, the primary author of spec, book, and reference implementation, still occasionally works on the FreeBSD sctp and has recently rewritten the FreeBSD TCP stack over the past few years. It's called RACK, and there's another extension in the pipe called BBR with incorporates ideas from Google's work. A very amazing person! * https://penturalabs.wordpress.com/2013/11/24/what-is-sigtran-ss7-sctp/ https://penturalabs.wordpress.com/2013/11/24/what-is-sigtran... * https://github.com/freebsd/freebsd/blob/master/sys/netinet/tcp_stacks/rack.c https://github.com/freebsd/freebsd/blob/master/sys/netinet/t...
lallysingh 8y ago
It's used in WebRTC data channels, atop of UDP and DTLS. It's not used above raw IP because few intermediate routers support it.
pilif 8y ago
> maybe if the quic people had built on it instead of udp Then QUIC would be as dead in the water as sctp is: between all the middleboxes currently running the internet it’s impossible to pass any low level protocol that wasn’t in use in the late 90ies when firewalls started to happen. As a matter of fact, quic is an attempt to get sctp like semantics going over a protocol that has a chance of being transmittable over the internet. In order for sctp to gain uptake, we need to collectively update all routers, firewalls, OSes and applications. In order for quic to gain uptake, we need to update applications.
- kev009 8y ago
  QUIC is kind of like a relaxed TCP over UDP. You can do SCTP over UDP (aka WebRTC), and TCP over UDP to gain those application-managed benefits with varying upper level trade offs. SCTP has pretty wide support, but it's kind of like an IPv6-lite in terms of re-education and that is the barrier to move forward if to supplant or scalably supplement TCP for the web.
peterwwillis 8y ago
> maybe if the quic people had built on it instead of udp I've noticed that all modern protocols are designed by lower-tier engineers in teams in departments in corporations that want new functionality in some product they make, but can't wait to propose a modification to an internet or web standard that will take 5 years to get adopted by all the vendors and service providers they want to use it, and they refuse to make a native app to do it. This gives you protocols like WebSockets, WebRTC, QUIC. On the application side you get things like WebAssembly. So a few companies are slowly cobbling together their distributed operating system for web applications called a "web browser". I think the unspoken plan is to subvert all native computing platforms by evolving web browsers into a reinvented Java-esque application platform. But ironically I don't think this was intentional. I think everyone's just lazy and reinventing every wheel to fit their niche, which just happens to be web apps. In 10+ years there may be a new platform and paradigm, and all this stuff will be obsolete, even though it'll do basically the same things.
- dagenix 8y ago
  I don't see how any of this is true. The problem with SCTP is that middleboxes don't understand it and tend to drop it. This create an unfortunate situation where middleboxes refuse to support it because no-one uses it and no-one uses it because middleboxes refuse to support it. You can't just create a standard that says everyone should support SCTP and make everyone do it - standards already say that middleboxes should support it and they don't. Its bizarre that you picked WebSockets, WebRTC, and QUIC as examples of how no one supports the standardization process. WebSockets were standardized via RFC 6455 in 2011, WebRTC via W3C, work with starting in ~2011, and QUIC implemented in 2012, with IETF standardization currently underway. These are all 6+ year investments by their sponsors.
  - wtallis 8y ago
    The sane development path would have been to build QUIC as a thin layer on top of SCTP, with a fallback to UDP implemented only if it proved necessary. I don't think the middlebox problem is a showstopper for a limited-audience experimental protocol like QUIC, and a successful QUIC-over-SCTP experiment could have gotten rid of the middlebox worries (by either motivating people to get rid of the broken middleboxes, or by proving they weren't common enough to care about), in much the same way that Apple's limited deployment of ECN has eliminated the worries about middleboxes that kill ECN flows. As for WebSockets, WebRTC, and WebAssembly, the fact that they've been standardized within their insular niche says nothing about the extent to which they duplicate functionality that has long existed outside the browser as part of the host OS or as libraries that use the low-level networking constructs that any native application has access to (but JavaScript didn't).
MertsA 8y ago
I really wish QUIC, WebRTC, etc would have standardized on SCTP over UDP for IPv4 and raw SCTP for IPv6. We don't need to reinvent the wheel, and we had (still have?) a bonafide opportunity to get out in front of the situation with IPv6. If we got actual adoption of SCTP before ye olde middlebox vendors started mucking things up on IPv6 like on IPv4 we wouldn't have to tunnel a protocol that was standardized over 18 years ago just to be able to use it on the internet. Things like multihoming make even more sense with IPv6 and the explosion of mobile devices. It just kills me inside that we already solved this problem in a way that's just about transparent to most all applications but it looks like we're never going to get widespread support because some middleboxes and NAT implementations couldn't be bothered to support it.

est 8y ago

How am I supposed to work with it? s=socket.socket(socket.AF_INET, socket.SOCK_SEQPACKET) [Errno 43] Protocol not supported

lallysingh 8y ago
You don't. Only the popular parts of TCP/IP are reliably implemented and relatively bug free. Use those, ignore the stuff in this article.
saurik 8y ago
This is a proposal, not documentation.

gwbas1c 8y ago

Just send either a 16-bit or 32-bit packet length before sending a packet. Much easier than abusing the protocol, and the reader knows how large of a buffer to allocate.

tonyg 8y ago

The urgent pointer mechanism is not reliable or well-implemented enough for widespread use. Excerpts from RFC 6093 (from 2011, https://tools.ietf.org/html/rfc6093 https://tools.ietf.org/html/rfc6093), "On the Implementation of the TCP Urgent Mechanism", which examines current implementation practices in detail, and ultimately "recommends against the use of urgent indications": Unfortunately, virtually all TCP implementations process TCP urgent indications differently. (sec. 3.1) All the popular implementations [...] interpret the semantics of the TCP Urgent Pointer as specified in Section 3.1 of RFC 793. This means that even when RFC 1122 formally updated RFC 793 to clarify the ambiguity [...] this clarification was never reflected in actual implementations (sec 3.2) Because implementations just never bothered to implement RFCs 1011 and 1122's update, later in the document they retract the advice given by RFCs 1011 and 1122, overriding the earlier overriding of RFC 793, effectively returning to RFC 793's definition for the urgent pointer! (!) There's more: [...] an application might be unable to interoperate with itself if both the TCP sender and the TCP receiver are running on the same host. (sec 3.2) [S]ome middleboxes clear the urgent indications by clearing the URG flag and setting the Urgent Pointer to zero. (sec 3.4) Ultimately: don't do it. You will not have a fun time. As a result of the issues discussed in Section 3.2 and Section 3.4, new applications SHOULD NOT employ the TCP urgent mechanism. (sec 5) Additionally, applications that still decide to use the urgent mechanism need to be designed for correct operation even when the URG flag is cleared by middleboxes. (sec 6)

tonyg 8y ago
It's a bit sad that, as a result of the urgent pointer being useless, there are 16 zero bits sent in every TCP header. They can't even be repurposed - the specs say they're meaningful only when the URG bit is set, and leave their condition when URG is clear unspecified, but various middleboxes take a clear URG bit as licence to explicitly zero the urgent pointer on the way through. (And of course some clear it no matter what.) This is one of those areas where IPv6 could have been something quite different and amazing. Separating TCP and IP was, in hindsight, a mistake; some alternate universe out there had the IPv6 process focus on finding a better way of carving up the problem.
AndyMcConachie 8y ago

jrockway 8y ago

I feel like when you start using protocol features you heard about from an RFC, you just get disappointed, because they never work as well as the author of the RFC envisioned. Certainly if you have two nodes both using software that you control, everything is going to be great. But that never happens. Someday, someone is going to want a load balancer in the middle. Does your favorite load balancer read the urgent flag in the TCP packet? Probably not. So now you are shaving that yak, or rewriting your app when you don't actually control the code (think cloud providers), wasting time and money because you read something you thought was "neat". If you want records in your TCP connection, define the first n octets of your stream to be a record length. Read the length, then read that many octets off the stream. There's your record. The next n octets of the stream are the length of the next record... and so on. This will work with every TCP stack everywhere, forever. Done. As for RFCs, I've recently been working on an aggregator for syslog messages. I read the RFC, and the protocol sounds quite nice, well-defined positions in the stream for all sorts of relevant data like the time, hostname, and even arbitrary structured data. Back in the real world, though, nobody read the RFC and a syslog message looks like "'<' <some numbers that are supposed to be the priority, but nobody can decide what the priority means so it's just some digits that you throw away> '>' <arbitrary bytes that have no standard meaning, sometimes there's something that looks like a timestamp or a hostname, but really trusting a device to know its own name is way too much to ask, and timestamps... don't get me started>". Sometimes the message ends with a \n, but I wouldn't count on it. TL;DR: it's nice that you read the RFC. Too bad nobody else did.