5 ms·

Just one more reason to redesign the tcp/ip stack. Can you imagine what our ABIs will look like if we're still hacking in kludges for another 40+ years? Opening

Just one more reason to redesign the tcp/ip stack. Can you imagine what our ABIs will look like if we're still hacking in kludges for another 40+ years? Opening a connection "the right way" will look like casting a spell with voodoo. Ooh, maybe we'll get cooler titles then, like Master of the Dark Network Arts.

dragontamer 5y ago
Is this really a tcp/ip stack issue, or is this a BSD-sockets issue? Since Cloudflare showed off how it is fixed with another API (aka: connectx) and/or ordering of bind/connect/etc. etc., it sounds more like a BSD-socket issue to me. ------ In fact, the closest thing to a "protocol error" to me is in the UDP section as far as I'm concerned?
- 0xbadcafebee 5y ago
  The source of the problem is the source port, which is a random number that only signifies what application running on a source host is requesting one particular connection. And it has to be a random number, because if it isn't it's a security issue. And re-use of ports is only necessary because we're using this finite range of numbers to signify a unique connection. The deeper down the rabbit hole you go, the more you realize that little of modern connections are by design, but rather are built on a series of kludges used to fix problems and use-cases that the original design didn't cover. A modern network stack could bring a lot of benefits. Here's a few things I would like in a new stack: - Service discovery. The reason nothing on the internet can use any port other than "443" is that we failed to build in the ability for TCP to relate to services rather than applications. TCP/443 refers to the application-layer of "HTTP", but also the transport-and-application layer of "TLS". And one IP can only ever host a single application on that host and port. If you want to run multiple applications, you have to run one application which talks the application protocol, deciphers what host you want from TLS, deciphers what host you want again from HTTP, and then opens up another connection to get you to the actual service you wanted. This is convoluted and unreliable - and decrypts the payload far from the actual application that should be looking at this data! Instead, the protocol's destination should be a combination of identifiers, and the stack should route the connection to the appropriate application. For example, instead of "host node01/port 443", you could open a connection to "host:node01/protocol:tls/protocol:http/service:public-web-pool-hackernews" (not that verbose though). All the application needs to do is open a listening socket with the same connection string, and the operating system will look up the application matching that connection string, unbundle any protocol layers it can, and connect a new file descriptor to the waiting application. You could even have multiple applications open listening sockets on the same connection string and the OS could round-robin connections to them automatically. And routers could pass connections to services distributed on pools of hosts without the need for an "application-specific load-balancing reverse-proxy". We can run all kinds of layers of protocols and services, and there is no limitation of tying up a single integer "443" with all these different concepts. This also allows us to expand the number of real internet protocols and not just keep stacking crap on top of HTTPS because we can't route any other traffic because everyone's hard-coded themselves into this one port number. - TLS becomes a part of the ABI. Any network application that wants to use TLS could open_tls_connection() and get a file descriptor. Just like applications shouldn't have to implement their own TCP/UDP packet parsers, they shouldn't have to bundle their own TLS implementations. This simplifies application code and increases speed by letting the kernel unbundle the payload from the encrypted session. - With the above two changes, add authentication and authorization layers to the stack. Now any network service can have encryption, privacy, authentication, and authorization, without needing every application to explicitly build in support for a particular method. This would allow you to write some arbitrary web app that requires login, and add an implicit trust of authorizations from host "github.com" with realm "public". If you get a connection request that's signed properly, you know the user is an authorized github.com user (and a payload with their metadata). Imagine authenticating every single network call by default. This would make it dead-simple to secure all network endpoints that often get left exposed on the internet with no authentication. - Distributed Tracing tags should be added to every layer and hop of a connection, the way an IP TTL is passed. This way you could know exactly where a connection is failing anywhere along its path. You could also pass tags forward along new connections to explicitly carry forward a trace, such as a database connection carrying a tag from a previous HTTP connection. No need for "DT infrastructure", every OS's stack would just come with tracing bundled and you could ls /proc/net/* to find out what was related to what.
  - dan-robertson 5y ago
    It’s true that TCP/IP don’t really follow the OSI 7 layer model but I think the solution is to get rid of the old model that is not fit for purpose rather than to rediscover the failures that led TCP/IP to succeed instead of the standards that were designed in tandem with the model. (Apart from the layers, the main things to come from that effort to today were x.509 and ldap)
  - zamadatix 5y ago
    Service discovery is already a function of DNS be it bare bones for decades like a SRV record saying which port the HTTP server is on to DNS-SD which is more catalog style infrastructure. Things like LDAP clients actually obey service discovery already. Things like HTTPS servers/browsers have resisted it because of the current security infrastructure (name only based) issues it would open not the network infrastructure. It's not like the security infrastructure side even needs to be redone from the ground up it's just the juice from changing the existing infrastructure hasn't been deemed worth the squeeze by anyone yet. TLS being deeply embedded has led to more problems than it avoids. Things like QUIC or HTTP/3 are the result of even just TCP sockets being OS managed. Getting anything new to happen when the old is 12 OS implementations updated independently of the program is never going to be popular with programs. That said Linux, FreeBSD, and Windows all support kTLS APIs already, in the case of the latter for about 2 decades. It can be useful in some situations but it's not a silver bullet. Those layers already exist, actually an option at each layer: IPsec, DTLS, TLS, HTTPS. The lack of bottom up integration and universal adoption isn't from lack of having another option, fighting that it's easy to control the auth the way you want in the HTTPS layer is probably the main restriction. Tracing tags are considered too problematic for privacy in modern protocols. E.g. QUIC offers an optional spin bit. Putting tracing tags about the full path of every packet certainly wouldn't fly, as convenient as it would make things. For debug packets only that's what ICMP already is. . There are a lot of cool things that could be done with a redesigned stack, mostly around preventing ossification of the lower layers, but it's not like it's ever been golden from day 1 or will ever be golden "if we just get it right next time"™. If the journey thus far has taught us anything it's that the internet protocol stack should design itself to avoid being everything included and instead try to be as transparent a transport it can so end applications can choose their own way of doing things without the assumptions of the transport getting in the way.
WJW 5y ago
The only reason the current way is not "voodoo" to you is because you've seen it a thousand times before. The current way is already 40 years of hacking in kludges.
turminal 5y ago
Yeah, everyone should move to Plan 9 already!
gpderetta 5y ago
> Can you imagine what our ABIs will look like if we're still hacking in kludges for another 40+ years? Stable, reliable and backward compatible? Extensions are a better way to grow widely used APIs. They put a burden on the implementor, but don't force thousands of consumers on the eternal treadmill of obsolescence.

dragontamer 5y ago

Is this really a tcp/ip stack issue, or is this a BSD-sockets issue? Since Cloudflare showed off how it is fixed with another API (aka: connectx) and/or ordering of bind/connect/etc. etc., it sounds more like a BSD-socket issue to me. ------ In fact, the closest thing to a "protocol error" to me is in the UDP section as far as I'm concerned?

0xbadcafebee 5y ago
The source of the problem is the source port, which is a random number that only signifies what application running on a source host is requesting one particular connection. And it has to be a random number, because if it isn't it's a security issue. And re-use of ports is only necessary because we're using this finite range of numbers to signify a unique connection. The deeper down the rabbit hole you go, the more you realize that little of modern connections are by design, but rather are built on a series of kludges used to fix problems and use-cases that the original design didn't cover. A modern network stack could bring a lot of benefits. Here's a few things I would like in a new stack: - Service discovery. The reason nothing on the internet can use any port other than "443" is that we failed to build in the ability for TCP to relate to services rather than applications. TCP/443 refers to the application-layer of "HTTP", but also the transport-and-application layer of "TLS". And one IP can only ever host a single application on that host and port. If you want to run multiple applications, you have to run one application which talks the application protocol, deciphers what host you want from TLS, deciphers what host you want again from HTTP, and then opens up another connection to get you to the actual service you wanted. This is convoluted and unreliable - and decrypts the payload far from the actual application that should be looking at this data! Instead, the protocol's destination should be a combination of identifiers, and the stack should route the connection to the appropriate application. For example, instead of "host node01/port 443", you could open a connection to "host:node01/protocol:tls/protocol:http/service:public-web-pool-hackernews" (not that verbose though). All the application needs to do is open a listening socket with the same connection string, and the operating system will look up the application matching that connection string, unbundle any protocol layers it can, and connect a new file descriptor to the waiting application. You could even have multiple applications open listening sockets on the same connection string and the OS could round-robin connections to them automatically. And routers could pass connections to services distributed on pools of hosts without the need for an "application-specific load-balancing reverse-proxy". We can run all kinds of layers of protocols and services, and there is no limitation of tying up a single integer "443" with all these different concepts. This also allows us to expand the number of real internet protocols and not just keep stacking crap on top of HTTPS because we can't route any other traffic because everyone's hard-coded themselves into this one port number. - TLS becomes a part of the ABI. Any network application that wants to use TLS could open_tls_connection() and get a file descriptor. Just like applications shouldn't have to implement their own TCP/UDP packet parsers, they shouldn't have to bundle their own TLS implementations. This simplifies application code and increases speed by letting the kernel unbundle the payload from the encrypted session. - With the above two changes, add authentication and authorization layers to the stack. Now any network service can have encryption, privacy, authentication, and authorization, without needing every application to explicitly build in support for a particular method. This would allow you to write some arbitrary web app that requires login, and add an implicit trust of authorizations from host "github.com" with realm "public". If you get a connection request that's signed properly, you know the user is an authorized github.com user (and a payload with their metadata). Imagine authenticating every single network call by default. This would make it dead-simple to secure all network endpoints that often get left exposed on the internet with no authentication. - Distributed Tracing tags should be added to every layer and hop of a connection, the way an IP TTL is passed. This way you could know exactly where a connection is failing anywhere along its path. You could also pass tags forward along new connections to explicitly carry forward a trace, such as a database connection carrying a tag from a previous HTTP connection. No need for "DT infrastructure", every OS's stack would just come with tracing bundled and you could ls /proc/net/* to find out what was related to what.
- dan-robertson 5y ago
  It’s true that TCP/IP don’t really follow the OSI 7 layer model but I think the solution is to get rid of the old model that is not fit for purpose rather than to rediscover the failures that led TCP/IP to succeed instead of the standards that were designed in tandem with the model. (Apart from the layers, the main things to come from that effort to today were x.509 and ldap)
- zamadatix 5y ago
  Service discovery is already a function of DNS be it bare bones for decades like a SRV record saying which port the HTTP server is on to DNS-SD which is more catalog style infrastructure. Things like LDAP clients actually obey service discovery already. Things like HTTPS servers/browsers have resisted it because of the current security infrastructure (name only based) issues it would open not the network infrastructure. It's not like the security infrastructure side even needs to be redone from the ground up it's just the juice from changing the existing infrastructure hasn't been deemed worth the squeeze by anyone yet. TLS being deeply embedded has led to more problems than it avoids. Things like QUIC or HTTP/3 are the result of even just TCP sockets being OS managed. Getting anything new to happen when the old is 12 OS implementations updated independently of the program is never going to be popular with programs. That said Linux, FreeBSD, and Windows all support kTLS APIs already, in the case of the latter for about 2 decades. It can be useful in some situations but it's not a silver bullet. Those layers already exist, actually an option at each layer: IPsec, DTLS, TLS, HTTPS. The lack of bottom up integration and universal adoption isn't from lack of having another option, fighting that it's easy to control the auth the way you want in the HTTPS layer is probably the main restriction. Tracing tags are considered too problematic for privacy in modern protocols. E.g. QUIC offers an optional spin bit. Putting tracing tags about the full path of every packet certainly wouldn't fly, as convenient as it would make things. For debug packets only that's what ICMP already is. . There are a lot of cool things that could be done with a redesigned stack, mostly around preventing ossification of the lower layers, but it's not like it's ever been golden from day 1 or will ever be golden "if we just get it right next time"™. If the journey thus far has taught us anything it's that the internet protocol stack should design itself to avoid being everything included and instead try to be as transparent a transport it can so end applications can choose their own way of doing things without the assumptions of the transport getting in the way.

WJW 5y ago

The only reason the current way is not "voodoo" to you is because you've seen it a thousand times before. The current way is already 40 years of hacking in kludges.

turminal 5y ago

Yeah, everyone should move to Plan 9 already!

gpderetta 5y ago

> Can you imagine what our ABIs will look like if we're still hacking in kludges for another 40+ years? Stable, reliable and backward compatible? Extensions are a better way to grow widely used APIs. They put a burden on the implementor, but don't force thousands of consumers on the eternal treadmill of obsolescence.