7 ms·

TCP is an underspecified two-node consensus algorithm

135 points by r4um 9y ago

lkarsten 9y ago
I applaud the effort to hate on "smart" middleware proxies! That being said, author gets no points for namedropping random distributed systems algorithms and using tcp keepalives (2 hours minimum!) as an argument against TLS terminating proxies. Is there a reason to (as he says) "fully implement the protocol" in the proxy? I battled with websockets through Pound last week, and it simply doesn't work because the author took a non-postel stand on protocol specifics. Having a protocol agnostic proxy like hitch (previously stud) fixed that without losing functionality, and I expect it to age better as well.
- jclulow 9y ago
  If more software would take an apostel stand, I feel we would have net fewer interoperability problems.
  - bringtheaction 9y ago
    By that you mean if more software was more strict about what it accepted instead of being liberal about it?
    - jclulow 9y ago
      Yes, that.
- zaarn 9y ago
  Sadly the internet isn't as nice. I've been to various hotels where the router would silently drop keep-alive packets (but god forbid informing the packet layer you do this!) and mangled DNS packets ("looking for mail.example.com? Here is the answer for example.com" and "looking for doesnotexist.com? Here is result for internalsearchengine.com which redirects you to a sponsored search page with ads")
  - pixl97 9y ago
    And this is why encrypted DNS is a must.
    - zaarn 9y ago
      Even encrypted DNS will suffer because middleboxes with captive portals will attempt to tamper it. Unless you pipe it over TLS or HTTP in which case you run into problems with not knowing why there is no connection in a captive portal (we obviously need to fix captive portals, they're source of 90% of problems)
      - geofft 9y ago
        I learned recently about RFC 7710 which specifies a DHCP (v4/v6) and RA option for "You're on a captive portal, here's the website you should visit before you get access": https://tools.ietf.org/html/rfc7710 https://tools.ietf.org/html/rfc7710 Do any of the major implementations of captive portals support it?
        zaarn 9y ago
        Not that I know. My pfSense firewall doesn't have it (IIRC), so my guess would be that poorly maintained router boxes in a hotel basement definitely don't have it. I'm not sure if the various DHCP clients communicate this properly to the OS or browser even (I wouldn't know how to query for it on Linux)
        geofft 9y ago
        If you're using NetworkManager you can get DHCP options by being mildly angry at the D-Bus API: $ python3 >>> import dbus >>> bus = dbus.SystemBus() >>> nm = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager") >>> conn = bus.get_object("org.freedesktop.NetworkManager", nm.Get("org.freedesktop.NetworkManager", "PrimaryConnection", dbus_interface="org.freedesktop.DBus.Properties")) >>> dhcp = bus.get_object("org.freedesktop.NetworkManager", conn.Get("org.freedesktop.NetworkManager.Connection.Active", "Dhcp4Config", dbus_interface="org.freedesktop.DBus.Properties")) >>> options = dhcp.Get("org.freedesktop.NetworkManager.DHCP4Config", "Options", dbus_interface="org.freedesktop.DBus.Properties") >>> str(options["subnet_mask"]) '255.255.255.240' I guess you can parse /var/lib/dhcp/dhclient.*.leases otherwise?
- gtirloni 9y ago
  For reference: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle "TCP implementations should follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others." -- Jon Postel
teknopaul 9y ago
I dont grok this, if tcp's model has fundamental problems how come the Internet works. :) The fact that a protocol technically is not perfect and causes jip for isps does not mean the application layer has to get involved. I've been writing tcp based apps for years and the stream abstraction has never failed me. After reading this I dont see why I should change that assumption? I have to rebuild connections occasionally but its never cost my application so much that an alternative more complicated abstraction layer made sense. I usually write req/response over tcp, an even more inaccurate abstraction. Occasionally nonblocking code. Never have I wanted more complexity than nio in my application layer. Devs do know that "tcp is not a stream of bytes" but deliberately do not want to get app code involved.
- nvarsj 9y ago
  > I dont grok this, if tcp's model has fundamental problems how come the Internet works. :) Depends on your definition of "works"! Just ask any gamer what happens when someone starts using netflix while they're playing a game... For a lot of people the internet doesn't work very well. TFA is describing, I think, the issue of stale / dead connections - which can be notoriously difficult to detect. Imagine you are building black boxes that sit in peoples' homes, with a long running TCP connection to push notifications to. With the state of the internet, it's very likely some percentage of those connections will die without either side realizing it. That's where the complexity comes in.
  - bringtheaction 9y ago
    Off-topic on my part here but I think it is so weird that people say “TFA” when referring to the article in the way you are doing. I see this a lot on HN. IMO it would be natural to use “TFA” in the same way you would use “RTFM” — when you are scolding someone for not having read what they should — but using “TFA” as just a synonym for “the article”... that always throws me off.
    - steveklabnik 9y ago
      You’re both right and wrong. That is, the... anger? you’re picking up isn’t incorrect, but instead lost in translation. Back in the /. days, which, as far as I know, is where this comes from, people would use it in this way. “Read TFA.” “Have you read TFA?” “Seems like you haven’t read TFA.” Eventually, that morphed into the usage you see on HN and everything else, as kind of a different version of OP. One useage grows, another dies off.
VMG 9y ago
title is borked
nh2 9y ago
shutdown() and half-closes are not "archaic" features. You need them to get even the basic stuff right (see https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-...) and you need it even more to implement "modern" application layer protocols like HTTP2 (if you don't use it, you get data loss bugs like this: https://trac.nginx.org/nginx/ticket/1250#comment:4 https://trac.nginx.org/nginx/ticket/1250#comment:4).
kenforthewin 9y ago
I suppose you could call it a two-node consensus algorithm, the same way plugging a flash drive into your laptop is. Even after reading, I don't see the benefit of viewing TCP this way.
- tonyarkles 9y ago
  It's a layer under the "it's a bidirectional stream" abstraction. Having had to debug gnarly TCP problems before (generally with semi-"intelligent" load balancers), the two-node consensus concept is good to keep in mind. Right now the "two-node consensus algorithm" idea has got me thinking more about some of the nasty problems I've debugged and how to look at them through that lens, especially considering partial partition tolerance. "Under what circumstances can the TCP state machine on either end lose consensus, and how will/won't it recover?"
peterwwillis 9y ago
The problem is in thinking of an HTTPS request-response through proxies as a single TCP connection. It isn't. A TLS proxy is not a normal part of a layered TCP/IP connection. It's literally in the name: "terminating" proxy. It stops the connection right there. Anything after the TLS proxy is outside the scope of the initial connection. Applications have to be engineered to pass on data from one connection to another. An example is stateful firewalls. Almost all stateful firewalls are NAT gateways with rules. NAT gateways are designed to pass certain things from one connection to another, but they are not simply unwrapping a layer from a connection and passing it on: they maintain separate connections. edit Apparently I'm wrong, as Netfiler apparently only defragments and then changes addresses and ports, but firewall vendors basically keep independent connections (for security reasons) TCP is specified just fine for consensus on a single TCP connection. It isn't specified for an HTTPS connection through middleware. Hence, such middleware is complicated.

135 points by r4um 9y ago

I applaud the effort to hate on "smart" middleware proxies! That being said, author gets no points for namedropping random distributed systems algorithms and using tcp keepalives (2 hours minimum!) as an argument against TLS terminating proxies. Is there a reason to (as he says) "fully implement the protocol" in the proxy? I battled with websockets through Pound last week, and it simply doesn't work because the author took a non-postel stand on protocol specifics. Having a protocol agnostic proxy like hitch (previously stud) fixed that without losing functionality, and I expect it to age better as well.

jclulow 9y ago
If more software would take an apostel stand, I feel we would have net fewer interoperability problems.
- bringtheaction 9y ago
  By that you mean if more software was more strict about what it accepted instead of being liberal about it?
  - jclulow 9y ago
    Yes, that.
zaarn 9y ago
Sadly the internet isn't as nice. I've been to various hotels where the router would silently drop keep-alive packets (but god forbid informing the packet layer you do this!) and mangled DNS packets ("looking for mail.example.com? Here is the answer for example.com" and "looking for doesnotexist.com? Here is result for internalsearchengine.com which redirects you to a sponsored search page with ads")
- pixl97 9y ago
  And this is why encrypted DNS is a must.
  - zaarn 9y ago
    Even encrypted DNS will suffer because middleboxes with captive portals will attempt to tamper it. Unless you pipe it over TLS or HTTP in which case you run into problems with not knowing why there is no connection in a captive portal (we obviously need to fix captive portals, they're source of 90% of problems)
    - geofft 9y ago
      I learned recently about RFC 7710 which specifies a DHCP (v4/v6) and RA option for "You're on a captive portal, here's the website you should visit before you get access": https://tools.ietf.org/html/rfc7710 https://tools.ietf.org/html/rfc7710 Do any of the major implementations of captive portals support it?
      - zaarn 9y ago
        Not that I know. My pfSense firewall doesn't have it (IIRC), so my guess would be that poorly maintained router boxes in a hotel basement definitely don't have it. I'm not sure if the various DHCP clients communicate this properly to the OS or browser even (I wouldn't know how to query for it on Linux)
        geofft 9y ago
        If you're using NetworkManager you can get DHCP options by being mildly angry at the D-Bus API: $ python3 >>> import dbus >>> bus = dbus.SystemBus() >>> nm = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager") >>> conn = bus.get_object("org.freedesktop.NetworkManager", nm.Get("org.freedesktop.NetworkManager", "PrimaryConnection", dbus_interface="org.freedesktop.DBus.Properties")) >>> dhcp = bus.get_object("org.freedesktop.NetworkManager", conn.Get("org.freedesktop.NetworkManager.Connection.Active", "Dhcp4Config", dbus_interface="org.freedesktop.DBus.Properties")) >>> options = dhcp.Get("org.freedesktop.NetworkManager.DHCP4Config", "Options", dbus_interface="org.freedesktop.DBus.Properties") >>> str(options["subnet_mask"]) '255.255.255.240' I guess you can parse /var/lib/dhcp/dhclient.*.leases otherwise?
gtirloni 9y ago
For reference: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle "TCP implementations should follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others." -- Jon Postel

teknopaul 9y ago

I dont grok this, if tcp's model has fundamental problems how come the Internet works. :) The fact that a protocol technically is not perfect and causes jip for isps does not mean the application layer has to get involved. I've been writing tcp based apps for years and the stream abstraction has never failed me. After reading this I dont see why I should change that assumption? I have to rebuild connections occasionally but its never cost my application so much that an alternative more complicated abstraction layer made sense. I usually write req/response over tcp, an even more inaccurate abstraction. Occasionally nonblocking code. Never have I wanted more complexity than nio in my application layer. Devs do know that "tcp is not a stream of bytes" but deliberately do not want to get app code involved.

nvarsj 9y ago
> I dont grok this, if tcp's model has fundamental problems how come the Internet works. :) Depends on your definition of "works"! Just ask any gamer what happens when someone starts using netflix while they're playing a game... For a lot of people the internet doesn't work very well. TFA is describing, I think, the issue of stale / dead connections - which can be notoriously difficult to detect. Imagine you are building black boxes that sit in peoples' homes, with a long running TCP connection to push notifications to. With the state of the internet, it's very likely some percentage of those connections will die without either side realizing it. That's where the complexity comes in.
- bringtheaction 9y ago
  Off-topic on my part here but I think it is so weird that people say “TFA” when referring to the article in the way you are doing. I see this a lot on HN. IMO it would be natural to use “TFA” in the same way you would use “RTFM” — when you are scolding someone for not having read what they should — but using “TFA” as just a synonym for “the article”... that always throws me off.
  - steveklabnik 9y ago
    You’re both right and wrong. That is, the... anger? you’re picking up isn’t incorrect, but instead lost in translation. Back in the /. days, which, as far as I know, is where this comes from, people would use it in this way. “Read TFA.” “Have you read TFA?” “Seems like you haven’t read TFA.” Eventually, that morphed into the usage you see on HN and everything else, as kind of a different version of OP. One useage grows, another dies off.

VMG 9y ago

title is borked

nh2 9y ago

shutdown() and half-closes are not "archaic" features. You need them to get even the basic stuff right (see https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-...) and you need it even more to implement "modern" application layer protocols like HTTP2 (if you don't use it, you get data loss bugs like this: https://trac.nginx.org/nginx/ticket/1250#comment:4 https://trac.nginx.org/nginx/ticket/1250#comment:4).

kenforthewin 9y ago

I suppose you could call it a two-node consensus algorithm, the same way plugging a flash drive into your laptop is. Even after reading, I don't see the benefit of viewing TCP this way.

tonyarkles 9y ago
It's a layer under the "it's a bidirectional stream" abstraction. Having had to debug gnarly TCP problems before (generally with semi-"intelligent" load balancers), the two-node consensus concept is good to keep in mind. Right now the "two-node consensus algorithm" idea has got me thinking more about some of the nasty problems I've debugged and how to look at them through that lens, especially considering partial partition tolerance. "Under what circumstances can the TCP state machine on either end lose consensus, and how will/won't it recover?"

peterwwillis 9y ago

The problem is in thinking of an HTTPS request-response through proxies as a single TCP connection. It isn't. A TLS proxy is not a normal part of a layered TCP/IP connection. It's literally in the name: "terminating" proxy. It stops the connection right there. Anything after the TLS proxy is outside the scope of the initial connection. Applications have to be engineered to pass on data from one connection to another. An example is stateful firewalls. Almost all stateful firewalls are NAT gateways with rules. NAT gateways are designed to pass certain things from one connection to another, but they are not simply unwrapping a layer from a connection and passing it on: they maintain separate connections. edit Apparently I'm wrong, as Netfiler apparently only defragments and then changes addresses and ports, but firewall vendors basically keep independent connections (for security reasons) TCP is specified just fine for consensus on a single TCP connection. It isn't specified for an HTTPS connection through middleware. Hence, such middleware is complicated.