10 ms·
Falsehoods programmers believe about TCP
- apyyxvIySl57 2y ago[flagged]
- 6EY7fcwLP3j0 2y ago[dead]
- CIQkhyvj7lFQ 2y ago[dead]
- peter_d_sherman 2y ago>"7. If we fail to connect to a well-known remote host, then we must be offline." Now that is a very interesting one! It's sort of related to the question: "How much of the Internet is accessible from any given point (location, locality, etc.) at any given point of time?" Which is sort of unknowable, at least, without attempting to connect with every possible connection point on the Internet, which (if it could be done) would still consist of a range of time, and every point in time following that point would bring changes, perhaps small relative to the whole -- but accruing over time -- more and more, as more time elapses... Observation: That same (or possibly similar!) phenomena would seem to be at play with respect to the measurment (observation) of quantum systems, i.e., the more certain you are of position, the less certain you are of velocity, and vice-versa... Well, the more you measure the connectivity to all points of the Internet at one point in time, the less certain you might be of the state of the entire system as more time elapses from that point in time... But now, why? Observation: Generally speaking, the larger a system is, the more degrees of freedom it has, in attempting to "lock down" (know by observation, be "certain" of) the entire state of that system at one point of time, the more the parts of the system with degrees of freedom (how many degrees of freedom does the entire Internet have?) will change/evolve/move/"be subject to change" as more time evolves the state of the system... in other words, if you can know position (instantaneous state) with certainty then you can't know velocity (where it's heading to and/or future state and/or that which predicts future state) with certainty! Sort of like you can know the instantaneous state of the Stock Market and its history... but no one can exactly predict its future (it has many, many degrees of freedom, all of which are subject to change in various unpredictable and bizarre ways!) Which brings us back to #7: >"7. If we fail to connect to a well-known remote host, then we must be offline." We might be offline... but then again, we might not be! (Ping, ICMP, UDP, Telnet and Gopher anyone?) But then again, we might be! The Internet's online/offline status (is it really off if it is off? Is it really on if its on?) -- is much like some modern relationships, that is, "It's complicated!" :-) The Internet is a Black Box! It's Schrodinger's Internet! You know, "if a TCP packet travelling at 99.44% of the speed of light on a westbound train track meets a UDP packet travelling at 99.43% of the speed of light on an eastbound train track, then when do they meet?" You know, "solve for x..." You know, "assume that the speed of light is constant and that quantum effects are not present!" :-)
- peter_d_sherman 2y agoRelated: https://en.wikipedia.org/wiki/Complementarity_(physics) https://en.wikipedia.org/wiki/Complementarity_(physics)
- saghm 2y ago> remember, all of the following statements are false at least some of the time, but for some of these, perhaps not very often > 5. There is a such thing as a TCP packet > 6. There is no such thing as a TCP packet I don't understand this at all. Either the concept of a TCP packet exists, or the concept does not exist. Even it's not being used in certain scenarios, I don't see how you can argue that "there's no such thing" any of the time. This might just be me misunderstanding whatever point they're trying to make, but I don't remember ever having such philosophical confusion from anything in any other "falsehoods programmers believe about..." article before.
- deathanatos 2y agoYeah, I agree with you here. I think the thing most related to that that I see people thinking is that send(2) & recv(2) calls translate 1:1 with packets send/recv. I.e., that they don't understand that the interface TCP exposes to applications is a byte stream. Which then results in things like thinking recv(2) will receive a complete "message" for some definition of message in the application protocol (i.e., the mistake belief that fragmentation won't happen).
- abnry 2y agoMy only guess how this could make sense is if there is some ambiguity in the definition of a TCP packet.
- o11c 2y agoWhat it really means is: Packets have well-defined boundaries between sufficiently-adjacent nodes. They are not guaranteed to keep those boundaries end-to-end over arbitrary middleware.
- marcosdumay 2y agoIt's about your abstraction level and the kinds of problem you are ignoring. It's true at the same time that you can't ignore the problems of stream communication nor the problems of package-based communication.
- jpollock 2y agoIf I had to guess, it would be an assumption that TCP was edge to edge with no translation in the middle. My guess is that this is talking about systems in the middle of the network, changing (for example) their sizes by combining and splitting packets to fit through various transits.
- richm44 2y ago1. A SYN will receive a SYN-ACK or a RST 2. A host from my machine is the same as from your machine 3. An IP from my machine is the same as from your machine
- eptcyka 2y ago1. A SYN may receive a SYN-ACK, RST or nothing at all.
- hinkley 2y agoI recall it blew my fiancée’s mind that I could unplug her ethernet cable, move it around an obstacle, plug it back in and all her connections were still alive. It’s designed to have bombs dropped on it.
- jancsika 2y agoWhat happens in that case? I'm going to speculate: 1. Remote keeps sending stuff to your unplugged connection 2. You plug your ethernet cable back in 3. Your computer's TCP acknowledges the last sequence number it received for each new sequence it receives from remote 4. Remote sees duplicate ACKs for same sequence number, interprets it as packet loss and resends the stuff
- deleted 2y ago[deleted]
- IgorPartola 2y agoWith timeouts. You can’t unplug it for an hour and have this happen. But a few seconds is exactly what this is designed for. As another commenter pointed out, your OS could also try to be “helpful”.
- toast0 2y agoYeah, that's pretty much it. If packets were sent while you were disconnected, they'll be gone, but if you're disconnected for only part of the burst, duplicate ACKing will trigger retransmits. If you were gone for the whole burst, you'll get put right by timer based retransmits. If you're gone for long enough, most peers will timeout on unacknowledged data (although that's not in the TCP RFC), and if there's no outstanding data, most peers eventually have some sort of periodic ping and timeout (tcp keep-alives is a reasonable fallback IMHO, if your application protocol doesn't have someything, although the default of IIRC 2 hours feels long in todays world of lots of NATs and much shorter timeouts).
- akira2501 2y agoIt may be your local gateway. Seeing no packets from your host. Attempting to refresh your MAC address via ARP. Getting no response. Generating an ICMP message as a result.
- dtaht 2y agoIt really is astounding to me how so many still do not understand that tcp is not a function call, or behaviors like slow start and congestion avoidance. Recently a new rate limiter for TCP went by that was so terribly, terribly broken, and I cannot help but imagine that most of the containers of the world suffer from Bufferbloat in general.
- dtaht 2y agoThe rate limiter in question: https://github.com/cilium/cilium/issues/29083 https://github.com/cilium/cilium/issues/29083
- solatic 2y agoRelated: you can get at most once delivery or at least once delivery; you cannot get exactly once delivery. If I had a dollar for every junior who thought that a lack of exactly once delivery guarantees was a bug...
- stanac 2y agoYup, I try to explain it with shouting a message to someone in a crowded room. You can yell at your boss "I fixed the bug", they can confirm it or ignore you, which is delivery at most once if you don't repeat the message. If you try to repeat the message until they confirm it, it is at least once delivery. edit: Point is in confirming that message is received. If you don't receive the confirmation the message was delivered at most once.
- bcoates 2y agoThis is a popular saying that is basically wrong. You have very limited guarantees around an arbitrarily bad partition, but this is also a detectable condition. Lots of defective systems exist, but in general non-defective systems generally guarantee "exactly once delivery or detected failure"
- rcxdude 2y agoThat sounds like "at most once" to me
- bcoates 2y agoIf you unplug the network you can't send messages, correct. Other people upthread have already gone over how you can't separate delivery from message processing and how TCP's attempt to do so makes it defective (unless you layer a whole additional system on top of it rendering most of TCP's design irrelevant) If you were trying to make a new, non-broken system on top of TCP or otherwise, allowing multiple delivery doesn't add any correctness/robustness benefits -- it just makes messages cheaper to send and receive. There is no "at most once or at least once" choice except in the pickwickian sense that if you don't require delivery or delivery confirmation you can save the effort of even trying.
- KaiserPro 2y agoSo TCP has slow start, and exponential fall off and shit. but you can get round that in a lot of cases by just having a load of TCP connections in parallel. TCP is cheap and well optimised, especially if you are keeping a bunch of connections open. (opening can be expensive) so if you have a high latency connection, or a bit of packet loss, and you want to reach line speed without having to figure out cornercases with UDP, just open up 100-1k TCP connections and multiplex them. bish bash bosh, mostly line speed over a high latency line (mind you this was in the days of 100m-500m cross atlantic internet, you'll probably need more connections to saturate a 10gig line.)
- nh2 2y agoSuch hack is often not necessary. Set larger kernel TCP send and receive buffers and enable BBR congestion control. Speed will usually be good also across high latency links, and no multiplexing logic needed. Especially if you control both sides of the connection.
- KaiserPro 2y ago> Set larger kernel TCP send and receive buffers and enable BBR congestion control I mean yeah, but that requires having access to the kernel config. so for most people multiplexing TCP is a useful way to maximise a link, without having to fiddle with stuff that is a pain to deploy. (politically as well as logistically) I deployed this "technique" before BBR was a thing. It worked well enough for what I needed it to do (move large images from London to California) It was pretty simple to engineer as well (mainly because I didn't have to make a fancy custom error detection/correction/rate limiting system over UDP )
- 6qVRwS5iw3fA 2y ago[flagged]
- 3KccUAJuWG1b 2y ago[dead]
- 2j8mJtnePjLM 2y ago[dead]
- koala_man 2y agoI find this "falsehoods programmers believe" format of making pointed claims that you intentionally don't clarify to be unhelpful and obnoxious
- IAmNotACellist 2y agoThe following list contains only falsehoods: 1. You're wrong 2. Okay, you're right 3. Okay maybe you're right or wrong but certainly not both
- efitz 2y agoYou’re doing it wrong. “Falsehoods programmers believe…” articles are designed to make you THINK about problematic assumptions. They are not like the 10 commandments and they are not decrees of absolute truth.
- Dylan16807 2y agoSaying a thing and then saying the opposite, without elaborating, is not good at making you THINK. This list is doing it wrong.
- Izkata 2y agoYep, the original "names" one was mostly written so negating each of the points gave you the exception you needed to handle. Even the cases written with both were done on a way it was obvious the negation didn't apply universally, so both worked.
- poorman 2y ago> Explainer for 1-4: https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem. TL;DR: If the connection breaks while an ACK is outstanding, the sender will have no way of knowing whether the segment was received, and this turns out to be an insoluble problem no matter how much complexity you pile on top of it. You need something resembling Paxos or Raft to get a guarantee like that The hashgraph algorithm is pretty sweet too and doesn't have the issue of a single write leader like Paxos and Raft. Basically multi-writers / leaderless https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf But to be fair, I'm not certain that CAP theorem and partition tolerance really belong in a conversation about TCP anyway
- kranuck 2y ago> 11. This is all low-level pedantry Yeah pretty much. maybe don't write contradictory unexplained nonsense.
- rR9292wTfIdP 2y ago[flagged]
- fZFKY14LpSbH 2y ago[dead]
- dasyatidprime 2y agoTo some of the critics here: did you or did you not notice the “Somebody ought to write one of those […] Here, I'll even get the ball rolling” framing? A polished such article this is not claiming itself to be! I would go as far as saying the HN submission title is misleading as a result.
- hamilyon2 2y agoI'll go out on a limb: inside datacenter on your own hardware, you can safely ignore low-level pedantry and mostly ignore “weird networks” and use TCP as two-way Unix pipe. “Mostly” because you still care about bandwidth limits and packet RPS limits and latency of course.
- toast0 2y agoI wouldn't, unless you've got a really solid understanding of your datacenter network and it's 100% good all the time. Which is unlikely, from my experience as a server person. If you've got dirty optics between two switches, now you're getting packet loss and TCP rears its head. Hopefully it's not an issue now, but diagnosing microbursting[1] was lots of fun, and really wigs TCP out. I've also run into 'fabric congestion'. My true favorite though is when you've got 2x aggregation on servers, and 4x aggregation for top of rack switches to spine switches, so there's 8 paths in each direction between two servers in adjacent racks, and only one path (sometimes in only one direction) is only running at 99.9%. That's a real PITA to track down unless you have visibility into switching metrics. [1] https://en.m.wikipedia.org/wiki/Micro-bursting_(networking) https://en.m.wikipedia.org/wiki/Micro-bursting_(networking)
- macintux 2y agoI was always suspicious about self-hosted high availability solutions (typically just diagrams, not yet implemented) that included redundant switches. Given how generally reliable switches are, I was inclined to believe that a misconfiguration or flaky network cable on one switch was more likely to cause a downtime (or significant degradation) than an outright switch failure, so adding another switch was doubling the chances of trouble and, as you note, making it harder to troubleshoot.
- toast0 2y agoIt kind of depends. You do get some weird stuff to debug, and more connections = more likely that one of them is broken. Otoh, if you ever do any scheduled maintenance on your switches (which is likely if they're doing anything fancy), having properly setup redundancy means you can announce a likely brief loss of redundancy, rather than a likely brief full loss of connectivity. If you have the right knobs, you can gracefully fail out the switch under maintenance and everything goes smoothly. Of course, sometimes you reboot the redundant switch and it confuses the other one and servers lose connectivity anyway.
- WUt2nuuNDqaW 2y ago[flagged]
- IAS4oB40A63x 2y ago[dead]
- HkJfJTle71cn 2y ago[dead]
- grishka 2y agoThis reminds me of a very particular problem that we tried to solve when I worked at VKontakte. It was about instant messaging and flaky mobile data connections. The problem: you're on a subway train and you send a message as it departs a station. The request does get to the server, but by the time the response arrives, the train is already in the tunnel and you don't have a signal any more. So the client thinks that the message failed to send, but it was, in fact, sent successfully. The client would retry when it's back online, and would send another copy of that message. The solution was to send a client-generated "random ID" with each request. I much later learned that this is conventionally called an "idempotency token". This worked, except there was now another problem: you sometimes receive your own message over the long-polling thing before the response to the request that sent it. You don't know for sure whether it's the message you just sent, or something else sent by a different client on the same account, because you don't know the ID of your message yet. This was solved by me delaying the processing of outgoing messages on the client side until all outstanding messages are fully sent and their IDs are known. Telegram solved this much more elegantly: when the client reconnects to the server, the server sends it all the responses that were not acknowledged during the previous connection. MTProto has its own acknowledgement mechanism in addition to TCP's. So yeah, instant messaging seems trivial at the first glance, but it turns out that TCP is a leaky enough abstraction that you need to somehow plug those leaks at the application level.
- zmj 2y agoI had to deal with the second problem in a file synchronization app. The solution was to propagate a “device id” through the request and poll/push, so the originating device could ignore changes that it originated.
- HDThoreaun 2y agoIve learned that just about nothing is trivial when it comes to distributed systems
- 8organicbits 2y ago> because you don't know the ID of your message yet Wouldn't the client know the ID if they randomly generated it? (honest question)
- deleted 2y ago[deleted]
- ijustlovemath 2y agoHas this author never heard of error correcting codes? The whole point of them is to assume there's lossiness and add bytes to allow correction (or at least detection) of tampered or missing bytes. That's why TCP (or maybe it's Ethernet?) frames include FEC bytes in their message format. Additionally, I'm sure they're aware that HTTP over TLS has encrypted data frames, which would be unreceivable in a lot of cases if these situations arose a bunch. And considering how much of the modern Internet is built on this paradigm, I think that many of these points are rare and probably extremely pedantic. This is coming from someone who agrees with much of the nuance implied (but not explained!) by the post. All great technical writing (which I assume these clickbait articles are at least attempting to be) is written with mutual discovery and deeper understanding in mind, and if you leave no actual explanation in the post, you can't really achieve either of those.
- okl 2y ago> That's why TCP (or maybe it's Ethernet?) frames include FEC bytes in their message format. Neither TCP nor Ethernet provide for forward error correction. Ethernet frames include a 32-bit CRC while TCP segments use the so called "internet checksum".
- ijustlovemath 2y agoCunningham's law at work!
- minetest2048 2y agoSome flavors of optical ethernet do have FEC: https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/implementation-optics-wp.html https://www.cisco.com/c/en/us/products/collateral/interfaces...
- ijustlovemath 2y agoSee above :)
- 2y ago
- mice69 2y ago[dead]
- derefr 2y ago> 11. This is all low-level pedantry. I can think of TCP like a two-way Unix pipe that goes over the network, and completely ignore how it is implemented. I mean, that's true, insofar as pipes have incredibly weak guarantees too — after all, the other end of a pipe might be a program reading from/writing to a network socket, or other unreliable transport. Whenever you let your program be plugged into an arbitrary pipe, you have to expect all that same flakiness and then some.
- paulddraper 2y ago> There is a such thing as a TCP packet In what way is that a falsehood?
- nirui 2y ago> If the connection breaks while an ACK is outstanding, the sender will have no way of knowing whether the segment was received The real question is, why this should be a problem that TCP must solve? TCP gives you a bidirectional waterflow-like pipe, and that's enough for you to create many useful applications. TCP never provided guarantee for correct delivery, that's your job. For example, if a HTTP request is interrupted before the respond is received, the sender should assume the request never reach the server and try again with a new connection, while the server should mitigate duplicated requests (reject or return a successful code). Well, maybe that's the point of the article, because many web pages gets confused if you send duplicated requests to them.
- erik_seaberg 2y agoThe server may or may not have seen the request, and the https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem proves it impossible to know in every case (no matter how many acks, the last could be dropped). A request that alters state should be retried using the same idempotency key, and the server should try to ack with whether the requested work already happened.
- halayli 2y agoThis post is meaningless without clearly defining what reliable means. Regarding ack not being received by sender when connection breaks, it's a weak and dishonest argument thinking it will strengthen their position, but completely ignoring the fact that TCP reliability is dependent on the simple and obvious fact that the connection exists!
- LtWorf 2y agoPlease note that it's not an article on lwn.net, it's a comment written by someone.
- dathinab 2y agoWhile I often do like "falsehoods <....> believe about <...>" format it doesn't always fit in well (and if placed alone without explanation often can at most help you to know where you have knowledge gaps but not which). A common problem are points which aren't really falsehoods, but where people frequently take false conclusions from it. E.g. if you ask if TCP is reliable, especially in a non CS paper context, the answer is yes. That is iff you take a reasonable definition of reliable (which doesn't expect literally impossible things) and a reasonable interpretation of mostly. And just listing it as a falsehood fails to point out that there are two potential issues with your understanding while making creating the risk of someone with expertise in that sub-field of IT potentially thinking TCP is quite unreliable when it isn't. I mean the most common usage of the word reliable is a gradient with its meaning in a yes/no question being a short form of "reliable _enough_". Furthermore for most use-cases the "unreliable" aspect of TCP isn't even the main relevant misunderstanding people can have with "TCP is mostly reliable" (through for some use case it is) The main troublesome misinterpretation is what mostly means. I.e. if you would give it a regious definition it would be "if sampling typical devices used in typical situations across some target audience then for most target audiences (weighted by audience relevance) most of the sampled devices will in a sufficient large long term moving average be reliable enough" or something like that. What that mainly means: - even if it's mostly reliable there will be devices for which it is reliable unreliable and anything in between - similar even if it's mostly reliable for a device that isn't necessary all the time - nor do we do statements about the patterns when the mostly doesn't apply, i.e. for a device TCP might be mostly reliable except every Sunday 3am for 30s, would still be mostly - there are use-cases where unreliability is much more common - there are audiences for which unreliability is much more common etc. Similar for point 5,6 about TCP packages, they are definitively a thing and there is no falsehood there. The falsehood is in believing you can reliable control them, that your OS or some middle ware isn't messing with them (e.g. splitting/combining/rewriting). So in some situations it's best to pretend there are non, but in some other situations you have to care and this might differ for different parts of the same protocol. So point 5 and 6 make sense, but don't point in a helpful direction. to be clear doesn't mean lists are bad, or this list being particular bad, but I which they had more references/details even if short and compact and more clearly separate things too