9 ms·

What they don't teach you about sockets

412 points by zdw 4y ago

zwieback 4y ago
In the early 90s I was working on AppleTalk-PC interoperability SW, which included "ATP", the AppleTalk Transaction Protocol, which is kind of a guaranteed delivery protocol. It even had "at least once" and "exactly once" options to make sure state wouldn't get messed up in client-server applications: https://developer.apple.com/library/archive/documentation/mac/pdf/Networking/ATP.pdf https://developer.apple.com/library/archive/documentation/ma... I was kind of sad to see that things like this fell by the wayside in favor of streaming protocols but of course it's way more efficient to not handle each transaction individually. The problem of users implementing TCP++ on top of TCP is a real problem, I think.
- NegativeLatency 4y ago
  Was there a driver or library for ATP or were you implementing that on PCs?
  - zwieback 4y ago
    No, we were the ones writing the PC drivers, for DOS, Windows and OS/2. Back then the lower layers were more or less standardized but there was no AppleTalk stack so that's what we developed as a product.
- bombela 4y ago
  I just want to point out that "exactly-once" is not physically possible. The documentation claims "exactly-once", and proceeds to explain that after a while, it will timeout. Which makes it "at-most-once", as expected. As far as I know; in communication; you can only have at-least-once (1..N) or at-most-once (0..1).
  - lilyball 4y ago
    According to the docs, the transaction release timeout is specified by the requester. The requester also specifies the duration to wait between retries and the maximum number of retries. So this does indeed appear to be exactly-once as long as the sender ensures the transaction release timeout is greater than the total time (retry timeout * max retries) plus any network delay.
  - erichocean 4y ago
    > As far as I know; in communication; you can only have at-least-once (1..N) or at-most-once (0..1). If you hash-chain all of your messages, you can trivially and reliably have exactly-once semantics. Just like a blockchain. Each message includes its predecessor hash, so if you get a message that you don't have the predecessor for, you can just request it. ACKs become "this is the latest message hash for which I have all of the predecessors" and can be done lazily, i.e. every nth message, or after not receiving a message in awhile. Source: Me, built a system that worked like this in 2014 (over UDP). Worked great, it had some similarities to how QUIC is designed (including the crypto).
dehrmann 4y ago
One of the big issues with TCP is a lot of communication isn't well-suited to the stream model and is message-oriented, so yes, like the author says, you have to go implement ACKs. And then you want multiplexing, which streams also fail at. Before you know it, you built a worse version of HTTP2.
- iTokio 4y ago
  And then you realize that sometimes it has become even worse due to tcp head of line blocking and you move to udp and build a worse version of http3
  - hinkley 4y ago
    I think Head of Line blocking is a more concrete problem but it seems that it's often encountered when running away from a different one: If you have different conversations going on between a single client and the server, you can make several connections to do it. You'll pay a little bit more at the start of the conversation of course, so be frugal with this design, and think about ways to 'hide' the delay during bootstrapping. But know that with funky enough network conditions, it's possible for one connection to error out but the other to continue to work. The problem for me always comes back to information architecture, and the pervasive lack of it, or at least lack of investment in it. If you really have two separate conversations with the server, losing one channel shouldn't make the whole application freak out. But we all know people take the path of least resistance, and soon you have 2.5 conversations on one channel and 1.5 on the other. The advice here is some of the same advice in more sophisticated treatises on RESTful APIs (it's all distributed computing, it's all the same problem set arranged in different orders). REST calls are generally supposed to be stateless. The client making what looks like a non sequitur call to the backend should Just Work. If you manage that, then having one channel inform the client of the existence of a resource and fetching that resource out of band isn't really coupling of the two channels. That subtlety is lost on some people who defend their actual coupling by pointing out that other people have done it too, when no, they really haven't. And if anything, multiplexing lets them get away with this bad behavior for much longer, allowing the bad patterns to become idiomatic.
  - 4y ago
andai 4y ago
I played around with making a little websocket layer for browser games a while back. I used a Windows tool called "clumsy" to simulate crappy network conditions and was surprised at how poor the results were with just WebSockets, despite all the overhead of being a protocol on top of a protocol. The result is that you need to build a protocol on top of the protocol on top of the protocol if you actually want your messages delivered...
- tuukkah 4y ago
  That's because WebSockets are more or less just sockets for web apps. You'lll want to use a protocol that deals with messages and their at-least-once delivery, such as MQTT (that can run on top of WebSocket if you need it).
- swagasaurus-rex 4y ago
  I built a javascript data synchronization library specifically for games https://github.com/siriusastrebe/jsynchronous https://github.com/siriusastrebe/jsynchronous A core part of the library is the ability to re-send data that's been lost due to connection interruption. Absolutely crucial for ensuring data is properly synchronized.
psyc 4y ago
I'm pleased to hear that anyone is still teaching anyone anything about sockets. The bit about Windows not treating sockets as files made me pause, since Windows does treat so many things as files. After thinking about it some, I suppose it's kernel32 that treats kernel32 things as files. Winsock has a separate history.
- amluto 4y ago
  Windows has a long and unfortunate history of encouraging programs to extend system behavior by injecting libraries into other programs’ address spaces. You can look up Winsock LSPs and Win32 hooks [0] for examples. This means that programs cannot rely on the public APIs actually interacting with the kernel in the way one would natively imagine — the implementation may be partially replaced with a call into user code from another vendor in the same progress. Eww! So, as I recall, a normal socket is a kernel object, but a third party LSP-provided socket might not be. This also means that any API, e.g. select, that may interact with more than one socket at once has problems. [1] [0] https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwindowshookexa https://docs.microsoft.com/en-us/windows/win32/api/winuser/n... — see the remarks section. [1] https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-winsock2-select https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/... — again, see the remarks.
- xeeeeeeeeeeenu 4y ago
  It's complicated. Sockets are mostly interchangeable with filehandles but there are many exceptions. For example, ReadFile() works with sockets, whereas DuplicateHandle() silently corrupts them. However, there's another problem: overlapped vs non-overlapped handles. socket() always creates overlapped sockets, while WSASocket() can create both types. Overlapped handles can't be read synchronously with standard APIs, which in turn means you can't read() a fd created from an overlapped handle. Naturally, in their infinite wisdom, Windows designers decided there's no need to provide an official API to introspect handles, so there's no documented way to tell them apart (there are unofficial ways, though). BTW, it's a proof of poor communication between teams in Microsoft, because their C runtime (especially its fd emulation) would greatly benefit from such an API. It's frustrating. I'm sure that if Windows was an open-source project, that mess would be fixed immediately.
weeksie 4y ago
Ah man, sockets. Real sockets. Yesterday's kiosk thread jogged my memory about all that but almost all of the network communication on my old kiosk installations was socket based. It was a pleasant way to work.
TYPE_FASTER 4y ago
What really helped me understand and troubleshoot network communications at the socket level was the book by W. Richard Stevens[0]. I think this is because he starts with an introduction to TCP, and builds from there. Knowing TCP states and transitions is important to reading packet dumps, and in general debugging networking. Also, no matter what platform you're using, there is probably a socket implementation somewhere at the core. You're best off understanding how sockets work, then understanding how the platform you're working with uses sockets[1]. Once I read the W. Richard Stevens book, I was able to read and understand RFCs for protocols like HTTP to know how things should work. Then you're better prepared to figure out if a behavior is due to your code, or an implementation of the protocol in your device, or the device on the other end of the network connection, or some intermediary device. [0] - http://www.kohala.com/start/unpv12e.html http://www.kohala.com/start/unpv12e.html [1] - https://docs.microsoft.com/en-us/windows/win32/winsock/porting-socket-applications-to-winsock https://docs.microsoft.com/en-us/windows/win32/winsock/porti...
- hinkley 4y ago
  There were many authors in that era who had practically compulsory books. Stevens is one of the few to have two. His Unix Programming book was referred to as the Bible.
  - sophacles 4y ago
    I still refer back to APUE often and push my junior folks to read it.
  - Agingcoder 4y ago
    The Linux programming interface, by Michael Kerrisk, is an excellent modern update.
- lanstin 4y ago
  We had to debug an issue with ARP during a deploy of docker containers a year or two ago. A nearly complete understanding of the lower layers is not that much knowledge and quite useful at times.
gabereiser 4y ago
This was a good write up but was kind of short of details. It would have been really awesome with some code examples. I love socket programming. Love, like I love going to the dentist. Having a state machine to handle basic I/O between client/server kind of blows when it comes to plenko machine switch statements of arbitrary state enums. Languages like Go or Python allow you to have a way of communicating with the client from the server in a more direct client->server oriented way. Write/Read input, do thing, write, read, do, repeat until you finish ops. Go is my favorite for this as I can spawn a goroutine to read/write to a socket. Rust is my second favorite for this but it's a bit trickier. Python's twisted framework has something like this with Protocols. I wish C++ had a standard socket implementation (std::network?) Anyway, this gave me a smile today so thanks.
jchw 4y ago
> I will also ask: how often are you writing applications that want to accept either files or sockets? In Go where there is an io.Reader/io.Writer abstraction where blocking interactions are OK and you're absolutely intended to handle all of the errors, it's really no problem at all to use a socket where you'd use a file. Unfortunately, this only works because the abstraction handles it well. You can't really do custom things with file descriptors, so the amount of useful things you can do by treating sockets as files is quite limited (though it certainly exists.) (I was going to say "TLS for example" though come to think of it this isn't even strictly true under Linux; https://docs.kernel.org/networking/tls.html https://docs.kernel.org/networking/tls.html ) Still, having TCP sockets and files at the same abstraction layer in general isn't all that bad; you should consider that writes to the filesystem could fail and do take time when blocking. When done right, this makes it much easier for apps to become transparent to the network and other media when it should be possible.
- jcranmer 4y ago
  > (I was going to say "TLS for example" though come to think of it this isn't even strictly true under Linux; https://docs.kernel.org/networking/tls.html https://docs.kernel.org/networking/tls.html ) TLS isn't that much harder to handle than regular TCP sockets--you're going to need a socket interface that lets you get reader and writer streams, and the extra roundtrips you need for negotiation are handled in the constructor for the TLS socket. It is more difficult if you want to support more advanced features of TLS, or especially if you want to support something like STARTTLS (negotiate TLS on an already-open socket). But this is already kind of true for sockets in general: the reader/writer abstraction breaks down relatively quickly if you need to do anything smarter than occasionally-flushed streams.
  - jchw 4y ago
    I think my original post was fairly unclear of what I meant. See, what I meant was like this. File descriptors are an OS abstraction; the "backends" you get are defined in the kernel. You can't really do custom behavior with FDs; for example, in Go, the Go TLS library can open a connection and return an io.Writer, and when you write, it will be symmetrically encrypted, transparent to you, as if the code spoke TLS. But when you're dealing with raw file descriptors, and the read and write syscalls, there's no way to make 'custom' read and write handlers, like you can with programming language abstractions. (I do acknowledge that you could in fact do some of this with pipes, though I have seldom seen it used this way outside of shell programming. It's kinda missing the point, anyway, since pipes are just another type of fd. You can do pipes on top of Go's abstraction too, but it would be very cumbersome in comparison.) But as a kind of quirk, Linux actually _does_ support TLS sockets. It will not do the handshake, so you still have to do that in userspace. But if you use the Linux kernel's TLS socket support, it will in fact give you an FD that you can read from and write to directly with transparency, as if it was any other file or socket; you don't have to handle the symmetric cryptography bits or use a separate interface. I think this is rather neat, although I'm not sure how practically useful it is. (Presumably it would be useful for hardware offloading, at least.)
hinkley 4y ago
The first bit of real code I ever wrote was an SO_LINGER bug fix for a game that couldn’t restart if users had disconnected due to loss of network. Then I had to explain it to several other people who had the same problems. Seems a lot of copy and paste went on among that community.
- CamperBob2 4y ago
  If that was truly the first bit of real code you wrote, it's pretty darned impressive. Comparable to learning to swim by jumping in at the deep end of a pool that's full of sharks, razor blades, and coliform bacteria.
  - hinkley 4y ago
    I was proud of it, but really homework assignments tend to be toy problems. As complex as they can get (which really isn’t that complex) they don’t do anything. It was a known problem, with snippets floating about. Understanding what it did and why it should work took a bit more. Then again, the first technical book I read for myself instead of for class what’s the Comer TCP/IP books, which hardly anybody did at the time and definitely nobody does now.
gwbas1c 4y ago
> If I send() a message, I have no guarantees that the other machine will recv() it if it is suddenly disconnected from the network. Again, this may be obvious to an experienced network programmer, but to an absolute beginner like me it was not. Uhh, that's 100% obvious. That's why it's not taught. Of course, I've also goofed on things that are obvious to other people. But, com'on, TCP isn't magic.
- tptacek 4y ago
  It's not 100% obvious. The mental model where send() blocks until recv() on the other side confirms it is coherent: the receiver sends ACKs with bumped ack numbers to acknowledge the bytes it's received, and could delay those ACKs until the application has taken the bytes out of the socket buffer. It doesn't work that way, of course, and shouldn't, but it could.
- hinkley 4y ago
  If it were so obvious we wouldn’t have so many concurrency bugs that appear time and again in new programs. If it’s not network flush it’s file system flush.
- jsmith45 4y ago
  > Uhh, that's 100% obvious. That's why it's not taught. Is it? You probably feel that way from knowing how TCP works. But it would be quite straightforward to make it true with a slightly modified version of TCP (that acknowledges all packets, rather than every second one) by having send() block until it receives back the ACK from the receiver. (Yeah this would kill transmission rates, but it would function!) And furthermore, while it would be a terrible idea, the ACK could even be delayed until the end application makes the recv() call for the packet. To somebody not familiar with the details (and why this would be a terrible tradeoff), something along those lines would be entirely plausible.
  - roelschroeven 4y ago
    Especially since TCP is often introduced like "TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network" or similar formulations (this one comes from Wikipedia). "Reliable" does seem to imply that all data sent is also received. It's of course impossible to guarantee that. Actually I think it's even a mistake to call TCP reliable. It's best-effort, and allows you to detect disconnects. That's all it does because that's all it can do.
10000truths 4y ago
Perhaps there should exist a flag for send() that would make it so that it doesn't return until all data in the send() call has been ACKed by the receiving side (with a user configurable timeout). Of course, it's still not bulletproof. The other side can receive the packets, stuff them in its receive buffer, send an ACK for those packets, and then fail before draining the receive buffer due to an OS crash or hardware failure. But computers and operating systems tend to be much more reliable than networks, so it would still provide a much stronger guarantee of delivery or failure.
- tenebrisalietum 4y ago
  There's a difference between ACK of the remote network stack (yeah, we got your packets, they're waiting in line) and ACK of the application (yeah, app X processed your requests composed of 1 or more packets) Compare with the classic OS optimization for spinning rust hard drives - write system calls will return immediately, but actual write requests to the hardware will occur sometime later. It's assumed most of the time your computer doesn't lose power, but that does happen sometimes, hence journaling.
- noselasd 4y ago
  Much stronger but of limited use still, for the reasons you listed. You very rarely care that the remote tcp/ip stack has acked the message, you care that the messages has been received by the program and processed - You're better off implementing your own acks in those cases, allowing you to report back any errors in that ack as well. Or you don't really care, and can just fire and forget those messages. And that also allows you to implement a system where you can pipeline messages - waiting for remote acks when allowing just 1 message in flight limits your throughput severely.
- spc476 4y ago
  UDP packets aren't ACKed by default---it's "best effort" (or "fire and forget," depending upon your view point).
Nezghul 4y ago
I very often see "reconnect loops" in various codebases and I wonder are they necessary? Wouldn't the same effect be achieved by for example increasing timeouts or some other connection parameter?
- pravus 4y ago
  For TCP the state required to maintain the socket in the kernel is invalidated on error and needs to be reset. The only way to do this is to explicitly perform the connection setup again. An extended timeout only delays this process since the remote side will have invalidated its state as well. UDP packets require no connection but you still might see some sort of re-synchronization code to reset application state which could be called "reconnect".
- pgorczak 4y ago
  They’re a bit of a feature of the connection-oriented nature of TCP as the other reply mentions. If the server process crashes and restarts for example, the client will be told that its previous connection is not valid anymore. Basically TCP lets client and server assume that all bytes put into the socket after connect()/accept() will end up at the other side in that same order. Each time there is an error that violates that assumption, the connection needs to be explicitly “reset”.
neilobremski 4y ago
This article may as well have been about writing to file streams does not block until a read occurs. Maybe the documentation (on sockets?!) could be more clear but at some point more words don't help with conceptual understanding.
larsonnn 4y ago
The title is confusing. I did learn it that way…
deathanatos 4y ago
> How you handle a file no longer existing vs. a socket disconnection are not likely to be very similar. I'm sure I'll get counter arguments to this, That's my cue! I think the "everything is a file" is somewhat misunderstood. I might even rephrase it as "everything is a file descriptor" first, but then if you need to give a name to it, or ACL that thing, that's what the file-system is for: that all the objects share a common hierarchy, and different things that need names can be put in say, the same directory. I.e., that there is one hierarchy, for pipes, devices, normal files, etc. I'd argue that the stuff that "is" a file (named or anonymous) or a file descriptor is actually rather small, and most operations are going to require knowing what kind of file you have. E.g., in Linux, read()/write() behave differently when operating on, e.g., an eventfd, read() requires a buffer of at least 8 B. Heck, "close" might really be the only thing that makes sense, generically. (I'd also argue that a (fallible¹) "link" call should be in that category, but alas, it isn't on Linux, and potentially unlink for similar reasons — i.e., give and remove names from a file. I think this is a design mistake personally, but Linux is what it is. What I think is a bigger mistake is having separate/isolated namespaces for separate types of objects. POSIX, and hence Linux, makes this error in a few spots.) But if you're just writing a TCP server, yeah, that probably doesn't matter. > and that you should write your applications to treat these the same. But I wouldn't argue that. A socket likely is a socket for some very specific purpose (whatever the app is, most likely), and that specific purpose is going mean it will be treated as that. In OOP terms, just b/c something might have a base class doesn't mean we always treat it as only an instance of the base class. Sometimes the stuff on the derived class is very important to the function of the program, and that's fine. ¹e.g., in the case of an anonymous normal file (an unlinked temp file, e.g., O_TMPFILE), attempting to link it on a different file-system would have to fail.
opportune 4y ago
If I could make a meta point, nothing about software is magic (except the fact it works at all). Computers do exactly as they are told and nothing more. If you’re using some underlying technology you should really know how it works at some level so you can understand what assumptions you can make and which you can’t. TCP doesn’t mean there is never data loss.
- dylan604 4y ago
  Except when literal bugs short out your system, or actual cosmic rays bit flip your data, or any plethora of other things that can cause your "correctly written" code to not function as expected. That's when the magicians come out and do their thing
Uptrenda 4y ago
It seems like the OP is mostly talking about 'blocking' sockets. Such sockets return when they're ready or there's an error. So send returns when its passed off its data to the network buffer (or if its full it will wait until it can pass off SOME data.) You might think that sounds excellent - but from memory send may not send all of the bytes you pass to it. So if you want to send out all of a given buffer with blocking sockets - you really need to write a loop that implements send_all with a count of the amount of bytes sent or quit on error. Blocking sockets are kind of shitty, not gonna lie. The counterpart to send is recv. Say you send a HTTP request to a web server and you want to get a response. With a blocking socket its quite possible that your application will simply wait forever. I am p sure the default 'timeout' for blocking sockets is 'None' so it just waits for success or failure. So a shitty web server can make your entire client hang. So how to solve this? Well, you might try setting a 'timeout' for blocking operations but this would also screw you. Any thread that calls that blocking operation is going to hang that entire time. Maybe that is fine for you -- should you design program to be multi-threaded and pass off sockets so they can wait like that -- and that is one such solution. Another solution -- and this is the one the OP uses -- is to use the 'select' call to check to see if a socket operation will 'block.' I believe it works on 'reading' and 'writing.' But wait a minute. Now you've got to implement some kind of performant loop that periodically does checks on your sockets. This may sound simple but its actually the subject of whole research projects to try build the most performant loops possible. Now we're really talking about event loops here and how to build them. So how to solve this... today... for real-world uses? Most people are just going to want to use asynchronous I/O. If you've never worked with async code before: its a way of doing event-based programming where you can suspend execution of a function if an event isn't ready. This allows other functions to 'run.' Note that this is a way to do 'concurrency' -- or switch between multiple tasks. A good async library may or may not also be 'parallel' -- the ability to execute functions simultaneously (like on multiple cores.) If we go back to the idea of the loop and using 'select' on our socket descriptors. This is really like a poor-persons async event loop. It can easily be implemented in a single thread, in a single core. But again -- for modern applications -- you're going to want to stay away from using the socket functions and go for async I/O instead. One last caveat to mention: Network code needs to be FAST. Not all software that we write needs to run as fast as possible. That's just a fact and indeed many warn against 'premature optimization.' I would say this advice doesn't bode well for network code. It's simply not acceptable to write crappy algorithms that add tens of milliseconds or nanoseconds to packet delivery time if you can avoid it. It can actually add up to costs a lot of money and make certain applications impossible. The thing is though -- profiling async code can be hard -- profiling network code even harder. A network is unreliable and to measure run-time of code you only care about how the code performs when its successful. So you're going to want to find tools that let you throw away erroneous results and measure how long 'coroutines' actually run for. Async network code may underlyingly use non-blocking sockets, select, and poll. But they are designed to be as efficient as possible. So if you have access to using them its probably what you want to use!
neonroku 4y ago
The following sentence in the article jumped out at me: “The difference is that you are not dealing with the unreliability of UDP like TCP is.” This reads to me like TCP is built on top of UDP, which at one time I thought to be the case, but it’s not. UDP and TCP are both transport layer, built on the internet layer, which is unreliable.
deleted 4y ago
[deleted]
quickthrower2 4y ago
Regarding his original expectations of TCP. Even if true, is there much difference between dropped data and data being delivered hours late? I imagine at an app level you would suspect any message that got sent 12 hours ago but kept in a queue. I imagine if that scenario is Ok you would explicitly use a queue system.
enriquto 4y ago
> How you handle a file no longer existing vs. a socket disconnection are not likely to be very similar. Why not? It seems that fopen(3) and fread(3) provide the perfect abstraction for that. The semantics to remove(3) a file that is open are very clear, and they represent exactly what you want to happen when a connection is lost. I never understood the need for "sockets" as a separate type of file. Why can't they be just be exposed as regular files?
dimman 4y ago
Interesting read. I’m quite curious of where all the initial misperceptions about sockets comes from. I can highly recommend Beej’s guide to network programming: https://beej.us/guide/bgnet/ https://beej.us/guide/bgnet/ That together with Linux/BSD man pages should be everything needed, some great documentation there.
- pgorczak 4y ago
  I definitely used to think TCP was more “high-level” than it actually is. Yes it does much more than UDP but still, its job is to get a sequence of bytes from A to B. You can tune it for higher throughput or more sensitive flow control but anything concerning message passing, request/response, … is beyond the scope of TCP.
  - dimman 4y ago
    Sure, but from a "high level" or "sockets" perspective, especially as a beginner it shouldn't be something you need to care about. A bit simplified, the basic stuff you need to know is: 1) UDP uses packages/messages which may or may not reach its destination. If it reaches its destination the data is intact. Normally connectionless. 2) TCP is a stream protocol. There is no package/message boundary unless you create it yourself (my tip is to do a simple binary TLV (type length value) protocol using say a fixed 4 byte header). Requires a connection to be setup first. 3) Network byte order - really important to read about. 4) Nagles algorithm (TCP_NODELAY) and SO_KEEPALIVE - those are a couple of things to read about. 5) Start with the simple select() approach to handle the socket activity. You can then go ahead and get more advanced by doing nonblocking I/O or do blocking I/O with each client in its own thread, figuring out pros and cons for your use case. You can add SSL/TLS on top of your TCP connection etc. EDIT: The SO_KEEPALIVE part is perhaps least important thing to start reading about. I'm a bit biased due to NAT traversal problems as I wrote a secure remote access solution for a major company several years back, utilising STUN/TURN servers, public key authentication (basically certificate pinning), TLS etc.
    - pgorczak 4y ago
      Yes and even at 2) some subtleties start. You can set up a connection, send a chunk of bytes, and close it. If you reach a clean connection close, you can be sure that all your bytes have reached the other side. As soon as you start sending multiple logical messages over a persistent connection and an error occurs, you need to write application logic to figure out where to pick up again after you reconnect. Even if you want to know which parts of your stream have already reached the other side, you need to add logic for that. This “multiple transactions over a persistent connection” may sound really straightforward but it’s not built into TCP itself.
throwaway2037 4y ago
I have two technical questions after reading this article. 1) The author writes: <<I expected that I would call e.g. isconnected() on the sockets after accept() tells me something happened to it.>> I cannot find any man page for this function. Google also fails me. Is this a hypothetical function or a non-POSIX call? 2) The author writes: <<On Linux, I also needed to disable signaling on the recv() so that I could handle the connection error inline rather than need to register a signal handler. I opted to add the MSG_NOSIGNAL to both send() and recv() and handle potential disconnect errors at each call.>> I checked the Linux man page for recv(). The flag MSG_NOSIGNAL does not exist (at least in my version). However, send() supports it. Do I misunderstand?
- makuto 4y ago
  Author here! Isconnected is a hypothetical.
bschwindHN 4y ago
Here's a fun one: TCP sockets can connect to themselves if you bind to an ephemeral port: https://sgros.blogspot.com/2013/08/tcp-client-self-connect.html https://sgros.blogspot.com/2013/08/tcp-client-self-connect.h...
solomatov 4y ago
Many of the problems with TCP are solved by QUIC: https://www.chromium.org/quic/ https://www.chromium.org/quic/ For example, it handles reconnect for you (it's based on UDP which is connectionless), it even survives IP address change, multiple streams over one connections and other thigns. There're several high quality libraries which you could use in your code.
vecter 4y ago
> I found it very confusing that I had to attempt to recv() from a socket and fail in order to even tell that the connection was no longer active. I expected that I would call e.g. isconnected() on the sockets after accept() tells me something happened to it. It does make sense to me now that it's better to have recv() fail and tell me about the disconnect. Otherwise, I might mistakenly assume that if I call isconnected() I am then guaranteed to have a good recv(). By keeping the disconnect tied to recv() failing, I know I need to handle potential disconnects at any recv() invocation. The same goes for send(). After thinking about it for a second, it makes sense since any check of isconnected() followed by a send() or recv() is subject to a race condition of the socket failing after the call to isconnected() but before the subsequent calls. You can never know without trying to send or receive, and even those may only fail "in the middle" of the transmission.
- spacechild1 4y ago
  I think that's exactly what the second half of the quoted paragraph is trying to say.
jeffffff 4y ago
"End-to-End Arguments in System Design" (https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf https://web.mit.edu/Saltzer/www/publications/endtoend/endtoe...) is a must read when it comes to application reliability and network programming
deleted 4y ago
[deleted]

412 points by zdw 4y ago

In the early 90s I was working on AppleTalk-PC interoperability SW, which included "ATP", the AppleTalk Transaction Protocol, which is kind of a guaranteed delivery protocol. It even had "at least once" and "exactly once" options to make sure state wouldn't get messed up in client-server applications: https://developer.apple.com/library/archive/documentation/mac/pdf/Networking/ATP.pdf https://developer.apple.com/library/archive/documentation/ma... I was kind of sad to see that things like this fell by the wayside in favor of streaming protocols but of course it's way more efficient to not handle each transaction individually. The problem of users implementing TCP++ on top of TCP is a real problem, I think.

NegativeLatency 4y ago
Was there a driver or library for ATP or were you implementing that on PCs?
- zwieback 4y ago
  No, we were the ones writing the PC drivers, for DOS, Windows and OS/2. Back then the lower layers were more or less standardized but there was no AppleTalk stack so that's what we developed as a product.
bombela 4y ago
I just want to point out that "exactly-once" is not physically possible. The documentation claims "exactly-once", and proceeds to explain that after a while, it will timeout. Which makes it "at-most-once", as expected. As far as I know; in communication; you can only have at-least-once (1..N) or at-most-once (0..1).
- lilyball 4y ago
  According to the docs, the transaction release timeout is specified by the requester. The requester also specifies the duration to wait between retries and the maximum number of retries. So this does indeed appear to be exactly-once as long as the sender ensures the transaction release timeout is greater than the total time (retry timeout * max retries) plus any network delay.
- erichocean 4y ago
  > As far as I know; in communication; you can only have at-least-once (1..N) or at-most-once (0..1). If you hash-chain all of your messages, you can trivially and reliably have exactly-once semantics. Just like a blockchain. Each message includes its predecessor hash, so if you get a message that you don't have the predecessor for, you can just request it. ACKs become "this is the latest message hash for which I have all of the predecessors" and can be done lazily, i.e. every nth message, or after not receiving a message in awhile. Source: Me, built a system that worked like this in 2014 (over UDP). Worked great, it had some similarities to how QUIC is designed (including the crypto).

dehrmann 4y ago

One of the big issues with TCP is a lot of communication isn't well-suited to the stream model and is message-oriented, so yes, like the author says, you have to go implement ACKs. And then you want multiplexing, which streams also fail at. Before you know it, you built a worse version of HTTP2.

iTokio 4y ago
And then you realize that sometimes it has become even worse due to tcp head of line blocking and you move to udp and build a worse version of http3
- hinkley 4y ago
  I think Head of Line blocking is a more concrete problem but it seems that it's often encountered when running away from a different one: If you have different conversations going on between a single client and the server, you can make several connections to do it. You'll pay a little bit more at the start of the conversation of course, so be frugal with this design, and think about ways to 'hide' the delay during bootstrapping. But know that with funky enough network conditions, it's possible for one connection to error out but the other to continue to work. The problem for me always comes back to information architecture, and the pervasive lack of it, or at least lack of investment in it. If you really have two separate conversations with the server, losing one channel shouldn't make the whole application freak out. But we all know people take the path of least resistance, and soon you have 2.5 conversations on one channel and 1.5 on the other. The advice here is some of the same advice in more sophisticated treatises on RESTful APIs (it's all distributed computing, it's all the same problem set arranged in different orders). REST calls are generally supposed to be stateless. The client making what looks like a non sequitur call to the backend should Just Work. If you manage that, then having one channel inform the client of the existence of a resource and fetching that resource out of band isn't really coupling of the two channels. That subtlety is lost on some people who defend their actual coupling by pointing out that other people have done it too, when no, they really haven't. And if anything, multiplexing lets them get away with this bad behavior for much longer, allowing the bad patterns to become idiomatic.
- 4y ago

andai 4y ago

I played around with making a little websocket layer for browser games a while back. I used a Windows tool called "clumsy" to simulate crappy network conditions and was surprised at how poor the results were with just WebSockets, despite all the overhead of being a protocol on top of a protocol. The result is that you need to build a protocol on top of the protocol on top of the protocol if you actually want your messages delivered...

tuukkah 4y ago
That's because WebSockets are more or less just sockets for web apps. You'lll want to use a protocol that deals with messages and their at-least-once delivery, such as MQTT (that can run on top of WebSocket if you need it).
swagasaurus-rex 4y ago
I built a javascript data synchronization library specifically for games https://github.com/siriusastrebe/jsynchronous https://github.com/siriusastrebe/jsynchronous A core part of the library is the ability to re-send data that's been lost due to connection interruption. Absolutely crucial for ensuring data is properly synchronized.

psyc 4y ago

I'm pleased to hear that anyone is still teaching anyone anything about sockets. The bit about Windows not treating sockets as files made me pause, since Windows does treat so many things as files. After thinking about it some, I suppose it's kernel32 that treats kernel32 things as files. Winsock has a separate history.

amluto 4y ago
Windows has a long and unfortunate history of encouraging programs to extend system behavior by injecting libraries into other programs’ address spaces. You can look up Winsock LSPs and Win32 hooks [0] for examples. This means that programs cannot rely on the public APIs actually interacting with the kernel in the way one would natively imagine — the implementation may be partially replaced with a call into user code from another vendor in the same progress. Eww! So, as I recall, a normal socket is a kernel object, but a third party LSP-provided socket might not be. This also means that any API, e.g. select, that may interact with more than one socket at once has problems. [1] [0] https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwindowshookexa https://docs.microsoft.com/en-us/windows/win32/api/winuser/n... — see the remarks section. [1] https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-winsock2-select https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/... — again, see the remarks.
xeeeeeeeeeeenu 4y ago
It's complicated. Sockets are mostly interchangeable with filehandles but there are many exceptions. For example, ReadFile() works with sockets, whereas DuplicateHandle() silently corrupts them. However, there's another problem: overlapped vs non-overlapped handles. socket() always creates overlapped sockets, while WSASocket() can create both types. Overlapped handles can't be read synchronously with standard APIs, which in turn means you can't read() a fd created from an overlapped handle. Naturally, in their infinite wisdom, Windows designers decided there's no need to provide an official API to introspect handles, so there's no documented way to tell them apart (there are unofficial ways, though). BTW, it's a proof of poor communication between teams in Microsoft, because their C runtime (especially its fd emulation) would greatly benefit from such an API. It's frustrating. I'm sure that if Windows was an open-source project, that mess would be fixed immediately.

weeksie 4y ago

Ah man, sockets. Real sockets. Yesterday's kiosk thread jogged my memory about all that but almost all of the network communication on my old kiosk installations was socket based. It was a pleasant way to work.

TYPE_FASTER 4y ago

What really helped me understand and troubleshoot network communications at the socket level was the book by W. Richard Stevens[0]. I think this is because he starts with an introduction to TCP, and builds from there. Knowing TCP states and transitions is important to reading packet dumps, and in general debugging networking. Also, no matter what platform you're using, there is probably a socket implementation somewhere at the core. You're best off understanding how sockets work, then understanding how the platform you're working with uses sockets[1]. Once I read the W. Richard Stevens book, I was able to read and understand RFCs for protocols like HTTP to know how things should work. Then you're better prepared to figure out if a behavior is due to your code, or an implementation of the protocol in your device, or the device on the other end of the network connection, or some intermediary device. [0] - http://www.kohala.com/start/unpv12e.html http://www.kohala.com/start/unpv12e.html [1] - https://docs.microsoft.com/en-us/windows/win32/winsock/porting-socket-applications-to-winsock https://docs.microsoft.com/en-us/windows/win32/winsock/porti...

hinkley 4y ago
There were many authors in that era who had practically compulsory books. Stevens is one of the few to have two. His Unix Programming book was referred to as the Bible.
- sophacles 4y ago
  I still refer back to APUE often and push my junior folks to read it.
- Agingcoder 4y ago
  The Linux programming interface, by Michael Kerrisk, is an excellent modern update.
lanstin 4y ago
We had to debug an issue with ARP during a deploy of docker containers a year or two ago. A nearly complete understanding of the lower layers is not that much knowledge and quite useful at times.

gabereiser 4y ago

This was a good write up but was kind of short of details. It would have been really awesome with some code examples. I love socket programming. Love, like I love going to the dentist. Having a state machine to handle basic I/O between client/server kind of blows when it comes to plenko machine switch statements of arbitrary state enums. Languages like Go or Python allow you to have a way of communicating with the client from the server in a more direct client->server oriented way. Write/Read input, do thing, write, read, do, repeat until you finish ops. Go is my favorite for this as I can spawn a goroutine to read/write to a socket. Rust is my second favorite for this but it's a bit trickier. Python's twisted framework has something like this with Protocols. I wish C++ had a standard socket implementation (std::network?) Anyway, this gave me a smile today so thanks.

jchw 4y ago

> I will also ask: how often are you writing applications that want to accept either files or sockets? In Go where there is an io.Reader/io.Writer abstraction where blocking interactions are OK and you're absolutely intended to handle all of the errors, it's really no problem at all to use a socket where you'd use a file. Unfortunately, this only works because the abstraction handles it well. You can't really do custom things with file descriptors, so the amount of useful things you can do by treating sockets as files is quite limited (though it certainly exists.) (I was going to say "TLS for example" though come to think of it this isn't even strictly true under Linux; https://docs.kernel.org/networking/tls.html https://docs.kernel.org/networking/tls.html ) Still, having TCP sockets and files at the same abstraction layer in general isn't all that bad; you should consider that writes to the filesystem could fail and do take time when blocking. When done right, this makes it much easier for apps to become transparent to the network and other media when it should be possible.

jcranmer 4y ago
> (I was going to say "TLS for example" though come to think of it this isn't even strictly true under Linux; https://docs.kernel.org/networking/tls.html https://docs.kernel.org/networking/tls.html ) TLS isn't that much harder to handle than regular TCP sockets--you're going to need a socket interface that lets you get reader and writer streams, and the extra roundtrips you need for negotiation are handled in the constructor for the TLS socket. It is more difficult if you want to support more advanced features of TLS, or especially if you want to support something like STARTTLS (negotiate TLS on an already-open socket). But this is already kind of true for sockets in general: the reader/writer abstraction breaks down relatively quickly if you need to do anything smarter than occasionally-flushed streams.
- jchw 4y ago
  I think my original post was fairly unclear of what I meant. See, what I meant was like this. File descriptors are an OS abstraction; the "backends" you get are defined in the kernel. You can't really do custom behavior with FDs; for example, in Go, the Go TLS library can open a connection and return an io.Writer, and when you write, it will be symmetrically encrypted, transparent to you, as if the code spoke TLS. But when you're dealing with raw file descriptors, and the read and write syscalls, there's no way to make 'custom' read and write handlers, like you can with programming language abstractions. (I do acknowledge that you could in fact do some of this with pipes, though I have seldom seen it used this way outside of shell programming. It's kinda missing the point, anyway, since pipes are just another type of fd. You can do pipes on top of Go's abstraction too, but it would be very cumbersome in comparison.) But as a kind of quirk, Linux actually _does_ support TLS sockets. It will not do the handshake, so you still have to do that in userspace. But if you use the Linux kernel's TLS socket support, it will in fact give you an FD that you can read from and write to directly with transparency, as if it was any other file or socket; you don't have to handle the symmetric cryptography bits or use a separate interface. I think this is rather neat, although I'm not sure how practically useful it is. (Presumably it would be useful for hardware offloading, at least.)

hinkley 4y ago

The first bit of real code I ever wrote was an SO_LINGER bug fix for a game that couldn’t restart if users had disconnected due to loss of network. Then I had to explain it to several other people who had the same problems. Seems a lot of copy and paste went on among that community.

CamperBob2 4y ago
If that was truly the first bit of real code you wrote, it's pretty darned impressive. Comparable to learning to swim by jumping in at the deep end of a pool that's full of sharks, razor blades, and coliform bacteria.
- hinkley 4y ago
  I was proud of it, but really homework assignments tend to be toy problems. As complex as they can get (which really isn’t that complex) they don’t do anything. It was a known problem, with snippets floating about. Understanding what it did and why it should work took a bit more. Then again, the first technical book I read for myself instead of for class what’s the Comer TCP/IP books, which hardly anybody did at the time and definitely nobody does now.

gwbas1c 4y ago

> If I send() a message, I have no guarantees that the other machine will recv() it if it is suddenly disconnected from the network. Again, this may be obvious to an experienced network programmer, but to an absolute beginner like me it was not. Uhh, that's 100% obvious. That's why it's not taught. Of course, I've also goofed on things that are obvious to other people. But, com'on, TCP isn't magic.

tptacek 4y ago
It's not 100% obvious. The mental model where send() blocks until recv() on the other side confirms it is coherent: the receiver sends ACKs with bumped ack numbers to acknowledge the bytes it's received, and could delay those ACKs until the application has taken the bytes out of the socket buffer. It doesn't work that way, of course, and shouldn't, but it could.
hinkley 4y ago
If it were so obvious we wouldn’t have so many concurrency bugs that appear time and again in new programs. If it’s not network flush it’s file system flush.
jsmith45 4y ago
> Uhh, that's 100% obvious. That's why it's not taught. Is it? You probably feel that way from knowing how TCP works. But it would be quite straightforward to make it true with a slightly modified version of TCP (that acknowledges all packets, rather than every second one) by having send() block until it receives back the ACK from the receiver. (Yeah this would kill transmission rates, but it would function!) And furthermore, while it would be a terrible idea, the ACK could even be delayed until the end application makes the recv() call for the packet. To somebody not familiar with the details (and why this would be a terrible tradeoff), something along those lines would be entirely plausible.
- roelschroeven 4y ago
  Especially since TCP is often introduced like "TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network" or similar formulations (this one comes from Wikipedia). "Reliable" does seem to imply that all data sent is also received. It's of course impossible to guarantee that. Actually I think it's even a mistake to call TCP reliable. It's best-effort, and allows you to detect disconnects. That's all it does because that's all it can do.

10000truths 4y ago

Perhaps there should exist a flag for send() that would make it so that it doesn't return until all data in the send() call has been ACKed by the receiving side (with a user configurable timeout). Of course, it's still not bulletproof. The other side can receive the packets, stuff them in its receive buffer, send an ACK for those packets, and then fail before draining the receive buffer due to an OS crash or hardware failure. But computers and operating systems tend to be much more reliable than networks, so it would still provide a much stronger guarantee of delivery or failure.

tenebrisalietum 4y ago
There's a difference between ACK of the remote network stack (yeah, we got your packets, they're waiting in line) and ACK of the application (yeah, app X processed your requests composed of 1 or more packets) Compare with the classic OS optimization for spinning rust hard drives - write system calls will return immediately, but actual write requests to the hardware will occur sometime later. It's assumed most of the time your computer doesn't lose power, but that does happen sometimes, hence journaling.
noselasd 4y ago
Much stronger but of limited use still, for the reasons you listed. You very rarely care that the remote tcp/ip stack has acked the message, you care that the messages has been received by the program and processed - You're better off implementing your own acks in those cases, allowing you to report back any errors in that ack as well. Or you don't really care, and can just fire and forget those messages. And that also allows you to implement a system where you can pipeline messages - waiting for remote acks when allowing just 1 message in flight limits your throughput severely.
spc476 4y ago
UDP packets aren't ACKed by default---it's "best effort" (or "fire and forget," depending upon your view point).

Nezghul 4y ago

I very often see "reconnect loops" in various codebases and I wonder are they necessary? Wouldn't the same effect be achieved by for example increasing timeouts or some other connection parameter?

pravus 4y ago
For TCP the state required to maintain the socket in the kernel is invalidated on error and needs to be reset. The only way to do this is to explicitly perform the connection setup again. An extended timeout only delays this process since the remote side will have invalidated its state as well. UDP packets require no connection but you still might see some sort of re-synchronization code to reset application state which could be called "reconnect".
pgorczak 4y ago
They’re a bit of a feature of the connection-oriented nature of TCP as the other reply mentions. If the server process crashes and restarts for example, the client will be told that its previous connection is not valid anymore. Basically TCP lets client and server assume that all bytes put into the socket after connect()/accept() will end up at the other side in that same order. Each time there is an error that violates that assumption, the connection needs to be explicitly “reset”.

neilobremski 4y ago

This article may as well have been about writing to file streams does not block until a read occurs. Maybe the documentation (on sockets?!) could be more clear but at some point more words don't help with conceptual understanding.

larsonnn 4y ago

The title is confusing. I did learn it that way…

deathanatos 4y ago

> How you handle a file no longer existing vs. a socket disconnection are not likely to be very similar. I'm sure I'll get counter arguments to this, That's my cue! I think the "everything is a file" is somewhat misunderstood. I might even rephrase it as "everything is a file descriptor" first, but then if you need to give a name to it, or ACL that thing, that's what the file-system is for: that all the objects share a common hierarchy, and different things that need names can be put in say, the same directory. I.e., that there is one hierarchy, for pipes, devices, normal files, etc. I'd argue that the stuff that "is" a file (named or anonymous) or a file descriptor is actually rather small, and most operations are going to require knowing what kind of file you have. E.g., in Linux, read()/write() behave differently when operating on, e.g., an eventfd, read() requires a buffer of at least 8 B. Heck, "close" might really be the only thing that makes sense, generically. (I'd also argue that a (fallible¹) "link" call should be in that category, but alas, it isn't on Linux, and potentially unlink for similar reasons — i.e., give and remove names from a file. I think this is a design mistake personally, but Linux is what it is. What I think is a bigger mistake is having separate/isolated namespaces for separate types of objects. POSIX, and hence Linux, makes this error in a few spots.) But if you're just writing a TCP server, yeah, that probably doesn't matter. > and that you should write your applications to treat these the same. But I wouldn't argue that. A socket likely is a socket for some very specific purpose (whatever the app is, most likely), and that specific purpose is going mean it will be treated as that. In OOP terms, just b/c something might have a base class doesn't mean we always treat it as only an instance of the base class. Sometimes the stuff on the derived class is very important to the function of the program, and that's fine. ¹e.g., in the case of an anonymous normal file (an unlinked temp file, e.g., O_TMPFILE), attempting to link it on a different file-system would have to fail.

opportune 4y ago

If I could make a meta point, nothing about software is magic (except the fact it works at all). Computers do exactly as they are told and nothing more. If you’re using some underlying technology you should really know how it works at some level so you can understand what assumptions you can make and which you can’t. TCP doesn’t mean there is never data loss.

dylan604 4y ago
Except when literal bugs short out your system, or actual cosmic rays bit flip your data, or any plethora of other things that can cause your "correctly written" code to not function as expected. That's when the magicians come out and do their thing

Uptrenda 4y ago

It seems like the OP is mostly talking about 'blocking' sockets. Such sockets return when they're ready or there's an error. So send returns when its passed off its data to the network buffer (or if its full it will wait until it can pass off SOME data.) You might think that sounds excellent - but from memory send may not send all of the bytes you pass to it. So if you want to send out all of a given buffer with blocking sockets - you really need to write a loop that implements send_all with a count of the amount of bytes sent or quit on error. Blocking sockets are kind of shitty, not gonna lie. The counterpart to send is recv. Say you send a HTTP request to a web server and you want to get a response. With a blocking socket its quite possible that your application will simply wait forever. I am p sure the default 'timeout' for blocking sockets is 'None' so it just waits for success or failure. So a shitty web server can make your entire client hang. So how to solve this? Well, you might try setting a 'timeout' for blocking operations but this would also screw you. Any thread that calls that blocking operation is going to hang that entire time. Maybe that is fine for you -- should you design program to be multi-threaded and pass off sockets so they can wait like that -- and that is one such solution. Another solution -- and this is the one the OP uses -- is to use the 'select' call to check to see if a socket operation will 'block.' I believe it works on 'reading' and 'writing.' But wait a minute. Now you've got to implement some kind of performant loop that periodically does checks on your sockets. This may sound simple but its actually the subject of whole research projects to try build the most performant loops possible. Now we're really talking about event loops here and how to build them. So how to solve this... today... for real-world uses? Most people are just going to want to use asynchronous I/O. If you've never worked with async code before: its a way of doing event-based programming where you can suspend execution of a function if an event isn't ready. This allows other functions to 'run.' Note that this is a way to do 'concurrency' -- or switch between multiple tasks. A good async library may or may not also be 'parallel' -- the ability to execute functions simultaneously (like on multiple cores.) If we go back to the idea of the loop and using 'select' on our socket descriptors. This is really like a poor-persons async event loop. It can easily be implemented in a single thread, in a single core. But again -- for modern applications -- you're going to want to stay away from using the socket functions and go for async I/O instead. One last caveat to mention: Network code needs to be FAST. Not all software that we write needs to run as fast as possible. That's just a fact and indeed many warn against 'premature optimization.' I would say this advice doesn't bode well for network code. It's simply not acceptable to write crappy algorithms that add tens of milliseconds or nanoseconds to packet delivery time if you can avoid it. It can actually add up to costs a lot of money and make certain applications impossible. The thing is though -- profiling async code can be hard -- profiling network code even harder. A network is unreliable and to measure run-time of code you only care about how the code performs when its successful. So you're going to want to find tools that let you throw away erroneous results and measure how long 'coroutines' actually run for. Async network code may underlyingly use non-blocking sockets, select, and poll. But they are designed to be as efficient as possible. So if you have access to using them its probably what you want to use!

neonroku 4y ago

The following sentence in the article jumped out at me: “The difference is that you are not dealing with the unreliability of UDP like TCP is.” This reads to me like TCP is built on top of UDP, which at one time I thought to be the case, but it’s not. UDP and TCP are both transport layer, built on the internet layer, which is unreliable.

deleted 4y ago

[deleted]

quickthrower2 4y ago

Regarding his original expectations of TCP. Even if true, is there much difference between dropped data and data being delivered hours late? I imagine at an app level you would suspect any message that got sent 12 hours ago but kept in a queue. I imagine if that scenario is Ok you would explicitly use a queue system.

enriquto 4y ago

> How you handle a file no longer existing vs. a socket disconnection are not likely to be very similar. Why not? It seems that fopen(3) and fread(3) provide the perfect abstraction for that. The semantics to remove(3) a file that is open are very clear, and they represent exactly what you want to happen when a connection is lost. I never understood the need for "sockets" as a separate type of file. Why can't they be just be exposed as regular files?

dimman 4y ago

Interesting read. I’m quite curious of where all the initial misperceptions about sockets comes from. I can highly recommend Beej’s guide to network programming: https://beej.us/guide/bgnet/ https://beej.us/guide/bgnet/ That together with Linux/BSD man pages should be everything needed, some great documentation there.

pgorczak 4y ago
I definitely used to think TCP was more “high-level” than it actually is. Yes it does much more than UDP but still, its job is to get a sequence of bytes from A to B. You can tune it for higher throughput or more sensitive flow control but anything concerning message passing, request/response, … is beyond the scope of TCP.
- dimman 4y ago
  Sure, but from a "high level" or "sockets" perspective, especially as a beginner it shouldn't be something you need to care about. A bit simplified, the basic stuff you need to know is: 1) UDP uses packages/messages which may or may not reach its destination. If it reaches its destination the data is intact. Normally connectionless. 2) TCP is a stream protocol. There is no package/message boundary unless you create it yourself (my tip is to do a simple binary TLV (type length value) protocol using say a fixed 4 byte header). Requires a connection to be setup first. 3) Network byte order - really important to read about. 4) Nagles algorithm (TCP_NODELAY) and SO_KEEPALIVE - those are a couple of things to read about. 5) Start with the simple select() approach to handle the socket activity. You can then go ahead and get more advanced by doing nonblocking I/O or do blocking I/O with each client in its own thread, figuring out pros and cons for your use case. You can add SSL/TLS on top of your TCP connection etc. EDIT: The SO_KEEPALIVE part is perhaps least important thing to start reading about. I'm a bit biased due to NAT traversal problems as I wrote a secure remote access solution for a major company several years back, utilising STUN/TURN servers, public key authentication (basically certificate pinning), TLS etc.
  - pgorczak 4y ago
    Yes and even at 2) some subtleties start. You can set up a connection, send a chunk of bytes, and close it. If you reach a clean connection close, you can be sure that all your bytes have reached the other side. As soon as you start sending multiple logical messages over a persistent connection and an error occurs, you need to write application logic to figure out where to pick up again after you reconnect. Even if you want to know which parts of your stream have already reached the other side, you need to add logic for that. This “multiple transactions over a persistent connection” may sound really straightforward but it’s not built into TCP itself.

throwaway2037 4y ago

I have two technical questions after reading this article. 1) The author writes: <<I expected that I would call e.g. isconnected() on the sockets after accept() tells me something happened to it.>> I cannot find any man page for this function. Google also fails me. Is this a hypothetical function or a non-POSIX call? 2) The author writes: <<On Linux, I also needed to disable signaling on the recv() so that I could handle the connection error inline rather than need to register a signal handler. I opted to add the MSG_NOSIGNAL to both send() and recv() and handle potential disconnect errors at each call.>> I checked the Linux man page for recv(). The flag MSG_NOSIGNAL does not exist (at least in my version). However, send() supports it. Do I misunderstand?

makuto 4y ago
Author here! Isconnected is a hypothetical.

bschwindHN 4y ago

Here's a fun one: TCP sockets can connect to themselves if you bind to an ephemeral port: https://sgros.blogspot.com/2013/08/tcp-client-self-connect.html https://sgros.blogspot.com/2013/08/tcp-client-self-connect.h...

solomatov 4y ago

Many of the problems with TCP are solved by QUIC: https://www.chromium.org/quic/ https://www.chromium.org/quic/ For example, it handles reconnect for you (it's based on UDP which is connectionless), it even survives IP address change, multiple streams over one connections and other thigns. There're several high quality libraries which you could use in your code.

vecter 4y ago

> I found it very confusing that I had to attempt to recv() from a socket and fail in order to even tell that the connection was no longer active. I expected that I would call e.g. isconnected() on the sockets after accept() tells me something happened to it. It does make sense to me now that it's better to have recv() fail and tell me about the disconnect. Otherwise, I might mistakenly assume that if I call isconnected() I am then guaranteed to have a good recv(). By keeping the disconnect tied to recv() failing, I know I need to handle potential disconnects at any recv() invocation. The same goes for send(). After thinking about it for a second, it makes sense since any check of isconnected() followed by a send() or recv() is subject to a race condition of the socket failing after the call to isconnected() but before the subsequent calls. You can never know without trying to send or receive, and even those may only fail "in the middle" of the transmission.

spacechild1 4y ago
I think that's exactly what the second half of the quoted paragraph is trying to say.

jeffffff 4y ago

"End-to-End Arguments in System Design" (https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf https://web.mit.edu/Saltzer/www/publications/endtoend/endtoe...) is a must read when it comes to application reliability and network programming

deleted 4y ago

[deleted]