3 ms·
And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received packa
by eru 2y ago
And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet.
That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet.
For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages.
Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)
- Aurornis 2y ago> And TCP wants to pretend that all your data arrives as a linear stream one after the other; That’s one of the reasons people use TCP. > For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. That’s a lot of complexity, engineering, debugging, and opportunities to introduce bugs, and for what gain? In the file transfer example, what would you actually gain by rolling your own entire transmission protocol with all this overhead? > Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.) Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space.
- eru 2y ago> It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. Yes, and you'd want to mostly build this extra protocol only once, and stick it into a library. > You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. Or you can use a fountain code, and just keep transmitting. > The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. Well, you can also use a simpler mechanism: you transmit all the packages from A to B once, then at the end B tells A which packages get lost, and A sends those again. Repeat until you have everything. That way needs more back-and-forth communication, but doesn't need any fancy error correcting code. > Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space. Yes, the loss model is different. However you can eg interleave your bits to get something that close enough to work. The main reason you don't need to use error correcting codes, is that transmitting feedback to the sender is typically a lot cheaper on the internet than in outer space. Not least because even bad latencies are typically measured in seconds at most, not minutes or hours. (You could however use these codes when for some reason you have a very asymmetrical link. Eg if you have internet via geo-stationary satellite, or if one sender is broadcast to lots and lots of different receivers, and doesn't want to deal with each of them individually.)
- jakjak123 2y ago> And TCP wants to pretend that all your data arrives as a linear stream one after the other; This is actually very nice for an application developer. Probably a tiny bit less nice if you are running popular services taking 100k req/s.
- eru 2y agoYes, that is convenient. But it's an abstraction you don't always need, but still always pay for. That's why I would suggest that were possible the kernel should only expose something at most as high level as UDP, and we build the TCP abstractions on top of that in user level. Of course, you wouldn't want to re-write everything from scratch all the time: you'd use libraries. The nice thing about user level libraries is that you can swap them out for better versions or for different abstractions without any privileged access.