9 ms·

TCP Puzzlers

468 points by jsnell 10y ago

ChuckMcM 10y ago
That is an awesome post. If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely. Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account, but these days with virtualization you can have a 500 node network all on the same freakin' host! And the law of large numbers says "rare things happen often, when you have a lot of things." Fun anecdote, at Blekko we had people who tried to scrape the search engine by fetching all 300 pages of results. They would do that with some script or code and it would be clear they weren't human because they would ask for each page right after the other. We sent them to a process that Greg wrote on a machine that did most of the TCP handshake and then went away. As a result the scrapers script would hang forever. We saw output that suggested some of these things sat their for months waiting for results that would never come.
- jsmthrowaway 10y ago
  This practice is a type of tarpitting, and is fairly common and achievable in several ways. I've seen it implemented with several-second delays between bytes, but you have to weigh such active techniques against having the connection and resources set up.
  - ChuckMcM 10y ago
    Or you can re-write enough of the TCP stack in user land to do all of the handshaking and then go away. No connections, no resources wasted, infinite numbers of clients waiting forever and ever in their read() call.
    - jsmthrowaway 10y ago
      Oh, sure, I actually liked your method and wish I had thought of it. I meant active techniques like mine.
- raarts 10y ago
  > Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account Not really. Writing reliable network programs always had to take this into account. Take the scenario (not mentioned in TFA) where a firewall somewhere in the network path suddenly starts blocking traffic in one direction. Or handle a patch cable being pulled from a switch (or from your computer which has a different result). These scenarios always were real and resulted in comparable connection error states. Do agree it's a good post though, explains it rather nicely.
  - bluejekyll 10y ago
    Completely agree. I've run into a lot of this problems on a single node.
- inpulse 10y ago
  So there's a way to hang the connection that won't be caught by a `timeout` setting in most HTTP clients? Or the bot writers were just too lazy to add timeouts in their bots?
  - ChuckMcM 10y ago
    I suspect the bot writers were too lazy to add timeouts.
    - diroussel 10y ago
      He had no way to know if they were acutally still waiting months later. Just because the peer has not sent a RST doesn't mean remote end still has a process/machine still in existence.
  - tedunangst 10y ago
    As one data point, Python feed parser has no timeout. I've seen it sit for days when the remote end disappeared.
- scottlamb 10y ago
  > If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely. Not necessarily. If you have a good RPC layer, it will abstract away most of these details. For example, gRPC automatically does periodic health checking so you never see a connection hang forever. (But you do see some CPU and network cost associated with open channels!) gRPC (or at least gRPC's predecessor, Stubby) has various other quirks which are more critical to understand if you build your system on top of it. I expect the same is true of other RPC layers. Some examples below: * a boolean controlling "initial reachability" of a channel before a TCP connection is established (when load-balancing between several servers, you generally want it to be false so your RPC goes to a known-working server) * situations in which existing channels are working but establishing new ones does not * the notion of "fail-fast" when a channel is unreachable * lame-ducking when a server is planning to shut down * deadlines being automatically adjusted for expected round-trip transit time * load-balanced channels: various algorithms, subsetting * quirky but extremely useful built-in debugging webpages It's valuable to understand all the layers underneath when abstractions leak, but the abstractions work most of the time. I'd say it's more essentially to understand the abstraction you're using than ones two or three layers down. People build (or at least contribute to) wildly successful distributed systems without a complete understanding of TCP or IP or Ethernet every day.
  - est 10y ago
    > For example, gRPC automatically does periodic health checking so you never see a connection hang forever. So, a simple setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE) call?
    - rdtsc 10y ago
      I found in practice that is usually not enough and every distributed system I designed or worked on ended up with heartbeats of some point. It could OS peculiarities, or could be inability to tweak the times of keepalives. Sometimes the heartbeats are sent by the server, sometimes by the client, it depends. But they always end up in the application layer protocol.
syncsynchalt 10y ago
One thing missing from this article is TCP heartbeats, which prevents the first two conditions from going on indefinitely. However since the minimum allowed heartbeat is (usually) two hours host reboot/shutdown is usually detected via a heartbeat at the application layer instead, which sends some no-op traffic every n seconds or minutes.
- betenoire 10y ago
  like you say, the heartbeat removes some instances, but not others. The article also says, "It's the responsibility of the application to handle the cases where these differ (often using a keep-alive mechanism)."
- ghusbands 10y ago
  The article mentions both. "It's possible to manage this problem using TCP or application-level keep-alive." (and did before you commented)
droopybuns 10y ago
On OSX: sudo dtruss -d -t bind,listen,accept,poll,read,write nc -l -p 8080 dtrace: failed to execute nc: dtrace cannot control executables signed with restricted entitlements If you copy nc over to /tmp/ and run dtruss again, it will work. Hope that helps others. Fun article, OP!
- sigjuice 10y ago
  I never liked dtruss much for complicated debugging because the output shows way too many things in hex. OTOH, output from strace on Linux tends to be much nicer. e.g. $ strace -tebind nc -l -p 8080 14:27:44 bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 $ strace -teconnect nc 10.88.88.140 8080 14:28:29 connect(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("10.88.88.140")}, 16) = -1 EINPROGRESS (Operation now in progress) Does anyone know if the various dtrace-based dtruss implementations can be made to do this?
  - rdtsc 10y ago
    Yeah same here. When I want to trace my code, boot up a VM just because am more familiar with strace and gdb rather than dtruss and lldb. I can see dtruss having more and fancier features, I but am just too lazy to learn two tools of the same type.
johnrob 10y ago
I used to mimimc the power off scenario using a wired connection and a router. The client machine ethernet connects to an up-linked router, and to mimic the server's poweroff, we just pull the uplink cable out from the router. As far as the client can tell, the network interface is still up (because the router is still alive). If you pull the client side cable from the router, the client's network stack detects the lost network interface and the socket closes immediately.
- caf 10y ago
  Some routers will send ICMP Destination Unreachable back in that situation.
webscalist 10y ago
Wow that modern CSS making it difficult to read longer than 2 seconds.
- eridius 10y ago
  What?
  - lightlyused 10y ago
    What he means is that the contrast between the text and background makes the article hard to read. If they would up the font-weight to 400 from 300 it would give a much better contrast.
    - eridius 10y ago
      Huh, I didn't even notice it. I wonder if this is due to text rendering differences between platforms? The page is pretty readable on Safari on OS X.
      - kiwijamo 10y ago
        Looks nice in Safari on iPhone too.
      - vacri 10y ago
        Any time you see mid- or light-grey text on white, you know that the designer is on OSX and doesn't care one jot about other viewers.
        forgetfullest 10y ago
        They may care, but they're simply unaware of how bad it looks, and also unaware of browsershots.org and similar services :)
      - lightlyused 10y ago
        Firefox on Ubuntu here.
Animats 10y ago
Idle but half-dead TCP connections will eventually time out. It can take hours, though, on some systems.[1] Early thinking was that if you have a Telnet session open, TCP itself should not time out. The Telnet server might have a timeout, but it wasn't TCP's job to decide how long you get for lunch. Today most servers have shorter timeouts, mostly as a defense against denial of service attacks. But it's often the HTTP server, not the TCP level, that times out first. [1] https://blogs.technet.microsoft.com/nettracer/2010/06/03/things-that-you-may-want-to-know-about-tcp-keepalives/ https://blogs.technet.microsoft.com/nettracer/2010/06/03/thi...
- andrewstuart2 10y ago
  setsockopt [1], though, lets you change the timeout on a per-socket basis via the SO_RCVTIMEO and SO_SNDTIMEO options. If you need to know sooner that your data isn't going to be sent, it's pretty trivial to set up a short timeout that overrides the system defaults. [1] https://www.freebsd.org/cgi/man.cgi?query=setsockopt&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=setsockopt&sektion...
  - caf 10y ago
    SO_RCVTIMEO and SO_SNDTIMEO are for setting a timeout on blocking socket operations. They don't tell you anything about whether the other end is still there or not. Setting those options isn't much different from setting a timeout in your poll() call.
- dap 10y ago
  > Idle but half-dead TCP connections will eventually time out. It can take hours, though, on some systems. This is not true, in general. I think you're describing connections which have TCP KeepAlive enabled on them.
  - bluejekyll 10y ago
    Correct. Connection read/write timeouts only apply when, wait for it, reading or writing. All TCP keep alive does is send a packet every so often, which is actually something which is rarely actually set.
richm44 10y ago
There are some other fun edge cases - for example: - If you have a port-forward to a machine that is switched off then you can get ICMP network unreachable or ICMP host unreachable as the response to the a SYN in the initial handshake. This can also happen at any point in the connection. Other ICMP messages can also occur like this (eg. admin prohibited). It's always worth remembering that the TCP connection is sitting on an underlying network stack that can also signal errors outside of the TCP protocol itself.
- rdtsc 10y ago
  Ooh fun! I just learned something new. I knew about how blocking ICMP could lead to MTU path discovery issues, but this is different. Do you know at the socket API level how the ICMP unreachagle is manifested. Looked at connect() error and saw ENETUNREACH -- guessing that's the one?
partycoder 10y ago
This would be in terms of system calls. In terms of messages being sent you can capture the traffic and see what was sent, with which flags, etc.
tbarbugli 10y ago
I did not really get the difference between 1. and 2. puzzlers, don't they test the same scenario?
- arielb1 10y ago
  Puzzler 1 is when a machine goes down and stays down, and you hang or retransmit until a timeout. Puzzler 2 is when a machine goes down and then comes back up, and you get reset.
  - rob-olmos 10y ago
    Puzzler #1 is an unclean reboot with a server-side TCP reset after the server is back online. Puzzler #2 is an unclean shutdown with a client-side TCP timeout since the server never responds.
    - arielb1 10y ago
      Yeah. The other way around.
arielb1 10y ago
Puzzler #3 does not have much with using a half-closed socket - you still have a socket that selects as writable, but is not connected to anything. For some reason I remember that when a process crashes the OS's TCP stack sends an RST to the opposing side, rather than a FIN, preventing puzzler #3.
- jtakkala 10y ago
  When a process exits abnormally or not and if there's unread data in the receive buffer then the kernel will send a RST, otherwise it would send a FIN.
shabble 10y ago
Other than using VMs and 'surprise' power-cycling them, what other ways are there to test these sorts of behaviours? Would an entry like `iptables -I forward -s ... -d ... -j DENY` on a host the 2 endpoints route over suffice to suddenly stop all traffic (or a pair of rules to cover a->b and b->a packets)? Are there any tools for 'and suddenly, the network disappears/goes to 99% packetloss/10s latency' etc that you can use to test existing software? I'm imagining some sort of iptables/tc wrappers that can take defined scenarios (or just chaos-monkey it) and apply them to traffic, possibly allowing assertions as to how the application should behave for various situations.
- beached_whale 10y ago
  would ipconfig eth0 down do the trick?
- jsnell 10y ago
  I wrote a tool for basically that exact use case. It causes deterministic network faults (on a per-flow rather than per-interface basis), and takes trace files from both sides of the fault. [0] https://www.snellman.net/blog/archive/2015-10-01-flow-disruptor/ https://www.snellman.net/blog/archive/2015-10-01-flow-disrup...
- deleted 10y ago
  [deleted]
- greenleafjacob 10y ago
  Jepsen [1]. For previous discussions on HN with MongoDB [2] Cassandra [3] RabbitMQ [4] and ETCD [5]. Others exist as well. [1]: https://github.com/aphyr/jepsen https://github.com/aphyr/jepsen [2]: https://news.ycombinator.com/item?id=9417773 https://news.ycombinator.com/item?id=9417773 [3]: https://news.ycombinator.com/item?id=6451885 https://news.ycombinator.com/item?id=6451885 [4]: https://news.ycombinator.com/item?id=7863856 https://news.ycombinator.com/item?id=7863856 [5]: https://news.ycombinator.com/item?id=7884640 https://news.ycombinator.com/item?id=7884640
deleted 10y ago
[deleted]
rob-olmos 10y ago
For Puzzler #2, did the kernel send any additional attempts on its own, beyond the initial attempt, before the 5min timeout?
- caf 10y ago
  Yes - but you only see those if you're watching on the network side.
saurik 10y ago
"When I type the message and hit enter, kodos gets woken up, reads the message from stdin, and sends it over the socket." <- From Puzzler #1, this is a typo: it should say "Kang", not "Kodos". Kang is sending a message to Kodos, and this message was typed into Kang's console (waking up Kang) which sent the message to Kodos.

468 points by jsnell 10y ago

That is an awesome post. If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely. Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account, but these days with virtualization you can have a 500 node network all on the same freakin' host! And the law of large numbers says "rare things happen often, when you have a lot of things." Fun anecdote, at Blekko we had people who tried to scrape the search engine by fetching all 300 pages of results. They would do that with some script or code and it would be clear they weren't human because they would ask for each page right after the other. We sent them to a process that Greg wrote on a machine that did most of the TCP handshake and then went away. As a result the scrapers script would hang forever. We saw output that suggested some of these things sat their for months waiting for results that would never come.

jsmthrowaway 10y ago
This practice is a type of tarpitting, and is fairly common and achievable in several ways. I've seen it implemented with several-second delays between bytes, but you have to weigh such active techniques against having the connection and resources set up.
- ChuckMcM 10y ago
  Or you can re-write enough of the TCP stack in user land to do all of the handshaking and then go away. No connections, no resources wasted, infinite numbers of clients waiting forever and ever in their read() call.
  - jsmthrowaway 10y ago
    Oh, sure, I actually liked your method and wish I had thought of it. I meant active techniques like mine.
raarts 10y ago
> Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account Not really. Writing reliable network programs always had to take this into account. Take the scenario (not mentioned in TFA) where a firewall somewhere in the network path suddenly starts blocking traffic in one direction. Or handle a patch cable being pulled from a switch (or from your computer which has a different result). These scenarios always were real and resulted in comparable connection error states. Do agree it's a good post though, explains it rather nicely.
- bluejekyll 10y ago
  Completely agree. I've run into a lot of this problems on a single node.
inpulse 10y ago
So there's a way to hang the connection that won't be caught by a `timeout` setting in most HTTP clients? Or the bot writers were just too lazy to add timeouts in their bots?
- ChuckMcM 10y ago
  I suspect the bot writers were too lazy to add timeouts.
  - diroussel 10y ago
    He had no way to know if they were acutally still waiting months later. Just because the peer has not sent a RST doesn't mean remote end still has a process/machine still in existence.
- tedunangst 10y ago
  As one data point, Python feed parser has no timeout. I've seen it sit for days when the remote end disappeared.
scottlamb 10y ago
> If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely. Not necessarily. If you have a good RPC layer, it will abstract away most of these details. For example, gRPC automatically does periodic health checking so you never see a connection hang forever. (But you do see some CPU and network cost associated with open channels!) gRPC (or at least gRPC's predecessor, Stubby) has various other quirks which are more critical to understand if you build your system on top of it. I expect the same is true of other RPC layers. Some examples below: * a boolean controlling "initial reachability" of a channel before a TCP connection is established (when load-balancing between several servers, you generally want it to be false so your RPC goes to a known-working server) * situations in which existing channels are working but establishing new ones does not * the notion of "fail-fast" when a channel is unreachable * lame-ducking when a server is planning to shut down * deadlines being automatically adjusted for expected round-trip transit time * load-balanced channels: various algorithms, subsetting * quirky but extremely useful built-in debugging webpages It's valuable to understand all the layers underneath when abstractions leak, but the abstractions work most of the time. I'd say it's more essentially to understand the abstraction you're using than ones two or three layers down. People build (or at least contribute to) wildly successful distributed systems without a complete understanding of TCP or IP or Ethernet every day.
- est 10y ago
  > For example, gRPC automatically does periodic health checking so you never see a connection hang forever. So, a simple setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE) call?
  - rdtsc 10y ago
    I found in practice that is usually not enough and every distributed system I designed or worked on ended up with heartbeats of some point. It could OS peculiarities, or could be inability to tweak the times of keepalives. Sometimes the heartbeats are sent by the server, sometimes by the client, it depends. But they always end up in the application layer protocol.

syncsynchalt 10y ago

One thing missing from this article is TCP heartbeats, which prevents the first two conditions from going on indefinitely. However since the minimum allowed heartbeat is (usually) two hours host reboot/shutdown is usually detected via a heartbeat at the application layer instead, which sends some no-op traffic every n seconds or minutes.

betenoire 10y ago
like you say, the heartbeat removes some instances, but not others. The article also says, "It's the responsibility of the application to handle the cases where these differ (often using a keep-alive mechanism)."
ghusbands 10y ago
The article mentions both. "It's possible to manage this problem using TCP or application-level keep-alive." (and did before you commented)

droopybuns 10y ago

On OSX: sudo dtruss -d -t bind,listen,accept,poll,read,write nc -l -p 8080 dtrace: failed to execute nc: dtrace cannot control executables signed with restricted entitlements If you copy nc over to /tmp/ and run dtruss again, it will work. Hope that helps others. Fun article, OP!

sigjuice 10y ago
I never liked dtruss much for complicated debugging because the output shows way too many things in hex. OTOH, output from strace on Linux tends to be much nicer. e.g. $ strace -tebind nc -l -p 8080 14:27:44 bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 $ strace -teconnect nc 10.88.88.140 8080 14:28:29 connect(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("10.88.88.140")}, 16) = -1 EINPROGRESS (Operation now in progress) Does anyone know if the various dtrace-based dtruss implementations can be made to do this?
- rdtsc 10y ago
  Yeah same here. When I want to trace my code, boot up a VM just because am more familiar with strace and gdb rather than dtruss and lldb. I can see dtruss having more and fancier features, I but am just too lazy to learn two tools of the same type.

johnrob 10y ago

I used to mimimc the power off scenario using a wired connection and a router. The client machine ethernet connects to an up-linked router, and to mimic the server's poweroff, we just pull the uplink cable out from the router. As far as the client can tell, the network interface is still up (because the router is still alive). If you pull the client side cable from the router, the client's network stack detects the lost network interface and the socket closes immediately.

caf 10y ago
Some routers will send ICMP Destination Unreachable back in that situation.

webscalist 10y ago

Wow that modern CSS making it difficult to read longer than 2 seconds.

eridius 10y ago
What?
- lightlyused 10y ago
  What he means is that the contrast between the text and background makes the article hard to read. If they would up the font-weight to 400 from 300 it would give a much better contrast.
  - eridius 10y ago
    Huh, I didn't even notice it. I wonder if this is due to text rendering differences between platforms? The page is pretty readable on Safari on OS X.
    - kiwijamo 10y ago
      Looks nice in Safari on iPhone too.
    - vacri 10y ago
      Any time you see mid- or light-grey text on white, you know that the designer is on OSX and doesn't care one jot about other viewers.
      - forgetfullest 10y ago
        They may care, but they're simply unaware of how bad it looks, and also unaware of browsershots.org and similar services :)
    - lightlyused 10y ago
      Firefox on Ubuntu here.

Animats 10y ago

Idle but half-dead TCP connections will eventually time out. It can take hours, though, on some systems.[1] Early thinking was that if you have a Telnet session open, TCP itself should not time out. The Telnet server might have a timeout, but it wasn't TCP's job to decide how long you get for lunch. Today most servers have shorter timeouts, mostly as a defense against denial of service attacks. But it's often the HTTP server, not the TCP level, that times out first. [1] https://blogs.technet.microsoft.com/nettracer/2010/06/03/things-that-you-may-want-to-know-about-tcp-keepalives/ https://blogs.technet.microsoft.com/nettracer/2010/06/03/thi...

andrewstuart2 10y ago
setsockopt [1], though, lets you change the timeout on a per-socket basis via the SO_RCVTIMEO and SO_SNDTIMEO options. If you need to know sooner that your data isn't going to be sent, it's pretty trivial to set up a short timeout that overrides the system defaults. [1] https://www.freebsd.org/cgi/man.cgi?query=setsockopt&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=setsockopt&sektion...
- caf 10y ago
  SO_RCVTIMEO and SO_SNDTIMEO are for setting a timeout on blocking socket operations. They don't tell you anything about whether the other end is still there or not. Setting those options isn't much different from setting a timeout in your poll() call.
dap 10y ago
> Idle but half-dead TCP connections will eventually time out. It can take hours, though, on some systems. This is not true, in general. I think you're describing connections which have TCP KeepAlive enabled on them.
- bluejekyll 10y ago
  Correct. Connection read/write timeouts only apply when, wait for it, reading or writing. All TCP keep alive does is send a packet every so often, which is actually something which is rarely actually set.

richm44 10y ago

There are some other fun edge cases - for example: - If you have a port-forward to a machine that is switched off then you can get ICMP network unreachable or ICMP host unreachable as the response to the a SYN in the initial handshake. This can also happen at any point in the connection. Other ICMP messages can also occur like this (eg. admin prohibited). It's always worth remembering that the TCP connection is sitting on an underlying network stack that can also signal errors outside of the TCP protocol itself.

rdtsc 10y ago
Ooh fun! I just learned something new. I knew about how blocking ICMP could lead to MTU path discovery issues, but this is different. Do you know at the socket API level how the ICMP unreachagle is manifested. Looked at connect() error and saw ENETUNREACH -- guessing that's the one?

partycoder 10y ago

This would be in terms of system calls. In terms of messages being sent you can capture the traffic and see what was sent, with which flags, etc.

tbarbugli 10y ago

I did not really get the difference between 1. and 2. puzzlers, don't they test the same scenario?

arielb1 10y ago
Puzzler 1 is when a machine goes down and stays down, and you hang or retransmit until a timeout. Puzzler 2 is when a machine goes down and then comes back up, and you get reset.
- rob-olmos 10y ago
  Puzzler #1 is an unclean reboot with a server-side TCP reset after the server is back online. Puzzler #2 is an unclean shutdown with a client-side TCP timeout since the server never responds.
  - arielb1 10y ago
    Yeah. The other way around.

arielb1 10y ago

Puzzler #3 does not have much with using a half-closed socket - you still have a socket that selects as writable, but is not connected to anything. For some reason I remember that when a process crashes the OS's TCP stack sends an RST to the opposing side, rather than a FIN, preventing puzzler #3.

jtakkala 10y ago
When a process exits abnormally or not and if there's unread data in the receive buffer then the kernel will send a RST, otherwise it would send a FIN.

shabble 10y ago

Other than using VMs and 'surprise' power-cycling them, what other ways are there to test these sorts of behaviours? Would an entry like `iptables -I forward -s ... -d ... -j DENY` on a host the 2 endpoints route over suffice to suddenly stop all traffic (or a pair of rules to cover a->b and b->a packets)? Are there any tools for 'and suddenly, the network disappears/goes to 99% packetloss/10s latency' etc that you can use to test existing software? I'm imagining some sort of iptables/tc wrappers that can take defined scenarios (or just chaos-monkey it) and apply them to traffic, possibly allowing assertions as to how the application should behave for various situations.

beached_whale 10y ago
would ipconfig eth0 down do the trick?
jsnell 10y ago
I wrote a tool for basically that exact use case. It causes deterministic network faults (on a per-flow rather than per-interface basis), and takes trace files from both sides of the fault. [0] https://www.snellman.net/blog/archive/2015-10-01-flow-disruptor/ https://www.snellman.net/blog/archive/2015-10-01-flow-disrup...
deleted 10y ago
[deleted]
greenleafjacob 10y ago
Jepsen [1]. For previous discussions on HN with MongoDB [2] Cassandra [3] RabbitMQ [4] and ETCD [5]. Others exist as well. [1]: https://github.com/aphyr/jepsen https://github.com/aphyr/jepsen [2]: https://news.ycombinator.com/item?id=9417773 https://news.ycombinator.com/item?id=9417773 [3]: https://news.ycombinator.com/item?id=6451885 https://news.ycombinator.com/item?id=6451885 [4]: https://news.ycombinator.com/item?id=7863856 https://news.ycombinator.com/item?id=7863856 [5]: https://news.ycombinator.com/item?id=7884640 https://news.ycombinator.com/item?id=7884640

deleted 10y ago

[deleted]

rob-olmos 10y ago

For Puzzler #2, did the kernel send any additional attempts on its own, beyond the initial attempt, before the 5min timeout?

caf 10y ago
Yes - but you only see those if you're watching on the network side.

saurik 10y ago

"When I type the message and hit enter, kodos gets woken up, reads the message from stdin, and sends it over the socket." <- From Puzzler #1, this is a typo: it should say "Kang", not "Kodos". Kang is sending a message to Kodos, and this message was typed into Kang's console (waking up Kang) which sent the message to Kodos.