3 ms·

As a former sunnie that worked on distributed systems, how would you cope with packet loss/out of order packets/weird socket states especially on Solaris? Would

by bitL 11y ago

As a former sunnie that worked on distributed systems, how would you cope with packet loss/out of order packets/weird socket states especially on Solaris? Would you do clock-per-socket and then merge them via internal buffers? How about when one/many of your nodes go down or you end up with a split brain? Or you'd just use 0MQ or similar super fast protocol, do some simple checks and if they fail, you'd just resend everything?

ChuckMcM 11y ago
NIS+ used "relative" timestamps, which were basically the delta between the server and the originator. So the originator would say "here is a packet and I think it is time 1415151515.5" and the server would get that packet and say Ok, since I think it is 1415151520 your delta is -5 and I'll subtract 5 from all of your times to put them into my time context. Then when you compared time stamps you did so in the servers frame of reference. But as the article points out it still suffered from jitter given the various subsystems between the packet and the service endpoint. As for out of order and lost packets the TCP layer prevent out of order, but retransmissions on lost packets resulted in big jitter spikes. Those were rare enough to pull out as a special case. And there was layered on top an optimistic transaction protocol where you could ask for the current transaction id, increment it by one and send your transaction with the assumption that if someone landed before you it would fail and you would have to restart. That worked well for read mostly applications (like a name service). The NoSQL database that Blekko designed uses a more complex promise system to preserve transaction ordering and it uses idempotent combinators which help manage time syncronization issues. But again, if we could wave a magic wand and get perfect synchronization it would be pretty interesting.

by bitL 11y ago

ChuckMcM 11y ago

NIS+ used "relative" timestamps, which were basically the delta between the server and the originator. So the originator would say "here is a packet and I think it is time 1415151515.5" and the server would get that packet and say Ok, since I think it is 1415151520 your delta is -5 and I'll subtract 5 from all of your times to put them into my time context. Then when you compared time stamps you did so in the servers frame of reference. But as the article points out it still suffered from jitter given the various subsystems between the packet and the service endpoint. As for out of order and lost packets the TCP layer prevent out of order, but retransmissions on lost packets resulted in big jitter spikes. Those were rare enough to pull out as a special case. And there was layered on top an optimistic transaction protocol where you could ask for the current transaction id, increment it by one and send your transaction with the assumption that if someone landed before you it would fail and you would have to restart. That worked well for read mostly applications (like a name service). The NoSQL database that Blekko designed uses a more complex promise system to preserve transaction ordering and it uses idempotent combinators which help manage time syncronization issues. But again, if we could wave a magic wand and get perfect synchronization it would be pretty interesting.