3 ms·

What caught my attention was where Twilio said the redis-slaves were timing out to the redis-master: http://www.twilio.com/blog/2013/07/billing-incident-post-m

What caught my attention was where Twilio said the redis-slaves were timing out to the redis-master: http://www.twilio.com/blog/2013/07/billing-incident-post-mortem.html http://www.twilio.com/blog/2013/07/billing-incident-post-mor... I think timeouts should be abolished for the vast majority of software today. The usual reasoning goes something like this: for a TCP connection, if you don't hear from the server for some period of time, you can assume that something is "wrong" and drop the connection. The fallacy is, the TCP connection is not really important to the shared state of two devices. From the very beginning (I'm talking 1970s!), devices should have been using tokens to identify one another, regardless of communication status. The tokens could be saved in nonvolatile memory on servers so that jobs could always continue where they left off. Instead we have a whole slew of nondeterministic pathological cases -exactly- like the one that hit Twilio. If you take on the burden of timeouts, you end up with dozens of places in your code (even more, potentially) where you just don't know what to do if you lose communication. If you don't take on the burden of timeouts, then you can just track each connection and all it costs you is storage space, which is practically free today and getting cheaper every year. With credentials from the client, you don't even have to worry about duplicate connections. You can write your client-server code deterministically and stick to the logic, and easily stress test failure modes.

donpdonp 13y ago
This resonated with me because I've been building services with zeromq lately. The zeromq bind and connect calls isolate the caller from managing disconnects. Zeromq will reestablish a dropped connection and the client code is none the wiser. Now I have some extra reasoning as to why this is a good idea. Thanks!
- dpe82 13y ago
  It's a good idea until the local send buffer fills up and starts silently dropping data. There's no free lunch.
  - tlrobinson 13y ago
    In theory don't programs already need to handle that case for slow/congested connections?
    - dpe82 13y ago
      Of course, but zeromq makes it feel like you don't. It provides a message massing system that you pass messages into on one end and they magically pop out the other. If the network is slow, congested or down it'll cache messages locally until it can reconnect and send them. The problem is you don't know what's going on and if the local cache fills up it'll silently drop messages. In some use cases that's perfectly fine- it depends on your application. Zeromq is great for some things - it's super flexible and FAST. It's just not, on its own, durable. If you need durability you have to build it on top. Doable, just not free.
      - 300bps 13y ago
        You're absolutely right. It just masks the timeout from the client for as long as it can and then fails just as ungracefully.
ironchef 13y ago
I've seen timeouts in similar situations where the IO was saturated (saving the RDB and writing to AOF at the same time).
Negitivefrags 13y ago
I don't really see how you are proposing we live without timeouts. You send a request to a server and no response comes back for a while. What do you do now? Wait forever?
- tyree732 13y ago
  He's a big fan of the C in the CAP theorem, but not so much the A.
  - zackmorris 13y ago
    Man, I wrote two rather long responses and then resigned myself to the fact that you are right. This is a very difficult problem. The only insight I gained was that since billing isn't a realtime task, availability shouldn't matter in this case. IMHO that's why what happened was considered a "mistake". Some thoughts: Ideally, they should be able to structure their database so that a high number of transactions can be queued (up to the limit of virtual memory) and open connections shouldn't have to be maintained for each one. Then they could get rid of the notion of timeouts, as long as the master is up a high enough percentage of the time to handle the load. If it isn't, then even the clients would block when they fill up their virtual memory until the master catches up. If they rewrite their code, they should start with the assumption that the master is normally down. They would have to solve the problem of two transactions conflicting on the client side. So for example, give each transaction a hash or digest that uniquely identifies it as a payment for a certain month. So if one client blocks for a day, and someone tries to start a payment on another client which also blocks, the master would resolve it when it comes back up by committing only one transaction for the hash and failing the other one. If they get all that logic right, then they could fire off a payment and when the client server says it's queued, they could move on and not need a receipt for the commit. Although I just realized if the client server crashes, they have no guarantee. So they would have to start the transactions on 2 or more client servers, whichever number statistically ensures a high enough guarantee, or possibly all of the clients. That sounds like a lot of overhead at first until you realize that each transaction only takes a few k of memory to represent. I think that possibly this points to a general solution. Give each transaction a unique ID based on its intent and context, and start it on enough client servers to provide a high enough statistical success rate. Make sure the clients store the pending transactions in nonvolatile memory so they can resume if the power goes out. I'm sure I've forgotten something but surely there is a way to solve this specific subset of problems so that companies could do a sanity check on themselves and avoid repeating history.

donpdonp 13y ago

This resonated with me because I've been building services with zeromq lately. The zeromq bind and connect calls isolate the caller from managing disconnects. Zeromq will reestablish a dropped connection and the client code is none the wiser. Now I have some extra reasoning as to why this is a good idea. Thanks!

dpe82 13y ago
It's a good idea until the local send buffer fills up and starts silently dropping data. There's no free lunch.
- tlrobinson 13y ago
  In theory don't programs already need to handle that case for slow/congested connections?
  - dpe82 13y ago
    Of course, but zeromq makes it feel like you don't. It provides a message massing system that you pass messages into on one end and they magically pop out the other. If the network is slow, congested or down it'll cache messages locally until it can reconnect and send them. The problem is you don't know what's going on and if the local cache fills up it'll silently drop messages. In some use cases that's perfectly fine- it depends on your application. Zeromq is great for some things - it's super flexible and FAST. It's just not, on its own, durable. If you need durability you have to build it on top. Doable, just not free.
    - 300bps 13y ago
      You're absolutely right. It just masks the timeout from the client for as long as it can and then fails just as ungracefully.

ironchef 13y ago

I've seen timeouts in similar situations where the IO was saturated (saving the RDB and writing to AOF at the same time).

Negitivefrags 13y ago

I don't really see how you are proposing we live without timeouts. You send a request to a server and no response comes back for a while. What do you do now? Wait forever?

tyree732 13y ago
He's a big fan of the C in the CAP theorem, but not so much the A.
- zackmorris 13y ago
  Man, I wrote two rather long responses and then resigned myself to the fact that you are right. This is a very difficult problem. The only insight I gained was that since billing isn't a realtime task, availability shouldn't matter in this case. IMHO that's why what happened was considered a "mistake". Some thoughts: Ideally, they should be able to structure their database so that a high number of transactions can be queued (up to the limit of virtual memory) and open connections shouldn't have to be maintained for each one. Then they could get rid of the notion of timeouts, as long as the master is up a high enough percentage of the time to handle the load. If it isn't, then even the clients would block when they fill up their virtual memory until the master catches up. If they rewrite their code, they should start with the assumption that the master is normally down. They would have to solve the problem of two transactions conflicting on the client side. So for example, give each transaction a hash or digest that uniquely identifies it as a payment for a certain month. So if one client blocks for a day, and someone tries to start a payment on another client which also blocks, the master would resolve it when it comes back up by committing only one transaction for the hash and failing the other one. If they get all that logic right, then they could fire off a payment and when the client server says it's queued, they could move on and not need a receipt for the commit. Although I just realized if the client server crashes, they have no guarantee. So they would have to start the transactions on 2 or more client servers, whichever number statistically ensures a high enough guarantee, or possibly all of the clients. That sounds like a lot of overhead at first until you realize that each transaction only takes a few k of memory to represent. I think that possibly this points to a general solution. Give each transaction a unique ID based on its intent and context, and start it on enough client servers to provide a high enough statistical success rate. Make sure the clients store the pending transactions in nonvolatile memory so they can resume if the power goes out. I'm sure I've forgotten something but surely there is a way to solve this specific subset of problems so that companies could do a sanity check on themselves and avoid repeating history.