4 ms·

Billing systems are not to be taken lightly, namely because money is inherently involved. When developing such systems, it is irresponsible to use new, unprove

Billing systems are not to be taken lightly, namely because money is inherently involved. When developing such systems, it is irresponsible to use new, unproven technologies without justification. When developing such systems, it is irresponsible to trade off the reliability and safety of the system for some "developer productivity". Such irresponsibility is just not acceptable. Failures due to such irresponsibility should not be tolerated, either.

taf2 13y ago
money can be refunded... we're not talking about a life - or a human life we're just talking about software. Reading how Twilio was using redis it sounds very reasonable (fsync). I've seen MySQL servers melt under extreme load running on very high end servers so if you're saying that somehow having had SQL in this case would have saved them... well... maybe you're right... but I wasn't there were you? I believe the engineers at Twilio are building and have built a platform to handle a scale that is going to continue to reach further than many traditional models will scale. It makes sense to me that they would need to look for in memory systems to push the limits - They are fsync'ing immediately - so while it's in memory for a read it's disk for writes so that sounds pretty safe to me... Also, a lot of people use Redis now, so I definitely wouldn't say it's unproven, that is as others have said like saying we should still be using typewriters to put ink on paper because it's "proven"... I'd answer that with... "good luck with that". And at the end of the day - "the shit will hit the fan" and it did... the team did great and we're all thankful.
- PommeDeTerre 13y ago
  Just because human lives aren't necessarily at stake it doesn't mean that poor (in my opinion) software system designs should be tolerated. And when we talk about "relational databases", we basically never mean MySQL. I'm not even sure why you'd bother to bring it up, given its poor reputation, its lack of proper functionality in some cases, and the many other (and far better) alternatives. As for scalability, some other commenter here posted a link to a twitter comment describing the scale in this case: https://twitter.com/dN0t/status/360119871318659074 https://twitter.com/dN0t/status/360119871318659074 It clearly states "tens of thousands" of transactions per second. That is well within the capabilities of the lower-end database systems on modern, low-end server hardware. Twilio will require a huge amount of growth just to reach the point where higher end database systems and hardware become appropriate, never mind "a scale that is going to continue to reach further than many traditional models will scale". Running into significant billing issues due to a likely unnecessary use (in my opinion) of a NoSQL database while still at a relatively small scale is not something that a team should be commended for, and nobody should be "thankful" that the incident did happen.
  - defen 13y ago
    The equivalent failure with PostgreSQL would be to update kernel.shmmax with a call to sysctl, and then forget to update /etc/sysctl.conf. Then if the machine reboots PostgreSQL will fail to start because it can't allocate enough memory. It mostly wasn't a redis problem as far as I can tell. That said, Twilio is probably at a big enough scale where they can afford to hire a couple full-time admins to deal with PostgreSQL.
jessaustin 13y ago
Failures due to such irresponsibility should not be tolerated, either. I appreciate that you are very committed to reliability. I also have worked on (telecom) billing systems. However, what are you suggesting here? Should all Twilio's customers find another provider? (Do you have a suggestion?) Should they just not use a service like Twilio's? Should Twilio quadruple their prices so they can throw up enough hardware to do all this in postgres? Should we all fly over to Sicily to protest outside antirez's house? Incidentally, in the late '90s I sat in on a few sales efforts in which Lucent pushed their "Datablitz" in-memory database for mediation and billing of call records. So, it's not as though nothing similar to this has ever been done.
deleted 13y ago
[deleted]

taf2 13y ago

money can be refunded... we're not talking about a life - or a human life we're just talking about software. Reading how Twilio was using redis it sounds very reasonable (fsync). I've seen MySQL servers melt under extreme load running on very high end servers so if you're saying that somehow having had SQL in this case would have saved them... well... maybe you're right... but I wasn't there were you? I believe the engineers at Twilio are building and have built a platform to handle a scale that is going to continue to reach further than many traditional models will scale. It makes sense to me that they would need to look for in memory systems to push the limits - They are fsync'ing immediately - so while it's in memory for a read it's disk for writes so that sounds pretty safe to me... Also, a lot of people use Redis now, so I definitely wouldn't say it's unproven, that is as others have said like saying we should still be using typewriters to put ink on paper because it's "proven"... I'd answer that with... "good luck with that". And at the end of the day - "the shit will hit the fan" and it did... the team did great and we're all thankful.

PommeDeTerre 13y ago
Just because human lives aren't necessarily at stake it doesn't mean that poor (in my opinion) software system designs should be tolerated. And when we talk about "relational databases", we basically never mean MySQL. I'm not even sure why you'd bother to bring it up, given its poor reputation, its lack of proper functionality in some cases, and the many other (and far better) alternatives. As for scalability, some other commenter here posted a link to a twitter comment describing the scale in this case: https://twitter.com/dN0t/status/360119871318659074 https://twitter.com/dN0t/status/360119871318659074 It clearly states "tens of thousands" of transactions per second. That is well within the capabilities of the lower-end database systems on modern, low-end server hardware. Twilio will require a huge amount of growth just to reach the point where higher end database systems and hardware become appropriate, never mind "a scale that is going to continue to reach further than many traditional models will scale". Running into significant billing issues due to a likely unnecessary use (in my opinion) of a NoSQL database while still at a relatively small scale is not something that a team should be commended for, and nobody should be "thankful" that the incident did happen.
- defen 13y ago
  The equivalent failure with PostgreSQL would be to update kernel.shmmax with a call to sysctl, and then forget to update /etc/sysctl.conf. Then if the machine reboots PostgreSQL will fail to start because it can't allocate enough memory. It mostly wasn't a redis problem as far as I can tell. That said, Twilio is probably at a big enough scale where they can afford to hire a couple full-time admins to deal with PostgreSQL.

jessaustin 13y ago

Failures due to such irresponsibility should not be tolerated, either. I appreciate that you are very committed to reliability. I also have worked on (telecom) billing systems. However, what are you suggesting here? Should all Twilio's customers find another provider? (Do you have a suggestion?) Should they just not use a service like Twilio's? Should Twilio quadruple their prices so they can throw up enough hardware to do all this in postgres? Should we all fly over to Sicily to protest outside antirez's house? Incidentally, in the late '90s I sat in on a few sales efforts in which Lucent pushed their "Datablitz" in-memory database for mediation and billing of call records. So, it's not as though nothing similar to this has ever been done.

deleted 13y ago

[deleted]