4 ms·

Totally agree. Need to clear up a developing misconception - Redis does not serve as the primary store for the account balance of Twilio's customers. The bil

Totally agree. Need to clear up a developing misconception - Redis does not serve as the primary store for the account balance of Twilio's customers. The billing system uses a double bookkeeping model common to many high volume designs with Redis as the in-flight data store (e.g. when a call or SMS message is created) with the transaction also stored independently to an RDBMS post-flight (e.g. when a call or SMS message is completed). Clearly however our implementation failed dangerously and did not recover in a manner that meets our customers' expectations. Totally get how such a misconception would occur from a cursory read of the incident report - just need to be clear.

justincormack 13y ago
But it does appear (if I understand correctly) that sometimes you used Redis as if it was the system of record rather than checking the actual one, hence the rebilling?
PommeDeTerre 13y ago
Is there actually a legitimate performance or scalability need to incorporate a NoSQL database in this case? Ever since NoSQL databases started receiving a lot of hype a few years back, I've witnessed a number of teams use them without any real justification. They'll build unnecessarily complex systems using one or more NoSQL database systems, all while a relational database would be more than sufficient for their needs. In several of these cases, some of the developers have been quite adamant that these NoSQL databases are essential. Then we rip them out, usually because they've been causing problems like described in this case. It quickly becomes obvious that they were never needed in the first place, and won't be needed even in the face of significantly increased load.
- AYBABTME 13y ago
  Some problems are better solved with key-value stores than with relational databases. It depends on the data model you need to map. Lot of people use relational databases as a big key value store, when really they should use a key value store. It's not necessarily about scalability or performance. =)
  - aidos 13y ago
    Yes. And Redis provides an interesting model with very specific performance characteristics depending on how you choose to store your data in each case. If you haven't worked with Redis it might be hard to appreciate how it's a bit more than just a key-value store and how it's a different tool from a relational db.
- threeseed 13y ago
  No offence but attitudes like this are the worst. If we all took your advice we would all be still using punch cards or writing everything in assembler. Sometimes you don't need a clear justification to use newer technologies. Perhaps developers just want to experience the significant developer productivity that comes with using many of the NoSQL databases. Also might be worth dropping the whole "SQL is better" insinuation. We have seen some pretty major data loss bugs in PostgreSQL and MySQL recently.
  - badclient 13y ago
    How are NoSql databases more productive than SQL?
  - PommeDeTerre 13y ago
    Billing systems are not to be taken lightly, namely because money is inherently involved. When developing such systems, it is irresponsible to use new, unproven technologies without justification. When developing such systems, it is irresponsible to trade off the reliability and safety of the system for some "developer productivity". Such irresponsibility is just not acceptable. Failures due to such irresponsibility should not be tolerated, either.
    - taf2 13y ago
      money can be refunded... we're not talking about a life - or a human life we're just talking about software. Reading how Twilio was using redis it sounds very reasonable (fsync). I've seen MySQL servers melt under extreme load running on very high end servers so if you're saying that somehow having had SQL in this case would have saved them... well... maybe you're right... but I wasn't there were you? I believe the engineers at Twilio are building and have built a platform to handle a scale that is going to continue to reach further than many traditional models will scale. It makes sense to me that they would need to look for in memory systems to push the limits - They are fsync'ing immediately - so while it's in memory for a read it's disk for writes so that sounds pretty safe to me... Also, a lot of people use Redis now, so I definitely wouldn't say it's unproven, that is as others have said like saying we should still be using typewriters to put ink on paper because it's "proven"... I'd answer that with... "good luck with that". And at the end of the day - "the shit will hit the fan" and it did... the team did great and we're all thankful.
- KirinDave 13y ago
  > Is there actually a legitimate performance or scalability need to incorporate a NoSQL database in this case? Usually the most compelling reason not to use a traditional SQL database is that they're expensive to field in distributed environments and require specialist engineers to support when you get out into the weeds. But I submit it's a category error to call Redis a "NoSQL database." It was also a category error to put data that needs to be redundant and durable into a storage system that is really specialized for high volume but low-reliability cache work. I mean, seriously. Minimum viable PostgresSQL deploy in EC2 is pretty damn expensive, but you may not have enough cash on hand to buy your own hardware and pay to put it in a "real datacenter." If you can find something cheaper and you're not even seed funded yet because no one funds your social product until you have 100k users, you're probably more likely to actually make it to where you can afford a different database by just using Redis. It's called "technical debt" and we accrue it both bitterly and often willingly.
- encoderer 13y ago
  I think you're making a number of mistakes in judgement here. Most importantly, you're conflating NoSQL with "not as reliable." As if there's no difference between Cassandra, HBase, Redis, etc. They're just "nosql" and "not needed" to you. That just seems highly ignorant of the relative merits of each database. And similarly, the attitude that a RDBMS is the right and proper data store, that all data should rightly prefer to live in an RDBMS aside from the loftiest heights of scalability. Finally, the assertion that using a key-value or column-based datastore is going to lead to an "unnecessarily complex" design. I'd not be surprised to find you disagree, but if we were working together on the same team, these are some things I'd try to work with you on.
  - fusiongyro 13y ago
    I can't speak for the person you're replying to, but I would like to point out that RDBMSes are built on theory designed explicitly to handle general data problems. Situations for which the relational model is not a good fit are almost by definition special cases. Similarly, while it is wise not to lump all of the non-SQL/non-relational databases together, to say that the bulk of them share a common obsession with scalability is not much of a reach. After all, we're not talking about Neo4J, FramerD, MetaKit, etc. We're talking about young systems that were born out of frustration with scalability issues. It's true that they all have different tradeoffs, strengths and weaknesses, but one significant advantage of Postgres is that it does so well in such a variety of situations. Maybe it isn't perfect for every scenario, but it's pretty good for most scenarios. This is partly an inherent advantage of age, and partly the benefit of a sound theoretical foundation. A big thing I dislike about the current crop of NoSQL databases is that you are expected to learn a great deal about their inner workings to make an informed decision about whether or not to use them. MongoDB really epitomizes this. But this is a pretty lazy complaint compared to random data loss and insane defaults (early days of Mongo), and I have to give you that. Your position sounds quite reasonable compared to the one I've been primed by, but it's going to be a few more years before these databases mature to the point I'd consider one a reasonable first choice for a generic database. I don't draw the line quite as far as the GP—sometimes you have a special case and you need special tools—but I can certainly empathize with questioning the wisdom of making a distributed non-relational database the primary store for a low-load, classic RDBMS problem. But Twilio has come forward and agreed with that position and denying that they do that, so the spark for this flamewar is conjecture. While people have come forward to defend such a configuration, few seem to be endorsing it by using it themselves. This says something about the reasonableness of the idea.
- hhuuggoo 13y ago
  I use nosql stores for the same reason that I use a dynamically typed language (python) I like the flexibility and agility. I don't use nosql stores for performance
newman314 13y ago
So what happens when there is a failed transaction mid-flight? It's already in redis but not in RDBMS. How do you rollback then?
marshray 13y ago
Regardlesss, something about the redis query returning a zero balance was triggering bad behavior in the billing system. Also, reading between the lines in the incident report made it sound as if there might have been multiple teams involved in the troubleshooting and not communicating perfectly. For example, were the redis admins informed that customers had been getting billed repeatedly when the decision was made to restart the billing system? Did they have access to the billing system logs which might have contained errors related to redis being read-only? All in all, big props to Twilio for starting to get customer accounts credited back within 11 hours of the first trouble and even more for their wonderful open disclosure.

justincormack 13y ago

But it does appear (if I understand correctly) that sometimes you used Redis as if it was the system of record rather than checking the actual one, hence the rebilling?

PommeDeTerre 13y ago

Is there actually a legitimate performance or scalability need to incorporate a NoSQL database in this case? Ever since NoSQL databases started receiving a lot of hype a few years back, I've witnessed a number of teams use them without any real justification. They'll build unnecessarily complex systems using one or more NoSQL database systems, all while a relational database would be more than sufficient for their needs. In several of these cases, some of the developers have been quite adamant that these NoSQL databases are essential. Then we rip them out, usually because they've been causing problems like described in this case. It quickly becomes obvious that they were never needed in the first place, and won't be needed even in the face of significantly increased load.

AYBABTME 13y ago
Some problems are better solved with key-value stores than with relational databases. It depends on the data model you need to map. Lot of people use relational databases as a big key value store, when really they should use a key value store. It's not necessarily about scalability or performance. =)
- aidos 13y ago
  Yes. And Redis provides an interesting model with very specific performance characteristics depending on how you choose to store your data in each case. If you haven't worked with Redis it might be hard to appreciate how it's a bit more than just a key-value store and how it's a different tool from a relational db.
threeseed 13y ago
No offence but attitudes like this are the worst. If we all took your advice we would all be still using punch cards or writing everything in assembler. Sometimes you don't need a clear justification to use newer technologies. Perhaps developers just want to experience the significant developer productivity that comes with using many of the NoSQL databases. Also might be worth dropping the whole "SQL is better" insinuation. We have seen some pretty major data loss bugs in PostgreSQL and MySQL recently.
- badclient 13y ago
  How are NoSql databases more productive than SQL?
- PommeDeTerre 13y ago
  Billing systems are not to be taken lightly, namely because money is inherently involved. When developing such systems, it is irresponsible to use new, unproven technologies without justification. When developing such systems, it is irresponsible to trade off the reliability and safety of the system for some "developer productivity". Such irresponsibility is just not acceptable. Failures due to such irresponsibility should not be tolerated, either.
  - taf2 13y ago
    money can be refunded... we're not talking about a life - or a human life we're just talking about software. Reading how Twilio was using redis it sounds very reasonable (fsync). I've seen MySQL servers melt under extreme load running on very high end servers so if you're saying that somehow having had SQL in this case would have saved them... well... maybe you're right... but I wasn't there were you? I believe the engineers at Twilio are building and have built a platform to handle a scale that is going to continue to reach further than many traditional models will scale. It makes sense to me that they would need to look for in memory systems to push the limits - They are fsync'ing immediately - so while it's in memory for a read it's disk for writes so that sounds pretty safe to me... Also, a lot of people use Redis now, so I definitely wouldn't say it's unproven, that is as others have said like saying we should still be using typewriters to put ink on paper because it's "proven"... I'd answer that with... "good luck with that". And at the end of the day - "the shit will hit the fan" and it did... the team did great and we're all thankful.
KirinDave 13y ago
> Is there actually a legitimate performance or scalability need to incorporate a NoSQL database in this case? Usually the most compelling reason not to use a traditional SQL database is that they're expensive to field in distributed environments and require specialist engineers to support when you get out into the weeds. But I submit it's a category error to call Redis a "NoSQL database." It was also a category error to put data that needs to be redundant and durable into a storage system that is really specialized for high volume but low-reliability cache work. I mean, seriously. Minimum viable PostgresSQL deploy in EC2 is pretty damn expensive, but you may not have enough cash on hand to buy your own hardware and pay to put it in a "real datacenter." If you can find something cheaper and you're not even seed funded yet because no one funds your social product until you have 100k users, you're probably more likely to actually make it to where you can afford a different database by just using Redis. It's called "technical debt" and we accrue it both bitterly and often willingly.
encoderer 13y ago
I think you're making a number of mistakes in judgement here. Most importantly, you're conflating NoSQL with "not as reliable." As if there's no difference between Cassandra, HBase, Redis, etc. They're just "nosql" and "not needed" to you. That just seems highly ignorant of the relative merits of each database. And similarly, the attitude that a RDBMS is the right and proper data store, that all data should rightly prefer to live in an RDBMS aside from the loftiest heights of scalability. Finally, the assertion that using a key-value or column-based datastore is going to lead to an "unnecessarily complex" design. I'd not be surprised to find you disagree, but if we were working together on the same team, these are some things I'd try to work with you on.
- fusiongyro 13y ago
  I can't speak for the person you're replying to, but I would like to point out that RDBMSes are built on theory designed explicitly to handle general data problems. Situations for which the relational model is not a good fit are almost by definition special cases. Similarly, while it is wise not to lump all of the non-SQL/non-relational databases together, to say that the bulk of them share a common obsession with scalability is not much of a reach. After all, we're not talking about Neo4J, FramerD, MetaKit, etc. We're talking about young systems that were born out of frustration with scalability issues. It's true that they all have different tradeoffs, strengths and weaknesses, but one significant advantage of Postgres is that it does so well in such a variety of situations. Maybe it isn't perfect for every scenario, but it's pretty good for most scenarios. This is partly an inherent advantage of age, and partly the benefit of a sound theoretical foundation. A big thing I dislike about the current crop of NoSQL databases is that you are expected to learn a great deal about their inner workings to make an informed decision about whether or not to use them. MongoDB really epitomizes this. But this is a pretty lazy complaint compared to random data loss and insane defaults (early days of Mongo), and I have to give you that. Your position sounds quite reasonable compared to the one I've been primed by, but it's going to be a few more years before these databases mature to the point I'd consider one a reasonable first choice for a generic database. I don't draw the line quite as far as the GP—sometimes you have a special case and you need special tools—but I can certainly empathize with questioning the wisdom of making a distributed non-relational database the primary store for a low-load, classic RDBMS problem. But Twilio has come forward and agreed with that position and denying that they do that, so the spark for this flamewar is conjecture. While people have come forward to defend such a configuration, few seem to be endorsing it by using it themselves. This says something about the reasonableness of the idea.
hhuuggoo 13y ago
I use nosql stores for the same reason that I use a dynamically typed language (python) I like the flexibility and agility. I don't use nosql stores for performance

newman314 13y ago

So what happens when there is a failed transaction mid-flight? It's already in redis but not in RDBMS. How do you rollback then?

marshray 13y ago

Regardlesss, something about the redis query returning a zero balance was triggering bad behavior in the billing system. Also, reading between the lines in the incident report made it sound as if there might have been multiple teams involved in the troubleshooting and not communicating perfectly. For example, were the redis admins informed that customers had been getting billed repeatedly when the decision was made to restart the billing system? Did they have access to the billing system logs which might have contained errors related to redis being read-only? All in all, big props to Twilio for starting to get customer accounts credited back within 11 hours of the first trouble and even more for their wonderful open disclosure.