4 ms·

I agree with the principle that good development practices will try to get the coder to think through the potential failure modes of the code they've written.

I agree with the principle that good development practices will try to get the coder to think through the potential failure modes of the code they've written. Unfortunately, I think you've come to the opposite conclusion in this case. When jedberg writes the code for a system backed by an eventually-consistent data store, he asks "What happens if this value is stale and how can I structure the code to cope with that in a sane and reasonable way?" But when $Random_Dev writes the code for such a system, he asks some combination of "What does eventually consistent mean?", "What do you mean I won't get back the right data?", and "Isn't that the DBA's problem?" The core issue being that the average dev will punctuate this line of questioning with things that shift the blame elsewhere; if not the DBA, then "Well, it's good enough for [INSERT_TRENDY_COMPANY_HERE] [ reddit? ;) ], so I think we'll be OK!" or "I don't think that will ever happen to us, our reads get consistent quickly enough". Just whatever will punt the issue down the road. To be honest, I believe that this impulse is the impetus behind the rise in NoSQL in the first place. Devs resent DBAs and SQL. They, very simply, want a place to shove their data, a place to read it out, and to not know anything else about the in between. I have seen many pitiful architectures that clearly turned on this consideration (and we're seeing it all over again with cloud orchestration). Good developers will dig in and search out the meaning of the data whether the failure case is handled at the DB level or not. Highly-structured systems like a traditional RDBMS are very good for resiliency because they require someone to put some definition and formality around the dataset to get basic functionality, and they are generally extremely careful about ensuring the things that come in and out of them are safe and accurate. That's the kind of backing that we really need, especially when there are many developers who can't be trusted to even try to understand more than the bare minimum necessary to get a positive quarterly review.

jedberg 9y ago
Your point is valid, but it makes me sad. We basically come from opposite viewpoints -- I think most devs will do the right thing, you think most will do the wrong thing. Honestly, you're probably right and I'm probably wrong, but I like to hope that I will always work in a place where I'm right. :) (It was Netflix BTW, not reddit. For the most part reddit used strongly consistent databases (Postgres), but we did use some eventual consistency in the caching layer).
- jopsen 9y ago
  Most people don't get consistency right... Outside of AWS, Google and Azure I see few cloud services that define their consistency models. And support always says: we are high availability and always consistent. Same goes for distributed software, few projects define their consistency model, and even fewer honor it.
  - StillBored 9y ago
    I agree with this 100%, the problem is the eventual consistency model opens up the failure windows. Its the case of the private toll agency here in TX that had a full blown court case/collections agency trying to collect (thousands for literally $2.75 in tolls) from me, while sending information to my 5 year out of date address. That despite the fact that I changed my address when I moved (says so in the TxDPS record, running on a "legacy" system), which got sent to the toll agency when I changed it. Of course somewhere in there the toll agency lost this information and never bothered to check the TxDPS record until I pointed out they were repeatedly sending notices to the wrong address. The point being that, this is only one of many problems they have had with their newfangled toll system because sometimes "eventually consistent" actually means "we threw the update away because a newer one came in, so we lost a critical piece of information.". Critical pieces of information like, people actually paying their tolls for example. http://www.texasturf.org/2012-06-01-03-09-30/latest-news/484-erroneous-toll-billing-by-txdot http://www.texasturf.org/2012-06-01-03-09-30/latest-news/484... http://story.kxan.com/txtag-troubles/ http://story.kxan.com/txtag-troubles/ https://www.reddit.com/r/Austin/comments/78wp9y/txtag_payment_processing_down_no_notices_anywhere/?st=jch2j8os&sh=6125cf32 https://www.reddit.com/r/Austin/comments/78wp9y/txtag_paymen...
- lulmerchant 9y ago
  I have a slightly different perspective. I think defaulting to NoSQL is simply removing a mature toolset from the developers kit, that they would have a lot of great use cases for. If you use it in the wrong places, then that almost certainly means increases complexity where you don't necessarily have to. Whether or not that results other positive behaviours seems a bit beside the point to me. If you managed a team of rock climbers, you could say "I prefer it when my rock climbers don't use safety ropes. It improves their climbing, because if they make a mistake, they'll fall to their death". It could be 100% true, but it doesn't mean you shouldn't simply be picking the right tools for the job.
scarface74 9y ago
Mongo with a strongly typed language also encourages someone to put some definition around the dataset. That someone is just the developer creating the PO*O (Plain Old [C#/Java, etc] object). In C# for instance using the standard Mongo driver, you're dealing with strongly typed collections and the compiler is keeping you from mistakingly going outside of the "schema". I don't know how good the other .Net drivers are for other Nosql databases.
lmm 9y ago
> Highly-structured systems like a traditional RDBMS are very good for resiliency because they require someone to put some definition and formality around the dataset to get basic functionality, and they are generally extremely careful about ensuring the things that come in and out of them are safe and accurate. That's not what happens in practice. Just because an RDBMS has transactions doesn't mean a programmer who isn't trying is going to use them correctly. And when you do enforce constraints at the DB level, you end up duplicating all your logic in two different languages, one of which isn't very nice for programming in. E.g. I've never found database timezone handling to be anything other than harmful. The application is just better at enforcing this kind of constraints; a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level.
- cookiecaper 9y ago
  >Just because an RDBMS has transactions doesn't mean a programmer who isn't trying is going to use them correctly. Definitely true, but RDBMS provide a lot of safety features beyond transactions, like a fixed schema. Mongo et al let developers stuff anything they want anywhere, creating an abominable bastardization between a key-value store and a database. That's their "killer feature" that has resulted in wide adoption so they can't really do away with it, even if some [likely reluctant] Mongo users continue to find ways to self-enforce a schema. It'd be one thing if Mongo was predominantly used as a staging ground for free-form, unpredictable data, but I think we all know that's not the case. People pretend like it's a replacement for a traditional RDBMS. >a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level. I really disagree here. The semantics and detail involved in _actually_ making sure a datum is securely stored, including a high likelihood of being recoverable in the case of a crash, is very complex in any non-trivial system. It's dismissive and naive to pretend that your average application developer, who generally doesn't even know what `fsync()` is, can do a better job at this than a real database, which will include advanced consistency features like intent logs.

jedberg 9y ago

Your point is valid, but it makes me sad. We basically come from opposite viewpoints -- I think most devs will do the right thing, you think most will do the wrong thing. Honestly, you're probably right and I'm probably wrong, but I like to hope that I will always work in a place where I'm right. :) (It was Netflix BTW, not reddit. For the most part reddit used strongly consistent databases (Postgres), but we did use some eventual consistency in the caching layer).

jopsen 9y ago
Most people don't get consistency right... Outside of AWS, Google and Azure I see few cloud services that define their consistency models. And support always says: we are high availability and always consistent. Same goes for distributed software, few projects define their consistency model, and even fewer honor it.
- StillBored 9y ago
  I agree with this 100%, the problem is the eventual consistency model opens up the failure windows. Its the case of the private toll agency here in TX that had a full blown court case/collections agency trying to collect (thousands for literally $2.75 in tolls) from me, while sending information to my 5 year out of date address. That despite the fact that I changed my address when I moved (says so in the TxDPS record, running on a "legacy" system), which got sent to the toll agency when I changed it. Of course somewhere in there the toll agency lost this information and never bothered to check the TxDPS record until I pointed out they were repeatedly sending notices to the wrong address. The point being that, this is only one of many problems they have had with their newfangled toll system because sometimes "eventually consistent" actually means "we threw the update away because a newer one came in, so we lost a critical piece of information.". Critical pieces of information like, people actually paying their tolls for example. http://www.texasturf.org/2012-06-01-03-09-30/latest-news/484-erroneous-toll-billing-by-txdot http://www.texasturf.org/2012-06-01-03-09-30/latest-news/484... http://story.kxan.com/txtag-troubles/ http://story.kxan.com/txtag-troubles/ https://www.reddit.com/r/Austin/comments/78wp9y/txtag_payment_processing_down_no_notices_anywhere/?st=jch2j8os&sh=6125cf32 https://www.reddit.com/r/Austin/comments/78wp9y/txtag_paymen...
lulmerchant 9y ago
I have a slightly different perspective. I think defaulting to NoSQL is simply removing a mature toolset from the developers kit, that they would have a lot of great use cases for. If you use it in the wrong places, then that almost certainly means increases complexity where you don't necessarily have to. Whether or not that results other positive behaviours seems a bit beside the point to me. If you managed a team of rock climbers, you could say "I prefer it when my rock climbers don't use safety ropes. It improves their climbing, because if they make a mistake, they'll fall to their death". It could be 100% true, but it doesn't mean you shouldn't simply be picking the right tools for the job.

scarface74 9y ago

Mongo with a strongly typed language also encourages someone to put some definition around the dataset. That someone is just the developer creating the PO*O (Plain Old [C#/Java, etc] object). In C# for instance using the standard Mongo driver, you're dealing with strongly typed collections and the compiler is keeping you from mistakingly going outside of the "schema". I don't know how good the other .Net drivers are for other Nosql databases.

lmm 9y ago

> Highly-structured systems like a traditional RDBMS are very good for resiliency because they require someone to put some definition and formality around the dataset to get basic functionality, and they are generally extremely careful about ensuring the things that come in and out of them are safe and accurate. That's not what happens in practice. Just because an RDBMS has transactions doesn't mean a programmer who isn't trying is going to use them correctly. And when you do enforce constraints at the DB level, you end up duplicating all your logic in two different languages, one of which isn't very nice for programming in. E.g. I've never found database timezone handling to be anything other than harmful. The application is just better at enforcing this kind of constraints; a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level.

cookiecaper 9y ago
>Just because an RDBMS has transactions doesn't mean a programmer who isn't trying is going to use them correctly. Definitely true, but RDBMS provide a lot of safety features beyond transactions, like a fixed schema. Mongo et al let developers stuff anything they want anywhere, creating an abominable bastardization between a key-value store and a database. That's their "killer feature" that has resulted in wide adoption so they can't really do away with it, even if some [likely reluctant] Mongo users continue to find ways to self-enforce a schema. It'd be one thing if Mongo was predominantly used as a staging ground for free-form, unpredictable data, but I think we all know that's not the case. People pretend like it's a replacement for a traditional RDBMS. >a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level. I really disagree here. The semantics and detail involved in _actually_ making sure a datum is securely stored, including a high likelihood of being recoverable in the case of a crash, is very complex in any non-trivial system. It's dismissive and naive to pretend that your average application developer, who generally doesn't even know what `fsync()` is, can do a better job at this than a real database, which will include advanced consistency features like intent logs.