3 ms·

My point is that I've seen very, very few situations (I can think of two in my entire career so far) where a "hyperscale NoSQL database" was actually the right

My point is that I've seen very, very few situations (I can think of two in my entire career so far) where a "hyperscale NoSQL database" was actually the right choice to solve the problem. I find that a lot of folks turn to these databases for imagined scale needs, not actual hard problems that need solving.

eklitzke 3y ago
Spanner has a SQL interface, fyi.
- nameless912 3y ago
  I don't think the API you interface with fundamentally changes the point that Spanner is hard to recommend from an engineering perspective at anything except the absolute most massive of scales, and even then it will create nearly as many problems as it solves. I'm not saying spanner is _wrong_ or shouldn't exist, but it's very difficult to be in the position where Spanner is the critical key to your application's success and not replaceable by <insert other, cheaper database here>.
  - endorphine 3y ago
    Care to elaborate on what are the problems that it will create? Honest question.
    - nameless912 3y ago
      Sure. Spanner is expensive, and your primary job as an engineer (if you work for an enterprise like most of us do) is to generate business value. So, if nothing else, you will run into the cost problems of Spanner. There are also other problems; iirc both DynamoDB and Spanner shard their key spaces, and each shard gets the same quota, and the key space shards all have to be the same size. This means that even though you might have paid for 1000rps, for example, that RPS volume is divided across all your shards, so if you have one part of the key space that gets way more volume than another you end up eating up the fractional capacity of that shard way faster than you intend and you have to either overprovision or queue requests, both of which are not ideal. At a previous job, we ended up creating a very complicated write through cache system in front of spanner that dynamically added memory/CPU capacity as needed to prevent hot shards; our application was extremely read heavy, and writes were relatively low RPS, so this ended up working OK, but we were paying tens of thousands of dollars a month for Spanner plus tens of thousands of dollars a month for all the compute sitting in front of it. I don't think we ended up doing much better than if we had bitten the bullet and run clustered Postgres because our write volume ended up being just a few hundred RPS, even though the read volume was 1000x that. Postgres behind this cache system would have handled the load just as well and cost less than half as much. The other thing that frustrates me personally about Spanner is that Google's docs are incomplete (as usual); there are lots of performance gotchas like this that exist throughout the entire service, and they aren't clearly documented (unlike, to their credit, AWS with Dynamo, who explains this entire problem very clearly and has an [expensive] prebuilt solution for it in the form of the DynamoDB accelerator).
0xbadcafebee 3y ago
DynamoDB is fantastic for not doing things at scale. It costs a few pennies, there is nothing to set up, it's all managed for you, it is insanely reliable, and it just works. I use it for all kinds of crap that an entire RDBMS is way overkill for.

eklitzke 3y ago

Spanner has a SQL interface, fyi.

nameless912 3y ago
I don't think the API you interface with fundamentally changes the point that Spanner is hard to recommend from an engineering perspective at anything except the absolute most massive of scales, and even then it will create nearly as many problems as it solves. I'm not saying spanner is _wrong_ or shouldn't exist, but it's very difficult to be in the position where Spanner is the critical key to your application's success and not replaceable by <insert other, cheaper database here>.
- endorphine 3y ago
  Care to elaborate on what are the problems that it will create? Honest question.
  - nameless912 3y ago
    Sure. Spanner is expensive, and your primary job as an engineer (if you work for an enterprise like most of us do) is to generate business value. So, if nothing else, you will run into the cost problems of Spanner. There are also other problems; iirc both DynamoDB and Spanner shard their key spaces, and each shard gets the same quota, and the key space shards all have to be the same size. This means that even though you might have paid for 1000rps, for example, that RPS volume is divided across all your shards, so if you have one part of the key space that gets way more volume than another you end up eating up the fractional capacity of that shard way faster than you intend and you have to either overprovision or queue requests, both of which are not ideal. At a previous job, we ended up creating a very complicated write through cache system in front of spanner that dynamically added memory/CPU capacity as needed to prevent hot shards; our application was extremely read heavy, and writes were relatively low RPS, so this ended up working OK, but we were paying tens of thousands of dollars a month for Spanner plus tens of thousands of dollars a month for all the compute sitting in front of it. I don't think we ended up doing much better than if we had bitten the bullet and run clustered Postgres because our write volume ended up being just a few hundred RPS, even though the read volume was 1000x that. Postgres behind this cache system would have handled the load just as well and cost less than half as much. The other thing that frustrates me personally about Spanner is that Google's docs are incomplete (as usual); there are lots of performance gotchas like this that exist throughout the entire service, and they aren't clearly documented (unlike, to their credit, AWS with Dynamo, who explains this entire problem very clearly and has an [expensive] prebuilt solution for it in the form of the DynamoDB accelerator).

0xbadcafebee 3y ago

DynamoDB is fantastic for not doing things at scale. It costs a few pennies, there is nothing to set up, it's all managed for you, it is insanely reliable, and it just works. I use it for all kinds of crap that an entire RDBMS is way overkill for.