5 ms·

I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MyS

I can't remember which db, but somebody a while back claimed that one of Amazon's "infinitely scalable" dbs was tons of abstraction on top a massive heap of MySQL instances. I don't trust anything outside core services on AWS. Regardless of whether the rumor I heard is true, it's clear they appreciate quantity over quality.

derwiki 6y ago
Sounds like Dynamo
newscom59 6y ago
What’s the problem with this? Anything that’s scalable is “just an abstraction” on top of a heap of shards/processes/nodes/data centers/whatever.
rubiquity 6y ago
Disclosure: I work at AWS, possibly near the system you're describing. Opinions are my own. If we're talking about the same thing then I think casting stones just because it is based on MySQL is severely misguided. MySQL has decades of optimizations and this particular system at Amazon has solved scaling problems and brought reliability to countless services without ever being the direct cause of an outage (to the best of my knowledge). Indeed, MySQL is not without its flaws but many of these are related to its quirks in transactions and replication which this system completely solves. The cherry on top is that you have a rock solid database with a familiar querying language and a massive knowledge base to get help from when needed. Oh, and did I mention this system supports multiple storage engines besides just MySQL/InnoDB? I for one wish we would open source this system though there are a ton of hurdles both technical and not. I think it would do wonders for the greater tech community by providing a much better option as your needs grow beyond a single node system. It has certainly served Amazon well in that role and I've heard Facebook and YouTube have similar systems based on MySQL. To further address your comment about Amazon/AWS lacking quality: this system is the epitome of our values of pragmatism and focusing our efforts on innovating where we can make the biggest impact. Hand rolling your own storage engines is fun and all but countless others have already spent decades doing so for marginal gains.
- panarky 6y ago
  > the system you're describing Why all the mysterious cloak-and-dagger?
  - rubiquity 6y ago
    It is possible that the person I replied to could be talking about an entirely different piece of software. Another reason is that the specifics of the system I'm referring to are not public knowledge. The more important takeaway is that building on top of MySQL/InnoDB is perfectly fine and that is what I tried to emphasize.
    - offbyone 6y ago
      Especially if it's the one I'm thinking of, too. It's a damn solid architecture.
blandflakes 6y ago
At more than one of my jobs, this has been exactly the right way to horizontally scale relational workloads and has gone very well.
ignoramous 6y ago
I believe someone allegedly from AWS said DynamoDB was written on top of MySQL (on top of InnoDB, really) [0] which would be similar to what Uber and Pinterest did as well. [1] [0] https://news.ycombinator.com/item?id=18426096 https://news.ycombinator.com/item?id=18426096 [1] https://news.ycombinator.com/item?id=12605026 https://news.ycombinator.com/item?id=12605026
WookieRushing 6y ago
This is really common and works pretty awesomely. MySQL is extremely battle tested and there’s lots of experts out there for it. FB built a similar system to maintain their graph: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nosql/ https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos... It’s a ton of tiny DBs that look like one massive eventually consistent DB
rantwasp 6y ago
if this is dynamo, this is just a bunch of BS. you cannot get the latencies DDB has with a ton of abstractions on top of MySql.

derwiki 6y ago

Sounds like Dynamo

newscom59 6y ago

What’s the problem with this? Anything that’s scalable is “just an abstraction” on top of a heap of shards/processes/nodes/data centers/whatever.

rubiquity 6y ago

Disclosure: I work at AWS, possibly near the system you're describing. Opinions are my own. If we're talking about the same thing then I think casting stones just because it is based on MySQL is severely misguided. MySQL has decades of optimizations and this particular system at Amazon has solved scaling problems and brought reliability to countless services without ever being the direct cause of an outage (to the best of my knowledge). Indeed, MySQL is not without its flaws but many of these are related to its quirks in transactions and replication which this system completely solves. The cherry on top is that you have a rock solid database with a familiar querying language and a massive knowledge base to get help from when needed. Oh, and did I mention this system supports multiple storage engines besides just MySQL/InnoDB? I for one wish we would open source this system though there are a ton of hurdles both technical and not. I think it would do wonders for the greater tech community by providing a much better option as your needs grow beyond a single node system. It has certainly served Amazon well in that role and I've heard Facebook and YouTube have similar systems based on MySQL. To further address your comment about Amazon/AWS lacking quality: this system is the epitome of our values of pragmatism and focusing our efforts on innovating where we can make the biggest impact. Hand rolling your own storage engines is fun and all but countless others have already spent decades doing so for marginal gains.

panarky 6y ago
> the system you're describing Why all the mysterious cloak-and-dagger?
- rubiquity 6y ago
  It is possible that the person I replied to could be talking about an entirely different piece of software. Another reason is that the specifics of the system I'm referring to are not public knowledge. The more important takeaway is that building on top of MySQL/InnoDB is perfectly fine and that is what I tried to emphasize.
  - offbyone 6y ago
    Especially if it's the one I'm thinking of, too. It's a damn solid architecture.

blandflakes 6y ago

At more than one of my jobs, this has been exactly the right way to horizontally scale relational workloads and has gone very well.

ignoramous 6y ago

I believe someone allegedly from AWS said DynamoDB was written on top of MySQL (on top of InnoDB, really) [0] which would be similar to what Uber and Pinterest did as well. [1] [0] https://news.ycombinator.com/item?id=18426096 https://news.ycombinator.com/item?id=18426096 [1] https://news.ycombinator.com/item?id=12605026 https://news.ycombinator.com/item?id=12605026

WookieRushing 6y ago

This is really common and works pretty awesomely. MySQL is extremely battle tested and there’s lots of experts out there for it. FB built a similar system to maintain their graph: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nosql/ https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos... It’s a ton of tiny DBs that look like one massive eventually consistent DB

rantwasp 6y ago

if this is dynamo, this is just a bunch of BS. you cannot get the latencies DDB has with a ton of abstractions on top of MySql.