4 ms·

AdWords was designed with sharding built into it's application layer. This essentially means that the developers were forced to implement a custom app-specific

by h1karu 12y ago

AdWords was designed with sharding built into it's application layer. This essentially means that the developers were forced to implement a custom app-specific persistence layer that rides on top of MySQL. This limits your ability to write SQL queries because you can only query within a particular shard and that decreases the usefulness of mysql and makes it feel more like a nosql data-store. At that point you find yourself asking why didn't we just use a NoSQL store ? And often times the answer is "because we wanted to use something we were already familiar with". Sometimes people are willing to add a lot more complexity to their application just to allow themselves to avoid having to learn something new.

balfirevic 12y ago
"At that point you find yourself asking why didn't we just use a NoSQL store" Because you still have the full ability to write general SQL within the shard. For many types of application this is useful.
- threeseed 12y ago
  Sure. But only if ALL the data within that query exists on the shard. It is rare that this would happen if you have anything resembling a normalised schema. In which case you would still be doing a lot of joins in your application layer. IMHO Sharded MySQL very much belongs in the NoSQL camp.
  - twic 12y ago
    I don't know that this is true. Imagine you're LeanKit, or Fog Creek, and you run a kanban board as a service. Or a bug tracker, CMS, whatever. You have many customers, each of whom has no more than thousands of users and millions of items. There are many relationships between objects belonging to a given customer, but precisely zero relationships between objects belonging to different customers. Shard using the customer identity as a key, and you have nicely spread-out data and the ability to do any query the application might need to, while still having a normalised schema. There are plenty of other application whose schemas have this property, or almost have it. In my company, we make financial applications, and a lot of the data has very similar siloed ownership structure. The one thing you can't do is reporting queries across your customers. That doesn't seem like a killer, though - it's normal to farm that stuff out to an offline reporting database even in single-server environments.
- h1karu 12y ago
  That's a fair point. If you need highly relational queries within a given shard then using a relational database there can make sense, but for many apps the kinds of queries that are used are not highly relational in nature so using a NoSQL store could have reduced the need for all of that app-specific sharding engineering while still allowing for all the queries that were needed by the app. I personally find that SQL relational databases are not flexible enough for my application so I go with a polyglot persistence architecture that ties together a nosql key-value store with a graph database technology. This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally. To each his own.
  - balfirevic 12y ago
    "This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally" That sounds pretty powerful. Do you mind sharing more information about the data storage/querying stack you use?
    - h1karu 12y ago
      NDA and all of that, but if you pick up this book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks http://pragprog.com/book/rwdata/seven-databases-in-seven-wee... and grep for the phrase "polyglot persistence" you'll find some great information to get you started. a video promo for the book: https://www.youtube.com/watch?v=bSAc56YCOaE https://www.youtube.com/watch?v=bSAc56YCOaE
      - twic 12y ago
        Can you talk about what sort of queries you do in Neo4j? Graphs are perhaps the only area where NoSQL may beat relational databases for expressiveness, rather than alleged scalability or ease of use. I've read the documentation for Neo4j and Cypher, but never used them in anger, so i'd be really interested to know more about how they are actually used. In particular, i would love to be able to make a comparison between a graph database and a relational database queried using recursive common table expressions, in the context of a real-world problem.

by h1karu 12y ago

balfirevic 12y ago

"At that point you find yourself asking why didn't we just use a NoSQL store" Because you still have the full ability to write general SQL within the shard. For many types of application this is useful.

threeseed 12y ago
Sure. But only if ALL the data within that query exists on the shard. It is rare that this would happen if you have anything resembling a normalised schema. In which case you would still be doing a lot of joins in your application layer. IMHO Sharded MySQL very much belongs in the NoSQL camp.
- twic 12y ago
  I don't know that this is true. Imagine you're LeanKit, or Fog Creek, and you run a kanban board as a service. Or a bug tracker, CMS, whatever. You have many customers, each of whom has no more than thousands of users and millions of items. There are many relationships between objects belonging to a given customer, but precisely zero relationships between objects belonging to different customers. Shard using the customer identity as a key, and you have nicely spread-out data and the ability to do any query the application might need to, while still having a normalised schema. There are plenty of other application whose schemas have this property, or almost have it. In my company, we make financial applications, and a lot of the data has very similar siloed ownership structure. The one thing you can't do is reporting queries across your customers. That doesn't seem like a killer, though - it's normal to farm that stuff out to an offline reporting database even in single-server environments.
h1karu 12y ago
That's a fair point. If you need highly relational queries within a given shard then using a relational database there can make sense, but for many apps the kinds of queries that are used are not highly relational in nature so using a NoSQL store could have reduced the need for all of that app-specific sharding engineering while still allowing for all the queries that were needed by the app. I personally find that SQL relational databases are not flexible enough for my application so I go with a polyglot persistence architecture that ties together a nosql key-value store with a graph database technology. This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally. To each his own.
- balfirevic 12y ago
  "This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally" That sounds pretty powerful. Do you mind sharing more information about the data storage/querying stack you use?
  - h1karu 12y ago
    NDA and all of that, but if you pick up this book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks http://pragprog.com/book/rwdata/seven-databases-in-seven-wee... and grep for the phrase "polyglot persistence" you'll find some great information to get you started. a video promo for the book: https://www.youtube.com/watch?v=bSAc56YCOaE https://www.youtube.com/watch?v=bSAc56YCOaE
    - twic 12y ago
      Can you talk about what sort of queries you do in Neo4j? Graphs are perhaps the only area where NoSQL may beat relational databases for expressiveness, rather than alleged scalability or ease of use. I've read the documentation for Neo4j and Cypher, but never used them in anger, so i'd be really interested to know more about how they are actually used. In particular, i would love to be able to make a comparison between a graph database and a relational database queried using recursive common table expressions, in the context of a real-world problem.