3 ms·

That's a fair point. If you need highly relational queries within a given shard then using a relational database there can make sense, but for many apps the ki

by h1karu 12y ago

That's a fair point. If you need highly relational queries within a given shard then using a relational database there can make sense, but for many apps the kinds of queries that are used are not highly relational in nature so using a NoSQL store could have reduced the need for all of that app-specific sharding engineering while still allowing for all the queries that were needed by the app. I personally find that SQL relational databases are not flexible enough for my application so I go with a polyglot persistence architecture that ties together a nosql key-value store with a graph database technology. This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally. To each his own.

balfirevic 12y ago
"This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally" That sounds pretty powerful. Do you mind sharing more information about the data storage/querying stack you use?
- h1karu 12y ago
  NDA and all of that, but if you pick up this book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks http://pragprog.com/book/rwdata/seven-databases-in-seven-wee... and grep for the phrase "polyglot persistence" you'll find some great information to get you started. a video promo for the book: https://www.youtube.com/watch?v=bSAc56YCOaE https://www.youtube.com/watch?v=bSAc56YCOaE
  - twic 12y ago
    Can you talk about what sort of queries you do in Neo4j? Graphs are perhaps the only area where NoSQL may beat relational databases for expressiveness, rather than alleged scalability or ease of use. I've read the documentation for Neo4j and Cypher, but never used them in anger, so i'd be really interested to know more about how they are actually used. In particular, i would love to be able to make a comparison between a graph database and a relational database queried using recursive common table expressions, in the context of a real-world problem.
    - h1karu 12y ago
      The approach I like is to use a key/value document store as the system of record (SOR). If any data conflicts arise between Redis, the document store, and the graph database, then the SOR wins. A good SOR should contain all of the data necessary to rebuild any other data source in its domain. Graph databases can scale write throughput fine when those writes are being fed to it from a single source so it's best to have a service who's sole purpose is to keep the graph database in sync with the SOR. The graph should only store the data you actually need in order to get the queries that you want. The specific queries in my domain are exactly the kind of thing I'm not supposed to talk about, but I will say you can do things like: Give me 50 users who who live near me (lat/long bounding box), who I'm not already following, who I have not already sent a message to, who have at least two friends who also live near me, and who have matching tags, order by number of followers. there are a bunch of great examples in this awesome book: http://www.amazon.com/Graph-Databases-Ian-Robinson/dp/1449356265 http://www.amazon.com/Graph-Databases-Ian-Robinson/dp/144935...
      - twic 12y ago
        Thanks. Your example sounds like something it would be pretty easy to express with a RCTE in a relational database. How the performance would compare, i have no idea.
        h1karu 12y ago
        Yea performance is the thing. You're correct that most of these queries can also be expressed in a relational DB. The underlying data structures involved in Neo4J do help with performance on these kinds of queries, but in addition to that I find that for me the Cypher query language feels like a more concise and elegant way to ask for the data. With RCTE it feels like I'm first asking the DB to construct a custom data structure, then I'm asking the DB to query that custom data-structure. With Cypher it feels like I'm simply asking for the exact data I need by specifying the relationships and attribute characteristics that I care about. This might just be a matter of taste, but if you start playing around with Cypher it just might grow on you.

by h1karu 12y ago

balfirevic 12y ago

"This way I can use the cypher query language to access data relationally across all my shards while scaling writes and Big-data horizontally" That sounds pretty powerful. Do you mind sharing more information about the data storage/querying stack you use?

h1karu 12y ago
NDA and all of that, but if you pick up this book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks http://pragprog.com/book/rwdata/seven-databases-in-seven-wee... and grep for the phrase "polyglot persistence" you'll find some great information to get you started. a video promo for the book: https://www.youtube.com/watch?v=bSAc56YCOaE https://www.youtube.com/watch?v=bSAc56YCOaE
- twic 12y ago
  Can you talk about what sort of queries you do in Neo4j? Graphs are perhaps the only area where NoSQL may beat relational databases for expressiveness, rather than alleged scalability or ease of use. I've read the documentation for Neo4j and Cypher, but never used them in anger, so i'd be really interested to know more about how they are actually used. In particular, i would love to be able to make a comparison between a graph database and a relational database queried using recursive common table expressions, in the context of a real-world problem.
  - h1karu 12y ago
    The approach I like is to use a key/value document store as the system of record (SOR). If any data conflicts arise between Redis, the document store, and the graph database, then the SOR wins. A good SOR should contain all of the data necessary to rebuild any other data source in its domain. Graph databases can scale write throughput fine when those writes are being fed to it from a single source so it's best to have a service who's sole purpose is to keep the graph database in sync with the SOR. The graph should only store the data you actually need in order to get the queries that you want. The specific queries in my domain are exactly the kind of thing I'm not supposed to talk about, but I will say you can do things like: Give me 50 users who who live near me (lat/long bounding box), who I'm not already following, who I have not already sent a message to, who have at least two friends who also live near me, and who have matching tags, order by number of followers. there are a bunch of great examples in this awesome book: http://www.amazon.com/Graph-Databases-Ian-Robinson/dp/1449356265 http://www.amazon.com/Graph-Databases-Ian-Robinson/dp/144935...
    - twic 12y ago
      Thanks. Your example sounds like something it would be pretty easy to express with a RCTE in a relational database. How the performance would compare, i have no idea.
      - h1karu 12y ago
        Yea performance is the thing. You're correct that most of these queries can also be expressed in a relational DB. The underlying data structures involved in Neo4J do help with performance on these kinds of queries, but in addition to that I find that for me the Cypher query language feels like a more concise and elegant way to ask for the data. With RCTE it feels like I'm first asking the DB to construct a custom data structure, then I'm asking the DB to query that custom data-structure. With Cypher it feels like I'm simply asking for the exact data I need by specifying the relationships and attribute characteristics that I care about. This might just be a matter of taste, but if you start playing around with Cypher it just might grow on you.