3 ms·

NoSQL Required Reading

55 points by techdog 17y ago

nicpottier 17y ago
Although it is sometimes hard to make work in all cases, I've recently moved a big project from MySQL to a Amazon's SimpleDB and S3 and it has turned out awesome. You have to design the system from the ground up to deal with the 'eventual consistency' problem, but that problem is fairly tractable and easy to wrap your head around. And the reward is 'infinite' scalability and no single points of failure. Good stuff. SQL is still great for the simple projects where you don't need to worry about uptime or scalability, but for everything else I'm sold on AWS (or BigTable)
- weeksie 17y ago
  I have a question regarding this actually. I can see NoSQL style data stores being really well suited to things like sets of permissions or user profiles (where there are a lot of fields that may or may not be blank and/or arbitrary). However, how do you model the eventual relationships that are necessary? All of the articles that come up seem to deal with _what_ a NoSQL data store might be, rather than how to build an application with one. Perhaps I'm missing something really obvious (which is often the case) but I'd love to hear more about how you modeled your data.
  - nicpottier 17y ago
    Can you give me an example of what you are thinking about? There's is no doubt SimpleDB especially can be a bit odd, it kind of forces you to create your own munged columns if you are doing queries across fields etc, but I've yet to run across anything I couldn't code around. (I've also built some libs to help to that end) Another thing which may not be obvious, is that SimpleDB in particular allows for more than one value per attribute, which often will save you the need for a subtable. IE, a user 'row' can have more than one value for the email 'column' or whatnot, and you can query on that intelligently.
    - weeksie 17y ago
      I guess an example of what I am talking about is a user's profile. Say I have a user account, which has a profile. That profile data is stored in a NoSQL document store. Now I want to get all of the profiles that are over 30 and match them with (for example, I'm reaching here) interests that are indexed by age bracket. How do I find and match my over 30 users with their interests? The example itself is a poor one but illustrates my confusion at how to go about building an entire app in a NoSQL db.
      - astine 17y ago
        I'd hand code a tagging system. I'd create arbitrary tag/category pairs, associate them with users and sort them into persistent maps stored by user, category and tag. Ultimately, if you want to add relational features to a non-relational store, you need to model the relations explicitly rather than relying on the underlying features of the store.
        weeksie 17y ago
        Cheers, thanks for that. I figured that was essentially the way to solve the problem. That said, I imagine that I'll be using a combination of both document stores and SQL databases on my next (largish) project.
      - nicpottier 17y ago
        Ya, I think there are definitely cases where it becomes very burdensome to use a NoSQL setup. Certainly anything where you are wanting to do analysis in the DB doesn't work very well. In your particular case I guess you'd have to do two queries. One to grab the users in the range you have, then another to grab the interests for that range. One interesting bit is that you could fire those off in // if you so wished, then do the merge. As I said earlier, it does kind of require you to really think through your access patterns ahead of time before going the NoSQL route. And at least in SimpleDB there is no doubt not being able to query across 'tables' can be a big letdown. But really it is those kinds of queries that eventually turn into problems, so to some extent it just forces you to think of the right way of doing things first. (I know, and agree that optimize last is usually a better strategy)

55 points by techdog 17y ago

Although it is sometimes hard to make work in all cases, I've recently moved a big project from MySQL to a Amazon's SimpleDB and S3 and it has turned out awesome. You have to design the system from the ground up to deal with the 'eventual consistency' problem, but that problem is fairly tractable and easy to wrap your head around. And the reward is 'infinite' scalability and no single points of failure. Good stuff. SQL is still great for the simple projects where you don't need to worry about uptime or scalability, but for everything else I'm sold on AWS (or BigTable)

weeksie 17y ago
I have a question regarding this actually. I can see NoSQL style data stores being really well suited to things like sets of permissions or user profiles (where there are a lot of fields that may or may not be blank and/or arbitrary). However, how do you model the eventual relationships that are necessary? All of the articles that come up seem to deal with _what_ a NoSQL data store might be, rather than how to build an application with one. Perhaps I'm missing something really obvious (which is often the case) but I'd love to hear more about how you modeled your data.
- nicpottier 17y ago
  Can you give me an example of what you are thinking about? There's is no doubt SimpleDB especially can be a bit odd, it kind of forces you to create your own munged columns if you are doing queries across fields etc, but I've yet to run across anything I couldn't code around. (I've also built some libs to help to that end) Another thing which may not be obvious, is that SimpleDB in particular allows for more than one value per attribute, which often will save you the need for a subtable. IE, a user 'row' can have more than one value for the email 'column' or whatnot, and you can query on that intelligently.
  - weeksie 17y ago
    I guess an example of what I am talking about is a user's profile. Say I have a user account, which has a profile. That profile data is stored in a NoSQL document store. Now I want to get all of the profiles that are over 30 and match them with (for example, I'm reaching here) interests that are indexed by age bracket. How do I find and match my over 30 users with their interests? The example itself is a poor one but illustrates my confusion at how to go about building an entire app in a NoSQL db.
    - astine 17y ago
      I'd hand code a tagging system. I'd create arbitrary tag/category pairs, associate them with users and sort them into persistent maps stored by user, category and tag. Ultimately, if you want to add relational features to a non-relational store, you need to model the relations explicitly rather than relying on the underlying features of the store.
      - weeksie 17y ago
        Cheers, thanks for that. I figured that was essentially the way to solve the problem. That said, I imagine that I'll be using a combination of both document stores and SQL databases on my next (largish) project.
    - nicpottier 17y ago
      Ya, I think there are definitely cases where it becomes very burdensome to use a NoSQL setup. Certainly anything where you are wanting to do analysis in the DB doesn't work very well. In your particular case I guess you'd have to do two queries. One to grab the users in the range you have, then another to grab the interests for that range. One interesting bit is that you could fire those off in // if you so wished, then do the merge. As I said earlier, it does kind of require you to really think through your access patterns ahead of time before going the NoSQL route. And at least in SimpleDB there is no doubt not being able to query across 'tables' can be a big letdown. But really it is those kinds of queries that eventually turn into problems, so to some extent it just forces you to think of the right way of doing things first. (I know, and agree that optimize last is usually a better strategy)