14 ms·

What The Heck Are You Actually Using NoSQL For?

121 points by abhijitr 16y ago

jeremyjarvis 16y ago
Most popular use right now is installing on [usually only] two nodes, then writing a blog post about how it's going to change the world, then going back to using good 'ol MySQL etc ;)
panarky 16y ago
I use key-value stores so I can implement the right data model for the right problem. The article cites graph databases as a better fit for graph applications than relational SQL databases. I'm using Redis data structures like sets and sorted sets that are 20x to 100x faster than relational databases. But I still use SQL for most CRUD operations where I care about consistency and volume isn't high enough to require sharding. There's a place for 'NoSQL', but it's not a panacea.
dgregd 16y ago
No ACID db is most useful in free web apps. Where users data is practically worthless. So who cares about occasionally lost data record. When people are paying for a service then ACID db is a must.
- rewind 16y ago
  But it's not always the data by itself that has value. Sure, it might be useless on its own, but if your app loses it and it causes the experience of using the app to be bad, then it doesn't matter how useless you think the data is; it's going to hurt the user's perception of your product. Also, "free web apps" doesn't translate to "worthless user data", although that's a different argument altogether.
- chrisaycock 16y ago
  Twitter is pretty lucky they were able to survive all of the negative press when they kept going down a couple years back.
- nickik 16y ago
  With alots of the big KeyValue store like cassandra you can make your data very save. You just have to traid of other stuff.
- rbranson 16y ago
  Electronic banking systems are far from ACID (think ATM operations, check and credit card transfer processes). Stock exchanges aren't ACID. Amazon.com isn't ACID. Logistics systems (FedEx, UPS, etc) aren't ACID. In fact, if you look at the information systems of Fortune 100 corporations, you'll find that almost every single one of them is non-ACID at the core.
  - jhugg 16y ago
    This is not really very accurate. At their core, each of these systems use ACID databases (or very nearly if you nitpick about isolation levels in Oracle). Between databases and between companies, they've developed "eventually correct" schemes to synchronize information. The lease patterns that many of these systems rely on require atomic operations at their core and offer stronger guarantees than the basic eventually consistent systems popularized by models like Amazon's Dynamo. It's not just that the account values have to agree between systems at the end of the day; they have to actually be correct at the end of the day. I know that consistency models vary between NoSQL systems (and even within a single NoSQL system). There's some great technology out there and plenty of problems to solve. There's are certainly plenty of use cases for NoSQL systems within banks and stock exchanges. But the "banks are eventually consistent" line of reasoning needs to die.
    - spenrose 16y ago
      I take your points. Here's the thing: if ACID is an aspect of developer tools and not a user experiences, and if ACID tools are the default, and if (as anyone who has held a checking account in the U.S. would surely agree) user-experience failures analogous to ACID violations occur regularly, isn't that evidence that we should continue to question what value we actually get from basing our systems on ACID/relational data stores?
      - jhugg 16y ago
        Asking what value a particular app gets from an ACID store vs a non-ACID store as part of a comprehensive analysis of two competing technologies is a GOOD THING. As for banks, exchanges, credit cards, packages, etc: When non-trivial sums of money are changing hands, ACID stores offer tremendous benefits as a building block in a larger system. Yes, things go wrong. Even given ACID building blocks, building huge banking systems is hard. Forcing bank developers to worry about consistency not just between systems, but also within systems isn't going to make that job easier.
benologist 16y ago
I use it mostly to store player-created levels and leaderboards which allows developers to efficiently + effortlessly include and filter by custom fields and data. I also use it for some smaller tasks like storing filenames/hashes for distributing content across my 5 servers (doesn't really need to be mongo for this), and some configuration information for games using me. I do it via MongoHQ, I did have it running and replicated across 2 of my servers but I'm just not familiar enough to keep it running and that bit me in the ass early on. With MongoHQ all I have to worry about is making sure my indexes are right. Next up I'll be using it for some non-aggregated analytics stuff, which won't be as big as Facebook's requirements but will still be in the billions-a-month range.
- vyrotek 16y ago
  Glad to hear that switching to Mongo worked out for you. I remember talking to you a while ago when most stuff was running on SqlServer. For our project we went the Windows Azure route and store our 'events' in their NoSql Table Storage. I must admit I have been tempted to switch to MongoHQ due to the lack of query support from Table Storage. Funny enough, we're also working on a custom leaderboards feature for our customers and the lack of secondary indexes really makes it difficult.
  - benologist 16y ago
    MongoHQ have a pretty awesome product. The only concern I have with them is I noticed they recently capped databases to 20 gig but that's 3x more than my leaderboards come to at the moment so that's future-me's problem. The custom data on the leaderboards and levels is just ridiculously easy, it's basically just copying object properties/values on the user end straight in to document properties/values in MongoDB.
    - benwyrosdick 16y ago
      Future-you should fear not. The 20gb limit is a soft limit and we have many who have gone past it. It is really there for two reasons ... 1) To give people some reasonable expectations about how much data they should be putting on the plan given it is a multi-tenant server and they are sharing resources. 2) To make room for an expanded offering, both a larger high-availability shared plan, and a dedicated server plan.
voxcogitatio 16y ago
The confusion about the use case of NoSQL probably stems from the term "NoSQL" being so vaguely defined. All you can say for sure about one is that it's not relational. But other than that there's not much in common between (for example) key-value stores, graph databases and object databases. Result: The author has to qualify all statements with "only applies to some NoSQL databases".
- cake 16y ago
  I thinks "NoSQL" a bad name too. With NoSQL you would expect the opposite of a db managed by SQL when in fact it's something completetly different.
- jfb 16y ago
  I would love a relational database with some less horrible language than SQL to manage it. That'd be NoSQL, no lie, but considerably different than a KV store.
psadauskas 16y ago
We use TokyoTyrant to store lots (TBs) of tiny (<100bytes), constantly updating bits of data. The record overhead on Mysql and Postgresql was huge, and the continual updates were leaving lots of dead tuples that had to be cleaned, and killed performance. I use MongoDB for greenfield app development and experiments, so I can quickly alter "tables" and "attributes", without having to bother with first thinking about schema, or having to write migrations.
- rbranson 16y ago
  Had the same experience with Tokyo Cabinet/Tyrant on a project. We had hundreds of millions of small records to store, and TCH could pack it into space fall enough to fit into disk cache on an XL EC2 instance.
- seunosewa 16y ago
  What kind of data was that, by the way? (in real-life terms).
die_sekte 16y ago
AFAIK Stylous.com uses vertexdb, a graph database.
rb2k_ 16y ago
I use Redis as a Cache and Queueing System at the same time. Workers connect and get new jobs, they also write the results to redis. Everything touches redis and redis is just so damn fast that I don't have to care about scaling for a while :)
- jfb 16y ago
  We do this as well. Having sets, sorted sets and hash tables built in makes grody hacks much less common, and Redis is seriously fast. We persist the important info back to a more conventional store (Postgres), but we can keep a large proportion of our current working set in memory, in Redis. That rocks.
  - dasil003 16y ago
    This is the very thing that attracted us to redis. We haven't come close to outgrowing a single MySQL server yet, but there was a handful of areas where we felt MySQL lacking. So far we have been able to make major performance improvements and decrease our database size dramatically, without worrying about redis persistence at all. That is, we use it purely in a caching capacity where any data it contains can be reconstructed at any time. The low-hanging fruit have been counter caches (lock contention with mysql counters was a problem wayyy too early), transient data (sessions, ip bans, etc), and picking random items from sets (other RDBMSes may have better ORDER BY random() properties, but MySQL sucks). In general I feel like 90% of our data is best served by a relational database. It's possible to shoehorn the rest in, but redis primitives allow highly targeted improvements to both performance and elegance.
    - jfb 16y ago
      Yeah, not having to implement (say) priority work queues YET AGAIN in a RDBMS was like checking your regulator at 40m and finding that you have enough air to get back to the surface, after all. We do a bunch of calculation with cached data in Redis, and it enables the naïve pattern of grabbing chunks of shit from the data store without having to cope with the horrible SQL generated by the ORM.
gyardley 16y ago
Writing and processing a ton of analytics data - all the raw logs sent into Flurry are stored in HDFS and then processed with map-reduce jobs.
tmcw 16y ago
PakistanSurvey.org is using MongoDB for general data storage of tons of survey responses and aggregation of them. Similar sites we're working on use CouchDB in the same capacity, because we can push simple analysis back into the database and represent more complex data in a natural form.
smhinsey 16y ago
I use a combination of semi-reliable stores, including a NoSql table storage provider, to implement durable messaging for distributed systems with nodes located in any number of data centers or clouds.
StavrosK 16y ago
I used MongoDB for my MSc thesis, and I really enjoyed the silent data corruption feature [1]. That said, I would use MongoDB again if it stops being horribly unstable. These days, I use redis for caching and as a celery backend, and I love it to bits. [1] not.
- mkramlich 16y ago
  please dish. esp what version of MongoDB you used when you had the corruption issue.
  - StavrosK 16y ago
    Here's part 2, there's a link to part 1: http://www.korokithakis.net/node/119 http://www.korokithakis.net/node/119 Also, because everyone is going to skim the post and say "you were using the 32-bit version": 1. Only for some of the corruptions. 2. IT SHOULDN'T CORRUPT DATA ANYWAY!
metamemetics 16y ago
I am investigating replacing MySQL with MongoDB in my model layer for my next prototype. My mind is still thinking in 3NF though. I understand dernormalization and avoiding joins will be useful from a performance standpoint. However I am unsure when to either go ahead and include a foreign key, retrieve it and perform a second query from the application layer, and when to go ahead and duplicate\embed all the field data. I'm leaning towards just making 2 simple sequential key lookup queries, the 2nd on the retrieved foreign key, rather than duplicating fields everywhere and keeping track of massively cascading changes. Instead of performing 1 MySQL Join. Although I usually think in terms of minimizing roundtrips to the database server. Wondering if anyone has a heuristic for this or suggested reading?
- rbranson 16y ago
  Please ask yourself why. In reality, MongoDB has almost all of the same limitations of MySQL and PostgreSQL, but lacks the production proven record. In addition, MongoDB has very weak durability guarantees on a single server, poor performance for data that is not in-memory, and continues some common SQL/ACID scalability pitfalls (use of arbitrary indexes and ad-hoc queries). Outside of this, you need to switch the question you ask from "what data do I need to capture?" to "what questions do I need to answer?"
  - metamemetics 16y ago
    >Please ask yourself why. Looking for a way to hands-free scale very cheaply without vendor lockin. Would be nice if I can simply add another machine to the database to the cluster. And not have to generate the ids in the application layer, use hashing algo to select correct machine, and have everything stop working when a single database goes down. Seems like it should be a solved problem by now. Investigating new tech won't set me back much time, and my MySQL queries aren't going to disappear if I don't like it. Furthermore looking at large sites such as Flickr that massively scaled MySQL it seems like they stopped using its relational features anyway.
    - rbranson 16y ago
jasonjei 16y ago
I've started using NoSQL with an ACID-based system by storing the document revisions (e.g, modified invoices, purchase orders) into a NoSQL database with the SQL database holding the master data. We store document revisions so if we were to ever audit modifications of the invoice, we could pull up the very first version to the second, third, nth revision of the document. Meanwhile, we still keep master data on SQL database.
- blantonl 16y ago
  Sounds like you are doing what we are doing. Testing the waters, evaluating how NoSQL technologies fit into your architecture, and testing your problems against NoSQL solutions. It is refreshing to see the approaches.
  - jasonjei 16y ago
    Thanks for the compliments! I'm still a bit wary of using NoSQL, and we store document revisions (e.g, if I modified invoice 1021, the lastest version would be updated in SQL, and the last version would be pushed into NoSQL) into SQL just in case (by using a draft_id column). We need to see if our software is still deemed auditable by accountants since they have very strict guidelines on how a document trail should be stored.
alecthomas 16y ago
We're using Redis because it's zero-configuration-required. This makes deploying on end-user networks much much easier.
jforman 16y ago
We're (Inkling) successfully using Cassandra in production to store most of our data. I wrote the first version of our data store modeled off of FriendFeed's architecture (back when we were five people). There is an excellent blog post about it here: http://bret.appspot.com/entry/how-friendfeed-uses-mysql http://bret.appspot.com/entry/how-friendfeed-uses-mysql When Cassandra came along, we had long debates internally about the risk of hitching ourselves to their wagon. Eventually we decided that an open source project with a similar (enough) architecture to what we'd been building ourselves was preferable to maintaining a python library in-house. So far we have no regrets. We have had no production issues and we have an architecture that is built to handle a large amount of incoming data (every note and highlight you take in an Inkling book is synched to our servers). We had already stripped away most SQL functionality by using FF's architecture, so we didn't lose much. And we gained a lot - the ability to tune the durability of different writes depending on the use case, far better fault tolerance, a conceptually simpler indexing strategy, etc. It definitely has a learning curve - error and status reporting is not wonderful, you generally need to be over-provisioned as a strategy, etc. But I'm happy we learned those lessons sooner rather than later - I'd hate to scramble to implement Cassandra in a strained production environment (my sympathies to a certain website who suffered through this).
mindcrime 16y ago
This blog entry ( http://www.engineyard.com/blog/2009/ldap-directories-the-forgotten-nosql/ http://www.engineyard.com/blog/2009/ldap-directories-the-for... ) is a little old now, but makes an interesting point - that LDAP was the original "NoSQL." If you buy that idea, lots of people have been using NoSQL - and using it for all sorts of things - for quite some time now.
- jfb 16y ago
  But LDAP is an Abomination unto the eyes of the Lord.
kingnothing 16y ago
Vitrue is currently using an autosharded mongodb cluster in production for writing and reading analytics data for our apps. We're still adding it across the board, but so far its going great.
mkramlich 16y ago
I'm using MongoDB as the default data store for a new geo LBS startup. Scalability and performance features were the main attraction, but also I just like the interfaces and ergonomics it presents to me as a programmer. Much prefer representing stored data structures in JSON, doing queries in RESTful HTTP-ish friendly form, and doing processing in Python, JavaScript, etc. Oh and not having to care about data schemas. At least early on in the project lifecycle, when there's no revenue and you may need to frequently evolve and/or significantly pivot the codebase. I love the fact that it tries to keep everything in memory if it can, and that it was designed from the ground up to be distributed. MongoDB is also rapidly becoming my "goto" data store when making prototypes to explore some new software project space. Because it's so amenable to rapid development and change. It's also replacing memcached in some situations where all I needed was a dumb memory cache. It can act like a dumb memory cache, except it has these extra features waiting in the wings I want, which is a bonus. I liked Redis in my evaluations and may use it more in the future but it lost out to MongoDB for the LBS project because it didn't fit the requirements as well or get enough little "taste wins" with me.
rantfoil 16y ago
At Posterous we use MySQL as a main data store but MongoDB for simple analytics and Redis for set operations around contacts and subscriptions. Redis is also great as a backing store for the Rails queuing system Resque.
fish2000 16y ago
SQL gives some people the howling fantods. I think a large part of the programming-nerd population looks at it and sees it as a kludgy chimera like JCL, or an inscrutably unfunny INTERCAL variant. Maybe they've been forced to use a shitty ORM leaked abstractions on a project of theirs which they had to clean up; maybe someone slipped them the hot SQL injection, back in the days of CGI; maybe their parents were tragically trampled to death by an elephant while a nearby dolphin laughed. Regardless of motiation, you have to concede to whom it riles that SQL is not the most likeable language/model/framework/paradigm out there. Personally, I actually love SQL. You asked me, nothing satisfies like nested right inner joins. But that's not who I am. I have needs. Sometimes, what I need is a schemaless eventually-consistent document-oriented persistent data store, because I am aggregating data from multiple web service APIs whose field names and structures change around like they were samples on a Girl Talk record. CouchDB ain't SlouchDB... I can dance to that. I can tell that some of my buddies are embracing databases from the ranks of the quote-unquote NoSQL movement because these databases' aesthetics are not at all like SQL. That's the catch with NoSQL -- it's a really pointless thing to talk about because it's not a thing; it is an un-thing, a classification of everything in the contemporary database world that is not SQL. It's the kind of classification that makes the most sense in the emotional context of how people feel about SQL.
dadro 16y ago
I'm using Mongodb to store Real Estate MLS data. MLS vendors all have their own schema for storing property data that can change on a regular basis. Rather than attempt to map every field we store each property as a document. We index the important fields (price, beds, baths, etc) but all other "metadata" is stored as a Dirty Attribute (in an embedded document). We are still beta testing and have 100K properties and ~1 million images stored. Query time is faster than our current LAMP site. We will shard based on MLS Vendor as we add more.

121 points by abhijitr 16y ago

Most popular use right now is installing on [usually only] two nodes, then writing a blog post about how it's going to change the world, then going back to using good 'ol MySQL etc ;)

panarky 16y ago

I use key-value stores so I can implement the right data model for the right problem. The article cites graph databases as a better fit for graph applications than relational SQL databases. I'm using Redis data structures like sets and sorted sets that are 20x to 100x faster than relational databases. But I still use SQL for most CRUD operations where I care about consistency and volume isn't high enough to require sharding. There's a place for 'NoSQL', but it's not a panacea.

dgregd 16y ago

No ACID db is most useful in free web apps. Where users data is practically worthless. So who cares about occasionally lost data record. When people are paying for a service then ACID db is a must.

rewind 16y ago
But it's not always the data by itself that has value. Sure, it might be useless on its own, but if your app loses it and it causes the experience of using the app to be bad, then it doesn't matter how useless you think the data is; it's going to hurt the user's perception of your product. Also, "free web apps" doesn't translate to "worthless user data", although that's a different argument altogether.
chrisaycock 16y ago
Twitter is pretty lucky they were able to survive all of the negative press when they kept going down a couple years back.
nickik 16y ago
With alots of the big KeyValue store like cassandra you can make your data very save. You just have to traid of other stuff.
rbranson 16y ago
Electronic banking systems are far from ACID (think ATM operations, check and credit card transfer processes). Stock exchanges aren't ACID. Amazon.com isn't ACID. Logistics systems (FedEx, UPS, etc) aren't ACID. In fact, if you look at the information systems of Fortune 100 corporations, you'll find that almost every single one of them is non-ACID at the core.
- jhugg 16y ago
  This is not really very accurate. At their core, each of these systems use ACID databases (or very nearly if you nitpick about isolation levels in Oracle). Between databases and between companies, they've developed "eventually correct" schemes to synchronize information. The lease patterns that many of these systems rely on require atomic operations at their core and offer stronger guarantees than the basic eventually consistent systems popularized by models like Amazon's Dynamo. It's not just that the account values have to agree between systems at the end of the day; they have to actually be correct at the end of the day. I know that consistency models vary between NoSQL systems (and even within a single NoSQL system). There's some great technology out there and plenty of problems to solve. There's are certainly plenty of use cases for NoSQL systems within banks and stock exchanges. But the "banks are eventually consistent" line of reasoning needs to die.
  - spenrose 16y ago
    I take your points. Here's the thing: if ACID is an aspect of developer tools and not a user experiences, and if ACID tools are the default, and if (as anyone who has held a checking account in the U.S. would surely agree) user-experience failures analogous to ACID violations occur regularly, isn't that evidence that we should continue to question what value we actually get from basing our systems on ACID/relational data stores?
    - jhugg 16y ago
      Asking what value a particular app gets from an ACID store vs a non-ACID store as part of a comprehensive analysis of two competing technologies is a GOOD THING. As for banks, exchanges, credit cards, packages, etc: When non-trivial sums of money are changing hands, ACID stores offer tremendous benefits as a building block in a larger system. Yes, things go wrong. Even given ACID building blocks, building huge banking systems is hard. Forcing bank developers to worry about consistency not just between systems, but also within systems isn't going to make that job easier.

benologist 16y ago

I use it mostly to store player-created levels and leaderboards which allows developers to efficiently + effortlessly include and filter by custom fields and data. I also use it for some smaller tasks like storing filenames/hashes for distributing content across my 5 servers (doesn't really need to be mongo for this), and some configuration information for games using me. I do it via MongoHQ, I did have it running and replicated across 2 of my servers but I'm just not familiar enough to keep it running and that bit me in the ass early on. With MongoHQ all I have to worry about is making sure my indexes are right. Next up I'll be using it for some non-aggregated analytics stuff, which won't be as big as Facebook's requirements but will still be in the billions-a-month range.

vyrotek 16y ago
Glad to hear that switching to Mongo worked out for you. I remember talking to you a while ago when most stuff was running on SqlServer. For our project we went the Windows Azure route and store our 'events' in their NoSql Table Storage. I must admit I have been tempted to switch to MongoHQ due to the lack of query support from Table Storage. Funny enough, we're also working on a custom leaderboards feature for our customers and the lack of secondary indexes really makes it difficult.
- benologist 16y ago
  MongoHQ have a pretty awesome product. The only concern I have with them is I noticed they recently capped databases to 20 gig but that's 3x more than my leaderboards come to at the moment so that's future-me's problem. The custom data on the leaderboards and levels is just ridiculously easy, it's basically just copying object properties/values on the user end straight in to document properties/values in MongoDB.
  - benwyrosdick 16y ago
    Future-you should fear not. The 20gb limit is a soft limit and we have many who have gone past it. It is really there for two reasons ... 1) To give people some reasonable expectations about how much data they should be putting on the plan given it is a multi-tenant server and they are sharing resources. 2) To make room for an expanded offering, both a larger high-availability shared plan, and a dedicated server plan.

voxcogitatio 16y ago

The confusion about the use case of NoSQL probably stems from the term "NoSQL" being so vaguely defined. All you can say for sure about one is that it's not relational. But other than that there's not much in common between (for example) key-value stores, graph databases and object databases. Result: The author has to qualify all statements with "only applies to some NoSQL databases".

cake 16y ago
I thinks "NoSQL" a bad name too. With NoSQL you would expect the opposite of a db managed by SQL when in fact it's something completetly different.
jfb 16y ago
I would love a relational database with some less horrible language than SQL to manage it. That'd be NoSQL, no lie, but considerably different than a KV store.

psadauskas 16y ago

We use TokyoTyrant to store lots (TBs) of tiny (<100bytes), constantly updating bits of data. The record overhead on Mysql and Postgresql was huge, and the continual updates were leaving lots of dead tuples that had to be cleaned, and killed performance. I use MongoDB for greenfield app development and experiments, so I can quickly alter "tables" and "attributes", without having to bother with first thinking about schema, or having to write migrations.

rbranson 16y ago
Had the same experience with Tokyo Cabinet/Tyrant on a project. We had hundreds of millions of small records to store, and TCH could pack it into space fall enough to fit into disk cache on an XL EC2 instance.
seunosewa 16y ago
What kind of data was that, by the way? (in real-life terms).

die_sekte 16y ago

AFAIK Stylous.com uses vertexdb, a graph database.

rb2k_ 16y ago

I use Redis as a Cache and Queueing System at the same time. Workers connect and get new jobs, they also write the results to redis. Everything touches redis and redis is just so damn fast that I don't have to care about scaling for a while :)

jfb 16y ago
We do this as well. Having sets, sorted sets and hash tables built in makes grody hacks much less common, and Redis is seriously fast. We persist the important info back to a more conventional store (Postgres), but we can keep a large proportion of our current working set in memory, in Redis. That rocks.
- dasil003 16y ago
  This is the very thing that attracted us to redis. We haven't come close to outgrowing a single MySQL server yet, but there was a handful of areas where we felt MySQL lacking. So far we have been able to make major performance improvements and decrease our database size dramatically, without worrying about redis persistence at all. That is, we use it purely in a caching capacity where any data it contains can be reconstructed at any time. The low-hanging fruit have been counter caches (lock contention with mysql counters was a problem wayyy too early), transient data (sessions, ip bans, etc), and picking random items from sets (other RDBMSes may have better ORDER BY random() properties, but MySQL sucks). In general I feel like 90% of our data is best served by a relational database. It's possible to shoehorn the rest in, but redis primitives allow highly targeted improvements to both performance and elegance.
  - jfb 16y ago
    Yeah, not having to implement (say) priority work queues YET AGAIN in a RDBMS was like checking your regulator at 40m and finding that you have enough air to get back to the surface, after all. We do a bunch of calculation with cached data in Redis, and it enables the naïve pattern of grabbing chunks of shit from the data store without having to cope with the horrible SQL generated by the ORM.

gyardley 16y ago

Writing and processing a ton of analytics data - all the raw logs sent into Flurry are stored in HDFS and then processed with map-reduce jobs.

tmcw 16y ago

PakistanSurvey.org is using MongoDB for general data storage of tons of survey responses and aggregation of them. Similar sites we're working on use CouchDB in the same capacity, because we can push simple analysis back into the database and represent more complex data in a natural form.

smhinsey 16y ago

I use a combination of semi-reliable stores, including a NoSql table storage provider, to implement durable messaging for distributed systems with nodes located in any number of data centers or clouds.

StavrosK 16y ago

I used MongoDB for my MSc thesis, and I really enjoyed the silent data corruption feature [1]. That said, I would use MongoDB again if it stops being horribly unstable. These days, I use redis for caching and as a celery backend, and I love it to bits. [1] not.

mkramlich 16y ago
please dish. esp what version of MongoDB you used when you had the corruption issue.
- StavrosK 16y ago
  Here's part 2, there's a link to part 1: http://www.korokithakis.net/node/119 http://www.korokithakis.net/node/119 Also, because everyone is going to skim the post and say "you were using the 32-bit version": 1. Only for some of the corruptions. 2. IT SHOULDN'T CORRUPT DATA ANYWAY!

metamemetics 16y ago

I am investigating replacing MySQL with MongoDB in my model layer for my next prototype. My mind is still thinking in 3NF though. I understand dernormalization and avoiding joins will be useful from a performance standpoint. However I am unsure when to either go ahead and include a foreign key, retrieve it and perform a second query from the application layer, and when to go ahead and duplicate\embed all the field data. I'm leaning towards just making 2 simple sequential key lookup queries, the 2nd on the retrieved foreign key, rather than duplicating fields everywhere and keeping track of massively cascading changes. Instead of performing 1 MySQL Join. Although I usually think in terms of minimizing roundtrips to the database server. Wondering if anyone has a heuristic for this or suggested reading?

rbranson 16y ago
Please ask yourself why. In reality, MongoDB has almost all of the same limitations of MySQL and PostgreSQL, but lacks the production proven record. In addition, MongoDB has very weak durability guarantees on a single server, poor performance for data that is not in-memory, and continues some common SQL/ACID scalability pitfalls (use of arbitrary indexes and ad-hoc queries). Outside of this, you need to switch the question you ask from "what data do I need to capture?" to "what questions do I need to answer?"
- metamemetics 16y ago
  >Please ask yourself why. Looking for a way to hands-free scale very cheaply without vendor lockin. Would be nice if I can simply add another machine to the database to the cluster. And not have to generate the ids in the application layer, use hashing algo to select correct machine, and have everything stop working when a single database goes down. Seems like it should be a solved problem by now. Investigating new tech won't set me back much time, and my MySQL queries aren't going to disappear if I don't like it. Furthermore looking at large sites such as Flickr that massively scaled MySQL it seems like they stopped using its relational features anyway.
  - rbranson 16y ago

jasonjei 16y ago

I've started using NoSQL with an ACID-based system by storing the document revisions (e.g, modified invoices, purchase orders) into a NoSQL database with the SQL database holding the master data. We store document revisions so if we were to ever audit modifications of the invoice, we could pull up the very first version to the second, third, nth revision of the document. Meanwhile, we still keep master data on SQL database.

blantonl 16y ago
Sounds like you are doing what we are doing. Testing the waters, evaluating how NoSQL technologies fit into your architecture, and testing your problems against NoSQL solutions. It is refreshing to see the approaches.
- jasonjei 16y ago
  Thanks for the compliments! I'm still a bit wary of using NoSQL, and we store document revisions (e.g, if I modified invoice 1021, the lastest version would be updated in SQL, and the last version would be pushed into NoSQL) into SQL just in case (by using a draft_id column). We need to see if our software is still deemed auditable by accountants since they have very strict guidelines on how a document trail should be stored.

alecthomas 16y ago

We're using Redis because it's zero-configuration-required. This makes deploying on end-user networks much much easier.

jforman 16y ago

We're (Inkling) successfully using Cassandra in production to store most of our data. I wrote the first version of our data store modeled off of FriendFeed's architecture (back when we were five people). There is an excellent blog post about it here: http://bret.appspot.com/entry/how-friendfeed-uses-mysql http://bret.appspot.com/entry/how-friendfeed-uses-mysql When Cassandra came along, we had long debates internally about the risk of hitching ourselves to their wagon. Eventually we decided that an open source project with a similar (enough) architecture to what we'd been building ourselves was preferable to maintaining a python library in-house. So far we have no regrets. We have had no production issues and we have an architecture that is built to handle a large amount of incoming data (every note and highlight you take in an Inkling book is synched to our servers). We had already stripped away most SQL functionality by using FF's architecture, so we didn't lose much. And we gained a lot - the ability to tune the durability of different writes depending on the use case, far better fault tolerance, a conceptually simpler indexing strategy, etc. It definitely has a learning curve - error and status reporting is not wonderful, you generally need to be over-provisioned as a strategy, etc. But I'm happy we learned those lessons sooner rather than later - I'd hate to scramble to implement Cassandra in a strained production environment (my sympathies to a certain website who suffered through this).

mindcrime 16y ago

This blog entry ( http://www.engineyard.com/blog/2009/ldap-directories-the-forgotten-nosql/ http://www.engineyard.com/blog/2009/ldap-directories-the-for... ) is a little old now, but makes an interesting point - that LDAP was the original "NoSQL." If you buy that idea, lots of people have been using NoSQL - and using it for all sorts of things - for quite some time now.

jfb 16y ago
But LDAP is an Abomination unto the eyes of the Lord.

kingnothing 16y ago

Vitrue is currently using an autosharded mongodb cluster in production for writing and reading analytics data for our apps. We're still adding it across the board, but so far its going great.

mkramlich 16y ago

I'm using MongoDB as the default data store for a new geo LBS startup. Scalability and performance features were the main attraction, but also I just like the interfaces and ergonomics it presents to me as a programmer. Much prefer representing stored data structures in JSON, doing queries in RESTful HTTP-ish friendly form, and doing processing in Python, JavaScript, etc. Oh and not having to care about data schemas. At least early on in the project lifecycle, when there's no revenue and you may need to frequently evolve and/or significantly pivot the codebase. I love the fact that it tries to keep everything in memory if it can, and that it was designed from the ground up to be distributed. MongoDB is also rapidly becoming my "goto" data store when making prototypes to explore some new software project space. Because it's so amenable to rapid development and change. It's also replacing memcached in some situations where all I needed was a dumb memory cache. It can act like a dumb memory cache, except it has these extra features waiting in the wings I want, which is a bonus. I liked Redis in my evaluations and may use it more in the future but it lost out to MongoDB for the LBS project because it didn't fit the requirements as well or get enough little "taste wins" with me.

rantfoil 16y ago

At Posterous we use MySQL as a main data store but MongoDB for simple analytics and Redis for set operations around contacts and subscriptions. Redis is also great as a backing store for the Rails queuing system Resque.

fish2000 16y ago

SQL gives some people the howling fantods. I think a large part of the programming-nerd population looks at it and sees it as a kludgy chimera like JCL, or an inscrutably unfunny INTERCAL variant. Maybe they've been forced to use a shitty ORM leaked abstractions on a project of theirs which they had to clean up; maybe someone slipped them the hot SQL injection, back in the days of CGI; maybe their parents were tragically trampled to death by an elephant while a nearby dolphin laughed. Regardless of motiation, you have to concede to whom it riles that SQL is not the most likeable language/model/framework/paradigm out there. Personally, I actually love SQL. You asked me, nothing satisfies like nested right inner joins. But that's not who I am. I have needs. Sometimes, what I need is a schemaless eventually-consistent document-oriented persistent data store, because I am aggregating data from multiple web service APIs whose field names and structures change around like they were samples on a Girl Talk record. CouchDB ain't SlouchDB... I can dance to that. I can tell that some of my buddies are embracing databases from the ranks of the quote-unquote NoSQL movement because these databases' aesthetics are not at all like SQL. That's the catch with NoSQL -- it's a really pointless thing to talk about because it's not a thing; it is an un-thing, a classification of everything in the contemporary database world that is not SQL. It's the kind of classification that makes the most sense in the emotional context of how people feel about SQL.

dadro 16y ago

I'm using Mongodb to store Real Estate MLS data. MLS vendors all have their own schema for storing property data that can change on a regular basis. Rather than attempt to map every field we store each property as a document. We index the important fields (price, beds, baths, etc) but all other "metadata" is stored as a Dirty Attribute (in an embedded document). We are still beta testing and have 100K properties and ~1 million images stored. Query time is faster than our current LAMP site. We will shard based on MLS Vendor as we add more.