18 ms·

Why I Migrated Away From MongoDB

203 points by svs 14y ago

stevencorona 14y ago
The downside, or challenge, with NoSQL (generally speaking) is that you need to handle your aggregations ahead of time - you need to know what queries you'll want to run in the future when you store your data. If you have some new aggregation you want to keep, you'll need to re-process the data (with Hadoop or something else). It's the trade-off of being able to scale reads and writes horizontally. And unless you need it, an RDBMS makes sense given the flexibility. Maybe, instead of looking at NoSQL as a full-on replacement for RDBMS, we can look at it as a better solution to sharding.
- aneth4 14y ago
  This is the opposite of "agile." It is difficult to know where your product will be in 2 months let alone 12, so it seems the advice to use SQL first is sound - unless you enjoy long distractions to solve simple JOINs.
  - bthomas 14y ago
    If you are refactoring your app enough to need a different class of joins, you'll probably need schema and data migrations too. I think no-schema fits agile quite well. For rapid prototyping, I prefer Mongo to even sqlite.
  - jeremyjh 14y ago
    You should at least know if you will likely have any analytic use cases.
- dkhenry 14y ago
  This is also true of SQL databases. It just depends on _what_ your aggregating. In this case if you need to aggregate the count of something then in a DB like MySQL you can lean on the index to get a quick count, but if you need to SUM or AVG you will be doing just as much CPU level work as a NoSQL solution. The difference is in MySQL its a simple query with the AVG operator, in MongoDB its a Map-Reduce query which is much harder to write. In general you need to know what your doing under the hood and how either solution effects your problem domain. Where I work we need to aggregate billions of data points on demand. This can't be done in real time without pre-aggregating the results ( and even then it takes tons of I/O just to process the aggregated data set )
blaines 14y ago
Very briefly put, I use MongoDB to start off most new projects. My primary objective is that my application fulfills it's use case. Data is malleable, so you should use the right tools for your needs. That being said, it sounds like the OP was trying to use a chisel as a replacement for a toolbox. Basically fighting the software (mongodb) to fit his requirements, instead of using additional tools. http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html http://blog.heroku.com/archives/2010/7/20/nosql/ http://blog.heroku.com/archives/2010/7/20/nosql/
dkhenry 14y ago
Looks like he jumped ship just a bit too early. http://docs.mongodb.org/manual/applications/aggregation/ http://docs.mongodb.org/manual/applications/aggregation/
- deleted 14y ago
  [deleted]
dccoolgai 14y ago
"To be honest, the decision to use MongoDb was an ill-thought out one. Lesson learned - thoroughly research any new technology you introduce into your stack, know well the strengths and weaknesses thereof and evaluate honestly whether it fits your needs or not - no matter how much hype there is surrounding said technology." I think you are not alone in learning this lesson with this particular technology. Fortunately it's one I learned by proxy from working adjacent to a team that decided to introduce Mongo into their stack...but I still wake up and hear the sceams at night of "You have to put the whole dataset on RAM?"...you weren't there, man...we lost a lot of good guys... You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack".
- mgkimsal 14y ago
  One of the issues is that if you actually do this - evaluate, look at all sides, and decide 'shiny new tech' IS NOT right for your situation/project, you're branded as something not good. "Not a team player", "stick in the mud", "not able to keep up with the times", etc. I'm not suggesting everyone should be sticking with 1966 COBOL - times change, new tech comes up which makes sense to adopt. But not jumping on the shiny new tech bandwagon can have social consequences you need to be aware of.
- stonemetal 14y ago
  You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack". Even if you have a clean line, when do you promote something across that line? It is easy for little bits of weirdness to escape detection(Mostly thinking about Cassandra write failures here.) in a fun to play with environment especially if they only come out when you are running a cluster.
- trafficlight 14y ago
  >> "You have to put the whole dataset on RAM?" I'm pretty new to the whole database thing, but how is MongoDB different from Postgres or Mysql in this respect? In a traditional database, the data is pulled directly from the hard drive. Why does Mongo suffer a performance hit and Mysql doesn't?
leothekim 14y ago
"I can only come to the conclusion that mongodb is a well-funded and elaborate troll." It's possible the reasoning he used to use mongodb is the same as the one he used to abandon it.
bunderbunder 14y ago
Fourthly, and this one completely blew my mind - somewhere along the stack of mongodb, mongoid and mongoid-map-reduce, somewhere there, type information was being lost. I thought we were scaling hard when one of our customers suddenly had 1111 documents overnight. Imagine my disappointment when I realised it was actually four 1s, added together. They’d become strings along the way. I've been having a similar problem with an SQLite data store, only the other way around. Strings were getting converted to numbers, with leading zeros that were significant and needed to be maintained being lost along the way. It sucked all the fun out of dynamic typing for me. At least in combination with automatic type conversions. Having to think about type and when to make transitions across type boundaries when you need to is just a little light busywork. Having to worry about type and transitions across type boundaries being made contrary to your intentions is a downright PITA and, it turns out, a serious quality control issue.
- parfe 14y ago
  "Any column in an SQLite version 3 database, except an INTEGER PRIMARY KEY column, may be used to store a value of any storage class." http://www.sqlite.org/datatype3.html http://www.sqlite.org/datatype3.html SQLite, for better or worse, is designed to do what you had an issue with. Pick a different DB if you need strict data types. Check out section 2.0 Type Affinity.
- zemo 14y ago
  Mongo accepts the data you give it. If you have a type-conversion error, it's in your application layer. I use Mongo daily and have never seen this problem, because I'm using a statically typed language. This seems like more of a complaint about Ruby than Mongo. I use Mongo daily on a Go project, and I actually think it's pretty annoying; I'm not trying to be a Mongo apologist, but ... this type conversion argument doesn't seem to be very fair to Mongo.
  - lucisferre 14y ago
    Bad code is bad code, in any language. I'm a little disappointed how any post about moving from X to Y (especially if X is Mongo) makes the top of the front page on HN. This is not really a very good or insightful post. It's one persons experience and anecdotes of the pain points of learning a new technology. Mildly interesting, but not really expository in any way.
harel 14y ago
You used a tool without researching it first, you jumped a bandwagon without finding out its destination, you most likely used it wrong because you didn't RTFM. Now you ditch and diss it. Grow up.
- mattmanser 14y ago
  Very harsh given that he points out he made these mistakes right at the beginning of his post.
  - harel 14y ago
    Fair enough, perhaps a bit harsh. But from that to call an entire system a Troll... It works for some, it doesn't for others. Usually its down to use case and knowledge of how things work. For me mongoDB was a perfect fix as a datastore along side a tranditional Postgres instance.
jaequery 14y ago
I've ran into similar issue as you described. Something that can be done so simple and quickly in SQL, was bewilderingly difficult to do in mongo. The schema-less database approach also seems attractive at first but updating your data whenever your "app schema" changes starts to become a pain real quick. Now I can't really live w/o having a schema first, it actually saves you a lot more time in the long run (even short run), being schema-less means you can't really do anything too fancy w/ your data (generate reports, advanced search, etc...)
- lmm 14y ago
  You still need to update your data when doing schema changes under SQL, and you have a lot less control over the process. And you can do anything to your data without a schema, you just need to build your app as a service that provides access to it. IME SQL schemas do more harm than good; usually you end up with a schema that's subtly weaker than what's actually valid for your application, and the difference between the two models will trip you up at the worst possible time. Have a small, distinct set of classes that you store, enforce that you store only those (and don't access the storage layer any other way), enforce that they remain backwards compatible, and enforce that you can't create invalid instances. But application code is the best place to do all these things.
  - einhverfr 14y ago
    IME SQL schemas do more harm than good Completely disagree here. The basic tradeoff is between flexible input and flexible output. Without a rigid schema, ad hoc reporting is impossible because you don't have an ability to articulate reporting criteria. I.e. no declarative schema means no declarative reporting query. I suppose that's ok as long as you never need to report on anything..... Might work....
    - lmm 14y ago
      I'm not saying don't have a rigid schema, just don't enforce it at the storage layer. If you're doing an ad-hoc report then you wouldn't have indexes in place for it in the SQL case, so I don't see how it's any worse or harder in mongodb.
jaimebuelta 14y ago
Mmm, not sure about some of the complains... - You can make case insensitive searches on the DB using regexes (http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-RegularExpressions http://www.mongodb.org/display/DOCS/Advanced+Queries#Advance...). A simple case-insensitive regex is not very bad performance-wise, but in general, case-insensitive searches should be avoided for search purposes (you can normalize to set everything to lower case or other equivalent trick) - The proper way of doing an audit (and search later) is to make an independent collection with a reference to the other(s) document in a different collection. Then you can index by user, date, or any other field and leave the main collection alone. The described embedded access collection doesn't look very scalable. - Making map-reduce queries is tricky (at least for me). I think the guys on 10gen realizes that and the new aggregation framework is a way of dealing with this. Anyway, the main advantage of SQL is this kind of things, the rich query capabilities. Even if MongoDB allows some compared with other NoSQL DBs, if there is a lot of work in defining new queries, probably a SQL DB is the best fit, as that is where SQL excel. I don't truly believe in this "you should research everything before starting" (I mean, I believe in research, but too many times the "you should do your homework" argument is overused. Sometimes you make a decision based in some data that changes later, or is incomplete), as there are a lot of situations where you find problems as you go, not in the first steps. But, according to the description, looks like PostgreSQL is a better match and the transition hasn't been too painful, so we can classify this into "bump in the road"/"success history". Also, probably right now the DB schema is way more known that in the beginning, which simplifies the use of a relational DB.
- datasage 14y ago
  - The proper way of doing an audit (and search later) is to make an independent collection with a reference to the other(s) document in a different collection. Then you can index by user, date, or any other field and leave the main collection alone. The described embedded access collection doesn't look very scalable. I think this point is very important even in the RDBMS side. There are cases, even with relational datastores that would preform better if the dataset was built to the query. The difficulty comes into play when you are trying to keep the denormalized data up to date based on changes within the base dataset.
se85 14y ago
The guy just jumped on the bandwagon without having a clue. Just reading this blog - it's clear that MongoDB was not a good fit for him, if he had bothered to do some research, he would have found this out on day one. Thats the real lesson he should be taking away from this and blogging about yet somehow MongoDB are trolls and it's all their fault because of a lack of features and they have bypassed 40 years of computer science and blah blah blah blah, excuses, excuses, excuses. edit: removed a few pointless sentences :-)
rbranson 14y ago
I'm no fan of MongoDB, but this same advice goes for any NoSQL data store. I am an Apache Cassandra contributor and community MVP, but my advice stays the same: it's best just to start with a SQL database and go from there. Read some books and learn it well: the "SQL Cookbook" from O'Reilly is great, and so is "The Art of SQL." Premature optimization continues to be the root of all evil.
- bunderbunder 14y ago
  Don't forget to read a book that's specific to your particular RDBMS. Because SQL databases are only trivially interchangeable for trivial cases. There have been more than a couple times when I was ready to blame the relational model, but further investigation revealed that the real root of the problem was that the existing schema or query used an approach that was optimized for one DBMS but performed terribly on the one we were actually using.
  - rbranson 14y ago
    This is why the "SQL Cookbook" is good, it has dozens and dozens of examples with permutations in Oracle, MS SQL, MySQL, and PostgreSQL.
  - gregjor 14y ago
    Wrong. SQL databases (except for SQLite) are almost completely interchangeable because they are all based on the same relational model and they all implement the ANSI SQL standard with only minor deviations. If you have a lot of stored procedures and triggers -- executable code embedded in the database -- you will have to rewrite that. Oracle is in a world of its own in a lot of ways, but if you are moving to or from Oracle you should have the required resources and expertise in your budget. If you are referring to the well-known cost of COUNT(*) in PostgreSQL (or MySQL with InnoDB, for that matter), or different ways to handle full-text searching, I agree that there are differences you have to deal with, but they are not usually a big deal. Contrast the fairly easy and routine process of migrating between MySQL, PostgreSQL, or SQL Server with the huge amount of work involved changing your application code from MongoDB to anything else.
armored_mammal 14y ago
Can someone confirm that there is no such thing as a case insensitive index/search in Mongo? If true it seems likely that the author's comments have some degree of truth, at least when it comes to its usefulness for web and mobile applications. Storing data only lowercase isn't a good a idea for obvious reasons, and storing two copies of the same data for searching only, while not the end of the world, seems a little silly.
- zemo 14y ago
  case-insensitive regex searches are supported.
  - lucasjans 14y ago
    Do you know why are case-insensitive searches not recommended? What's the realistic work-around?
    - parfe 14y ago
      http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-RegularExpressions http://www.mongodb.org/display/DOCS/Advanced+Queries#Advance... Regex queries ending with /i (case-insensitive flag) cannot efficiently use indexes, but must do full index scans.
    - ericcholis 14y ago
      I've found that a simple {"lastname":/cholis/i} works great. However, trying to do the same thing for a multi-key search isn't ideal. Specifically, searching for 3 words in a title using $and with multiple regex queries on a collection with 100,000+ documents took about 520 ms. The mongodb documentation suggests that you could have an array with your keywords, generated from the field you wish to search. Using indexes on multikeys would make this faster, but your index size would be much larger.[1] For my project, I'm likely going to institute solution like elasticsearch or SOLR. [1] http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mong...
manorasa 14y ago
I think the real lesson here is use the right tool for the right job.
jamesli 14y ago
I am both a database guy and a software engineer. Being a software engineer, i kind of understand the hype behind NoSQL. Being a database guy spending years in studying how database engine works under the hood, many NoSQL implementations make me wonder how powerful marketing can be. In general, I love the ideas behind NoSQL. I can still feel the excitement when reading the BigTable and MapReduce papers. HBase, Hadoop, Radis, etc. are awesome products. I use some of them in my work. But some other NoSQL products? Being engineers, we must understand the implementation and be full aware of its limitations, instead of believing their marketing materials. Well, if all you want is to test a toy product, to build a prototype, or your product is of low concurrency and low data size and you have no concern on operation, it certainly looks that they make your development easier. But in these scenarios, any good relational databases won't add significant burden either.
- taligent 14y ago
  > Being engineers, we must understand the implementation and be full aware of its limitations, instead of believing their marketing materials. And as engineers we must understand that most other engineers do take their role seriously and evaluate products on their merits. Implying that they are falling for "marketing" just because you don't agree with their choice and then lecturing them for their choice doesn't make you come across well.
  - einhverfr 14y ago
    I do think that the popularity of MySQL, however, owes a lot to it being used by non-engineers for simple web apps, though ;-)
dkarl 14y ago
You have to load every document in the database and extract the audit trail from it, then filter it in your app for the user you’re looking for. Just the thought of what that would do to my hardware was enough to turn me off the whole idea. Naive question from somebody who has done a little reading on and dabbling with key/document-with-MapReduce style datastores, but who hasn't tackled a real production problem: I thought running queries over the entire dataset was one of the assumptions of horizontally scalable document stores? In terms of avoiding computation, you can only limit queries by document key, which even if you're clever/lucky doesn't always encode the parameters you're querying on, or doesn't encode them in the right order, so you should be prepared to run queries over your entire dataset. Hopefully the queries you run often are optimized (e.g., using indexes or clever use of key ranges), but in the general case, you have to be prepared to scan the whole shebang, and that's supposed to be okay because of horizontal scalability, right?
- enjo 14y ago
  Yep, or you need to build some other construct to support it (IE: keeping running tallies and the like). It's a tradeoff between the benefits of the document store vs the loss of relational data. The blog author here clearly didn't understand the trade-offs he was making. As always with these discussions: it's important to use the right tool for the job. I'm a big fan of what Mongo is doing. I've used it two higher scale projects and have no complaints. Of course, I'm using it in the context for which Mongo excels.
daveman 14y ago
As an analytics professional who was pressured into a MongoDB environment, I feel the OP's pain. If you want to do gymnastics with your data, (aggregations of aggregations, joining result sets back onto data), SQL expressions are a 1000 times easier than Mongo constructs (e.g. map reduces). We usually ended up scraping out data from Mongo and dumping records into a SQL database before doing our transformations. All that said, our developers loved the ease of simple retrieval and insertion, and of course the scalability. So I guess you ultimately need to base your decisions on your priorities. I don't fault the OP though, since it's hard to know just how limiting NoSQL will be until you try to do all the things you used to assume were database tablestakes (no pun intended).
itaborai83 14y ago
To be fair, some NoSQL solutions were being sold marketing wise as the be-all and end-all of data solutions. Just google "mongodb mysql migration" and look how everyone is/was so eager to jump on the non-relational bandwagon. Some backlash was to be expected, after all, we might have reached the Trough of Disillusionment
adambard 14y ago
As a relative idiot when it comes to this sort of thing, I'd like to insert the following supplementary question: what is the sort of application/dataset for which Mongo is particularly suited? I've used it on small projects, and have enjoyed it. Perhaps my data has just been simple/loosely-coupled enough to never run into these problems? I read a lot of posts like this on HN before every trying Mongo, so I've at least been convinced to always implement schema at the application layer. Others seem to keep learning that lesson in harder ways.
- stingraycharles 14y ago
  "what is the sort of application/dataset for which Mongo is particularly suited" The majority of the NoSQL databases are based on Amazon's Dynamo: loosely coupled replication. MongoDB is one of the few (next to Hbase and a few others) that adopts Google BigTable's architecture: data is divided in Ranges, and each mongod node serves multiple Ranges. This means MongoDB is able to provide atomicity where it's harder with other SQL databases. In particular, we need to be able to do some sort of "compare and swap" operation that is guaranteed to be atomic/consistent, while still being able to have our mongod nodes distributed over multiple datacenters. In Dynamo-based architectures, in order to provide the same amount of atomicity, you always end up writing to at least half + 1 the amount of replicated nodes you have available in your cluster. This is more awkward, and reduces the flexibility of the whole (the atomicity guarantee Mongo provides also works for stored javascript procedures, for example). Having said that, we're using MongoDB about 3 years in production at this point, but we're far from happy about the availability it provides (issues like MongoDB not detecting that a node has gone down, failing to fail over, etc). We run a HA service, and to date all of our failures in uptime have been either the fault of our hosting provider or mongodb not failing over when it should. As such, we're always looking for a better alternative to move to, but at the moment MongoDB is about as good as it gets.
  - edwardcapriolo 14y ago
    MongoDB is not even remotely close big table architecture. It has a different data model and a different sharding model, and just about a different everything.
chaostheory 14y ago
For me, what killed my enthusiasm for mongodb is the write locks. Yes they have been greatly improved in the 2.x release but it's still not good enough (for me).
redler 14y ago
digiDoc is all about converting paper documents like receipts and business cards into searchable database, and so a document database seemed like a logical fit(!). It looks like this single initial assumption is where things started going wrong: conflating the pieces of paper that happen to be called "documents" in the real world with the concept of a "document" in the context of a system like MongoDB.
- josephcooney 14y ago
  It's an easy mistake to make - to assume two things with the same name might be similar. Especially when paper documents have been called thus for a very long time. It seems like another instance where all the good names were taken. http://jcooney.net/post/2012/04/03/Transaction-Argument-Class-Message-Service-Agent-Method.aspx http://jcooney.net/post/2012/04/03/Transaction-Argument-Clas...
aliks 14y ago
Reasons why drop mongodb: 1. try $or $and with $near 2. No b-tree index :: count() 10.000 rows = 100% CPU usage ;) Type google.com then:: site:jira.mongodb.org/browse/ planned but not scheduled
bassemali 14y ago
I've never used Ruby on the application layer, but I'd be wary of using an ODM with MongoDB. The single most shocking issue seems to be at the application / ODM level. Using the official 10gen-supported drivers gives you more control and a better understanding of what's going on every step of the way. Also, a thorough understanding of MongoDB indexing, advanced queries and schema design would have squashed all of these issues. Has anybody had a more pleasant experience with a MongoDB ODM?
gregjor 14y ago
You were fortunate to recognize that MongoDB was the wrong tool for your job, and lucky to be able to move to Postgres instead of continuing to throw your time and effort away. I see the ad hominem "you're an ignorant idiot" attacks already started, along with advice like using regexes to do case-insensitive searches. Watching the NoSQL "movement" encounter the problems RDBMSs fixed 20 years ago and then hand-wave and kludge them away is frustrating. I wrote about some of this in http://typicalprogrammer.com/?p=14 http://typicalprogrammer.com/?p=14. Look at the bright side: programmers who are writing NoSQL-backed apps are creating the fossil fuel that will keep programmers who know RDBMs working into our retirement years. I already have more work than I can do fixing web apps that were built around crap data management tools that failed to scale beyond a few thousand users. Your Postgres expertise will still be a money-making skill long after MongoDB is forgotten.
- gaius 14y ago
  The thing with the NoSQL guys is that many of them seem not to be in a position to make an educated comparison. For example, an, uhh, enthusiastic MongoDB advocate recently informed me that MongoDB was superior to Oracle because in Oracle you had to poll a table to see if it changed. Except, no, that isn't actually true: http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_dcn.htm#BGBBHGAH http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_... - and that document is from 2005. And you could do a trigger and an AQ message/callback 5 years before that (at least). You haven't needed to poll an Oracle database for changes in a loooong time. Basically, every evangelism point, you have to double-check and cross-reference, because as you say, the NoSQL guys are encountering issues the RDBMS community addressed years ago (7 in my example, but the sharding stuff, 20+ years) - except they think they are discovering it for the first time!
  - gregjor 14y ago
    Programming is more like fashion than science in this regard. Every decade or so something truly new happens in the software world. All the rest is mostly sound and fury, signifying nothing. If you're young or new to programming it's easy to mistake the buzz around things like NoSQL for innovation when they are usually re-discoveries of old (and often discarded or obsolete) ideas dressed up in new clothes. There's also the tendency to favor new shiny things and reject old crufty (but proven) things, to want to be part of what seems like the leading edge, to be that guy in the cube farm who is playing with the cool new stuff. I have been programming longer than RDBMSs have been available, so I know from experience what it's like to manage large databases in application code, and how hard it can be to maintain consistency or do accurate queries and aggregation with half-baked tools. It's frustrating to see a new generation of programmers go through this, but it's human nature to ignore the past. My fourteen year old son wears his pants pulled down below his waist, Vans shoes, hoodies, lots of hair. He looks pretty much like I did when I was fourteen back in the 1970s. The underlying technologies are the same: pants, shirt, shoes, sweater, hair. The only differences are superficial. To him that style is edgy and contemporary and something his parents don't get. NoSQL is the gangster fashion of programming right now.
DonnyV 14y ago
If he just did 10min of research he would of realized MongoDB isn't for him. Also by doing that research he would've realized that MongoDB has no data constraints. Thats all done in your model in your application.
jeremyjh 14y ago
Upvoted for?
ilaksh 14y ago
On your home page you imply that you can automatically OCR arbitrary handwritten receipts into an analyzable format. No one can do that. That is your problem, not MongoDB. As far as aggregation, use the new Aggregation Framework http://docs.mongodb.org/manual/tutorial/aggregation-examples/ http://docs.mongodb.org/manual/tutorial/aggregation-examples...: db.zipcodes.aggregate $group: _id: "$state" totalPop: $sum: "$pop" , $match: totalPop: $gte: 10 * 1000 * 1000 As far as "losing the independence of your data access paths", no you don't. You are free to use linking instead of embedding wherever you want. http://www.mongodb.org/display/DOCS/Schema+Design#SchemaDesign-EmbeddingandLinking http://www.mongodb.org/display/DOCS/Schema+Design#SchemaDesi... MongoDB doesn't have a built-in full text search? So what. Most systems with large amounts of text to search do not rely on the text search capabilities built into relational databases anyway. People use actual full-text search engines like Lucene/Solr, Sphinx, reds, etc. Having said that, if you just wanted to support lowercase keyword queries with MongoDB, would it really be so hard to extract and store lowercase keywords from your text, as suggested here? http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mong... If you are trying to add four 1s and get '1111' instead of 4, that is an error in your application code which has nothing to do with MongoDB. Very common problem with JavaScript. If it is JavaScript, try finding the code where you are attempting numeric addition and change it so that instead of saying for example 'total += newNumber' it says 'total += (newNumber * 1)' .
- nnnnnnnn 14y ago
  "On your home page you imply that you can automatically OCR arbitrary handwritten receipts into an analyzable format. No one can do that. That is your problem" Jeez, lay off the confrontational tone. He doesn't say anything about OCR. Maybe he's using humans to do data entry? In any event, it's completely irrelevant to the topic of databases.
  - ilaksh 14y ago
    I notice you ignored all of my several very specific points directly related to his issues with the database system and your only comment was a criticism about the tone you perceived. OK, maybe he is using humans to do data entry. The home page to me implies that the process is automatic, but I guess it doesn't rule out the possibility of humans doing data entry when he says 'tag and categorize'. But if he is using humans to do data entry instead of some automatic OCR, that is still his main business problem, rather than MongoDB. The application is relevant to the database discussion, and Hacker News is about all aspects of startups.
nashequilibrium 14y ago
Taking all these comments into account, if startups have to prototype quickly while trying to find market fit, does it makes sense to start off using something like mongodb but with the plan to migrate to another database when you business starts growing? The database space is so confusing right now. It seems like Postgres is the safest choice and i also like this post fron Adam D'Angelo - 'http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use-MySQL-as-the-data-store-instead-of-NoSQLs-such-as-Cassandra-MongoDB-or-CouchDB http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use...
nashequilibrium 14y ago
Taking all these comments into account, if startups have to prototype quickly while trying to find market fit, does it makes sense to start off using something like mongodb but with the plan ti migrate to another database when you business starts growing? The database space is so confusing right now.
nashequilibrium 14y ago
Taking all these comments into account, if startups have to prototype quickly while trying to find market fit, does it makes sense to start off using something like mongodb but with the plan ti migrate to another database when you business starts growing? The database space is so confusing right now.
programminggeek 14y ago
Look, there are some places where document DB's solve problems easier/better than SQL, other places kind of suck. For example, plain old object mapping is easier with a document DB. Relational DB's tend to make your code look/feel/act more relational and less object oriented. Your object model tends to look just like your table structure. This can be good or bad depending on your viewpoint. There are some approaches to solve some of the author's problems that end up making the Mongo system look and feel a lot more like a SQL system because sometimes data is actually related. The author could have also taken a different approach to his data schema that would have fit more of a non-relational worldview. Software development and architecture is about making choices and working with and around the limitations of your tools. It doesn't matter if PostgreSQL or MongoDB are "better". It's about solving a problem using a set of tools you are comfortable with.
bitdiffusion 14y ago
It's not necessarily all or nothing - I have worked on several projects now each using multiple database-type options: mongodb for read-intensive, loose-schema type stuff where the growth is generally predictable (e.g. products, suppliers, logs), postgres for relational-type stuff (orders) and solr for searching (I know solr isn't a database but people seem hung up on whether mongodb supports case-insensitive searching - hint: don't use any database for search). I doubt that, unless it's extremely simple, any set of requirements are an exact match to only one of these technologies... mix and match is the future :P
Teef 14y ago
There are 3 reasons I have gone running and screaming from and RDBMS. 1. Software gets large / complex to get meaning full work done. I am all about data consistency but at some point it is time to break things up into services and not have a single database. 2. If the software is popular enough everyone is running to use NoSQL (cache is NoSQL). 3. Clearly it is not a good storage solution either because for example in an address book nested list greatly simplifies everything. (right tool for the job) I spent many years hammering away with RDBMS and by and large it was great until it wasn't. I try to look at data storage more holistically now based on best guess of the problem. I have tried to convert an application from Postgresql to MongoDB and it failed but that wasn't MongoDB's falt it was because I didn't change the data model to fit a document storage system. I have also tried to use PostgreSQL for a realtime reporting system and failed horrifically and that was not Postgesql fault it was mine. Amazing what happens when you stop pushing a chain and start pull it!
- einhverfr 14y ago
  >1. Software gets large / complex to get meaning full work done. I am all about data consistency but at some point it is time to break things up into services and not have a single database. This is true. Managing complexity is always an important task. I am not sure that NoSQL solves this however. Also the best way to break things up is to loosely couple things, and this requires to some extent that you have ACID compliance. A good RDBMS, like PostgreSQL or Oracle, will provide tools for managing that loose coupling. >"2. If the software is popular enough everyone is running to use NoSQL (cache is NoSQL)." Like proverbial lemmings over a cliff.... >"3. Clearly it is not a good storage solution either because for example in an address book nested list greatly simplifies everything. (right tool for the job)" Funny, I thought nesting was what WITH RECURSIVE was for.... I am not saying there aren't use cases for MongoDB or reasons to switch some applications. For example I can think of a few really cool apps, like maybe a network back-plane for a huge LDAP directory. Also content management might be a good fit. But despite your years of experience, it doesn't sound like you have really looked at how to solve these with good RDBMS's.....
tonynero 14y ago
The guy is getting such hate on the comments on his site, yet his opening line is that his choice was ill thought. Let him express his issues right? I choose MongoDB for my last side project and while it was awesome working schema-less and developing the client facing part of the project was certainly quicker to deliver, i feel pretty lost on the analytics/BI side of it and couldn't say it better than him: "Not having JOINs makes your data an intractable lump of mud" So coming from a relational/SQL background I found MongoDb awesome upfront, but frustrating later on... and yes I'm off to learn http://docs.mongodb.org/manual/applications/aggregation/ http://docs.mongodb.org/manual/applications/aggregation/
ww520 14y ago
Wow, the first comment on the blog is so vile. He angrily blamed the "victim" (OP) as a talentless developer. Tools are enablers and supposed to make ordinary people rock star. If it takes a rock star to use a tool, the tool fails.
voidr 14y ago
Relational databases are awesome if you are not dealing with huge amounts of data that your current hardware can't handle the relational way. There are some cases where you have a ridiculous amount of data(rows) and you simply can't store that in a relational database and you are happy to live without the benefits of relational databases. If you have millions of rows, you are probably better off with something like MongoDB, if you need to search that, you should probably use something like Sphinx or Lucene anyway. But if you know that you won't have too much data for the forceable future, you should use relational databases. OR you could simply use both.
- gregjor 14y ago
  There may be cases where non-relational databases are the right solution, but unless you are using a toy RDBMS millions of rows is not considered a lot of data. Millions of rows added every day... now you're talking. Oracle handles that kind of thing just fine.
- jeltz 14y ago
  A relational database at very modest hardware can handle millions of rows, and with a solid database, good hardware and a DBA who knows his shit you can handle billions. OpenStreetMap has over a billion nodes stored in a PostgreSQL database. http://www.openstreetmap.org/stats/data_stats.html http://www.openstreetmap.org/stats/data_stats.html My point is that you can get very far with a classic relational database before you have to scale vertically.
  - einhverfr 14y ago
    Billions of rows btw is not a problem even on modest hardware, with a half-decent db, etc. It's analytics on this where it become possibly an issue. Retrieving one row out of 1 billion is not that much more complex than retrieving one row out of 1 million, nor is it that much more expensive computationally assuming the right index is in place. The problem comes with high concurrency, in particular very high write concurrency, or with very complex queries which require a lot of RAM to do properly. But that's where you need a solid db, good hardware, and a solid DBA.
mrinterweb 14y ago
I find this article to be more a reflection of a NoSQL newbie's failed foray with a document database that later realized that the grass is not as green as originally perceived. The developer realized that he does not like map-reduce and missed not having joins. I don't see how this person's failed experience with MongoDB is a reflection on MongoDB. I think the recent popularity of MongoDB bashing is maybe a testament to MongoDBs popularity. I'd guess that because MongoDB is probably the closest NoSQL database to a RDBMS with its ad hoc queries, that it is attracting many newcomers.
- jeremyjh 14y ago
  Yes, MongoDB is more of a general-purpose database with lots of features that remind us of relational databases. It is a purpose-built application database for applications that would otherwise almost certainly be built with a relational database. As they get deeper into their project and find out how some of the trade-offs play-out their self-doubt is always about "should we just go relational" - no one is staying up all night wondering if they should migrate to Riak. If you start out with Riak you almost certainly know why and are using it in a very specific context.
effinjames 14y ago
calm down redditors, he just need a basic non majestic scale solution, SQL fitted him very well. large scale data aggregation needs to address disk and network latency, that's where NoSQL shines. and if you operate at large scale no 1 single simple tool will do justice, remember quote from google, 'at scale everything breaks'?
- Karunamon 14y ago
  calm down redditors Please knock this crap off.
deleted 14y ago
[deleted]
dkhenry 14y ago
Competly aside from the Article. The level of vitriolic discourse in this topic is astounding. I am amazed that as a community discussions of Database engines can draw out such mean spirited anger. I have never down voted as many comments on HN in a single thread then I have on this topic. I don't care which side of the debate you come down on. There is no excuse for belittleing and insulting others in a technical forum. Thats right I am looking at you gregjor, gaius, and zemo In this case it appears to mostly be those arguing for Postgre, but I wouldn't care if you were arguing for sunshine and unicorns there is a way to behave civilly and your not doing it.
- gregjor 14y ago
  You can find this level of discourse in plenty of topics every day. Programmers draw blood over indenting with tabs or spaces. It's geek entertainment. I have never been upvoted so much on HN. I admit to strong opinions and light sarcasm but you'll have to show me where I've been uncivil, belittling or insulting, except perhaps in response to people who insulted me.
- einhverfr 14y ago
  In this case it appears to mostly be those arguing for Postgre Curious. I have never found a database named "Postgre" to be used by anybody. Perhaps you can direct me to the download site. I think the bigger issue is that there isn't a lot of discussion from the NoSQL crowd about what you give up when you go to a NoSQL solution. I think that sort of disclosure would help people weigh the options a lot better. From the comments of some people here you'd almost think they would build an ERP app in Mongo....
firemanx 14y ago
I work for a company that operates in the energy industry. We utilize both RDBMS and "NoSQL", both have their purposes that they fit in well. We store customer account and configuration data in Postgres, and use Cassandra to store time-series statistics and high write volume data. I have a background in data warehousing in both Oracle and SQL Server, and was part of the decision to use a polyglot persistence model. I've got at least a decade's worth of experience in the DW world, and more as a general developer before that, so I like to think I've got a relatively credible background in a variety of data stores. I haven't looked at Mongo much - it's durability concerns and the write lock stuff pushed me away from it early on (I don't mean to disparage it, but that was where it was at when I evaluated it), but Cassandra's configurable consistency levels and operational story at a cluster level are what sold us for our time-series data (that, and the ability to construct a sparse timeline and multiplex reads/writes). For anything we need flexible querying with, we push it into specialized Postgres dbs. The level of willful ignorance and vitrol in this thread is kind of amazing. Most of the really experienced DW guys I know are all looking at HBase, Cassandra and others because they fit a niche that we've all been looking for in certain data sets at really large scale. It doesn't mean we're ditching our relational data stores, it just means we're augmenting them with other tools because they fit the job at hand. To suggest that one tool is absolutely perfect for every scenarios seems a little short-sighted to me, possibly driven out of inexperience. I don't mean that as an insult - I know a lot of guys who've been working on the same data sets for 30 years who really do just need the one tool - however, you've got to realize there are other data sets and problems for which your hammer just won't fit.
anthony_barker 14y ago
Made the same mistake on a banking project 10 years ago (with domino). The project in question was a project tracking database. For accounting type problems use a relational database. For document driven items - e.g. a resume database - nosql works great. For a hybrid pick your battles... or use both.

203 points by svs 14y ago

The downside, or challenge, with NoSQL (generally speaking) is that you need to handle your aggregations ahead of time - you need to know what queries you'll want to run in the future when you store your data. If you have some new aggregation you want to keep, you'll need to re-process the data (with Hadoop or something else). It's the trade-off of being able to scale reads and writes horizontally. And unless you need it, an RDBMS makes sense given the flexibility. Maybe, instead of looking at NoSQL as a full-on replacement for RDBMS, we can look at it as a better solution to sharding.

aneth4 14y ago
This is the opposite of "agile." It is difficult to know where your product will be in 2 months let alone 12, so it seems the advice to use SQL first is sound - unless you enjoy long distractions to solve simple JOINs.
- bthomas 14y ago
  If you are refactoring your app enough to need a different class of joins, you'll probably need schema and data migrations too. I think no-schema fits agile quite well. For rapid prototyping, I prefer Mongo to even sqlite.
- jeremyjh 14y ago
  You should at least know if you will likely have any analytic use cases.
dkhenry 14y ago
This is also true of SQL databases. It just depends on _what_ your aggregating. In this case if you need to aggregate the count of something then in a DB like MySQL you can lean on the index to get a quick count, but if you need to SUM or AVG you will be doing just as much CPU level work as a NoSQL solution. The difference is in MySQL its a simple query with the AVG operator, in MongoDB its a Map-Reduce query which is much harder to write. In general you need to know what your doing under the hood and how either solution effects your problem domain. Where I work we need to aggregate billions of data points on demand. This can't be done in real time without pre-aggregating the results ( and even then it takes tons of I/O just to process the aggregated data set )

blaines 14y ago

Very briefly put, I use MongoDB to start off most new projects. My primary objective is that my application fulfills it's use case. Data is malleable, so you should use the right tools for your needs. That being said, it sounds like the OP was trying to use a chisel as a replacement for a toolbox. Basically fighting the software (mongodb) to fit his requirements, instead of using additional tools. http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html http://blog.heroku.com/archives/2010/7/20/nosql/ http://blog.heroku.com/archives/2010/7/20/nosql/

dkhenry 14y ago

Looks like he jumped ship just a bit too early. http://docs.mongodb.org/manual/applications/aggregation/ http://docs.mongodb.org/manual/applications/aggregation/

deleted 14y ago
[deleted]

dccoolgai 14y ago

"To be honest, the decision to use MongoDb was an ill-thought out one. Lesson learned - thoroughly research any new technology you introduce into your stack, know well the strengths and weaknesses thereof and evaluate honestly whether it fits your needs or not - no matter how much hype there is surrounding said technology." I think you are not alone in learning this lesson with this particular technology. Fortunately it's one I learned by proxy from working adjacent to a team that decided to introduce Mongo into their stack...but I still wake up and hear the sceams at night of "You have to put the whole dataset on RAM?"...you weren't there, man...we lost a lot of good guys... You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack".

mgkimsal 14y ago
One of the issues is that if you actually do this - evaluate, look at all sides, and decide 'shiny new tech' IS NOT right for your situation/project, you're branded as something not good. "Not a team player", "stick in the mud", "not able to keep up with the times", etc. I'm not suggesting everyone should be sticking with 1966 COBOL - times change, new tech comes up which makes sense to adopt. But not jumping on the shiny new tech bandwagon can have social consequences you need to be aware of.
stonemetal 14y ago
You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack". Even if you have a clean line, when do you promote something across that line? It is easy for little bits of weirdness to escape detection(Mostly thinking about Cassandra write failures here.) in a fun to play with environment especially if they only come out when you are running a cluster.
trafficlight 14y ago
>> "You have to put the whole dataset on RAM?" I'm pretty new to the whole database thing, but how is MongoDB different from Postgres or Mysql in this respect? In a traditional database, the data is pulled directly from the hard drive. Why does Mongo suffer a performance hit and Mysql doesn't?

leothekim 14y ago

"I can only come to the conclusion that mongodb is a well-funded and elaborate troll." It's possible the reasoning he used to use mongodb is the same as the one he used to abandon it.

bunderbunder 14y ago

Fourthly, and this one completely blew my mind - somewhere along the stack of mongodb, mongoid and mongoid-map-reduce, somewhere there, type information was being lost. I thought we were scaling hard when one of our customers suddenly had 1111 documents overnight. Imagine my disappointment when I realised it was actually four 1s, added together. They’d become strings along the way. I've been having a similar problem with an SQLite data store, only the other way around. Strings were getting converted to numbers, with leading zeros that were significant and needed to be maintained being lost along the way. It sucked all the fun out of dynamic typing for me. At least in combination with automatic type conversions. Having to think about type and when to make transitions across type boundaries when you need to is just a little light busywork. Having to worry about type and transitions across type boundaries being made contrary to your intentions is a downright PITA and, it turns out, a serious quality control issue.

parfe 14y ago
"Any column in an SQLite version 3 database, except an INTEGER PRIMARY KEY column, may be used to store a value of any storage class." http://www.sqlite.org/datatype3.html http://www.sqlite.org/datatype3.html SQLite, for better or worse, is designed to do what you had an issue with. Pick a different DB if you need strict data types. Check out section 2.0 Type Affinity.
zemo 14y ago
Mongo accepts the data you give it. If you have a type-conversion error, it's in your application layer. I use Mongo daily and have never seen this problem, because I'm using a statically typed language. This seems like more of a complaint about Ruby than Mongo. I use Mongo daily on a Go project, and I actually think it's pretty annoying; I'm not trying to be a Mongo apologist, but ... this type conversion argument doesn't seem to be very fair to Mongo.
- lucisferre 14y ago
  Bad code is bad code, in any language. I'm a little disappointed how any post about moving from X to Y (especially if X is Mongo) makes the top of the front page on HN. This is not really a very good or insightful post. It's one persons experience and anecdotes of the pain points of learning a new technology. Mildly interesting, but not really expository in any way.

harel 14y ago

You used a tool without researching it first, you jumped a bandwagon without finding out its destination, you most likely used it wrong because you didn't RTFM. Now you ditch and diss it. Grow up.

mattmanser 14y ago
Very harsh given that he points out he made these mistakes right at the beginning of his post.
- harel 14y ago
  Fair enough, perhaps a bit harsh. But from that to call an entire system a Troll... It works for some, it doesn't for others. Usually its down to use case and knowledge of how things work. For me mongoDB was a perfect fix as a datastore along side a tranditional Postgres instance.

jaequery 14y ago

I've ran into similar issue as you described. Something that can be done so simple and quickly in SQL, was bewilderingly difficult to do in mongo. The schema-less database approach also seems attractive at first but updating your data whenever your "app schema" changes starts to become a pain real quick. Now I can't really live w/o having a schema first, it actually saves you a lot more time in the long run (even short run), being schema-less means you can't really do anything too fancy w/ your data (generate reports, advanced search, etc...)

lmm 14y ago
You still need to update your data when doing schema changes under SQL, and you have a lot less control over the process. And you can do anything to your data without a schema, you just need to build your app as a service that provides access to it. IME SQL schemas do more harm than good; usually you end up with a schema that's subtly weaker than what's actually valid for your application, and the difference between the two models will trip you up at the worst possible time. Have a small, distinct set of classes that you store, enforce that you store only those (and don't access the storage layer any other way), enforce that they remain backwards compatible, and enforce that you can't create invalid instances. But application code is the best place to do all these things.
- einhverfr 14y ago
  IME SQL schemas do more harm than good Completely disagree here. The basic tradeoff is between flexible input and flexible output. Without a rigid schema, ad hoc reporting is impossible because you don't have an ability to articulate reporting criteria. I.e. no declarative schema means no declarative reporting query. I suppose that's ok as long as you never need to report on anything..... Might work....
  - lmm 14y ago
    I'm not saying don't have a rigid schema, just don't enforce it at the storage layer. If you're doing an ad-hoc report then you wouldn't have indexes in place for it in the SQL case, so I don't see how it's any worse or harder in mongodb.

jaimebuelta 14y ago

Mmm, not sure about some of the complains... - You can make case insensitive searches on the DB using regexes (http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-RegularExpressions http://www.mongodb.org/display/DOCS/Advanced+Queries#Advance...). A simple case-insensitive regex is not very bad performance-wise, but in general, case-insensitive searches should be avoided for search purposes (you can normalize to set everything to lower case or other equivalent trick) - The proper way of doing an audit (and search later) is to make an independent collection with a reference to the other(s) document in a different collection. Then you can index by user, date, or any other field and leave the main collection alone. The described embedded access collection doesn't look very scalable. - Making map-reduce queries is tricky (at least for me). I think the guys on 10gen realizes that and the new aggregation framework is a way of dealing with this. Anyway, the main advantage of SQL is this kind of things, the rich query capabilities. Even if MongoDB allows some compared with other NoSQL DBs, if there is a lot of work in defining new queries, probably a SQL DB is the best fit, as that is where SQL excel. I don't truly believe in this "you should research everything before starting" (I mean, I believe in research, but too many times the "you should do your homework" argument is overused. Sometimes you make a decision based in some data that changes later, or is incomplete), as there are a lot of situations where you find problems as you go, not in the first steps. But, according to the description, looks like PostgreSQL is a better match and the transition hasn't been too painful, so we can classify this into "bump in the road"/"success history". Also, probably right now the DB schema is way more known that in the beginning, which simplifies the use of a relational DB.

datasage 14y ago
- The proper way of doing an audit (and search later) is to make an independent collection with a reference to the other(s) document in a different collection. Then you can index by user, date, or any other field and leave the main collection alone. The described embedded access collection doesn't look very scalable. I think this point is very important even in the RDBMS side. There are cases, even with relational datastores that would preform better if the dataset was built to the query. The difficulty comes into play when you are trying to keep the denormalized data up to date based on changes within the base dataset.

se85 14y ago

The guy just jumped on the bandwagon without having a clue. Just reading this blog - it's clear that MongoDB was not a good fit for him, if he had bothered to do some research, he would have found this out on day one. Thats the real lesson he should be taking away from this and blogging about yet somehow MongoDB are trolls and it's all their fault because of a lack of features and they have bypassed 40 years of computer science and blah blah blah blah, excuses, excuses, excuses. edit: removed a few pointless sentences :-)

rbranson 14y ago

I'm no fan of MongoDB, but this same advice goes for any NoSQL data store. I am an Apache Cassandra contributor and community MVP, but my advice stays the same: it's best just to start with a SQL database and go from there. Read some books and learn it well: the "SQL Cookbook" from O'Reilly is great, and so is "The Art of SQL." Premature optimization continues to be the root of all evil.

bunderbunder 14y ago
Don't forget to read a book that's specific to your particular RDBMS. Because SQL databases are only trivially interchangeable for trivial cases. There have been more than a couple times when I was ready to blame the relational model, but further investigation revealed that the real root of the problem was that the existing schema or query used an approach that was optimized for one DBMS but performed terribly on the one we were actually using.
- rbranson 14y ago
  This is why the "SQL Cookbook" is good, it has dozens and dozens of examples with permutations in Oracle, MS SQL, MySQL, and PostgreSQL.
- gregjor 14y ago
  Wrong. SQL databases (except for SQLite) are almost completely interchangeable because they are all based on the same relational model and they all implement the ANSI SQL standard with only minor deviations. If you have a lot of stored procedures and triggers -- executable code embedded in the database -- you will have to rewrite that. Oracle is in a world of its own in a lot of ways, but if you are moving to or from Oracle you should have the required resources and expertise in your budget. If you are referring to the well-known cost of COUNT(*) in PostgreSQL (or MySQL with InnoDB, for that matter), or different ways to handle full-text searching, I agree that there are differences you have to deal with, but they are not usually a big deal. Contrast the fairly easy and routine process of migrating between MySQL, PostgreSQL, or SQL Server with the huge amount of work involved changing your application code from MongoDB to anything else.

armored_mammal 14y ago

Can someone confirm that there is no such thing as a case insensitive index/search in Mongo? If true it seems likely that the author's comments have some degree of truth, at least when it comes to its usefulness for web and mobile applications. Storing data only lowercase isn't a good a idea for obvious reasons, and storing two copies of the same data for searching only, while not the end of the world, seems a little silly.

zemo 14y ago
case-insensitive regex searches are supported.
- lucasjans 14y ago
  Do you know why are case-insensitive searches not recommended? What's the realistic work-around?
  - parfe 14y ago
    http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-RegularExpressions http://www.mongodb.org/display/DOCS/Advanced+Queries#Advance... Regex queries ending with /i (case-insensitive flag) cannot efficiently use indexes, but must do full index scans.
  - ericcholis 14y ago
    I've found that a simple {"lastname":/cholis/i} works great. However, trying to do the same thing for a multi-key search isn't ideal. Specifically, searching for 3 words in a title using $and with multiple regex queries on a collection with 100,000+ documents took about 520 ms. The mongodb documentation suggests that you could have an array with your keywords, generated from the field you wish to search. Using indexes on multikeys would make this faster, but your index size would be much larger.[1] For my project, I'm likely going to institute solution like elasticsearch or SOLR. [1] http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mong...

manorasa 14y ago

I think the real lesson here is use the right tool for the right job.

jamesli 14y ago

I am both a database guy and a software engineer. Being a software engineer, i kind of understand the hype behind NoSQL. Being a database guy spending years in studying how database engine works under the hood, many NoSQL implementations make me wonder how powerful marketing can be. In general, I love the ideas behind NoSQL. I can still feel the excitement when reading the BigTable and MapReduce papers. HBase, Hadoop, Radis, etc. are awesome products. I use some of them in my work. But some other NoSQL products? Being engineers, we must understand the implementation and be full aware of its limitations, instead of believing their marketing materials. Well, if all you want is to test a toy product, to build a prototype, or your product is of low concurrency and low data size and you have no concern on operation, it certainly looks that they make your development easier. But in these scenarios, any good relational databases won't add significant burden either.

taligent 14y ago
> Being engineers, we must understand the implementation and be full aware of its limitations, instead of believing their marketing materials. And as engineers we must understand that most other engineers do take their role seriously and evaluate products on their merits. Implying that they are falling for "marketing" just because you don't agree with their choice and then lecturing them for their choice doesn't make you come across well.
- einhverfr 14y ago
  I do think that the popularity of MySQL, however, owes a lot to it being used by non-engineers for simple web apps, though ;-)

dkarl 14y ago

You have to load every document in the database and extract the audit trail from it, then filter it in your app for the user you’re looking for. Just the thought of what that would do to my hardware was enough to turn me off the whole idea. Naive question from somebody who has done a little reading on and dabbling with key/document-with-MapReduce style datastores, but who hasn't tackled a real production problem: I thought running queries over the entire dataset was one of the assumptions of horizontally scalable document stores? In terms of avoiding computation, you can only limit queries by document key, which even if you're clever/lucky doesn't always encode the parameters you're querying on, or doesn't encode them in the right order, so you should be prepared to run queries over your entire dataset. Hopefully the queries you run often are optimized (e.g., using indexes or clever use of key ranges), but in the general case, you have to be prepared to scan the whole shebang, and that's supposed to be okay because of horizontal scalability, right?

enjo 14y ago
Yep, or you need to build some other construct to support it (IE: keeping running tallies and the like). It's a tradeoff between the benefits of the document store vs the loss of relational data. The blog author here clearly didn't understand the trade-offs he was making. As always with these discussions: it's important to use the right tool for the job. I'm a big fan of what Mongo is doing. I've used it two higher scale projects and have no complaints. Of course, I'm using it in the context for which Mongo excels.

daveman 14y ago

As an analytics professional who was pressured into a MongoDB environment, I feel the OP's pain. If you want to do gymnastics with your data, (aggregations of aggregations, joining result sets back onto data), SQL expressions are a 1000 times easier than Mongo constructs (e.g. map reduces). We usually ended up scraping out data from Mongo and dumping records into a SQL database before doing our transformations. All that said, our developers loved the ease of simple retrieval and insertion, and of course the scalability. So I guess you ultimately need to base your decisions on your priorities. I don't fault the OP though, since it's hard to know just how limiting NoSQL will be until you try to do all the things you used to assume were database tablestakes (no pun intended).

itaborai83 14y ago

To be fair, some NoSQL solutions were being sold marketing wise as the be-all and end-all of data solutions. Just google "mongodb mysql migration" and look how everyone is/was so eager to jump on the non-relational bandwagon. Some backlash was to be expected, after all, we might have reached the Trough of Disillusionment

adambard 14y ago

As a relative idiot when it comes to this sort of thing, I'd like to insert the following supplementary question: what is the sort of application/dataset for which Mongo is particularly suited? I've used it on small projects, and have enjoyed it. Perhaps my data has just been simple/loosely-coupled enough to never run into these problems? I read a lot of posts like this on HN before every trying Mongo, so I've at least been convinced to always implement schema at the application layer. Others seem to keep learning that lesson in harder ways.

stingraycharles 14y ago
"what is the sort of application/dataset for which Mongo is particularly suited" The majority of the NoSQL databases are based on Amazon's Dynamo: loosely coupled replication. MongoDB is one of the few (next to Hbase and a few others) that adopts Google BigTable's architecture: data is divided in Ranges, and each mongod node serves multiple Ranges. This means MongoDB is able to provide atomicity where it's harder with other SQL databases. In particular, we need to be able to do some sort of "compare and swap" operation that is guaranteed to be atomic/consistent, while still being able to have our mongod nodes distributed over multiple datacenters. In Dynamo-based architectures, in order to provide the same amount of atomicity, you always end up writing to at least half + 1 the amount of replicated nodes you have available in your cluster. This is more awkward, and reduces the flexibility of the whole (the atomicity guarantee Mongo provides also works for stored javascript procedures, for example). Having said that, we're using MongoDB about 3 years in production at this point, but we're far from happy about the availability it provides (issues like MongoDB not detecting that a node has gone down, failing to fail over, etc). We run a HA service, and to date all of our failures in uptime have been either the fault of our hosting provider or mongodb not failing over when it should. As such, we're always looking for a better alternative to move to, but at the moment MongoDB is about as good as it gets.
- edwardcapriolo 14y ago
  MongoDB is not even remotely close big table architecture. It has a different data model and a different sharding model, and just about a different everything.

chaostheory 14y ago

For me, what killed my enthusiasm for mongodb is the write locks. Yes they have been greatly improved in the 2.x release but it's still not good enough (for me).

redler 14y ago

digiDoc is all about converting paper documents like receipts and business cards into searchable database, and so a document database seemed like a logical fit(!). It looks like this single initial assumption is where things started going wrong: conflating the pieces of paper that happen to be called "documents" in the real world with the concept of a "document" in the context of a system like MongoDB.

josephcooney 14y ago
It's an easy mistake to make - to assume two things with the same name might be similar. Especially when paper documents have been called thus for a very long time. It seems like another instance where all the good names were taken. http://jcooney.net/post/2012/04/03/Transaction-Argument-Class-Message-Service-Agent-Method.aspx http://jcooney.net/post/2012/04/03/Transaction-Argument-Clas...

aliks 14y ago

Reasons why drop mongodb: 1. try $or $and with $near 2. No b-tree index :: count() 10.000 rows = 100% CPU usage ;) Type google.com then:: site:jira.mongodb.org/browse/ planned but not scheduled

bassemali 14y ago

I've never used Ruby on the application layer, but I'd be wary of using an ODM with MongoDB. The single most shocking issue seems to be at the application / ODM level. Using the official 10gen-supported drivers gives you more control and a better understanding of what's going on every step of the way. Also, a thorough understanding of MongoDB indexing, advanced queries and schema design would have squashed all of these issues. Has anybody had a more pleasant experience with a MongoDB ODM?

gregjor 14y ago

You were fortunate to recognize that MongoDB was the wrong tool for your job, and lucky to be able to move to Postgres instead of continuing to throw your time and effort away. I see the ad hominem "you're an ignorant idiot" attacks already started, along with advice like using regexes to do case-insensitive searches. Watching the NoSQL "movement" encounter the problems RDBMSs fixed 20 years ago and then hand-wave and kludge them away is frustrating. I wrote about some of this in http://typicalprogrammer.com/?p=14 http://typicalprogrammer.com/?p=14. Look at the bright side: programmers who are writing NoSQL-backed apps are creating the fossil fuel that will keep programmers who know RDBMs working into our retirement years. I already have more work than I can do fixing web apps that were built around crap data management tools that failed to scale beyond a few thousand users. Your Postgres expertise will still be a money-making skill long after MongoDB is forgotten.

gaius 14y ago
The thing with the NoSQL guys is that many of them seem not to be in a position to make an educated comparison. For example, an, uhh, enthusiastic MongoDB advocate recently informed me that MongoDB was superior to Oracle because in Oracle you had to poll a table to see if it changed. Except, no, that isn't actually true: http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_dcn.htm#BGBBHGAH http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_... - and that document is from 2005. And you could do a trigger and an AQ message/callback 5 years before that (at least). You haven't needed to poll an Oracle database for changes in a loooong time. Basically, every evangelism point, you have to double-check and cross-reference, because as you say, the NoSQL guys are encountering issues the RDBMS community addressed years ago (7 in my example, but the sharding stuff, 20+ years) - except they think they are discovering it for the first time!
- gregjor 14y ago
  Programming is more like fashion than science in this regard. Every decade or so something truly new happens in the software world. All the rest is mostly sound and fury, signifying nothing. If you're young or new to programming it's easy to mistake the buzz around things like NoSQL for innovation when they are usually re-discoveries of old (and often discarded or obsolete) ideas dressed up in new clothes. There's also the tendency to favor new shiny things and reject old crufty (but proven) things, to want to be part of what seems like the leading edge, to be that guy in the cube farm who is playing with the cool new stuff. I have been programming longer than RDBMSs have been available, so I know from experience what it's like to manage large databases in application code, and how hard it can be to maintain consistency or do accurate queries and aggregation with half-baked tools. It's frustrating to see a new generation of programmers go through this, but it's human nature to ignore the past. My fourteen year old son wears his pants pulled down below his waist, Vans shoes, hoodies, lots of hair. He looks pretty much like I did when I was fourteen back in the 1970s. The underlying technologies are the same: pants, shirt, shoes, sweater, hair. The only differences are superficial. To him that style is edgy and contemporary and something his parents don't get. NoSQL is the gangster fashion of programming right now.

DonnyV 14y ago

If he just did 10min of research he would of realized MongoDB isn't for him. Also by doing that research he would've realized that MongoDB has no data constraints. Thats all done in your model in your application.

jeremyjh 14y ago

Upvoted for?

ilaksh 14y ago

On your home page you imply that you can automatically OCR arbitrary handwritten receipts into an analyzable format. No one can do that. That is your problem, not MongoDB. As far as aggregation, use the new Aggregation Framework http://docs.mongodb.org/manual/tutorial/aggregation-examples/ http://docs.mongodb.org/manual/tutorial/aggregation-examples...: db.zipcodes.aggregate $group: _id: "$state" totalPop: $sum: "$pop" , $match: totalPop: $gte: 10 * 1000 * 1000 As far as "losing the independence of your data access paths", no you don't. You are free to use linking instead of embedding wherever you want. http://www.mongodb.org/display/DOCS/Schema+Design#SchemaDesign-EmbeddingandLinking http://www.mongodb.org/display/DOCS/Schema+Design#SchemaDesi... MongoDB doesn't have a built-in full text search? So what. Most systems with large amounts of text to search do not rely on the text search capabilities built into relational databases anyway. People use actual full-text search engines like Lucene/Solr, Sphinx, reds, etc. Having said that, if you just wanted to support lowercase keyword queries with MongoDB, would it really be so hard to extract and store lowercase keywords from your text, as suggested here? http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mong... If you are trying to add four 1s and get '1111' instead of 4, that is an error in your application code which has nothing to do with MongoDB. Very common problem with JavaScript. If it is JavaScript, try finding the code where you are attempting numeric addition and change it so that instead of saying for example 'total += newNumber' it says 'total += (newNumber * 1)' .

nnnnnnnn 14y ago
"On your home page you imply that you can automatically OCR arbitrary handwritten receipts into an analyzable format. No one can do that. That is your problem" Jeez, lay off the confrontational tone. He doesn't say anything about OCR. Maybe he's using humans to do data entry? In any event, it's completely irrelevant to the topic of databases.
- ilaksh 14y ago
  I notice you ignored all of my several very specific points directly related to his issues with the database system and your only comment was a criticism about the tone you perceived. OK, maybe he is using humans to do data entry. The home page to me implies that the process is automatic, but I guess it doesn't rule out the possibility of humans doing data entry when he says 'tag and categorize'. But if he is using humans to do data entry instead of some automatic OCR, that is still his main business problem, rather than MongoDB. The application is relevant to the database discussion, and Hacker News is about all aspects of startups.

nashequilibrium 14y ago

Taking all these comments into account, if startups have to prototype quickly while trying to find market fit, does it makes sense to start off using something like mongodb but with the plan to migrate to another database when you business starts growing? The database space is so confusing right now. It seems like Postgres is the safest choice and i also like this post fron Adam D'Angelo - 'http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use-MySQL-as-the-data-store-instead-of-NoSQLs-such-as-Cassandra-MongoDB-or-CouchDB http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use...

nashequilibrium 14y ago

programminggeek 14y ago

Look, there are some places where document DB's solve problems easier/better than SQL, other places kind of suck. For example, plain old object mapping is easier with a document DB. Relational DB's tend to make your code look/feel/act more relational and less object oriented. Your object model tends to look just like your table structure. This can be good or bad depending on your viewpoint. There are some approaches to solve some of the author's problems that end up making the Mongo system look and feel a lot more like a SQL system because sometimes data is actually related. The author could have also taken a different approach to his data schema that would have fit more of a non-relational worldview. Software development and architecture is about making choices and working with and around the limitations of your tools. It doesn't matter if PostgreSQL or MongoDB are "better". It's about solving a problem using a set of tools you are comfortable with.

bitdiffusion 14y ago

It's not necessarily all or nothing - I have worked on several projects now each using multiple database-type options: mongodb for read-intensive, loose-schema type stuff where the growth is generally predictable (e.g. products, suppliers, logs), postgres for relational-type stuff (orders) and solr for searching (I know solr isn't a database but people seem hung up on whether mongodb supports case-insensitive searching - hint: don't use any database for search). I doubt that, unless it's extremely simple, any set of requirements are an exact match to only one of these technologies... mix and match is the future :P

Teef 14y ago

There are 3 reasons I have gone running and screaming from and RDBMS. 1. Software gets large / complex to get meaning full work done. I am all about data consistency but at some point it is time to break things up into services and not have a single database. 2. If the software is popular enough everyone is running to use NoSQL (cache is NoSQL). 3. Clearly it is not a good storage solution either because for example in an address book nested list greatly simplifies everything. (right tool for the job) I spent many years hammering away with RDBMS and by and large it was great until it wasn't. I try to look at data storage more holistically now based on best guess of the problem. I have tried to convert an application from Postgresql to MongoDB and it failed but that wasn't MongoDB's falt it was because I didn't change the data model to fit a document storage system. I have also tried to use PostgreSQL for a realtime reporting system and failed horrifically and that was not Postgesql fault it was mine. Amazing what happens when you stop pushing a chain and start pull it!

einhverfr 14y ago
>1. Software gets large / complex to get meaning full work done. I am all about data consistency but at some point it is time to break things up into services and not have a single database. This is true. Managing complexity is always an important task. I am not sure that NoSQL solves this however. Also the best way to break things up is to loosely couple things, and this requires to some extent that you have ACID compliance. A good RDBMS, like PostgreSQL or Oracle, will provide tools for managing that loose coupling. >"2. If the software is popular enough everyone is running to use NoSQL (cache is NoSQL)." Like proverbial lemmings over a cliff.... >"3. Clearly it is not a good storage solution either because for example in an address book nested list greatly simplifies everything. (right tool for the job)" Funny, I thought nesting was what WITH RECURSIVE was for.... I am not saying there aren't use cases for MongoDB or reasons to switch some applications. For example I can think of a few really cool apps, like maybe a network back-plane for a huge LDAP directory. Also content management might be a good fit. But despite your years of experience, it doesn't sound like you have really looked at how to solve these with good RDBMS's.....

tonynero 14y ago

The guy is getting such hate on the comments on his site, yet his opening line is that his choice was ill thought. Let him express his issues right? I choose MongoDB for my last side project and while it was awesome working schema-less and developing the client facing part of the project was certainly quicker to deliver, i feel pretty lost on the analytics/BI side of it and couldn't say it better than him: "Not having JOINs makes your data an intractable lump of mud" So coming from a relational/SQL background I found MongoDb awesome upfront, but frustrating later on... and yes I'm off to learn http://docs.mongodb.org/manual/applications/aggregation/ http://docs.mongodb.org/manual/applications/aggregation/

ww520 14y ago

Wow, the first comment on the blog is so vile. He angrily blamed the "victim" (OP) as a talentless developer. Tools are enablers and supposed to make ordinary people rock star. If it takes a rock star to use a tool, the tool fails.

voidr 14y ago

Relational databases are awesome if you are not dealing with huge amounts of data that your current hardware can't handle the relational way. There are some cases where you have a ridiculous amount of data(rows) and you simply can't store that in a relational database and you are happy to live without the benefits of relational databases. If you have millions of rows, you are probably better off with something like MongoDB, if you need to search that, you should probably use something like Sphinx or Lucene anyway. But if you know that you won't have too much data for the forceable future, you should use relational databases. OR you could simply use both.

gregjor 14y ago
There may be cases where non-relational databases are the right solution, but unless you are using a toy RDBMS millions of rows is not considered a lot of data. Millions of rows added every day... now you're talking. Oracle handles that kind of thing just fine.
jeltz 14y ago
A relational database at very modest hardware can handle millions of rows, and with a solid database, good hardware and a DBA who knows his shit you can handle billions. OpenStreetMap has over a billion nodes stored in a PostgreSQL database. http://www.openstreetmap.org/stats/data_stats.html http://www.openstreetmap.org/stats/data_stats.html My point is that you can get very far with a classic relational database before you have to scale vertically.
- einhverfr 14y ago
  Billions of rows btw is not a problem even on modest hardware, with a half-decent db, etc. It's analytics on this where it become possibly an issue. Retrieving one row out of 1 billion is not that much more complex than retrieving one row out of 1 million, nor is it that much more expensive computationally assuming the right index is in place. The problem comes with high concurrency, in particular very high write concurrency, or with very complex queries which require a lot of RAM to do properly. But that's where you need a solid db, good hardware, and a solid DBA.

mrinterweb 14y ago

I find this article to be more a reflection of a NoSQL newbie's failed foray with a document database that later realized that the grass is not as green as originally perceived. The developer realized that he does not like map-reduce and missed not having joins. I don't see how this person's failed experience with MongoDB is a reflection on MongoDB. I think the recent popularity of MongoDB bashing is maybe a testament to MongoDBs popularity. I'd guess that because MongoDB is probably the closest NoSQL database to a RDBMS with its ad hoc queries, that it is attracting many newcomers.

jeremyjh 14y ago
Yes, MongoDB is more of a general-purpose database with lots of features that remind us of relational databases. It is a purpose-built application database for applications that would otherwise almost certainly be built with a relational database. As they get deeper into their project and find out how some of the trade-offs play-out their self-doubt is always about "should we just go relational" - no one is staying up all night wondering if they should migrate to Riak. If you start out with Riak you almost certainly know why and are using it in a very specific context.

effinjames 14y ago

calm down redditors, he just need a basic non majestic scale solution, SQL fitted him very well. large scale data aggregation needs to address disk and network latency, that's where NoSQL shines. and if you operate at large scale no 1 single simple tool will do justice, remember quote from google, 'at scale everything breaks'?

Karunamon 14y ago
calm down redditors Please knock this crap off.

deleted 14y ago

[deleted]

dkhenry 14y ago

Competly aside from the Article. The level of vitriolic discourse in this topic is astounding. I am amazed that as a community discussions of Database engines can draw out such mean spirited anger. I have never down voted as many comments on HN in a single thread then I have on this topic. I don't care which side of the debate you come down on. There is no excuse for belittleing and insulting others in a technical forum. Thats right I am looking at you gregjor, gaius, and zemo In this case it appears to mostly be those arguing for Postgre, but I wouldn't care if you were arguing for sunshine and unicorns there is a way to behave civilly and your not doing it.

gregjor 14y ago
You can find this level of discourse in plenty of topics every day. Programmers draw blood over indenting with tabs or spaces. It's geek entertainment. I have never been upvoted so much on HN. I admit to strong opinions and light sarcasm but you'll have to show me where I've been uncivil, belittling or insulting, except perhaps in response to people who insulted me.
einhverfr 14y ago
In this case it appears to mostly be those arguing for Postgre Curious. I have never found a database named "Postgre" to be used by anybody. Perhaps you can direct me to the download site. I think the bigger issue is that there isn't a lot of discussion from the NoSQL crowd about what you give up when you go to a NoSQL solution. I think that sort of disclosure would help people weigh the options a lot better. From the comments of some people here you'd almost think they would build an ERP app in Mongo....

firemanx 14y ago

I work for a company that operates in the energy industry. We utilize both RDBMS and "NoSQL", both have their purposes that they fit in well. We store customer account and configuration data in Postgres, and use Cassandra to store time-series statistics and high write volume data. I have a background in data warehousing in both Oracle and SQL Server, and was part of the decision to use a polyglot persistence model. I've got at least a decade's worth of experience in the DW world, and more as a general developer before that, so I like to think I've got a relatively credible background in a variety of data stores. I haven't looked at Mongo much - it's durability concerns and the write lock stuff pushed me away from it early on (I don't mean to disparage it, but that was where it was at when I evaluated it), but Cassandra's configurable consistency levels and operational story at a cluster level are what sold us for our time-series data (that, and the ability to construct a sparse timeline and multiplex reads/writes). For anything we need flexible querying with, we push it into specialized Postgres dbs. The level of willful ignorance and vitrol in this thread is kind of amazing. Most of the really experienced DW guys I know are all looking at HBase, Cassandra and others because they fit a niche that we've all been looking for in certain data sets at really large scale. It doesn't mean we're ditching our relational data stores, it just means we're augmenting them with other tools because they fit the job at hand. To suggest that one tool is absolutely perfect for every scenarios seems a little short-sighted to me, possibly driven out of inexperience. I don't mean that as an insult - I know a lot of guys who've been working on the same data sets for 30 years who really do just need the one tool - however, you've got to realize there are other data sets and problems for which your hammer just won't fit.

anthony_barker 14y ago

Made the same mistake on a banking project 10 years ago (with domino). The project in question was a project tracking database. For accounting type problems use a relational database. For document driven items - e.g. a resume database - nosql works great. For a hybrid pick your battles... or use both.