6 ms·

Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one

Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one or more of: * didn't read the manual * poor schema * didn't maintain the database (compactions, etc.) In this case, they hit several: " Its volume on disk is growing 3-4 times faster than the real volume of data it store;" They should be doing compactions and are not. Using PostgreSQL does not avoid administration; it simply changes the administration to be done. "it eats up all the memory without the possibility to limit this" That's the idea -- that memory isn't actually used though; it's just memory mapping the file. It will swap out for something else that needs the space unless you are actively using all the data, in which case you really are using all your memory. Which is why you should put it on its own server... "it begins to slow down the application because of frequent disk access" "Finally we sleep quietly, and don’t fear that mongodb will drive out redis to swap once again." You should be running Mongo on a server by itself. At the very least, if you're having disk contention issues, don't run it on the same server as your other database. I'm not sure you always need to read the manual for everything, but for your production database, it's probably worth it.

dev360 13y ago
In all fairness, the compaction is a major pain in Mongo. I get a little worked up about this because I cant think of another database that handles compaction this poorly, but feel free to correct me if Im wrong.
- ehwizard 13y ago
  Have you tried turning on power of 2 allocation? In general, it makes compaction much less important. Though online compaction is definitely needed.
  - dev360 13y ago
    Yes we switched to it a month ago, which improved it but like you said, we are still having to compact frequently and having the hassle of switching the primary. Cassandra has turned out to be much more performant and easier to maintain for our use case.
lucian1900 13y ago
Mongo's disk format is extremely wasteful, the database files are gigantic. That is a real problem and there is no way to compact this to anywhere near the size something like Postgres would have for the same data. Mongo is very bad at managing used memory. In fact it doesn't actually manage memory since it just mmaps its database file. It also touches disk much more often than would be reasonable, especially for how much memory it uses. It's a terrible database and it is perfectly legitimate to be annoyed at it being this terrible.
- jeltz 13y ago
  And let's not forget the fact that it has a per database lock, which is a really strange choice for a document database.
  - mkennedy 13y ago
    The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.
    - lucian1900 13y ago
      It is a problem in practice if you have any amount of load. You'll see latency quickly go up with lock times, which go up with concurrency. Why bother, when you can use another db without this problem, right now?
- remon 13y ago
  Although it is true that MongoDB uses a lot of disk compared to your average RDMS there are reasons for that. 1) MongoDB (and various other NoSQL solution) are schemaless and thus have to store document fields along with the values for each document. This alone usually results in roughly twice as much actual disk space being used compared to an RDBMS. 2) MongoDB preallocates fairly large chunks of disk for their mmap based storage (2Gb per database files by default). This means there will be up to 2Gb * N where N is the number of logical databases in "wasted" (more accurately, unused) space. This can be addressed somewhat through the --smallfiles option. 3) The biggest issue that I actually consider an design flaw is the ineffective reuse of disk space previously occupied by deleted or, more commonly, moved documents. MongoDB reserves a bit of padding for each document but since a lot of documents can grow over time these documents will be moved around on disk leaving holes in the data files. These holes are not adequately re-used and a compaction is required to make that space available again. Compaction is NOT a background process at the moment and blocks writes. The "usePowerOf2Sizes" option will help with this issue at the expense of always using a power of 2 size in bytes per document. The above are factual reasons why MongoDB uses a lot of disk space. It's certainly a relatively young database and some issues do need to be addressed but this whole polarizing "it's terrible booo!" nonsense has to stop. Inform yourself, choose the tech appropriate for your project and post mortem aftwards. Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree). Tokutek offers what seems to be a better storage engine but does lag behind a bit on releases. I'm not affiliated with them but if you're interested you can check out http://www.tokutek.com/products/tokumx-for-mongodb/ http://www.tokutek.com/products/tokumx-for-mongodb/
jlouis 13y ago
For what it is worth, I would think people actually try different things in the existing setup before they decide on doing a switch like this. It is not easy to pull off at all. My guess would be that if you have way more Postgres knowledge in the house, then it is more sensible to run Postgres. This also drives the amount of administrative overhead needed.
rlpb 13y ago
> Seriously, another case of using Mongo incorrectly? If a large proportion of MongoDB users are using it incorrectly, then I'd argue that it is a MongoDB problem, if only a documentation and messaging one. Clarity on what is and is not an appropriate use should be prominent. So, what is this proportion?
- derefr 13y ago
  Or, to be even more specific--if there's a Right Way to use a program, that Right Way should be encoded as defaults you have to override (if you know what you're doing), and automated actions you have to disable (if you know what you're doing.)
  - threeseed 13y ago
    It has nothing to do with the defaults. It is all about people forgetting that MongoDB is a document database and not a relational one. I can write apps that will be 10x faster with MongoDB and 10x faster with PostgreSQL. It's all about matching your domain model to your database.
    - macspoofing 13y ago
      >t is all about people forgetting that MongoDB is a document database and not a relational one. It's been a while since I looked into Mongo, but that was a Mongo marketing problem. They used to (still?) advertised themselves as a RDBMS replacement, literally.
      - raverbashing 13y ago
        Yes, but it's more complex than for example, replacing MySQL with PostgreSQL, where the basic structure is going to be the same. You have to think and adapt to convert from a RDBS to MongoDB
      - threeseed 13y ago
        What did you expect them to do ? Of course they are going to position themselves as a RDBMS solution. They want people to switch in order to make money. It doesn't mean as a developer you're going to be stupid or incompetent enough to ignore every piece of technical documentation and just focus on the marketing lines.
chaostheory 13y ago
> Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one or more of: > * poor schema You're right. If people read the awesome mongodb docs before using it, they'd figure out that mongodb's ideal, good for performance schema has limitations that doesn't fit with a lot of projects. Of course this may have changed since mongodb evolves pretty quickly.
jtchang 13y ago
> * didn't read the manual > * poor schema > * didn't maintain the database (compactions, etc.) The real world dictates that this happens more often than not. You know why I like Postgres? When I don't read the manual, create a crappy schema, and forgot to maintain the database it STILL seems to work okay.
- randomdata 13y ago
  To be fair, Postgres has automatic vacuuming now, but it is a relatively new feature. Both projects seem to agree that it is not a high-priority item, though there is certainly something to be said about using a mature product, which Mongo is most certainly not. Your comment has made me quite curious to know what people using mature databases of the time were saying about Postgres 19 years ago, when it was roughly the same age Mongo is today.
  - makomk 13y ago
    Back when Postgres didn't have automatic vacuuming, the need for manual vacuuming was one of the commonly-suggested reasons why MySQL was a better option - after all, MySQL just worked out of the box.
  - deleted 13y ago
    [deleted]
  - jeltz 13y ago
    Yeah, autovacuum was added in 7.4 (2003) and not really to be trusted to do its job without monitoring until 8.3 (2008). But the priorities of the PostgreSQL project have changed in the last about 5 years. Today usability is important.
  - sergiosgc 13y ago
    There's a huge difference. Postgresql was formally correct first, then it became fast, then it became easy to use. I always had the feeling I could trust it with my data. Postgresql, if wrong, would be wrong towards correctedness and towards data safety, at the expense of speed. I definitely don't get the same vibe from MongoDB.
jeffdavis 13y ago
> They should be doing compactions and are not. https://jira.mongodb.org/browse/SERVER-11763 https://jira.mongodb.org/browse/SERVER-11763 It looks like compaction is an offline process. That really puts the user between a rock and a hard place.
- functional_test 13y ago
  In a proper production environment, you just compact each slave one at a time because you have a replica set rather than a single instance. Of course, if you aren't replicating your business's production database, you have a whole world of problems.
  - nl 13y ago
    Of course, if you aren't replicating your business's production database, you have a whole world of problems. Not really. In RDBMS you can have offline backups (assuming you aren't building for high availability). I don't know if that exists for MongoDB. Even pre-auto-vacuuming Postgres (ie, before Postgres v8) would allow you to vacuum the database while online. You had a performance hit, but there was no need to switch to a backup server or anything.
  - jeffdavis 13y ago
    Good point, but that would make me a little nervous. What happens if the compaction takes a while and the replica gets far behind? And wouldn't the compaction time just keep getting longer and longer as data grows? Can you repartition the data online to keep the cleanup work bounded?
  - darkarmani 13y ago
    > In a proper production environment, you just compact each slave one at a time because you have a replica set rather than a single instance. That's the solution?? That sounds like a workaround in a production environment.
    - lmm 13y ago
      Yes and no. Deleting is a hard problem that many databases don't handle well. Explicitly deciding to punt the problem like this is a much better approach than allowing performance to degrade unpredictably.
blablabla123 13y ago
What about locking? I heard that Mongo has a locking with only DB-granularity.
- mkennedy 13y ago
  The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.
jb007 13y ago
MongoDB therefore, is not a general purpose database. I recommend http://www.amisalabs.com http://www.amisalabs.com
bsg75 13y ago
> "Finally we sleep quietly, and don’t fear that mongodb will drive out redis to swap once again." MongoDB and Redis on the same box? Two data stores that need working set / all of the data to reside in RAM for performance? That is a recipe bound for failure. Everyone seems to learn about physical separation the hard way.
rdtsc 13y ago
> Seriously, another case of using Mongo incorrectly? If everyone uses Mongo incorrectly, the problem is not Mongo. It is like the person crying out how everyone in the world is crazy.
- functional_test 13y ago
  I seem to have been able to use it correctly. In fact, I ran a cluster for years in production without any issues. I know of several other groups that have used it successfully as well. As far as I can tell, a lot of people assumed it worked like a SQL database. It doesn't, which disappointed them. I'll even admit that some of the original defaults like the write concerns didn't really make sense as defaults. But that was all in the introductory documentation. Major subsystems like databases deserve at least a skim of the documentation if not a full read; if not up front then at least before putting them into production.
nailer 13y ago
The current stable node drivers silently throws away exceptions. Seriously, mongodb inc acknowledge it. Is this also a case of not using mongo correctly?

dev360 13y ago

In all fairness, the compaction is a major pain in Mongo. I get a little worked up about this because I cant think of another database that handles compaction this poorly, but feel free to correct me if Im wrong.

ehwizard 13y ago
Have you tried turning on power of 2 allocation? In general, it makes compaction much less important. Though online compaction is definitely needed.
- dev360 13y ago
  Yes we switched to it a month ago, which improved it but like you said, we are still having to compact frequently and having the hassle of switching the primary. Cassandra has turned out to be much more performant and easier to maintain for our use case.

lucian1900 13y ago

Mongo's disk format is extremely wasteful, the database files are gigantic. That is a real problem and there is no way to compact this to anywhere near the size something like Postgres would have for the same data. Mongo is very bad at managing used memory. In fact it doesn't actually manage memory since it just mmaps its database file. It also touches disk much more often than would be reasonable, especially for how much memory it uses. It's a terrible database and it is perfectly legitimate to be annoyed at it being this terrible.

jeltz 13y ago
And let's not forget the fact that it has a per database lock, which is a really strange choice for a document database.
- mkennedy 13y ago
  The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.
  - lucian1900 13y ago
    It is a problem in practice if you have any amount of load. You'll see latency quickly go up with lock times, which go up with concurrency. Why bother, when you can use another db without this problem, right now?
remon 13y ago
Although it is true that MongoDB uses a lot of disk compared to your average RDMS there are reasons for that. 1) MongoDB (and various other NoSQL solution) are schemaless and thus have to store document fields along with the values for each document. This alone usually results in roughly twice as much actual disk space being used compared to an RDBMS. 2) MongoDB preallocates fairly large chunks of disk for their mmap based storage (2Gb per database files by default). This means there will be up to 2Gb * N where N is the number of logical databases in "wasted" (more accurately, unused) space. This can be addressed somewhat through the --smallfiles option. 3) The biggest issue that I actually consider an design flaw is the ineffective reuse of disk space previously occupied by deleted or, more commonly, moved documents. MongoDB reserves a bit of padding for each document but since a lot of documents can grow over time these documents will be moved around on disk leaving holes in the data files. These holes are not adequately re-used and a compaction is required to make that space available again. Compaction is NOT a background process at the moment and blocks writes. The "usePowerOf2Sizes" option will help with this issue at the expense of always using a power of 2 size in bytes per document. The above are factual reasons why MongoDB uses a lot of disk space. It's certainly a relatively young database and some issues do need to be addressed but this whole polarizing "it's terrible booo!" nonsense has to stop. Inform yourself, choose the tech appropriate for your project and post mortem aftwards. Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree). Tokutek offers what seems to be a better storage engine but does lag behind a bit on releases. I'm not affiliated with them but if you're interested you can check out http://www.tokutek.com/products/tokumx-for-mongodb/ http://www.tokutek.com/products/tokumx-for-mongodb/

jlouis 13y ago

For what it is worth, I would think people actually try different things in the existing setup before they decide on doing a switch like this. It is not easy to pull off at all. My guess would be that if you have way more Postgres knowledge in the house, then it is more sensible to run Postgres. This also drives the amount of administrative overhead needed.

rlpb 13y ago

> Seriously, another case of using Mongo incorrectly? If a large proportion of MongoDB users are using it incorrectly, then I'd argue that it is a MongoDB problem, if only a documentation and messaging one. Clarity on what is and is not an appropriate use should be prominent. So, what is this proportion?

derefr 13y ago
Or, to be even more specific--if there's a Right Way to use a program, that Right Way should be encoded as defaults you have to override (if you know what you're doing), and automated actions you have to disable (if you know what you're doing.)
- threeseed 13y ago
  It has nothing to do with the defaults. It is all about people forgetting that MongoDB is a document database and not a relational one. I can write apps that will be 10x faster with MongoDB and 10x faster with PostgreSQL. It's all about matching your domain model to your database.
  - macspoofing 13y ago
    >t is all about people forgetting that MongoDB is a document database and not a relational one. It's been a while since I looked into Mongo, but that was a Mongo marketing problem. They used to (still?) advertised themselves as a RDBMS replacement, literally.
    - raverbashing 13y ago
      Yes, but it's more complex than for example, replacing MySQL with PostgreSQL, where the basic structure is going to be the same. You have to think and adapt to convert from a RDBS to MongoDB
    - threeseed 13y ago
      What did you expect them to do ? Of course they are going to position themselves as a RDBMS solution. They want people to switch in order to make money. It doesn't mean as a developer you're going to be stupid or incompetent enough to ignore every piece of technical documentation and just focus on the marketing lines.

chaostheory 13y ago

> Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one or more of: > * poor schema You're right. If people read the awesome mongodb docs before using it, they'd figure out that mongodb's ideal, good for performance schema has limitations that doesn't fit with a lot of projects. Of course this may have changed since mongodb evolves pretty quickly.

jtchang 13y ago

> * didn't read the manual > * poor schema > * didn't maintain the database (compactions, etc.) The real world dictates that this happens more often than not. You know why I like Postgres? When I don't read the manual, create a crappy schema, and forgot to maintain the database it STILL seems to work okay.

randomdata 13y ago
To be fair, Postgres has automatic vacuuming now, but it is a relatively new feature. Both projects seem to agree that it is not a high-priority item, though there is certainly something to be said about using a mature product, which Mongo is most certainly not. Your comment has made me quite curious to know what people using mature databases of the time were saying about Postgres 19 years ago, when it was roughly the same age Mongo is today.
- makomk 13y ago
  Back when Postgres didn't have automatic vacuuming, the need for manual vacuuming was one of the commonly-suggested reasons why MySQL was a better option - after all, MySQL just worked out of the box.
- deleted 13y ago
  [deleted]
- jeltz 13y ago
  Yeah, autovacuum was added in 7.4 (2003) and not really to be trusted to do its job without monitoring until 8.3 (2008). But the priorities of the PostgreSQL project have changed in the last about 5 years. Today usability is important.
- sergiosgc 13y ago
  There's a huge difference. Postgresql was formally correct first, then it became fast, then it became easy to use. I always had the feeling I could trust it with my data. Postgresql, if wrong, would be wrong towards correctedness and towards data safety, at the expense of speed. I definitely don't get the same vibe from MongoDB.

jeffdavis 13y ago

> They should be doing compactions and are not. https://jira.mongodb.org/browse/SERVER-11763 https://jira.mongodb.org/browse/SERVER-11763 It looks like compaction is an offline process. That really puts the user between a rock and a hard place.

functional_test 13y ago
In a proper production environment, you just compact each slave one at a time because you have a replica set rather than a single instance. Of course, if you aren't replicating your business's production database, you have a whole world of problems.
- nl 13y ago
  Of course, if you aren't replicating your business's production database, you have a whole world of problems. Not really. In RDBMS you can have offline backups (assuming you aren't building for high availability). I don't know if that exists for MongoDB. Even pre-auto-vacuuming Postgres (ie, before Postgres v8) would allow you to vacuum the database while online. You had a performance hit, but there was no need to switch to a backup server or anything.
- jeffdavis 13y ago
  Good point, but that would make me a little nervous. What happens if the compaction takes a while and the replica gets far behind? And wouldn't the compaction time just keep getting longer and longer as data grows? Can you repartition the data online to keep the cleanup work bounded?
- darkarmani 13y ago
  > In a proper production environment, you just compact each slave one at a time because you have a replica set rather than a single instance. That's the solution?? That sounds like a workaround in a production environment.
  - lmm 13y ago
    Yes and no. Deleting is a hard problem that many databases don't handle well. Explicitly deciding to punt the problem like this is a much better approach than allowing performance to degrade unpredictably.

blablabla123 13y ago

What about locking? I heard that Mongo has a locking with only DB-granularity.

mkennedy 13y ago
The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.

jb007 13y ago

MongoDB therefore, is not a general purpose database. I recommend http://www.amisalabs.com http://www.amisalabs.com

bsg75 13y ago

> "Finally we sleep quietly, and don’t fear that mongodb will drive out redis to swap once again." MongoDB and Redis on the same box? Two data stores that need working set / all of the data to reside in RAM for performance? That is a recipe bound for failure. Everyone seems to learn about physical separation the hard way.

rdtsc 13y ago

> Seriously, another case of using Mongo incorrectly? If everyone uses Mongo incorrectly, the problem is not Mongo. It is like the person crying out how everyone in the world is crazy.

functional_test 13y ago
I seem to have been able to use it correctly. In fact, I ran a cluster for years in production without any issues. I know of several other groups that have used it successfully as well. As far as I can tell, a lot of people assumed it worked like a SQL database. It doesn't, which disappointed them. I'll even admit that some of the original defaults like the write concerns didn't really make sense as defaults. But that was all in the introductory documentation. Major subsystems like databases deserve at least a skim of the documentation if not a full read; if not up front then at least before putting them into production.

nailer 13y ago

The current stable node drivers silently throws away exceptions. Seriously, mongodb inc acknowledge it. Is this also a case of not using mongo correctly?