5 ms·

Mongo's disk format is extremely wasteful, the database files are gigantic. That is a real problem and there is no way to compact this to anywhere near the size

by lucian1900 13y ago

Mongo's disk format is extremely wasteful, the database files are gigantic. That is a real problem and there is no way to compact this to anywhere near the size something like Postgres would have for the same data. Mongo is very bad at managing used memory. In fact it doesn't actually manage memory since it just mmaps its database file. It also touches disk much more often than would be reasonable, especially for how much memory it uses. It's a terrible database and it is perfectly legitimate to be annoyed at it being this terrible.

jeltz 13y ago
And let's not forget the fact that it has a per database lock, which is a really strange choice for a document database.
- mkennedy 13y ago
  The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.
  - lucian1900 13y ago
    It is a problem in practice if you have any amount of load. You'll see latency quickly go up with lock times, which go up with concurrency. Why bother, when you can use another db without this problem, right now?
remon 13y ago
Although it is true that MongoDB uses a lot of disk compared to your average RDMS there are reasons for that. 1) MongoDB (and various other NoSQL solution) are schemaless and thus have to store document fields along with the values for each document. This alone usually results in roughly twice as much actual disk space being used compared to an RDBMS. 2) MongoDB preallocates fairly large chunks of disk for their mmap based storage (2Gb per database files by default). This means there will be up to 2Gb * N where N is the number of logical databases in "wasted" (more accurately, unused) space. This can be addressed somewhat through the --smallfiles option. 3) The biggest issue that I actually consider an design flaw is the ineffective reuse of disk space previously occupied by deleted or, more commonly, moved documents. MongoDB reserves a bit of padding for each document but since a lot of documents can grow over time these documents will be moved around on disk leaving holes in the data files. These holes are not adequately re-used and a compaction is required to make that space available again. Compaction is NOT a background process at the moment and blocks writes. The "usePowerOf2Sizes" option will help with this issue at the expense of always using a power of 2 size in bytes per document. The above are factual reasons why MongoDB uses a lot of disk space. It's certainly a relatively young database and some issues do need to be addressed but this whole polarizing "it's terrible booo!" nonsense has to stop. Inform yourself, choose the tech appropriate for your project and post mortem aftwards. Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree). Tokutek offers what seems to be a better storage engine but does lag behind a bit on releases. I'm not affiliated with them but if you're interested you can check out http://www.tokutek.com/products/tokumx-for-mongodb/ http://www.tokutek.com/products/tokumx-for-mongodb/
- efuquen 13y ago
  "Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree)." This. Aside from general reliability issues I've had in the past, which are definitely fixable and might be now, this design decision is the thing that will cripple, and continue to cripple the db. The idea that some how the kernel is going to better at managing a db's memory is ridiculous. Kernel paging was designed for a completely different use case, it's not optimized to manage the memory of a database. The idea that somehow this line of reasoning: "oh hey guys, those Linux kernel hackers are smart and they can do it way better then use so let's just use their memory management system" was ok in the mind of the MongoDB creators really puts me off. It doesn't matter how many geniuses work on kernel memory management when they are solving a completely different problem then you need to solve, all memory management schemes aren't equal. When providing a db you need full control to be able to probably tune memory management to fit your needs, there is no magical fix for this.
  - remon 13y ago
    Why "continue to cripple"? It's actually one of the things that can be changed quite easily at some point in time and doesn't introduce insurmountable backwards compatibility issues. And as I mentioned (and you cut out of your quote) there is at least one alternative available already.
- lucian1900 13y ago
  1. That's not necessarily the only possible implementation. It would be trivial to assign a number to each key and keep this map in the header of the db file. 2. That's not really the issue, I don't care about the size of small dbs. Large dbs have gigantic sizes. 3. That is absolutely abysmal, yes. It doesn't just use 2x as much space as other dbs, in practice that can be up to 20-30x as much in bad cases. It's commonly at least 5x. It really is absolutely terrible when compared to pretty much anything. It's slower than safer dbs that can do the same and more things. There is no selling point. Just don't use it.
  - remon 13y ago

jeltz 13y ago

And let's not forget the fact that it has a per database lock, which is a really strange choice for a document database.

mkennedy 13y ago
The guys at MongoDB are working on moving this to a more fine-grained lock in the future. In practice, it's not usually a problem but it'll be less so going forward.
- lucian1900 13y ago
  It is a problem in practice if you have any amount of load. You'll see latency quickly go up with lock times, which go up with concurrency. Why bother, when you can use another db without this problem, right now?

remon 13y ago

Although it is true that MongoDB uses a lot of disk compared to your average RDMS there are reasons for that. 1) MongoDB (and various other NoSQL solution) are schemaless and thus have to store document fields along with the values for each document. This alone usually results in roughly twice as much actual disk space being used compared to an RDBMS. 2) MongoDB preallocates fairly large chunks of disk for their mmap based storage (2Gb per database files by default). This means there will be up to 2Gb * N where N is the number of logical databases in "wasted" (more accurately, unused) space. This can be addressed somewhat through the --smallfiles option. 3) The biggest issue that I actually consider an design flaw is the ineffective reuse of disk space previously occupied by deleted or, more commonly, moved documents. MongoDB reserves a bit of padding for each document but since a lot of documents can grow over time these documents will be moved around on disk leaving holes in the data files. These holes are not adequately re-used and a compaction is required to make that space available again. Compaction is NOT a background process at the moment and blocks writes. The "usePowerOf2Sizes" option will help with this issue at the expense of always using a power of 2 size in bytes per document. The above are factual reasons why MongoDB uses a lot of disk space. It's certainly a relatively young database and some issues do need to be addressed but this whole polarizing "it's terrible booo!" nonsense has to stop. Inform yourself, choose the tech appropriate for your project and post mortem aftwards. Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree). Tokutek offers what seems to be a better storage engine but does lag behind a bit on releases. I'm not affiliated with them but if you're interested you can check out http://www.tokutek.com/products/tokumx-for-mongodb/ http://www.tokutek.com/products/tokumx-for-mongodb/

efuquen 13y ago
"Small note on the mmap thing; a lot of people consider the mmap based storage engine a big issue (I tend to agree)." This. Aside from general reliability issues I've had in the past, which are definitely fixable and might be now, this design decision is the thing that will cripple, and continue to cripple the db. The idea that some how the kernel is going to better at managing a db's memory is ridiculous. Kernel paging was designed for a completely different use case, it's not optimized to manage the memory of a database. The idea that somehow this line of reasoning: "oh hey guys, those Linux kernel hackers are smart and they can do it way better then use so let's just use their memory management system" was ok in the mind of the MongoDB creators really puts me off. It doesn't matter how many geniuses work on kernel memory management when they are solving a completely different problem then you need to solve, all memory management schemes aren't equal. When providing a db you need full control to be able to probably tune memory management to fit your needs, there is no magical fix for this.
- remon 13y ago
  Why "continue to cripple"? It's actually one of the things that can be changed quite easily at some point in time and doesn't introduce insurmountable backwards compatibility issues. And as I mentioned (and you cut out of your quote) there is at least one alternative available already.
lucian1900 13y ago
1. That's not necessarily the only possible implementation. It would be trivial to assign a number to each key and keep this map in the header of the db file. 2. That's not really the issue, I don't care about the size of small dbs. Large dbs have gigantic sizes. 3. That is absolutely abysmal, yes. It doesn't just use 2x as much space as other dbs, in practice that can be up to 20-30x as much in bad cases. It's commonly at least 5x. It really is absolutely terrible when compared to pretty much anything. It's slower than safer dbs that can do the same and more things. There is no selling point. Just don't use it.
- remon 13y ago