4 ms·

A lot of hype... And we still have db level locking. If document level is too difficult, at LEAST do collection level (not that it is too much better, but least

A lot of hype... And we still have db level locking. If document level is too difficult, at LEAST do collection level (not that it is too much better, but least it some real improvement).

z92 13y ago
I was reading somewhere that Mongo can't do document level/record level locking because of mmap'ed files. The whole database is memory mapped. And mmap doesn't understand underlined data structure, it views the whole file as a large single blob. Ditching mmap will not be that easy, cause most of the speed and simplicity of Mongo comes from using mmap.
- arkham 13y ago
  It doesn't use mmap for locking though - and in any case one database is already multiple files (2GB max). There are internal data structures used by MongoDB and the database itself takes care of locking, mmap just gets the data into memory.
- AlisdairO 13y ago
  mmap does complicate things. A traditional database often works using write ahead logs. The log holds changes made to the database over time - so when you want to write a change to your DB, you put 'I'm changing value x.y to 50' in your WAL. Sometime later the actual data pages holding the modified x data structure can be written out to disk. If you have a crash before the data pages get written out, you can 'replay' your WAL file to redo all the lost changes. Unfortunately, a central requirement of a write ahead log is that the data pages must not get written before the WAL. If that were to occur, and there were a crash, the system wouldn't know that it had to undo the changes to the data pages, leaving you with corrupt data. mmap generally doesn't provide the ability to pin your dirty pages in memory - they're subject to getting flushed any time the system is under memory pressure - so it makes logging a lot more complicated to get right.
  - arkham 13y ago
    Just to note, the MongoDB journal is basically a write ahead log and has been around since 1.8. It's not simple, you are correct, and involved remapping portions of memory privately, and leads to some inflated numbers on the virtual memory reporting side. There's a great write up here: http://blog.mongodb.org/post/33700094220/how-mongodbs-journaling-works http://blog.mongodb.org/post/33700094220/how-mongodbs-journa...
    - AlisdairO 13y ago
      That's interesting, thanks!
- bashcoder 13y ago
  Yes, and an important limitation stems from this as well. For a tool like Redis, it's fairly easy for me to accept the limitation that my data size can't exceed available RAM. But for an indexed document store with full-text query capabilities, it's a lot harder to for me to accept that limitation.
  - thedufer 13y ago
    Are you implying that a performant Mongo DB must keep all data in RAM? That's not true. We've got almost 1TB in a Mongo DB, and we sure don't have that much RAM. You do have to be able to keep your indexes in RAM, but that's much less limiting.
    - bashcoder 13y ago
      Not 1:1, no - thanks for clarifying that. But depending on the content and the indexing, there is a strong correlation between the size of the database and the memory requirements under Mongo DB.
Kiro 13y ago
I don't know anything about MongoDB but can you give an example where DB level locking is a problem?
- mikegioia 13y ago
  High volume of writes
  - angersock 13y ago
    Isn't that supposed to be the ideal use case for Mongo though?
    - zardosht 13y ago
      Not for single-server performance. The database level lock severely limits MongoDB's single server performance. Just look up the sysbench benchmark comparing MongoDB with TokuMX (which I work on)
  - brianwawok 13y ago
    High volume of writes from MANY producers. From 1 producer, it doesn't matter. Though the point of mongo is to be webscale which implies to me many writers.
    - bsg75 13y ago
      What makes the single producer (writer?) case different from multiple, in the context of the effect on readers? If there is a single high-volume data pump, for example machine generated data, will readers be affected by a continuous "fire hose" of incoming data?
    - mikegioia 13y ago
      In my experience, even 1 producer caused a full database lock for hours on our production server. We have 1 mongo server and an erroneous scheduled task of ours started at about 6am. It's only one process but the task basically re-syncs an entire collection (which was about 80,000 writes). That single producer caused wide-scale locking/hanging for all readers on the website and I had to manually stop the task during business hours because of that. Oy!
sigzero 13y ago
"Finally, MongoDB 2.6 lays the foundation for massive improvements to concurrency in MongoDB 2.8, including document-level locking." They hear you and are working on it.
- portmanteaufu 13y ago
  It was about a year between the first release of 2.4 and the 2.6 release. Should we expect another full year for 2.8?
  - leif 13y ago
    I believe they are on a ~yearly schedule now.
disbelief 13y ago
Every Mongo release announcement I hold out hope for an improvement to database-level write lock, and every time I'm disappointing. Considering this is probably the #1 or #2 complaint people have with Mongo, I'm surprised it hasn't been addressed ahead of some of the other features that have made it in to recent releases.
rogerbinns 13y ago
Heck I'd even be happy for them to do locking on a per file basis. (For those unfamiliar with mongodb, the actual database storage is broken up into 2GB files.)
- citruspi 13y ago
  I believe the 2 GB database limit is only applicable to 32-bit instances[0]. [0]: http://blog.mongodb.org/post/137788967/32-bit-limitations http://blog.mongodb.org/post/137788967/32-bit-limitations
  - rogerbinns 13y ago
    You are confusing two things. I am talking about the files making up a database. MongoDB doesn't use one file per database - instead it allocates files up to 2GB in size (the first one starts at 64MB and then they double in size until the 2GB limit). So for example a database that is 200GB will consist of (roughly) 100 2GB files. What you linked to is a consequence of memory mapping. Mapping a single 2GB file in a 32 bit process will use up virtually all the address space and you couldn't map more than one at a time.

A lot of hype... And we still have db level locking. If document level is too difficult, at LEAST do collection level (not that it is too much better, but least it some real improvement).

z92 13y ago

I was reading somewhere that Mongo can't do document level/record level locking because of mmap'ed files. The whole database is memory mapped. And mmap doesn't understand underlined data structure, it views the whole file as a large single blob. Ditching mmap will not be that easy, cause most of the speed and simplicity of Mongo comes from using mmap.

arkham 13y ago
It doesn't use mmap for locking though - and in any case one database is already multiple files (2GB max). There are internal data structures used by MongoDB and the database itself takes care of locking, mmap just gets the data into memory.
AlisdairO 13y ago
mmap does complicate things. A traditional database often works using write ahead logs. The log holds changes made to the database over time - so when you want to write a change to your DB, you put 'I'm changing value x.y to 50' in your WAL. Sometime later the actual data pages holding the modified x data structure can be written out to disk. If you have a crash before the data pages get written out, you can 'replay' your WAL file to redo all the lost changes. Unfortunately, a central requirement of a write ahead log is that the data pages must not get written before the WAL. If that were to occur, and there were a crash, the system wouldn't know that it had to undo the changes to the data pages, leaving you with corrupt data. mmap generally doesn't provide the ability to pin your dirty pages in memory - they're subject to getting flushed any time the system is under memory pressure - so it makes logging a lot more complicated to get right.
- arkham 13y ago
  Just to note, the MongoDB journal is basically a write ahead log and has been around since 1.8. It's not simple, you are correct, and involved remapping portions of memory privately, and leads to some inflated numbers on the virtual memory reporting side. There's a great write up here: http://blog.mongodb.org/post/33700094220/how-mongodbs-journaling-works http://blog.mongodb.org/post/33700094220/how-mongodbs-journa...
  - AlisdairO 13y ago
    That's interesting, thanks!
bashcoder 13y ago
Yes, and an important limitation stems from this as well. For a tool like Redis, it's fairly easy for me to accept the limitation that my data size can't exceed available RAM. But for an indexed document store with full-text query capabilities, it's a lot harder to for me to accept that limitation.
- thedufer 13y ago
  Are you implying that a performant Mongo DB must keep all data in RAM? That's not true. We've got almost 1TB in a Mongo DB, and we sure don't have that much RAM. You do have to be able to keep your indexes in RAM, but that's much less limiting.
  - bashcoder 13y ago
    Not 1:1, no - thanks for clarifying that. But depending on the content and the indexing, there is a strong correlation between the size of the database and the memory requirements under Mongo DB.

Kiro 13y ago

I don't know anything about MongoDB but can you give an example where DB level locking is a problem?

mikegioia 13y ago
High volume of writes
- angersock 13y ago
  Isn't that supposed to be the ideal use case for Mongo though?
  - zardosht 13y ago
    Not for single-server performance. The database level lock severely limits MongoDB's single server performance. Just look up the sysbench benchmark comparing MongoDB with TokuMX (which I work on)
- brianwawok 13y ago
  High volume of writes from MANY producers. From 1 producer, it doesn't matter. Though the point of mongo is to be webscale which implies to me many writers.
  - bsg75 13y ago
    What makes the single producer (writer?) case different from multiple, in the context of the effect on readers? If there is a single high-volume data pump, for example machine generated data, will readers be affected by a continuous "fire hose" of incoming data?
  - mikegioia 13y ago
    In my experience, even 1 producer caused a full database lock for hours on our production server. We have 1 mongo server and an erroneous scheduled task of ours started at about 6am. It's only one process but the task basically re-syncs an entire collection (which was about 80,000 writes). That single producer caused wide-scale locking/hanging for all readers on the website and I had to manually stop the task during business hours because of that. Oy!

sigzero 13y ago

"Finally, MongoDB 2.6 lays the foundation for massive improvements to concurrency in MongoDB 2.8, including document-level locking." They hear you and are working on it.

portmanteaufu 13y ago
It was about a year between the first release of 2.4 and the 2.6 release. Should we expect another full year for 2.8?
- leif 13y ago
  I believe they are on a ~yearly schedule now.

disbelief 13y ago

Every Mongo release announcement I hold out hope for an improvement to database-level write lock, and every time I'm disappointing. Considering this is probably the #1 or #2 complaint people have with Mongo, I'm surprised it hasn't been addressed ahead of some of the other features that have made it in to recent releases.

rogerbinns 13y ago

Heck I'd even be happy for them to do locking on a per file basis. (For those unfamiliar with mongodb, the actual database storage is broken up into 2GB files.)

citruspi 13y ago
I believe the 2 GB database limit is only applicable to 32-bit instances[0]. [0]: http://blog.mongodb.org/post/137788967/32-bit-limitations http://blog.mongodb.org/post/137788967/32-bit-limitations
- rogerbinns 13y ago
  You are confusing two things. I am talking about the files making up a database. MongoDB doesn't use one file per database - instead it allocates files up to 2GB in size (the first one starts at 64MB and then they double in size until the 2GB limit). So for example a database that is 200GB will consist of (roughly) 100 2GB files. What you linked to is a consequence of memory mapping. Mapping a single 2GB file in a 32 bit process will use up virtually all the address space and you couldn't map more than one at a time.