4 ms·

With one hand you giveth; "Fast asynchronous writes" and the other taketh away; "Concurrency is so-so: the entire database is locked during writes.".

rbranson 15y ago
It's like unsuspecting developers are walking down Developer Lane and see a candy bar sitting there that says "FAST, ASYNCHRONOUS WRITES." So they pick it up, bite into it, and then the candy bar grows a giant mutant global lock head that eats their face.
old-gregg 15y ago
Actually I don't believe the two are related. And they're working on the concurrency improvements all the time: https://jira.mongodb.org/browse/SERVER-1240 https://jira.mongodb.org/browse/SERVER-1240
- philjackson 15y ago
  Can you explain how they're not related? I'm not poking you to start a fight, I'm genuinely curious.
  - samstokes 15y ago
    Agreed, I don't understand how writes can both be fast and asynchronous, and also lock the whole database. Does it depend on what kind of write you're doing?
    - old-gregg 15y ago
      The notion of memory-mapped files (mmap on POSIX) is central to how MongoDB works. It doesn't really manage its own writes or buffers. They just update data structures in memory and let the kernel figure out how to map that to disk. However, IIRC they're forcing it to write dirty pages every 10 seconds by default. Thus I don't see how's that related to locking. My understanding is that the reason they still use a global lock are: a) They're young and haven't gotten around implementing this yet. b) Since writes are fast, this hasn't been as big of an issue as people might expect (it was for us, though) And everything works as expected as long as you're fine with your writes being asynchronous. But once you call getLastError() requesting your writes to go through after every insert, then you start seeing the global lock bite.
      - rbranson 15y ago
        The global lock is something that will always exist with their architecture. It isn't a feature to implement, it's a total redesign and rethinking of the fundamentals. Their much-vaunted in-place, memory-mapped I/O architecture becomes useless because concurrency means using lock-free data structures. MyISAM in MySQL locks the entire table because it does in-place updates. Row locks can't just be added into MyISAM. It'd be such a redesign to add it, it'd end up being something much more like InnoDB than MyISAM. 10gen would have to eat an awful lot of their own words if they decided to go that route.
  - latch 15y ago
    My current wait % on the global lock is 0.02%. Deletes and updates will yield. Inserts won't, but the insert happen to memory, which is fast. I think disk saturation on long sync delays is really the only major pain point now (and even that seems better than it used to be)
    - rbranson 15y ago
      The problem is: "a pending write lock acquisition will prevent further read lock acquisitions until fulfilled." Inserts aren't actually guaranteed to be in memory, as they're done in a file-backed memory region. A write and page fault can and frequently does cause a flush and hang while that lock is held, particularly as the hot dataset size increases relative to physical RAM.

rbranson 15y ago

It's like unsuspecting developers are walking down Developer Lane and see a candy bar sitting there that says "FAST, ASYNCHRONOUS WRITES." So they pick it up, bite into it, and then the candy bar grows a giant mutant global lock head that eats their face.

old-gregg 15y ago

Actually I don't believe the two are related. And they're working on the concurrency improvements all the time: https://jira.mongodb.org/browse/SERVER-1240 https://jira.mongodb.org/browse/SERVER-1240

philjackson 15y ago
Can you explain how they're not related? I'm not poking you to start a fight, I'm genuinely curious.
- samstokes 15y ago
  Agreed, I don't understand how writes can both be fast and asynchronous, and also lock the whole database. Does it depend on what kind of write you're doing?
  - old-gregg 15y ago
    The notion of memory-mapped files (mmap on POSIX) is central to how MongoDB works. It doesn't really manage its own writes or buffers. They just update data structures in memory and let the kernel figure out how to map that to disk. However, IIRC they're forcing it to write dirty pages every 10 seconds by default. Thus I don't see how's that related to locking. My understanding is that the reason they still use a global lock are: a) They're young and haven't gotten around implementing this yet. b) Since writes are fast, this hasn't been as big of an issue as people might expect (it was for us, though) And everything works as expected as long as you're fine with your writes being asynchronous. But once you call getLastError() requesting your writes to go through after every insert, then you start seeing the global lock bite.
    - rbranson 15y ago
      The global lock is something that will always exist with their architecture. It isn't a feature to implement, it's a total redesign and rethinking of the fundamentals. Their much-vaunted in-place, memory-mapped I/O architecture becomes useless because concurrency means using lock-free data structures. MyISAM in MySQL locks the entire table because it does in-place updates. Row locks can't just be added into MyISAM. It'd be such a redesign to add it, it'd end up being something much more like InnoDB than MyISAM. 10gen would have to eat an awful lot of their own words if they decided to go that route.
- latch 15y ago
  My current wait % on the global lock is 0.02%. Deletes and updates will yield. Inserts won't, but the insert happen to memory, which is fast. I think disk saturation on long sync delays is really the only major pain point now (and even that seems better than it used to be)
  - rbranson 15y ago
    The problem is: "a pending write lock acquisition will prevent further read lock acquisitions until fulfilled." Inserts aren't actually guaranteed to be in memory, as they're done in a file-backed memory region. A write and page fault can and frequently does cause a flush and hang while that lock is held, particularly as the hot dataset size increases relative to physical RAM.