5 ms·

When you have so many writes that sharding isn't enough. When you make changes so fast you need a liquid schema. When you want to make your boss learn map-red

by elchief 14y ago

When you have so many writes that sharding isn't enough. When you make changes so fast you need a liquid schema. When you want to make your boss learn map-reduce so he can query the data. When the application can take care of integration and not the database itself.

lwat 14y ago
> When you have so many writes that sharding isn't enough. Does Mongo still have a global write lock?
- elchief 14y ago
  no longer (but only a few months)
  - mgummelt 14y ago
    Though I believe it still has a per-collection lock.
    - Jare 14y ago
      It has per-database locking. Per-collection is being worked on but no planned date yet. [1] [1] http://www.mongodb.org/display/DOCS/How+does+concurrency+work#Howdoesconcurrencywork-mongod http://www.mongodb.org/display/DOCS/How+does+concurrency+wor...
leif 14y ago
This will be obvious, but I work at Tokutek. > When you have so many writes that sharding isn't enough. TokuDB does indexed insertions very fast. [1] We've even plugged ourselves in underneath MongoDB (just for fun) and we beat them too. [2] > When you make changes so fast you need a liquid schema. TokuDB supports lots of schema changes with zero downtime. [3] > When you want to make your boss learn map-reduce so he can query the data. Can't help you there, but I can make you not need to torture your boss that way. > When the application can care of integration and not the database. What if it didn't need to? We also get fantastic compression [4], retain full transactional semantics, and lots of other fun stuff. Email us if you're curious! [1]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#iiBench http://www.tokutek.com/resources/benchmark-results/benchmark... [2]: http://www.tokutek.com/2012/08/10x-insertion-performance-increase-for-mongodb-with-fractal-tree-indexes/ http://www.tokutek.com/2012/08/10x-insertion-performance-inc... [3]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#HotSchema http://www.tokutek.com/resources/benchmark-results/benchmark... [4]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#Compression http://www.tokutek.com/resources/benchmark-results/benchmark...
- DanWaterworth 14y ago
  In my experience, it's not just throughput that's important, but also 99th percentile latency. If I understand fractal trees correctly, you sometimes need to rewrite all of your elements on disk. How do you do this without causing lag?
  - leif 14y ago
    That's happily not how fractal trees work, at all. We have a few talks online describing how they work. Zardosht has one here http://vimeo.com/m/26471692 http://vimeo.com/m/26471692 . I thought Bradley had a more detailed one at MIT but I can't find it right now. (EDIT: found it! http://video.mit.edu/watch/lecture-19-how-tokudb-fractal-tree-indexes-work-1361/ http://video.mit.edu/watch/lecture-19-how-tokudb-fractal-tre...) Basically, I think you're thinking of the COLA. What we implement does have a literal tree structure, with nodes and children and the whole thing, so at any point you're just writing out new copies of individual nodes, which are on the order of a megabyte. At no point do we have to rewrite a large portion of the tree, so there aren't any latency issues.
    - DanWaterworth 14y ago
      Thank you very much for the links. Is my understanding correct? Essentially, a fractal tree is a B-Tree (or perhaps a B+Tree?) with buffers on each branch (per child). Operations get added to these buffers and when one becomes full, the operations get passed to the corresponding child node. Operations are applied when they reach the node that is responsible for the data concerned.
      - leif 14y ago
        That's about it! It's like a B+Tree in that the data resides in the leaves (well, and in the buffers), except that the fanout is a lot lower because we save room in the internal nodes for the buffers.
        DanWaterworth 14y ago

by elchief 14y ago

lwat 14y ago

> When you have so many writes that sharding isn't enough. Does Mongo still have a global write lock?

elchief 14y ago
no longer (but only a few months)
- mgummelt 14y ago
  Though I believe it still has a per-collection lock.
  - Jare 14y ago
    It has per-database locking. Per-collection is being worked on but no planned date yet. [1] [1] http://www.mongodb.org/display/DOCS/How+does+concurrency+work#Howdoesconcurrencywork-mongod http://www.mongodb.org/display/DOCS/How+does+concurrency+wor...

leif 14y ago

This will be obvious, but I work at Tokutek. > When you have so many writes that sharding isn't enough. TokuDB does indexed insertions very fast. [1] We've even plugged ourselves in underneath MongoDB (just for fun) and we beat them too. [2] > When you make changes so fast you need a liquid schema. TokuDB supports lots of schema changes with zero downtime. [3] > When you want to make your boss learn map-reduce so he can query the data. Can't help you there, but I can make you not need to torture your boss that way. > When the application can care of integration and not the database. What if it didn't need to? We also get fantastic compression [4], retain full transactional semantics, and lots of other fun stuff. Email us if you're curious! [1]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#iiBench http://www.tokutek.com/resources/benchmark-results/benchmark... [2]: http://www.tokutek.com/2012/08/10x-insertion-performance-increase-for-mongodb-with-fractal-tree-indexes/ http://www.tokutek.com/2012/08/10x-insertion-performance-inc... [3]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#HotSchema http://www.tokutek.com/resources/benchmark-results/benchmark... [4]: http://www.tokutek.com/resources/benchmark-results/benchmarks-vs-innodb-hdds/#Compression http://www.tokutek.com/resources/benchmark-results/benchmark...

DanWaterworth 14y ago
In my experience, it's not just throughput that's important, but also 99th percentile latency. If I understand fractal trees correctly, you sometimes need to rewrite all of your elements on disk. How do you do this without causing lag?
- leif 14y ago
  That's happily not how fractal trees work, at all. We have a few talks online describing how they work. Zardosht has one here http://vimeo.com/m/26471692 http://vimeo.com/m/26471692 . I thought Bradley had a more detailed one at MIT but I can't find it right now. (EDIT: found it! http://video.mit.edu/watch/lecture-19-how-tokudb-fractal-tree-indexes-work-1361/ http://video.mit.edu/watch/lecture-19-how-tokudb-fractal-tre...) Basically, I think you're thinking of the COLA. What we implement does have a literal tree structure, with nodes and children and the whole thing, so at any point you're just writing out new copies of individual nodes, which are on the order of a megabyte. At no point do we have to rewrite a large portion of the tree, so there aren't any latency issues.
  - DanWaterworth 14y ago
    Thank you very much for the links. Is my understanding correct? Essentially, a fractal tree is a B-Tree (or perhaps a B+Tree?) with buffers on each branch (per child). Operations get added to these buffers and when one becomes full, the operations get passed to the corresponding child node. Operations are applied when they reach the node that is responsible for the data concerned.
    - leif 14y ago
      That's about it! It's like a B+Tree in that the data resides in the leaves (well, and in the buffers), except that the fanout is a lot lower because we save room in the internal nodes for the buffers.
      - DanWaterworth 14y ago