6 ms·

Sysbench Benchmark for MongoDB – Performance Update

63 points by plasma 13y ago

plasma 13y ago
Received an email from TokoDB about this new tech, available to play now. Perf Blog: http://www.tokutek.com/2013/03/sysbench-benchmark-for-mongodb/ http://www.tokutek.com/2013/03/sysbench-benchmark-for-mongod... Download Source: https://github.com/Tokutek/mongo https://github.com/Tokutek/mongo TokuTek say to contact them for using it. TokuTek made TokuDB, a storage engine for MySQL that uses Fractal Indexes (as opposed to B-Tree), read more at http://www.tokutek.com/products/tokudb-for-mysql/ http://www.tokutek.com/products/tokudb-for-mysql/
- danielbarla 13y ago
  I'm curious about the fractal tree indexing referred to, but so far all the information I'm running across seems to be fairly one-sided. Performance is rarely so one-dimensional that I can accept statements like "When used in a database, they can be used in any setting where a B-tree is used, with improved performance" (from http://en.wikipedia.org/wiki/TokuDB http://en.wikipedia.org/wiki/TokuDB) on face value. The algorithm claims to have strong insert performance, as well as higher compressability when compared to B-trees, and all the performance results play heavily on these. What about read-heavy workloads, etc? Can someone with a bit more insight provide an objective overview?
  - leif 13y ago
    I am an engineer at Tokutek. Just talking about TokuMX, I have only seen MongoDB beat us on single-threaded, read-only workloads on extremely small data sets (like under 8K, or one MongoDB B-tree bucket). The statement is really pretty much true. The best B-tree indexes (InnoDB) beat Fractal Tree indexes a little bit on in-memory reads, but we're not done tuning our implementation. On out-of-memory reads though, we usually beat B-trees because we compress so much better that we simply need to read less off disk. Benchmarking is a tricky business though. Workloads can be varied, and you can probably find some corner cases where InnoDB beats TokuDB. I think we have some outstanding issues where the MySQL optimizer doesn't plan our queries properly, and we're still working on that. I'm a little out of touch with the MySQL side these days though so that could be wrong. But algorithmically, there are no cases where a B-tree index has a significant advantage over a Fractal Tree index. Generally though, the read performance advantage we see is that if your indexes are Fractal Tree indexes, you can afford to maintain a richer set of indexes than you could have on InnoDB or MongoDB, and these extra indexes make your queries orders of magnitude faster. I think this is the most important (non-obvious) point to understand. I gave a talk about it here: http://www.youtube.com/watch?v=q6BnG74FZMQ http://www.youtube.com/watch?v=q6BnG74FZMQ
    - StefanKarpinski 13y ago
      This data structure sounds as mythical as a unicorn. Are there any peer-reviewed publications about this data structure or is it a trade secret? The name seems pretty vacuous/hype-laden since any well-balanced tree is fractal.
      - esmet 13y ago
        "Fractal Tree" is indeed a marketing trademark. But the algorithms used are not necessarily secret. The implementation is similar to the cache oblivious streaming btree: http://supertech.csail.mit.edu/papers/sbtree.pdf‎ http://supertech.csail.mit.edu/papers/sbtree.pdf‎ Of course what's not described in the paper include: transactions, mvcc, logging/recovery, parallelism, etc. Those are really interesting topics as well.
      - leif 13y ago
        The name is a marketing term, you're right. The structure is somewhere in the realm of buffer trees, LSM-trees, $B^\epsilon$ trees, and COLAs. It's hard to give it an academic name precisely because the implementation takes hints from many places, but we hope to publish more soon. B-trees are on the optimal tradeoff curve but there are many other points on that curve, and B-trees heavily favor reads so there's plenty of room to win on the write side. I have a couple of blogs about this actually: http://www.tokutek.com/2011/09/write-optimization-myths-comparison-clarifications/ http://www.tokutek.com/2011/09/write-optimization-myths-comp... http://www.tokutek.com/2011/10/write-optimization-myths-comparison-clarifications-part-2/ http://www.tokutek.com/2011/10/write-optimization-myths-comp...
    - danielbarla 13y ago
      Thanks for the information, with your link and the ones others have posted, I've been able to get a better idea of how it works. Regarding benchmarking, my recommendation would be to present other workloads as well; if what you say is true, you have nothing to fear, and rough parity with B-trees in your worst case scenarios would only make your argument stronger (it would definitely impress me more if it was included). My point is that developers who are worried about index algorithm performance don't normally believe in free lunches, and your current benchmarks seem to be slightly cherry picked. Some of the wording is also a bit of a hard sell (and it sounds like your algorithm is good enough to stand on its own legs, so it's unnecessary). This will put off a lot of the more skeptical developers immediately.
  - mjs 13y ago
    The best explanation I've found of how it works (slides, PDF): http://cdn.oreillystatic.com/en/assets/1/event/36/How%20TokuDB%20Fractal%20Tree%20Databases%20Work%20Presentation.pdf http://cdn.oreillystatic.com/en/assets/1/event/36/How%20Toku...
- nemothekid 13y ago
  What version of MongoDB was TokuDB forked from? Could I use this as a drop in replacement for a 2.4 deployment, or mix a replica set with 2.4 and Toku?
  - zardosht 13y ago
    TokuMX was forked from 2.2.4, but we have and will continue to add features from 2.4. For instance, hash based sharding is in TokuMX. One cannot mix a replica set with TokuMX and MongoDB.
    - nasalgoat 13y ago
      Could I start a TokuMX replica to get the data across then shut off the master and switch to using it? We have a very large install and migrating the data on production systems would be very difficult otherwise.
      - leif 13y ago
        Unfortunately not, the replication format is completely different. A tool to do this would not be difficult to write, though.
programminggeek 13y ago
So, TokuMX is web scale? In all seriousness, this looks super cool, but I'm easily fooled by benchmarks and I don't have any projects right now that really push MongoDB that hard, so it doesn't impact me other than giving us a potentially faster Mongo, which is cool.
- zardosht 13y ago
  I work for Tokutek. In addition to performance improvements and compression, another cool thing about TokuMX: it's fully transactional. We hope this makes development of applications simpler.
  - programminggeek 13y ago
    That's awesome! I hope you didn't mind the joke too much. I really do use/love Mongo and I appreciate anyone who contributes to it.
  - kapilvt 13y ago
    could you clarify this statement. mongodb native is consistent and atomic wrt to single documents. Are you saying that tokumx does something additional wrt to transactions (which also implies client api changes)?
    - leif 13y ago
      TokuMX offers multi-document transactional semantics without application changes (snapshot reads), as well as protocol support for multi-statement (read-modify-write style) transactions, within a single shard. We are still designing how we want to present transactions in a sharded cluster.
  - deleted 13y ago
    [deleted]
toolslive 13y ago
is to be expected: you have a data structure where half of the inserts can be done in O(1).
philsnow 13y ago
From what I understand, what mongodb needs is not more speed, but better acid guarantees by default.
- zardosht 13y ago
  From Leif earlier, "TokuMX offers multi-document transactional semantics without application changes (snapshot reads), as well as protocol support for multi-statement (read-modify-write style) transactions, within a single shard. We are still designing how we want to present transactions in a sharded cluster."
  - philsnow 13y ago
    All built on the rock-solid webscale foundation of Mongo?
nasalgoat 13y ago
I've signed up for this beta release - even without ACID transactions on a cluster-wide basis, the other features alone are worth it, especially the compression.
malkia 13y ago
I'm working with PostgreSQL right now, and it supports various indices (I'm sticking with btree for everything I have for now). Just made me wonder, whether the folks behind TokuDB cannot pursue PGSQL variant of their solution? never mind - found some good answers here - http://postgresql.1045698.n5.nabble.com/Fractal-tree-indexing-td5744987.html http://postgresql.1045698.n5.nabble.com/Fractal-tree-indexin...

63 points by plasma 13y ago

Received an email from TokoDB about this new tech, available to play now. Perf Blog: http://www.tokutek.com/2013/03/sysbench-benchmark-for-mongodb/ http://www.tokutek.com/2013/03/sysbench-benchmark-for-mongod... Download Source: https://github.com/Tokutek/mongo https://github.com/Tokutek/mongo TokuTek say to contact them for using it. TokuTek made TokuDB, a storage engine for MySQL that uses Fractal Indexes (as opposed to B-Tree), read more at http://www.tokutek.com/products/tokudb-for-mysql/ http://www.tokutek.com/products/tokudb-for-mysql/

danielbarla 13y ago
I'm curious about the fractal tree indexing referred to, but so far all the information I'm running across seems to be fairly one-sided. Performance is rarely so one-dimensional that I can accept statements like "When used in a database, they can be used in any setting where a B-tree is used, with improved performance" (from http://en.wikipedia.org/wiki/TokuDB http://en.wikipedia.org/wiki/TokuDB) on face value. The algorithm claims to have strong insert performance, as well as higher compressability when compared to B-trees, and all the performance results play heavily on these. What about read-heavy workloads, etc? Can someone with a bit more insight provide an objective overview?
- leif 13y ago
  I am an engineer at Tokutek. Just talking about TokuMX, I have only seen MongoDB beat us on single-threaded, read-only workloads on extremely small data sets (like under 8K, or one MongoDB B-tree bucket). The statement is really pretty much true. The best B-tree indexes (InnoDB) beat Fractal Tree indexes a little bit on in-memory reads, but we're not done tuning our implementation. On out-of-memory reads though, we usually beat B-trees because we compress so much better that we simply need to read less off disk. Benchmarking is a tricky business though. Workloads can be varied, and you can probably find some corner cases where InnoDB beats TokuDB. I think we have some outstanding issues where the MySQL optimizer doesn't plan our queries properly, and we're still working on that. I'm a little out of touch with the MySQL side these days though so that could be wrong. But algorithmically, there are no cases where a B-tree index has a significant advantage over a Fractal Tree index. Generally though, the read performance advantage we see is that if your indexes are Fractal Tree indexes, you can afford to maintain a richer set of indexes than you could have on InnoDB or MongoDB, and these extra indexes make your queries orders of magnitude faster. I think this is the most important (non-obvious) point to understand. I gave a talk about it here: http://www.youtube.com/watch?v=q6BnG74FZMQ http://www.youtube.com/watch?v=q6BnG74FZMQ
  - StefanKarpinski 13y ago
    This data structure sounds as mythical as a unicorn. Are there any peer-reviewed publications about this data structure or is it a trade secret? The name seems pretty vacuous/hype-laden since any well-balanced tree is fractal.
    - esmet 13y ago
      "Fractal Tree" is indeed a marketing trademark. But the algorithms used are not necessarily secret. The implementation is similar to the cache oblivious streaming btree: http://supertech.csail.mit.edu/papers/sbtree.pdf‎ http://supertech.csail.mit.edu/papers/sbtree.pdf‎ Of course what's not described in the paper include: transactions, mvcc, logging/recovery, parallelism, etc. Those are really interesting topics as well.
    - leif 13y ago
      The name is a marketing term, you're right. The structure is somewhere in the realm of buffer trees, LSM-trees, $B^\epsilon$ trees, and COLAs. It's hard to give it an academic name precisely because the implementation takes hints from many places, but we hope to publish more soon. B-trees are on the optimal tradeoff curve but there are many other points on that curve, and B-trees heavily favor reads so there's plenty of room to win on the write side. I have a couple of blogs about this actually: http://www.tokutek.com/2011/09/write-optimization-myths-comparison-clarifications/ http://www.tokutek.com/2011/09/write-optimization-myths-comp... http://www.tokutek.com/2011/10/write-optimization-myths-comparison-clarifications-part-2/ http://www.tokutek.com/2011/10/write-optimization-myths-comp...
  - danielbarla 13y ago
    Thanks for the information, with your link and the ones others have posted, I've been able to get a better idea of how it works. Regarding benchmarking, my recommendation would be to present other workloads as well; if what you say is true, you have nothing to fear, and rough parity with B-trees in your worst case scenarios would only make your argument stronger (it would definitely impress me more if it was included). My point is that developers who are worried about index algorithm performance don't normally believe in free lunches, and your current benchmarks seem to be slightly cherry picked. Some of the wording is also a bit of a hard sell (and it sounds like your algorithm is good enough to stand on its own legs, so it's unnecessary). This will put off a lot of the more skeptical developers immediately.
- mjs 13y ago
  The best explanation I've found of how it works (slides, PDF): http://cdn.oreillystatic.com/en/assets/1/event/36/How%20TokuDB%20Fractal%20Tree%20Databases%20Work%20Presentation.pdf http://cdn.oreillystatic.com/en/assets/1/event/36/How%20Toku...
nemothekid 13y ago
What version of MongoDB was TokuDB forked from? Could I use this as a drop in replacement for a 2.4 deployment, or mix a replica set with 2.4 and Toku?
- zardosht 13y ago
  TokuMX was forked from 2.2.4, but we have and will continue to add features from 2.4. For instance, hash based sharding is in TokuMX. One cannot mix a replica set with TokuMX and MongoDB.
  - nasalgoat 13y ago
    Could I start a TokuMX replica to get the data across then shut off the master and switch to using it? We have a very large install and migrating the data on production systems would be very difficult otherwise.
    - leif 13y ago
      Unfortunately not, the replication format is completely different. A tool to do this would not be difficult to write, though.

programminggeek 13y ago

So, TokuMX is web scale? In all seriousness, this looks super cool, but I'm easily fooled by benchmarks and I don't have any projects right now that really push MongoDB that hard, so it doesn't impact me other than giving us a potentially faster Mongo, which is cool.

zardosht 13y ago
I work for Tokutek. In addition to performance improvements and compression, another cool thing about TokuMX: it's fully transactional. We hope this makes development of applications simpler.
- programminggeek 13y ago
  That's awesome! I hope you didn't mind the joke too much. I really do use/love Mongo and I appreciate anyone who contributes to it.
- kapilvt 13y ago
  could you clarify this statement. mongodb native is consistent and atomic wrt to single documents. Are you saying that tokumx does something additional wrt to transactions (which also implies client api changes)?
  - leif 13y ago
    TokuMX offers multi-document transactional semantics without application changes (snapshot reads), as well as protocol support for multi-statement (read-modify-write style) transactions, within a single shard. We are still designing how we want to present transactions in a sharded cluster.
- deleted 13y ago
  [deleted]

toolslive 13y ago

is to be expected: you have a data structure where half of the inserts can be done in O(1).

philsnow 13y ago

From what I understand, what mongodb needs is not more speed, but better acid guarantees by default.

zardosht 13y ago
From Leif earlier, "TokuMX offers multi-document transactional semantics without application changes (snapshot reads), as well as protocol support for multi-statement (read-modify-write style) transactions, within a single shard. We are still designing how we want to present transactions in a sharded cluster."
- philsnow 13y ago
  All built on the rock-solid webscale foundation of Mongo?

nasalgoat 13y ago

I've signed up for this beta release - even without ACID transactions on a cluster-wide basis, the other features alone are worth it, especially the compression.

malkia 13y ago

I'm working with PostgreSQL right now, and it supports various indices (I'm sticking with btree for everything I have for now). Just made me wonder, whether the folks behind TokuDB cannot pursue PGSQL variant of their solution? never mind - found some good answers here - http://postgresql.1045698.n5.nabble.com/Fractal-tree-indexing-td5744987.html http://postgresql.1045698.n5.nabble.com/Fractal-tree-indexin...