3 ms·
TokuDB is a very modern engine that uses a data-structure called "Fractal Trees". Imagine something like a B-Tree with buffers in the non-leaf nodes. It also he
by bratao 11y ago
TokuDB is a very modern engine that uses a data-structure called "Fractal Trees". Imagine something like a B-Tree with buffers in the non-leaf nodes. It also heavily compress the row data.
The main point of TokuDB is not writing everything to the disk thanks to the buffers, and with the very efficient compression, more data fits in the memory and saves the space for big-data applications. ( please someone correct me if I´m wrong)
In our tests TokuDB was vital for our startup. With it, we can use cheap dedicated servers and our performance is amazing. We tried PostgreSQL, MongoDB 3.0 and MySQL 5.7 and they can´t fit our data in a 2TB disk or were slow in our tests.
- tomcam 11y agoNot writing everything to disk ASAP makes me... queasy...
- ddorian43 11y agoThe log is always written, what's not written asap is changes on the tree (changes go into the buffers first and then trickle down when buffers are full).
- leif 11y agoThis is the critical insight. TokuDB uses a write-ahead log which is synced according to the configuration, and can be made as immediate as full fsync on commit. This provides the strongest durability available on a single machine. Where TokuDB gets its speed boost is by delaying the random reads associated with updating the indexing structure (the Fractal Tree). The buffers are written to disk on checkpoint, but because they're buffers, the potentially random writes are localized to a smaller number of nodes high in the tree, which minimizes the number of disk seeks required. Since sequential I/O is cheaper than random, the sequential writes to the write-ahead log are very fast, so even in very strict durability configurations, TokuDB can easily outperform databases which use random writes to update the indexing structures, such as the B-trees used by InnoDB and most other RDBMSes. More details here: https://www.percona.com/blog/2011/09/22/write-optimization-myths-comparison-clarifications/ https://www.percona.com/blog/2011/09/22/write-optimization-m...
- rch 11y agoIt's fine for people who use RDBMSs for data processing, like loading daily log data into tables and running one or more massive queries to yield averages, etc. etc. Not ideal, but simple and valid enough.
- nickpsecurity 11y agoIt's fine for regular loads, too, so long as they do regular saves to disk and have a high availability setup. Plenty of setups out there using in-memory, checkpointed DB's and RAM drives on reliable hardware without serious problems.
- mfisher87 11y agoI don't have a good understanding of TokuDB, but I'm not sure the buffers are necessarily in memory. The problem the buffers solve is the cost of rebuilding the index when you insert or delete data -- buffers allow you to delay the rebuild so as to batch many updates together. Even if the buffers were in memory, you'd still have the old (inconsistent) index on disk (presumably the index is marked inconsistent until the writes in the buffers are applied). You just need to rebuild it to match the new data.
- ucho 11y agoIsn't delaying index updates until more operations accumulate a standard for all DB for 20 or more years? Without it write operation would be painfully slow.
- deleted 11y ago[deleted]
- colanderman 11y agoThe GP is slightly incorrect. The buffers are on disk; the benefit comes because the buffers are laid out in transaction order, so they can be written sequentially under any workload; whereas a standard B-tree is laid out in sorted order, so can only be written sequentially under sequential workloads. The benefit is similar to that of log-structured databases.