4 ms·

All the reasons given boil down to “we didn’t want to mess with TokuDB”. The build process (requiring specific GCC versions, CMake, and a bunch of packages incl

All the reasons given boil down to “we didn’t want to mess with TokuDB”. The build process (requiring specific GCC versions, CMake, and a bunch of packages including Valgrind), the errno patch to task_struct, and the half dozen libc stub functions are all there because TokuDB expects userspace libc. I don’t know whether the authors ever attempted to patch TokuDB itself (or streamline it down to the essentials for the kernel). Instead they appear to have just taken the entire userspace-designed library and hacked the kernel until it fit. It’s a halfway-decent strategy if your goal is to get the thing off the ground as quickly as possible, but obviously a real implementation would have to ship a modified TokuDB instead (it’s OSS, so it should be hackable!) This is pretty common in academia, sadly. As an academic who has released some academic OSS code myself, I can say that often there’s just not enough time or motivation to fix a blob of code into a generally usable format. Often it’s released just so that we can say “hey we open-sourced it so other researchers can build on it/replicate our results”. This may be one of the reasons why academic ideas don’t make it out to the real world that quickly.

ramses0 8y ago
It makes sense, V1 is the hack (speed) and if it gets traction then you push for a V2 with different constraints and more market information (leverage).
espeed 8y ago
Note that some of the people involved are also the original authors of TokuDB [0], including Bradley Kuszmaul [1] , Michael Bender [2], and Martin Farach-Colton [3] who were the founders of Tokutek (acquired by Percona in 2015). Previous discussion with the authors: A Comparison of Log-Structured Merge (LSM) and Fractal Tree Indexing https://news.ycombinator.com/item?id=9248298 https://news.ycombinator.com/item?id=9248298 [0] TokuDB Fractal Tree Index https://github.com/Tokutek/ft-index https://github.com/Tokutek/ft-index TokuDB Engine https://github.com/percona/tokudb-engine https://github.com/percona/tokudb-engine [1] Bradley Kuszmaul http://people.csail.mit.edu/bradley/startups.html http://people.csail.mit.edu/bradley/startups.html [2] Michael Bender https://www3.cs.stonybrook.edu/~bender/ https://www3.cs.stonybrook.edu/~bender/ [3] Martin Farach-Colton https://www.cs.rutgers.edu/~farach/ https://www.cs.rutgers.edu/~farach/
tobias3 8y ago
TokuDB seems to be written in C++. It's neat that they managed to make that work in-kernel. Also there are patent notices in the TokuDB files. So probability of it being in Linux at some point approaches zero. And with the patents they even made the whole technique unviable for any Linux fs (haven't looked at them in detail obviously).
- espeed 8y ago
  See Bradley's comment wrt to licensing the fractal tree code as GPLv2 w/ a patent provision https://news.ycombinator.com/item?id=18208209 https://news.ycombinator.com/item?id=18208209
adrianmonk 8y ago
Perhaps TokuDB was evolving rapidly at the time they were trying to incorporate it into this filesystem. It seems like the choices are: 1. Fork TokuDB. 2. Fork the kernel. If the goal is research, the first one of those would then slow down the research. The second one only gets you stuck on an old kernel, which while not great for end users, isn't as big of a problem for a researcher as having their researcher slowed down.
loeg 8y ago
This isn't completely fair, although most of it is. Some of the reasons for modifying Linux are performance enhancements to the filesystem layer that only make sense with a Bε-tree filesystem (and are not possible without patching Linux). They cover this in the hour-long talk at MS research linked elsewhere in this thread.[1] (Yes, it's long, but it's a pretty good presentation.) E.g., they describe modifying the page cache to write-through small modifications to file data, rather than dirtying the entire page and writing it back later (a form of write amplification). [1]: https://www.youtube.com/watch?v=fBt5NuNsoII https://www.youtube.com/watch?v=fBt5NuNsoII
- wtallis 8y ago
  > E.g., they describe modifying the page cache to write-through small modifications to file data, rather than dirtying the entire page and writing it back later (a form of write amplification). Considering that all the storage on the market now has sectors at least as large as a 4k page, this isn't actually reducing write amplification. At most, in some cases it might save a tiny bit of bus traffic.
  - loeg 8y ago
    They batch small edits into a log. Many edits to one sector written.

ramses0 8y ago

It makes sense, V1 is the hack (speed) and if it gets traction then you push for a V2 with different constraints and more market information (leverage).

espeed 8y ago

Note that some of the people involved are also the original authors of TokuDB [0], including Bradley Kuszmaul [1] , Michael Bender [2], and Martin Farach-Colton [3] who were the founders of Tokutek (acquired by Percona in 2015). Previous discussion with the authors: A Comparison of Log-Structured Merge (LSM) and Fractal Tree Indexing https://news.ycombinator.com/item?id=9248298 https://news.ycombinator.com/item?id=9248298 [0] TokuDB Fractal Tree Index https://github.com/Tokutek/ft-index https://github.com/Tokutek/ft-index TokuDB Engine https://github.com/percona/tokudb-engine https://github.com/percona/tokudb-engine [1] Bradley Kuszmaul http://people.csail.mit.edu/bradley/startups.html http://people.csail.mit.edu/bradley/startups.html [2] Michael Bender https://www3.cs.stonybrook.edu/~bender/ https://www3.cs.stonybrook.edu/~bender/ [3] Martin Farach-Colton https://www.cs.rutgers.edu/~farach/ https://www.cs.rutgers.edu/~farach/

tobias3 8y ago

TokuDB seems to be written in C++. It's neat that they managed to make that work in-kernel. Also there are patent notices in the TokuDB files. So probability of it being in Linux at some point approaches zero. And with the patents they even made the whole technique unviable for any Linux fs (haven't looked at them in detail obviously).

espeed 8y ago
See Bradley's comment wrt to licensing the fractal tree code as GPLv2 w/ a patent provision https://news.ycombinator.com/item?id=18208209 https://news.ycombinator.com/item?id=18208209

adrianmonk 8y ago

Perhaps TokuDB was evolving rapidly at the time they were trying to incorporate it into this filesystem. It seems like the choices are: 1. Fork TokuDB. 2. Fork the kernel. If the goal is research, the first one of those would then slow down the research. The second one only gets you stuck on an old kernel, which while not great for end users, isn't as big of a problem for a researcher as having their researcher slowed down.

loeg 8y ago

This isn't completely fair, although most of it is. Some of the reasons for modifying Linux are performance enhancements to the filesystem layer that only make sense with a Bε-tree filesystem (and are not possible without patching Linux). They cover this in the hour-long talk at MS research linked elsewhere in this thread.[1] (Yes, it's long, but it's a pretty good presentation.) E.g., they describe modifying the page cache to write-through small modifications to file data, rather than dirtying the entire page and writing it back later (a form of write amplification). [1]: https://www.youtube.com/watch?v=fBt5NuNsoII https://www.youtube.com/watch?v=fBt5NuNsoII

wtallis 8y ago
> E.g., they describe modifying the page cache to write-through small modifications to file data, rather than dirtying the entire page and writing it back later (a form of write amplification). Considering that all the storage on the market now has sectors at least as large as a 4k page, this isn't actually reducing write amplification. At most, in some cases it might save a tiny bit of bus traffic.
- loeg 8y ago
  They batch small edits into a log. Many edits to one sector written.