4 ms·

This is really interesting... ... so I'm really a bit bewildered and sad by some of the engineering choices they made along the way, like requiring a modified

This is really interesting... ... so I'm really a bit bewildered and sad by some of the engineering choices they made along the way, like requiring a modified kernel: https://github.com/oscarlab/betrfs/blob/master/README.md#compiling-the-code https://github.com/oscarlab/betrfs/blob/master/README.md#com... The reasons are detailed just above that link target, but are somewhat absurd IMHO: they modified the kernel's `struct task_struct` to pass error values, rather than fixing one of their libraries to handle its own error values. In return, nobody ever gets to use this without patching a kernel (which is, by the way, so ancient of a fork at this point that Forget It). I know it's unfair of a random internetfolk to complain about the engineering choices of a project like this, I'm sure I don't know the influences and tradeoffs first hand, etc, etc, but... Ow. I would've tried this. But not on a 3.11 kernel fork. I hope someone can take all the awesome research here and get it to veer towards something slightly closer to product.

yjftsjthsd-h 8y ago
I wouldn't actually apply such an invasive patch on any system I ever cared about anyway, but I wonder if it would work on the RHEL/CentOS kernel, which has a supposed version of 3.10?
- wazoox 8y ago
  It has very little to do with a 3.10, for instance it includes XFS v5 which was introduced in kernel 3.16.
  - yjftsjthsd-h 8y ago
    Hence "supposed version"; it's my impression that they backported a non-trivial amount from as far forward as the 4.x series.
nialv7 8y ago
Feels more like an academic project to prove a point, than an actual usable file system.
- escape_goat 8y ago
  I don't think any reasonable academic wants anyone to actually use the prototype of their file system with irreplaceable (i.e. personal) data. Perhaps requiring a patched kernel is this project's way of making sure that that doesn't happen.
ploxiln 8y ago
But also, you have to build TokuDB, using gcc-4.7 specifically, and then > We import TokuDB as a binary blob, and overwrite TokuDB symbols using symbols from these files. Pretty weird ...
nneonneo 8y ago
All the reasons given boil down to “we didn’t want to mess with TokuDB”. The build process (requiring specific GCC versions, CMake, and a bunch of packages including Valgrind), the errno patch to task_struct, and the half dozen libc stub functions are all there because TokuDB expects userspace libc. I don’t know whether the authors ever attempted to patch TokuDB itself (or streamline it down to the essentials for the kernel). Instead they appear to have just taken the entire userspace-designed library and hacked the kernel until it fit. It’s a halfway-decent strategy if your goal is to get the thing off the ground as quickly as possible, but obviously a real implementation would have to ship a modified TokuDB instead (it’s OSS, so it should be hackable!) This is pretty common in academia, sadly. As an academic who has released some academic OSS code myself, I can say that often there’s just not enough time or motivation to fix a blob of code into a generally usable format. Often it’s released just so that we can say “hey we open-sourced it so other researchers can build on it/replicate our results”. This may be one of the reasons why academic ideas don’t make it out to the real world that quickly.
- ramses0 8y ago
  It makes sense, V1 is the hack (speed) and if it gets traction then you push for a V2 with different constraints and more market information (leverage).
- espeed 8y ago
  Note that some of the people involved are also the original authors of TokuDB [0], including Bradley Kuszmaul [1] , Michael Bender [2], and Martin Farach-Colton [3] who were the founders of Tokutek (acquired by Percona in 2015). Previous discussion with the authors: A Comparison of Log-Structured Merge (LSM) and Fractal Tree Indexing https://news.ycombinator.com/item?id=9248298 https://news.ycombinator.com/item?id=9248298 [0] TokuDB Fractal Tree Index https://github.com/Tokutek/ft-index https://github.com/Tokutek/ft-index TokuDB Engine https://github.com/percona/tokudb-engine https://github.com/percona/tokudb-engine [1] Bradley Kuszmaul http://people.csail.mit.edu/bradley/startups.html http://people.csail.mit.edu/bradley/startups.html [2] Michael Bender https://www3.cs.stonybrook.edu/~bender/ https://www3.cs.stonybrook.edu/~bender/ [3] Martin Farach-Colton https://www.cs.rutgers.edu/~farach/ https://www.cs.rutgers.edu/~farach/

yjftsjthsd-h 8y ago

I wouldn't actually apply such an invasive patch on any system I ever cared about anyway, but I wonder if it would work on the RHEL/CentOS kernel, which has a supposed version of 3.10?

wazoox 8y ago
It has very little to do with a 3.10, for instance it includes XFS v5 which was introduced in kernel 3.16.
- yjftsjthsd-h 8y ago
  Hence "supposed version"; it's my impression that they backported a non-trivial amount from as far forward as the 4.x series.

nialv7 8y ago

Feels more like an academic project to prove a point, than an actual usable file system.

escape_goat 8y ago
I don't think any reasonable academic wants anyone to actually use the prototype of their file system with irreplaceable (i.e. personal) data. Perhaps requiring a patched kernel is this project's way of making sure that that doesn't happen.

ploxiln 8y ago

But also, you have to build TokuDB, using gcc-4.7 specifically, and then > We import TokuDB as a binary blob, and overwrite TokuDB symbols using symbols from these files. Pretty weird ...

nneonneo 8y ago

All the reasons given boil down to “we didn’t want to mess with TokuDB”. The build process (requiring specific GCC versions, CMake, and a bunch of packages including Valgrind), the errno patch to task_struct, and the half dozen libc stub functions are all there because TokuDB expects userspace libc. I don’t know whether the authors ever attempted to patch TokuDB itself (or streamline it down to the essentials for the kernel). Instead they appear to have just taken the entire userspace-designed library and hacked the kernel until it fit. It’s a halfway-decent strategy if your goal is to get the thing off the ground as quickly as possible, but obviously a real implementation would have to ship a modified TokuDB instead (it’s OSS, so it should be hackable!) This is pretty common in academia, sadly. As an academic who has released some academic OSS code myself, I can say that often there’s just not enough time or motivation to fix a blob of code into a generally usable format. Often it’s released just so that we can say “hey we open-sourced it so other researchers can build on it/replicate our results”. This may be one of the reasons why academic ideas don’t make it out to the real world that quickly.

ramses0 8y ago
It makes sense, V1 is the hack (speed) and if it gets traction then you push for a V2 with different constraints and more market information (leverage).
espeed 8y ago
Note that some of the people involved are also the original authors of TokuDB [0], including Bradley Kuszmaul [1] , Michael Bender [2], and Martin Farach-Colton [3] who were the founders of Tokutek (acquired by Percona in 2015). Previous discussion with the authors: A Comparison of Log-Structured Merge (LSM) and Fractal Tree Indexing https://news.ycombinator.com/item?id=9248298 https://news.ycombinator.com/item?id=9248298 [0] TokuDB Fractal Tree Index https://github.com/Tokutek/ft-index https://github.com/Tokutek/ft-index TokuDB Engine https://github.com/percona/tokudb-engine https://github.com/percona/tokudb-engine [1] Bradley Kuszmaul http://people.csail.mit.edu/bradley/startups.html http://people.csail.mit.edu/bradley/startups.html [2] Michael Bender https://www3.cs.stonybrook.edu/~bender/ https://www3.cs.stonybrook.edu/~bender/ [3] Martin Farach-Colton https://www.cs.rutgers.edu/~farach/ https://www.cs.rutgers.edu/~farach/