9 ms·

Libsqlfs: A POSIX-style file system on top of an SQLite database

100 points by networked 12y ago

Alupis 12y ago
Interesting project. Out of curiosity, when/why would someone want to use something like this? SQLite is a file database, in that the database is literally a file, which means it will reside on another already existing filesystem - so you would have: ` Abstract Filesystem ------------------- SQLiteDB ------------------- OS Filesystem
- liveoneggs 12y ago
  It could be very convenient as an alternative to tarfs where you want to package up an entire X into a mount point to distribute.
- fiatmoney 12y ago
  It looks like they wanted to implement a encrypted userspace filesystem, presumably in an environment without something like EncFS.
  - craigching 12y ago
    This was exactly how I thought it might be interesting too.
    - eighthave 12y ago
      Yup, exactly, it is used in IOCipher to provide encrypted virtual disks that don't need root access or special permissions to mount.
- shawn-butler 12y ago
  A portable, encrypted, transactional "abstract" filesystem that works with existing POSIX semantics and code. Would be useful where you have legacy codebase and want to deploy it new scenarios where POSIX filesystem access is not guaranteed.
- koenigdavidmj 12y ago
  DMG (a disk image that can be mounted) is the main way to distribute OS X applications, and something this could have fulfilled the same use case if Linux wasn't as fond of centrally distributed packages.
  - icebraining 12y ago
    It's not like Linux lacks disk image formats, though, like cloop (used by Knoppix). It's just that when you don't have to distribute weird metadata like resource forks, what's the point of using a whole filesystem versus a tar file?
    - mikeash 12y ago
      Even when you do have weird metadata, it's easy enough to retrofit archive formats to handle it, which Mac OS X did years ago. Disk image distribution is a complete anachronism now, and it's odd how persistent it is.
      - lvillani 12y ago
        DMGs still have the advantage that you can customise the Finder windows that opens when you double click one [1]. So, from an UX standpoint, they still make a lot of sense to distribute application bundles since it compels users to "install" the application by dragging the bundle icon to the appropriate directory, which is conveniently symlinked in that Finder window. It's also easier to produce than a self-installing package. With a .zip or .tar.gz users would be left with an app bundle and no idea of where to put it. Generally they would just throw it somewhere random across the filesystem. At least, that's what non-technical Mac users I know happen to do. [1]: https://support.cdn.mozilla.net/media/uploads/gallery/images/2012-01-26-17-04-06-2b7732.png https://support.cdn.mozilla.net/media/uploads/gallery/images...
        mikeash 12y ago
        I think the proper solution to that, if there's actually a problem with "somewhere random", is to have apps that move themselves on first launch. It's pretty easy to do, and IMO easier than creating a consistent build process for a good-looking dmg.
- bane 12y ago
  I could see this being a ridiculously easy way to distribute portable applications.
linuxhansl 12y ago
Why not use a local loop mount? You'd still have a filesystem in a single file. At least on Linux.
- akira2501 12y ago
  Storage overhead. The loopback device is still a block device, if you skip this step, you don't have to waste the block overhead when storing a file that's smaller than the block size. If you're going to store a large number of small-sized files, this fuse <-> sql bridge could be a win.
  - shawnz 12y ago
    But this has block overhead too. Why not just use a smaller block size?
    - Dylan16807 12y ago
      It uses blocks but records get packed on disk. (I think; it's certainly capable of it.) More obvious is that the database as a whole is dynamically sized.
  - justincormack 12y ago
    Some filesystems can compress in this case.
- krylon 12y ago
  I do not know if the library takes advantage of it, but using an ACID-compliant database as the backend, one could make use of transactions at the filesystem level.
  - rakoo 12y ago
    Something that's lacking in the Linux world is an efficient way to determine "what file has changed since that time". Using SQLite db could be really helpful in this situation.
edlebert 12y ago
Time to put a SQLite database on this new filesystem!
- izietto 12y ago
  lol they created a recursive monster *.+
  - chris_wot 12y ago
    It would be more fun to see Oracle run on it.
pwr22 12y ago
Any performance benchmarks?
peterwwillis 12y ago
There's a universe of archive formats that do this without the need for a structured query language or FUSE. But i'm sure somebody has a need for this...
- Dylan16807 12y ago
  Archive formats are for archiving. They barely support writes at all, let alone atomic random writes.
malkia 12y ago
Must port this to windows - there is DOKAN and CbFS (commercial) that are basically FUSE for Windows...
- VikingCoder 12y ago
  The reports I see is that DOKAN is more trouble than it's worth. And the problem I have with CbFS is it's fairly expensive for a hobbyist like me to consider developing a file system on top of it. :-/ I wish there were a good, free FUSE for Windows.
  - 7952 12y ago
    Funny, I have just spent most of the morning trying to find a user mode file system for windows. The best I can come up with is mounting WebDav or using a .net SMB implementation. I wonder if it would be better to just build a software iSCSI target?
    - malkia 12y ago
      Try Dokan, you might like it. It's not perfect, but free and might just work for you.
narrator 12y ago
A good use case for this might be a filesystem with a large number of files where one could quickly find recently changed files without having to sequential scan the whole system. Presumably the access and modified time would be stored as separate indexed attributes.
- e12e 12y ago
  I was going to reply with something along the lines of "modern filesystems should already have meta-data indexes".... because I thought I'd read about that in connection to ext4, btrfs and/or zfs. Now I'm no longer certain ... all I came up with was the following, which might be of interest: BabuDB (related to extreemfs.org): http://dl.acm.org/citation.cfm?id=1849822 http://dl.acm.org/citation.cfm?id=1849822 Spyglass: http://www.ssrc.ucsc.edu/pub/leung09-fast.html http://www.ssrc.ucsc.edu/pub/leung09-fast.html And more from the same researchers: "Scalable File System Indexing": http://www.ssrc.ucsc.edu/proj/fsindexing.html http://www.ssrc.ucsc.edu/proj/fsindexing.html Perhaps I'm thinking of reiser4? See also: TokuFS: https://www.usenix.org/conference/hotstorage12/workshop-program/presentation/esmet https://www.usenix.org/conference/hotstorage12/workshop-prog... https://github.com/esmet/tokufs https://github.com/esmet/tokufs I'm still certain I've seen talk of indexing metadata, and I thought it was in an actual, open, system...
  - 286c8cb04bda 12y ago
    > I'm still certain I've seen talk of indexing metadata... This reminded me of BFS: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System
    - e12e 12y ago
      That could be it... thanks. I went as far afield as Hammer (DragonflyBSD) -- but didn't think to look at BeFS...
  - XorNot 12y ago
    Metadata indexing isn't really a filesystem-level problem though, depending what you're trying to do. An index can be anywhere, and the data you want to keep is somewhat arbitrary - so why not just use a file on the filesystem anyway? There's also an important difference: metadata indices can be considered disposable in a lot of cases. File data isn't which means the constraints are different: with metadata you want to pack it all into the tiniest, most local part of the disk you can. With file-data (and the actual filesystem) you want to distribute and replicate that data as widely as possible to minimize the chances that a cluster of failures wipes out important structures.
fragmede 12y ago
From the README: Copyright 2006, Palmsource, Inc., an ACCESS company. I wonder if this has anything to do with when PalmOS got a 'filesystem' even though the OS originally only gave programs a database to interface with, way back when...
- guessbest 12y ago
  Good catch. It might have something to do with the Palm Lifedrive[1] and its 4gb limit or the NVFS[2] of the Palm 650 and other models. Who knows? It seems like it could be used by a software services company to make a thin client with an encrypted operating system that could detect tampering or cloning. I'm guessing that what the Guardian Project[3] does and why it gets mentioned along with SQLCipher. [1] http://how-to.wikia.com/wiki/Howto_replace_microdrive_with_compactflash_in_LifeDrive http://how-to.wikia.com/wiki/Howto_replace_microdrive_with_c... [2] http://en.wikipedia.org/wiki/Non-Volatile_File_System http://en.wikipedia.org/wiki/Non-Volatile_File_System [3] https://guardianproject.info/ https://guardianproject.info/
gioele 12y ago
Presentation by Hans Reiser on Reiser4. Reiser just explained the new Linux VFS layer introduced by Reiser4 and the ability to see each file a small directory or a set of records. Audience: Then what is the difference between a DB and a filesystem? Reiser: Marketing.
- agumonkey 12y ago
  I too would like to remove unnecessary tersm and concepts. But it seems that splitting into FS / DB / Apps gives people a way to optimize their own way. At the cost of communication/reintegration. ps: I always found the IBM/COBOL record oriented FS a good idea. You remove a lot of ad-hoc parsing code from loading and writing data.
- agumonkey 12y ago
  I too would like to remove unnecessary tersm and concepts. But it seems that splitting into FS / DB / Apps gives people a way to optimize their own way. At the cost of communication/reintegration.
jrapdx3 12y ago
Though not exactly the same thing, Tcl can use its VFS extension to provide a "file system" for an application backed by a database or ZIP archive. The VFS allows bundling up an app as a self-contained executable (a "starpak") or runnable with the Tcl/Tk interpreter (aka "tclkit"). FS access is transparent to the app--read/write operations are the same for all FS types. Some work has been done to use sqlite as a VFS data storage medium. I haven't yet tried it myself, but in principle, it's not too hard to accomplish. I'm putting that project on my list...
fit2rule 12y ago
In the "why use this?" department, I think the answer to the question is that you want to distribute massive amounts of content in a single-file (making content updates easy) while also maintaining a POSIX-level control over the content in the container.. so if you've got a massive SQL database full of content - say a dictionary or wikipedia, or whatever - doing a content transform that spits out a single .db to your paid/registered/updating customers is quite useful..
fsiefken 12y ago
reminds me of the abandoned WinFS project (Windows Future Storage). http://en.wikipedia.org/wiki/WinFS http://en.wikipedia.org/wiki/WinFS
bnolsen 12y ago
I would really like it if they did lgpl with a static link exception. dynamic can be a pain to deal with cross platform.
- eighthave 12y ago
  You'll have to talk to PalmSource about that, Guardian Project inherited the LGPL license from them.
Animats 12y ago
Tandem Computers actually did implement files on top of a database. Their OS didn't have files. Their system was a replicated database running directly on the hard disk, with no file system. There's something to be said for this. Your files get ACID properties. If we were serious about file integrity, we'd have file systems that worked like this: - Unit files. The unit of data is the entire file. Files are written once, and when closed successfully, the file transaction commits and others can read the file. Any update replaces the entire file as an atomic operation. (Many applications need this, and try to do it with various move and rename operations, usually leaving files behind if things fail at the wrong moment.) - Log files. You can only add at the end. Writes are atomic. In the event of a crash, the file is valid up to some recently completed write. (On many systems, log files can tail off into junk or contain truncated records.) - Temporary files. When the process or process group exits, they're gone. Random access is OK. (You shouldn't have to clean up junk temporary files.) - Managed files. These support a database or something with complex structure. There are extra I/O functions for locking, flushing and being sure a write has been committed to disk. That covers most of the use cases for files. There have been file systems which did some of this, but not in recent years.
- Alupis 12y ago
  OS/400 systems (by IBM, also called AS/400, System i, iSeries, iOS, etc)are a database operating system, there are no files, only "libraries". The entire thing sits on top of a customized DB2 database.
hartzell 12y ago
Reminds me of Mike Olson's [Inversion Filesystem](http://db.cs.berkeley.edu/papers/S2K-93-28.pdf http://db.cs.berkeley.edu/papers/S2K-93-28.pdf) from the early 90's.
jtseng 12y ago
Many people say that DBs are not good for large blobs, hence the advice to put large files such as images in the files system, not in the database. What's different here that makes it acceptable to put files in the database?
- eighthave 12y ago
  libsqlfs is a filesystem, so it stores blocks of data that are indexed. A file is then stored as references to the relevant blocks. SQL databases work fine with 8k blobs, like we use for libsqlfs in IOCipher.

100 points by networked 12y ago

Interesting project. Out of curiosity, when/why would someone want to use something like this? SQLite is a file database, in that the database is literally a file, which means it will reside on another already existing filesystem - so you would have: ` Abstract Filesystem ------------------- SQLiteDB ------------------- OS Filesystem

liveoneggs 12y ago
It could be very convenient as an alternative to tarfs where you want to package up an entire X into a mount point to distribute.
fiatmoney 12y ago
It looks like they wanted to implement a encrypted userspace filesystem, presumably in an environment without something like EncFS.
- craigching 12y ago
  This was exactly how I thought it might be interesting too.
  - eighthave 12y ago
    Yup, exactly, it is used in IOCipher to provide encrypted virtual disks that don't need root access or special permissions to mount.
shawn-butler 12y ago
A portable, encrypted, transactional "abstract" filesystem that works with existing POSIX semantics and code. Would be useful where you have legacy codebase and want to deploy it new scenarios where POSIX filesystem access is not guaranteed.
koenigdavidmj 12y ago
DMG (a disk image that can be mounted) is the main way to distribute OS X applications, and something this could have fulfilled the same use case if Linux wasn't as fond of centrally distributed packages.
- icebraining 12y ago
  It's not like Linux lacks disk image formats, though, like cloop (used by Knoppix). It's just that when you don't have to distribute weird metadata like resource forks, what's the point of using a whole filesystem versus a tar file?
  - mikeash 12y ago
    Even when you do have weird metadata, it's easy enough to retrofit archive formats to handle it, which Mac OS X did years ago. Disk image distribution is a complete anachronism now, and it's odd how persistent it is.
    - lvillani 12y ago
      DMGs still have the advantage that you can customise the Finder windows that opens when you double click one [1]. So, from an UX standpoint, they still make a lot of sense to distribute application bundles since it compels users to "install" the application by dragging the bundle icon to the appropriate directory, which is conveniently symlinked in that Finder window. It's also easier to produce than a self-installing package. With a .zip or .tar.gz users would be left with an app bundle and no idea of where to put it. Generally they would just throw it somewhere random across the filesystem. At least, that's what non-technical Mac users I know happen to do. [1]: https://support.cdn.mozilla.net/media/uploads/gallery/images/2012-01-26-17-04-06-2b7732.png https://support.cdn.mozilla.net/media/uploads/gallery/images...
      - mikeash 12y ago
        I think the proper solution to that, if there's actually a problem with "somewhere random", is to have apps that move themselves on first launch. It's pretty easy to do, and IMO easier than creating a consistent build process for a good-looking dmg.
bane 12y ago
I could see this being a ridiculously easy way to distribute portable applications.

linuxhansl 12y ago

Why not use a local loop mount? You'd still have a filesystem in a single file. At least on Linux.

akira2501 12y ago
Storage overhead. The loopback device is still a block device, if you skip this step, you don't have to waste the block overhead when storing a file that's smaller than the block size. If you're going to store a large number of small-sized files, this fuse <-> sql bridge could be a win.
- shawnz 12y ago
  But this has block overhead too. Why not just use a smaller block size?
  - Dylan16807 12y ago
    It uses blocks but records get packed on disk. (I think; it's certainly capable of it.) More obvious is that the database as a whole is dynamically sized.
- justincormack 12y ago
  Some filesystems can compress in this case.
krylon 12y ago
I do not know if the library takes advantage of it, but using an ACID-compliant database as the backend, one could make use of transactions at the filesystem level.
- rakoo 12y ago
  Something that's lacking in the Linux world is an efficient way to determine "what file has changed since that time". Using SQLite db could be really helpful in this situation.

edlebert 12y ago

Time to put a SQLite database on this new filesystem!

izietto 12y ago
lol they created a recursive monster *.+
- chris_wot 12y ago
  It would be more fun to see Oracle run on it.

pwr22 12y ago

Any performance benchmarks?

peterwwillis 12y ago

There's a universe of archive formats that do this without the need for a structured query language or FUSE. But i'm sure somebody has a need for this...

Dylan16807 12y ago
Archive formats are for archiving. They barely support writes at all, let alone atomic random writes.

malkia 12y ago

Must port this to windows - there is DOKAN and CbFS (commercial) that are basically FUSE for Windows...

VikingCoder 12y ago
The reports I see is that DOKAN is more trouble than it's worth. And the problem I have with CbFS is it's fairly expensive for a hobbyist like me to consider developing a file system on top of it. :-/ I wish there were a good, free FUSE for Windows.
- 7952 12y ago
  Funny, I have just spent most of the morning trying to find a user mode file system for windows. The best I can come up with is mounting WebDav or using a .net SMB implementation. I wonder if it would be better to just build a software iSCSI target?
  - malkia 12y ago
    Try Dokan, you might like it. It's not perfect, but free and might just work for you.

narrator 12y ago

A good use case for this might be a filesystem with a large number of files where one could quickly find recently changed files without having to sequential scan the whole system. Presumably the access and modified time would be stored as separate indexed attributes.

e12e 12y ago
I was going to reply with something along the lines of "modern filesystems should already have meta-data indexes".... because I thought I'd read about that in connection to ext4, btrfs and/or zfs. Now I'm no longer certain ... all I came up with was the following, which might be of interest: BabuDB (related to extreemfs.org): http://dl.acm.org/citation.cfm?id=1849822 http://dl.acm.org/citation.cfm?id=1849822 Spyglass: http://www.ssrc.ucsc.edu/pub/leung09-fast.html http://www.ssrc.ucsc.edu/pub/leung09-fast.html And more from the same researchers: "Scalable File System Indexing": http://www.ssrc.ucsc.edu/proj/fsindexing.html http://www.ssrc.ucsc.edu/proj/fsindexing.html Perhaps I'm thinking of reiser4? See also: TokuFS: https://www.usenix.org/conference/hotstorage12/workshop-program/presentation/esmet https://www.usenix.org/conference/hotstorage12/workshop-prog... https://github.com/esmet/tokufs https://github.com/esmet/tokufs I'm still certain I've seen talk of indexing metadata, and I thought it was in an actual, open, system...
- 286c8cb04bda 12y ago
  > I'm still certain I've seen talk of indexing metadata... This reminded me of BFS: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System
  - e12e 12y ago
    That could be it... thanks. I went as far afield as Hammer (DragonflyBSD) -- but didn't think to look at BeFS...
- XorNot 12y ago
  Metadata indexing isn't really a filesystem-level problem though, depending what you're trying to do. An index can be anywhere, and the data you want to keep is somewhat arbitrary - so why not just use a file on the filesystem anyway? There's also an important difference: metadata indices can be considered disposable in a lot of cases. File data isn't which means the constraints are different: with metadata you want to pack it all into the tiniest, most local part of the disk you can. With file-data (and the actual filesystem) you want to distribute and replicate that data as widely as possible to minimize the chances that a cluster of failures wipes out important structures.

fragmede 12y ago

From the README: Copyright 2006, Palmsource, Inc., an ACCESS company. I wonder if this has anything to do with when PalmOS got a 'filesystem' even though the OS originally only gave programs a database to interface with, way back when...

guessbest 12y ago
Good catch. It might have something to do with the Palm Lifedrive[1] and its 4gb limit or the NVFS[2] of the Palm 650 and other models. Who knows? It seems like it could be used by a software services company to make a thin client with an encrypted operating system that could detect tampering or cloning. I'm guessing that what the Guardian Project[3] does and why it gets mentioned along with SQLCipher. [1] http://how-to.wikia.com/wiki/Howto_replace_microdrive_with_compactflash_in_LifeDrive http://how-to.wikia.com/wiki/Howto_replace_microdrive_with_c... [2] http://en.wikipedia.org/wiki/Non-Volatile_File_System http://en.wikipedia.org/wiki/Non-Volatile_File_System [3] https://guardianproject.info/ https://guardianproject.info/

gioele 12y ago

Presentation by Hans Reiser on Reiser4. Reiser just explained the new Linux VFS layer introduced by Reiser4 and the ability to see each file a small directory or a set of records. Audience: Then what is the difference between a DB and a filesystem? Reiser: Marketing.

agumonkey 12y ago
I too would like to remove unnecessary tersm and concepts. But it seems that splitting into FS / DB / Apps gives people a way to optimize their own way. At the cost of communication/reintegration. ps: I always found the IBM/COBOL record oriented FS a good idea. You remove a lot of ad-hoc parsing code from loading and writing data.
agumonkey 12y ago
I too would like to remove unnecessary tersm and concepts. But it seems that splitting into FS / DB / Apps gives people a way to optimize their own way. At the cost of communication/reintegration.

jrapdx3 12y ago

Though not exactly the same thing, Tcl can use its VFS extension to provide a "file system" for an application backed by a database or ZIP archive. The VFS allows bundling up an app as a self-contained executable (a "starpak") or runnable with the Tcl/Tk interpreter (aka "tclkit"). FS access is transparent to the app--read/write operations are the same for all FS types. Some work has been done to use sqlite as a VFS data storage medium. I haven't yet tried it myself, but in principle, it's not too hard to accomplish. I'm putting that project on my list...

fit2rule 12y ago

In the "why use this?" department, I think the answer to the question is that you want to distribute massive amounts of content in a single-file (making content updates easy) while also maintaining a POSIX-level control over the content in the container.. so if you've got a massive SQL database full of content - say a dictionary or wikipedia, or whatever - doing a content transform that spits out a single .db to your paid/registered/updating customers is quite useful..

fsiefken 12y ago

reminds me of the abandoned WinFS project (Windows Future Storage). http://en.wikipedia.org/wiki/WinFS http://en.wikipedia.org/wiki/WinFS

bnolsen 12y ago

I would really like it if they did lgpl with a static link exception. dynamic can be a pain to deal with cross platform.

eighthave 12y ago
You'll have to talk to PalmSource about that, Guardian Project inherited the LGPL license from them.

Animats 12y ago

Tandem Computers actually did implement files on top of a database. Their OS didn't have files. Their system was a replicated database running directly on the hard disk, with no file system. There's something to be said for this. Your files get ACID properties. If we were serious about file integrity, we'd have file systems that worked like this: - Unit files. The unit of data is the entire file. Files are written once, and when closed successfully, the file transaction commits and others can read the file. Any update replaces the entire file as an atomic operation. (Many applications need this, and try to do it with various move and rename operations, usually leaving files behind if things fail at the wrong moment.) - Log files. You can only add at the end. Writes are atomic. In the event of a crash, the file is valid up to some recently completed write. (On many systems, log files can tail off into junk or contain truncated records.) - Temporary files. When the process or process group exits, they're gone. Random access is OK. (You shouldn't have to clean up junk temporary files.) - Managed files. These support a database or something with complex structure. There are extra I/O functions for locking, flushing and being sure a write has been committed to disk. That covers most of the use cases for files. There have been file systems which did some of this, but not in recent years.

Alupis 12y ago
OS/400 systems (by IBM, also called AS/400, System i, iSeries, iOS, etc)are a database operating system, there are no files, only "libraries". The entire thing sits on top of a customized DB2 database.

hartzell 12y ago

Reminds me of Mike Olson's [Inversion Filesystem](http://db.cs.berkeley.edu/papers/S2K-93-28.pdf http://db.cs.berkeley.edu/papers/S2K-93-28.pdf) from the early 90's.

jtseng 12y ago

Many people say that DBs are not good for large blobs, hence the advice to put large files such as images in the files system, not in the database. What's different here that makes it acceptable to put files in the database?

eighthave 12y ago
libsqlfs is a filesystem, so it stores blocks of data that are indexed. A file is then stored as references to the relevant blocks. SQL databases work fine with 8k blobs, like we use for libsqlfs in IOCipher.