11 ms·
Show HN: VectorVFS, your filesystem as a vector database
- malcolmgreaves 1y agoFun idea storing embeddings in inodes! Very clever! I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.py https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.... Still, fun idea :)
- perone 1y agoThanks. There is a bit of a nuance there, for example: you can build an index in first pass which will indeed be linear, but then later keep it in an open prompt for subsequent queries, I'm planning to implement that mode soon. But agree, it is not intended to search 10 million files, but you seldom have this use case in local use anyways.
- binarymax 1y agoO(n) is still OK for vector search if n isn't too large. Filesystem search solutions are currently terrible, with background indexing jobs and poor relevance. This won't scale for every file on your system but anything in your working documents folder would easily work well.
- PaulHoule 1y agoThe lack of an index is not bad at all if you have it stored contiguously in RAM: the mechanical sympathy is great, SIMD will spin like a top not to mention multithreaded programming, etc. Circa 2014 or so I worked on a search engine that scanned maybe 2GB worth of vectors for 10 million documents, queries were turned around in much less than a second, nobody complained about the speed. If you gotta gather the data from a lot of different inodes, it is a different story.
- ori_b 1y agoIt's not stored continuously in ram. It's stored in extended attributes.
- esafak 1y agothanks for saving readers time. If so this is not a viable tool for production.
- int_19h 1y agoAn index could be built on top of this though if desired. No need to have it in the FS itself.
- yencabulator 1y agoBut then there's no point in storing anything in xattrs.
- int_19h 1y agoThe reason would be that it's there as the source of truth, and when files e.g. get copied around, so does the metadata. The indexer doesn't need to be synchronous wrt such operations though, it can just watch the FS for changes and spin up reindexing as needed asynchronously.
- natas 1y agothis is actually a great idea
- iugtmkbdfil834 1y agoAssuming I understand it correctly, the idea is to be able to have LLMs get through file systems more easily with some interesting benefits to human users as well. The idea is interesting and I want to try it out.
- perone 1y agoHi, there are no LLMs involved, it is all local and an embedding (vector representation) of the data is created and then that is used for search later, nothing is sent to cloud from your files and there are no local LLMs running as well, only the encoders (I use the Perception Encoder from Meta released a few weeks ago).
- anotherpaul 1y agoGreat idea indeed. The documentation needs a bit more information to be useful. What GPU backends are supported for example? How do I delete the embedding information after I decide to uninstall it? Will give it a try though.
- perone 1y agoThanks, I'm working on implementing the commands to clean the embeddings (you can now do that with Linux xattr command-line tool). I'm supporting CPU or GPU (NVIDIA) for the encoders and it only supports Linux at the moment.
- 3abiton 1y agoI am curious why Python, and not rust for example?
- danudey 1y agoNot OP, but despite working in an all-Go shop I just wrote a utility in Python the other week and caught some flak for it. The reason I gave (which was accepted) was that the process of creating a proof of concept and iterating on it rapidly is vastly easier in Python (for me) than it is in Go. In essence, it would have taken me at least a week, possibly more, to write the program I ended up with in Golang, but it only took me a day to write it in Python, and, now that I understand the problem and have a working (production-ready) prototype, it would probably only take me another day to rewrite it in Golang. Also, a large chunk of the functionality in this Python script seems to be libraries - pillow for image processing, but also pytorch and related vision/audio/codec libraries. Even if similar production-ready Rust crates are available (I'm not sure if they are), this kind of thing is something Python excels at and which these modules are already optimized for. Most of the "work" happening here isn't happening in Python, by and large.
- hadlock 1y agoSure, but now your all-go shop now needs to support two languages, two sets of linters, ci/cd etc for a single utility. It might be faster for you but if the utility is going to be used for more than a couple of weeks now it's a real hassle to get a go developer to make sure they have the right version of the interpeter, remember all the ins and outs of python etc.
- badmonster 1y agointeresting
- Ericson2314 1y agoThe idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.
- 7qW24A 1y agoI’m a database guy, not an OS guy, so I agree, obviously… But what is the micro-kernel angle?
- packetlost 1y agoLikely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine. Persistent file systems are essentially key-value stores, usually with optimizations for enumerating keys under a namespace (also known as listing the files in a directory). IMO a big problem with POSIX filesystems is the lack of atomicity and lock guarantees when editing a file. This and a complete lack of consistent networked API are the key reasons few treat file systems as KV stores. It's a pity, really.
- mrlongroots 1y ago> "Likely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine." "Userspace vs not" is a different argument from "consistency vs not" or "atomicity vs not" or "POSIX vs not". Someone still needs to solve that problem. Sure instead of SQLite over POSIX you could implement POSIX over SQLite over raw blocks. But you haven't gained anything meaningful. > Persistent file systems are essentially key-value stores I think this is reductive enough to be equivalent to "a key-value store is a thin wrapper over the block abstraction, as it already provides a key-value interface, which is just a thin layer over taking a magnet and pointing it at an offset". Persistent filesystems can be built over key-value stores. This is especially common in distributed filesystems. But they also circumvent a key-value abstraction entirely. > IMO a big problem with POSIX filesystems is the lack of atomicity Atomicity requires write-ahead logging + flushing a cache. I fail to see why this needs to be mandatory, when it can be effectively implemented at a higher layer. > This and a complete lack of consistent networked API A consistent networked API would require you to hit the metadata server for every operation. No caching. Your system would grind to a halt. Finally, nothing in the POSIX spec prohibits an atomic filesystem or consistency guarantees. It is just that no one wants to implement these things that way because it overprovisions for one property at the expense of others.
- b0a04gl 1y agoIf VectorVFS obscures retrieval logic behind opaque embeddings, how do users debug why a file surfaced—or worse, why one didn’t?
- deleted 1y ago[deleted]
- refulgentis 1y agoWhat is a non-opaque embedding? Does VectorVFS do retrieval, or store embeddings in EXT4? Is retrieval logic obscured by VectorVFS? If VectorVFS did retrieval with non-opaque embeddings, how would one debug why a file surfaced?
- perone 1y agoHi, not sure if I understood what you meant by opaque embeddings as well, but the reason why files surface or not is due to the similarity score (which is basically the dot product of embeddings).
- jlhawn 1y agoHow much work do you think it would be to also have a separate xattr which has a human-readable description of the file contents? I wonder if it that might already be an intermediate product of some of the embedding tools, like "arbitrary media" -> "text description of media" -> "embedding vector". You could store both of those as xattrs and you could debug by comparing your text query with the text description of the file contents as they should produce similar embedding vectors. You could even audit any file, assuming you know what its contents are, by checking the text description xattr generated by this program.
- b0a04gl 1y ago[dead]
- deleted 1y ago[deleted]
- esafak 1y agoFiles-as-vector stores is LanceDB's value proposition. How do you compare in performance, etc.?
- perone 1y agoThis is quite different than LanceDB. In VectorVFS I'm using the inodes directly to store the embeddings, there is no external file with metadata and db, the db is your filesystem itself, that's the key difference.
- esafak 1y agoThat's an implementation detail, and it sounds more like a liability than a selling point, to have such tight coupling. (Why) do you see not using files as a good thing? Let me ask another question: is this intended for production use, or is it more of a research project? Because as a user I care about things like speed, simplicity, flexibility, and robustness.
- adenta 1y agoI wonder if I could use this locally on my macbook. The finder applications built-in search is kinda meh.
- perone 1y agoI'm planning to support MacOS, the only issue is with the encoders that I'm using now, I will probably work more on it next week to try to make a release that works on MacOS as well. Thanks !
- tzury 1y agoI’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2]. When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that: 1. uploaded the raw logs to Cloud Storage, and 2. tracked state with three folders: `pending/`, `processing/`, `done/`. A cron job cycled through those directories and quietly pushed petabytes every week without dropping a byte. Later, Google’s own pipelines—and third-party stacks like Logstash—never matched that script’s throughput or reliability. Lesson: reach for the filesystem first; add services only once you’ve proven you actually need them. [1] https://en.wikipedia.org/wiki/Everything_is_a_file https://en.wikipedia.org/wiki/Everything_is_a_file [2] https://en.wikipedia.org/wiki/Unix_philosophy https://en.wikipedia.org/wiki/Unix_philosophy
- dominicq 1y agoCan you say more about the use case? What problem were you solving? How did it work exactly? Sounds interesting so I'd like to learn more.
- tzury 1y agoSure. We were building Reblaze (started 2011), a cloud WAF / DDoS-mitigation platform. Every HTTP request—good, bad, or ugly—had to be stored for offline anomaly-detection and clustering. Traffic profile - Baseline: ≈ 15 B requests/day - Under attack: the same 15 B can arrive in 2-3 hours Why BigQuery (even in alpha)? It was the only thing that could swallow that firehose and stay query-able minutes later — crucial when you’re under attack and your data source must not melt down. Pipeline (all shell + cron) Edge nodes → write JSON logs locally and a local cron push to Cloud Storage Tiny VM with a cron loop - Scans `pending/`, composes many small blobs into one “max-size” blob in `processing/`. - Executes `bq load …` into the customer’s isolated dataset. - On success, moves the blob to `done/`; on failure, drops it back to `pending/`. Downstream ML/alerting* pulls straight from BigQuery That handful of `gsutil`, `bq`, and `mv` commands moved multiple petabytes a week without losing a byte. Later pipelines—Dataflow, Logstash, etc.—never matched its throughput or reliability.
- bullen 1y agoI did something similar, but I use these EXT4 requirements: - hard links (only tar works for backup) - small file size (or inodes run out before disk space) http://root.rupy.se http://root.rupy.se It's very useful for global distributed real-time data that don't need the P in CAP for writes. (no new data can be created if one node is offline = you can login, but not register)
- jlhawn 1y agoIf I understand correctly, this is attaching metadata to files in a format that LLMs (or any tool that can understand the semantic embedding vector) can leverage to understand what a file is without having to actually read the contents of the file. That obviously has a lot of interesting use cases, but my first assumption was that this could be used to quickly/easily search your filesystem with some prompt like "Play the video from last month where we went camping and saw a flock of turkeys". But that would require having an actual vector DB running on your system which you could use to quickly look up files using an embedding of your query, no?
- lstodd 1y agoso, like magic(5)?
- mywittyname 1y agoWhat is magic(5) and how is it similar to what was described?
- simcop2387 1y agoI think they're referring to this, https://linux.die.net/man/5/magic https://linux.die.net/man/5/magic given the notation. That said I don't really see how it'd be all that relevant to the discussion so maybe i'm missing something else.
- danudey 1y agomagic(5) is a system for determining the type of a file by examining the 'magic bytes' at or near the start of a file. For example, POSIX tar files have a defined file format that starts with a header struct: https://www.gnu.org/software/tar/manual/html_node/Standard.html https://www.gnu.org/software/tar/manual/html_node/Standard.h... You can see that at byte offset 257 is `char magic[6]`, which contains `TMAGIC`, which is the byte string "ustar\0". Thus, if a file has the bytes 'ustar\0' at offset 257 we can reasonably assume that it's a tar file. Almost every defined file type has some kind of string of 'magic' predefined bytes at a predefined location that lets a program know "yes, this is in fact a JPEG file" rather than just asserting "it says .jpg so let's try to interpret this bytestring and see what happens". As for how it's similar: I don't think it actually is, I think that's a misunderstanding. The metadata that this vector FS is storing is more than "this is a a JPEG" or "this is a word document", as I understand it, so comparing it to magic(5) is extremely reductionist. I could be mistaken, however.
- colordrops 1y agoRt
- pseudosavant 1y agoThis immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to easily tap into. I'd be surprised if cloud storage services like OneDrive don't already do some kind of vector for every file you store. But an online web service isn't the same as being built into the core of the OS.
- perone 1y agoI share the same feeling, I think filesystems will have to reinvent themselves given the pace of how useful ML models became in the past years.
- didgetmaster 1y agoI built a local object store that was designed to replace file systems. You can create hundreds of millions of objects (e.g. files) and attach a variety of metadata tags to each one. A tag could be a number, string, or other data type (including vector info). Searches for objects with certain tags is exceptionally fast. I invented it because I found searching conventional file systems that support extended attributes to be unbearably slow.
- tugdual 1y agoGot a demo ?
- didgetmaster 1y agoTons of demo videos on my YouTube channel. Free beta available for download on my website. Links in my profile.
- p_ing 1y agoWinFS wasn't a file system laid down on hardware, it was just a SQL database that stored arbitrary data.
- asadawadia 1y agois the embedding for the whole file? or each 1024/512 byte chunk?
- javier2 1y agoi looked into something similar a few years ago, where i stored embeddings in xattrs
- quantadev 1y agoI've been wondering for about 20 years why File Systems basically died and stopped innovating. For example we have lots of hierarchical data structures in the world, and no one seems to have figured out how to let a folder be the storage, instead of always just databases. For example, if we simply had the ability to have "ordered" files inside folders, that would instantly make it practical for a folder structure to represent "Documents". After all, documents are nothing but a list of paragraphs and images, so if we simply had ordering in file systems we could have document editors which are using individual files for each paragraph of text or image. It would be amazing. Also think about use cases like Jupyter Notebooks. We could stop using the XML file format, and just make it a folder structure instead. Each cell (node) being in a file. All social media messages and chatbot conversations could be easily saved as folders structures. I've heard many file copy tools ignore XATTR so I've never tried to use it for this purpose, so maybe we've had the capability all along and just nobody thought to use it in a big way that became popular yet. Maybe I should consider XATTR and take it seriously.
- wfn 1y agoI agree! I'm sort of exploring "programmable filesystem" concept (using FuseFS) (for some notes, see [1]). Re: ordered files: depends on FS. e.g. filesystems which use B+ trees will tend to have files (in directories) in lexical order. So in some cases you may not need a new FS: echo 'for f in *.txt; do cat "$f"; done' > doc.sh; chmod +x doc.sh => `doc.sh` in dir produces 'documents' (add newlines / breaks as needed, or add piping through Markdown processor); symlink to some standardized filename 'Process', etc... That said... wouldn't it be nice to have ridiculous easily pluggable features like echo "finish this poem: roses are red," > /auto-llm/poem.txt; cat .. :) [1]: chaotic notes: https://kfs.mkj.lt/#welcome https://kfs.mkj.lt/#welcome (see bullet point list below)
- quantadev 1y agoInteresting stuff. Thanks for posting that.
- thirdtrigger 1y agoMight be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well). 1. https://weaviate.io/developers/weaviate/installation/embedded https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat https://weaviate.io/developers/academy/py/vector_index/flat
- binarymax 1y agoWhy weaviate and not FAISS? The latter is faster and lighter.
- lysp 1y agoI think they are associated with the project
- bobvanluijt 1y agoIt depends on additional filters and whether you want to use vector search only. The upside of using Faiss would be storing the ID as file metadata and embedding it in the Faiss index. However, if you need any other filters or data, you would need to store it somewhere else.
- gitroom 1y agoGotta say, the old school debate on filesystems vs databases will never get old for me - I always end up with more questions than answers after reading stuff like this.
- j45 1y agoEverything's old school, everything's new. It's important to remember that the cloud is also invented by the old school and understanding the oscillation between client/server architectures vs local, and it's implication on topics of data and files is interesting too. More questions means more learning until I learned there's no one right or wrong, just what works best, where, when, for how long, and what the tradeoffs are. Quick wins/decisions are often bandaids that pile up in an different way.
- PeterZaitsev 1y agoI think comparing it to Vector Database is confusing as database would typically mean indexes and some sort of query support. Storing Embeddings with File is interesting concept... we already do it for some file formats (ie EXIF), where this one is generalized... yet you would need to have some actual database to load this data into to process at scale. Another issue I see is support for different models and embedding formats to make this data really portable - like I can take my file drop it into any system and its embedding "seamlessly" integrates
- lovelysoni03 1y ago[flagged]
- PeterStuer 1y agoIf there is no indexing, how will your search time not increase linear or worse with the number of files?
- ndsipa_pomu 1y agoI've long wanted to have a linux filesystem that robustly supported "tags" for files so that I didn't have to rely on the filesystem hierarchy to represent media files etc. e.g. I might want to tag a particular films as "Scifi" and also "Horror". Of course, for films, NFO files are typically used for this kind of metadata, but I'd like a similar facility that could be applied to any type of file.
- sneak 1y agoThat is literally what xattrs are for.
- ndsipa_pomu 1y agoYes, but they seem fairly limited in terms of userspace programs. How would you use xattrs to produce a filesystem hierarchy that say, listed the same file in multiple folders according to the attributes?
- sneak 1y agoThe xattrs are for storing the tag metadata, you’d use other tools (easily composed from shell utilities) to find files that match tags. If you really want it to be in multiple locations, you could make a fuse interface that shows directories full of files matching specific tags.
- ndsipa_pomu 1y ago> If you really want it to be in multiple locations, you could make a fuse interface that shows directories full of files matching specific tags Yeah, that's the kind of thing that I've wanted, but not really had the programming skill/experience/patience to make. There have been a couple of similar projects, but nothing that seems popular enough to be worthwhile spending time using.
- sneak 1y ago
- yencabulator 1y ago> Zero-overhead indexing Embeddings are stored as extended attributes (xattrs) on each file, eliminating the need for external index files or services. Ain't no such thing as zero-overhead indexing. Just because you can't articulate where the overhead is doesn't make it disappear.