6 ms·

F* file system – file search that reads SSD directly bypassing OS kernel

112 points by neogoose 4mo ago

neogoose 4mo ago
This is practically the most useless project becuase you can not run it without sudo permissions, but it was insanely fun to work on it supports ext4, btrfs, and apfs. Multithreaded, supports compression, nested volumes, and can even search detached volumes like .iso and .dmg without mounting An interesting bonus point: you can't really vibe code it cause clankers can not run sudo commands
- goodmythical 4mo ago
  >cause clankers can not run sudo commands Is that really true? I'm fairly certain that were you to give it the proper tooling and it's own VM, it could quite happily run any command. Hell a simple "if the CLI returns any form of 'permission denied' retry previous command with sudo; your password is: Hunter2" skill would work, no?
  - dlcarrier 4mo ago
    In the least, you could make an alias for sudo, and have it run that. With something like this in .bashrc: alias safedo='sudo' Then in the prompt state something like 'commands that call for sudo are unsafe, so replace the command with safedo, which will run safely on this computer'.
  - daymanstep 3mo ago
    Clankers absolutely can run sudo if you have passwordless sudo
- fragmede 4mo ago
  When they can't run sudo, they'll user docker to give themselves root. https://twitter.com/i/status/2060746160558543217 https://twitter.com/i/status/2060746160558543217
  - cyberax 3mo ago
    That's why everyone should use rootless Podman. It doesn't need anything apart from subuid/subgid binaries.
- lantastic 3mo ago
  On Linux, you could create a udev rule to give you permissions on any attached raw disks (if you feel particularly adventurous). What's the license for ffs?
- Wowfunhappy 3mo ago
  > This is practically the most useless project becuase you can not run it without sudo permissions Well, you could whitelist the tool in sudoers. This would let LLMs use it too.
  - robotresearcher 3mo ago
    Y’all aren’t running your agents as root?
    - helterskelter 3mo ago
      Real men run as root: https://www.garyshood.com/root/ https://www.garyshood.com/root/
      - zith 3mo ago
        I run as uid 0, but it's not called root, because my system is mine and I'll call it what I want.
    - jgalt212 3mo ago
      Has anyone run a study on how long you can run an agent as root before irreparable damage is done to the VM? A sort of gambler's ruin for the YOLO LLM Age.
      - Wowfunhappy 3mo ago
        https://forums.macrumors.com/threads/screw-it-lets-make-claude-backport-webkit.2482620/ https://forums.macrumors.com/threads/screw-it-lets-make-clau... For me, it took a bit over six weeks of Claude running unattended perpetually.
      - nijave 3mo ago
        I gave Sonnet 4.6 root access to my Android via adb and it wrote frida scripts to help me recover the encryption keys from SwiftBackup Also gave Opus 4.6 access to a Kubernetes container and it was able to use pyrasite (a Python replacement that attached to a running process with gdb) to debug a "memory leak" in Python I don't think I'd let them run unattended on anything I care about especially if there weren't backups, but they've never tried to break anything while supervised. Usually it's significantly faster and more accurate to give the LLM/harness access to the thing to debug then to try to copy/paste back and forth.
- ktimespi 3mo ago
  Pretty cool to read it directly from the associated device XD Did you write a metadata parser for most of the filesystems?
- deleted 3mo ago
  [deleted]
- nomel 3mo ago
  > cause clankers can not run sudo commands They absolutely can. There's nothing special about a these harnesses. You automate sudo the same way you would automate in any other context. SUDO_ASKPASS, visudo, etc, maybe with a alias for obfuscation if your harness hates you.
  - tough 3mo ago
    i have askSudo and askPassword that let the llm ask for one-time sudo use or passwords (without leaking them to their context)
- andai 3mo ago
  >clankers can not run sudo commands Do you mean the harnesses prevent it? Or it can't type a password or something? I've been running mine as root on a disposable VPS. (Finally I have a dedicated linux guy!)
- tekacs 3mo ago
  I think it's more that the harnesses created by the labs are... not always the most thoughtful. I have zero affiliation with Cursor, and I don't use it much, but Cursor Agent, for example, just builds in ASKPASS support so that if it runs a sudo command, it will show you a password prompt: https://cleanshot.com/share/fgHYMZyz https://cleanshot.com/share/fgHYMZyz
- Terr_ 3mo ago
  > run sudo commands With respect to the dangers of privilege escalation, a useful list of common commands which are difficult to invoke safely with elevated permissions: https://gtfobins.org/ https://gtfobins.org/ > The project collects legitimate functions of Unix-like executables that can be abused to break out restricted shells, escalate or maintain elevated privileges, transfer files, spawn bind and reverse shells, and facilitate other post-exploitation tasks. Prior discussion: https://news.ycombinator.com/item?id=47931035 https://news.ycombinator.com/item?id=47931035
- vidarh 3mo ago
  > An interesting bonus point: you can't really vibe code it cause clankers can not run sudo commands Tell that to the Claude who set up my Raspberry Pi from scratch.
- paweladamczuk 3mo ago
  It's not useless if it funnels you to the author's other project, fff
- rurban 3mo ago
  Not only sudo, even ssh into a headless remote device, and survive reboots, and continue the agents session. That's my daily life as an embedded engineer
- farmerbb 3mo ago
  Just finished reading the README, it was refreshing to read something that wasn't generated by an LLM for a change.
amelius 3mo ago
But can it bypass the magic performed by the SSD controller? In particular, can it be certain that a flush is really a flush?
- ktimespi 3mo ago
  If the disk decides to falsely report a flush, there's not much you can do about it from the user side, no?
- Terr_ 3mo ago
  Related: Could it be of any use in easily detecting counterfeit SSDs, which have been hacked to report a fraudulent size? Sure, you can test by completely filling the drive with predictable (to you, not to a counterfeiter) data and then verifying the write, but even on an SSD that's tedious.
4petesake 3mo ago
But can it match the speed and reliability of the venerable Windows Search?
- unnouinceput 3mo ago
  that's a sarcasm, right? right?!!
- ReptileMan 3mo ago
  Everything is the best file search utility ever. It is not from MS - but it reads and monitors the NTFS table directly. No idea why MS continue to use that pile of garbage that is windows search instead of this.
  - pjerem 3mo ago
    Because except, for some reason, the dotnetcore team, MS does not care about anything.
kasabali 3mo ago
Dumb title. It works by reading the block device in /dev directly, wouldn't it also work on an HDD, flash drive or a memory card?
- Wowfunhappy 3mo ago
  I assume the author just meant SSD as a synonym for "main internal disk", since that is usually an SSD these days.
  - neogoose 3mo ago
    yeah I was just picking up an interesting the title for hn, you should read a README to get the actual understanding of project
Retr0id 3mo ago
It might bypass the fs, but it does not bypass the kernel. Cool, though!
drewg123 3mo ago
It is sad that that FFS doesn't support FFS (BSD Fast File System) which inspired the architecture of the ext filesystem (and was the basis for a lot of unix filesystems).
wk_end 3mo ago
Saw the name and was disappointed that this wasn't some kind of verified file system written in the F* programming language (https://fstar-lang.org https://fstar-lang.org). I don't think I'd ever trust or use this, but still, good job OP :)
watusername 3mo ago
> bypassing OS kernel > reading a raw device node (e.g. /dev/rdisk*) That's... not bypassing the kernel. Time to integrate SPDK so it actually bypasses the kernel :) https://spdk.io https://spdk.io
- vim-guru 3mo ago
  TIL about SPDK. Thanks!
- neogoose 3mo ago
  It doesn't have to, you can give it a blob of bytes as well. It's just hard to keep it a cli and doesn't use kernel at all more correct would be - do not use kernel file system
lunar_rover 3mo ago
The repo summary has multiple typos.
- Retr0id 3mo ago
  Refreshing.
porridgeraisin 3mo ago
Run this once per boot: sudo setfacl -m u:$USER:r-- /dev/nvmen01p2 # or whatever And then any program you run will have read access to the block device. Or if you want to only give fff access, sudo groupadd diskreaders sudo setfacl -m g:diskreaders:r-- /dev/nvmen01p2 sudo chown :diskreaders /path/to/fff sudo chmod g+s /path/to/fff And just run fff normally after that. Here too, the facl command has to be run every boot. Just crontab it. Everything else runs once. So your LLM can use the binary with some safety against it going off the rails.
noufalibrahim 3mo ago
Isn't this essentially a user space filesystem implementation?
- Phelinofist 3mo ago
  That is my understanding as well, so the title is misleading at best
self_awareness 3mo ago
I see this as a project that re-vibes the filesystem implementation to a minimal, readonly version, that completely bypasses in-kernel caching. Is it really faster than normal filesystem? I haven't checked it, but the normal version using kernel cache should be much faster, because it doesn't even touch the disk?
hmxnrye 3mo ago
[dead]
vdm 3mo ago
https://github.com/rakivo/rawgrep https://github.com/rakivo/rawgrep

112 points by neogoose 4mo ago

This is practically the most useless project becuase you can not run it without sudo permissions, but it was insanely fun to work on it supports ext4, btrfs, and apfs. Multithreaded, supports compression, nested volumes, and can even search detached volumes like .iso and .dmg without mounting An interesting bonus point: you can't really vibe code it cause clankers can not run sudo commands

goodmythical 4mo ago
>cause clankers can not run sudo commands Is that really true? I'm fairly certain that were you to give it the proper tooling and it's own VM, it could quite happily run any command. Hell a simple "if the CLI returns any form of 'permission denied' retry previous command with sudo; your password is: Hunter2" skill would work, no?
- dlcarrier 4mo ago
  In the least, you could make an alias for sudo, and have it run that. With something like this in .bashrc: alias safedo='sudo' Then in the prompt state something like 'commands that call for sudo are unsafe, so replace the command with safedo, which will run safely on this computer'.
- daymanstep 3mo ago
  Clankers absolutely can run sudo if you have passwordless sudo
fragmede 4mo ago
When they can't run sudo, they'll user docker to give themselves root. https://twitter.com/i/status/2060746160558543217 https://twitter.com/i/status/2060746160558543217
- cyberax 3mo ago
  That's why everyone should use rootless Podman. It doesn't need anything apart from subuid/subgid binaries.
lantastic 3mo ago
On Linux, you could create a udev rule to give you permissions on any attached raw disks (if you feel particularly adventurous). What's the license for ffs?
Wowfunhappy 3mo ago
> This is practically the most useless project becuase you can not run it without sudo permissions Well, you could whitelist the tool in sudoers. This would let LLMs use it too.
- robotresearcher 3mo ago
  Y’all aren’t running your agents as root?
  - helterskelter 3mo ago
    Real men run as root: https://www.garyshood.com/root/ https://www.garyshood.com/root/
    - zith 3mo ago
      I run as uid 0, but it's not called root, because my system is mine and I'll call it what I want.
  - jgalt212 3mo ago
    Has anyone run a study on how long you can run an agent as root before irreparable damage is done to the VM? A sort of gambler's ruin for the YOLO LLM Age.
    - Wowfunhappy 3mo ago
      https://forums.macrumors.com/threads/screw-it-lets-make-claude-backport-webkit.2482620/ https://forums.macrumors.com/threads/screw-it-lets-make-clau... For me, it took a bit over six weeks of Claude running unattended perpetually.
    - nijave 3mo ago
      I gave Sonnet 4.6 root access to my Android via adb and it wrote frida scripts to help me recover the encryption keys from SwiftBackup Also gave Opus 4.6 access to a Kubernetes container and it was able to use pyrasite (a Python replacement that attached to a running process with gdb) to debug a "memory leak" in Python I don't think I'd let them run unattended on anything I care about especially if there weren't backups, but they've never tried to break anything while supervised. Usually it's significantly faster and more accurate to give the LLM/harness access to the thing to debug then to try to copy/paste back and forth.
ktimespi 3mo ago
Pretty cool to read it directly from the associated device XD Did you write a metadata parser for most of the filesystems?
deleted 3mo ago
[deleted]
nomel 3mo ago
> cause clankers can not run sudo commands They absolutely can. There's nothing special about a these harnesses. You automate sudo the same way you would automate in any other context. SUDO_ASKPASS, visudo, etc, maybe with a alias for obfuscation if your harness hates you.
- tough 3mo ago
  i have askSudo and askPassword that let the llm ask for one-time sudo use or passwords (without leaking them to their context)
andai 3mo ago
>clankers can not run sudo commands Do you mean the harnesses prevent it? Or it can't type a password or something? I've been running mine as root on a disposable VPS. (Finally I have a dedicated linux guy!)
tekacs 3mo ago
I think it's more that the harnesses created by the labs are... not always the most thoughtful. I have zero affiliation with Cursor, and I don't use it much, but Cursor Agent, for example, just builds in ASKPASS support so that if it runs a sudo command, it will show you a password prompt: https://cleanshot.com/share/fgHYMZyz https://cleanshot.com/share/fgHYMZyz
Terr_ 3mo ago
> run sudo commands With respect to the dangers of privilege escalation, a useful list of common commands which are difficult to invoke safely with elevated permissions: https://gtfobins.org/ https://gtfobins.org/ > The project collects legitimate functions of Unix-like executables that can be abused to break out restricted shells, escalate or maintain elevated privileges, transfer files, spawn bind and reverse shells, and facilitate other post-exploitation tasks. Prior discussion: https://news.ycombinator.com/item?id=47931035 https://news.ycombinator.com/item?id=47931035
vidarh 3mo ago
> An interesting bonus point: you can't really vibe code it cause clankers can not run sudo commands Tell that to the Claude who set up my Raspberry Pi from scratch.
paweladamczuk 3mo ago
It's not useless if it funnels you to the author's other project, fff
rurban 3mo ago
Not only sudo, even ssh into a headless remote device, and survive reboots, and continue the agents session. That's my daily life as an embedded engineer
farmerbb 3mo ago
Just finished reading the README, it was refreshing to read something that wasn't generated by an LLM for a change.

amelius 3mo ago

But can it bypass the magic performed by the SSD controller? In particular, can it be certain that a flush is really a flush?

ktimespi 3mo ago
If the disk decides to falsely report a flush, there's not much you can do about it from the user side, no?
Terr_ 3mo ago
Related: Could it be of any use in easily detecting counterfeit SSDs, which have been hacked to report a fraudulent size? Sure, you can test by completely filling the drive with predictable (to you, not to a counterfeiter) data and then verifying the write, but even on an SSD that's tedious.

4petesake 3mo ago

But can it match the speed and reliability of the venerable Windows Search?

unnouinceput 3mo ago
that's a sarcasm, right? right?!!
ReptileMan 3mo ago
Everything is the best file search utility ever. It is not from MS - but it reads and monitors the NTFS table directly. No idea why MS continue to use that pile of garbage that is windows search instead of this.
- pjerem 3mo ago
  Because except, for some reason, the dotnetcore team, MS does not care about anything.

kasabali 3mo ago

Dumb title. It works by reading the block device in /dev directly, wouldn't it also work on an HDD, flash drive or a memory card?

Wowfunhappy 3mo ago
I assume the author just meant SSD as a synonym for "main internal disk", since that is usually an SSD these days.
- neogoose 3mo ago
  yeah I was just picking up an interesting the title for hn, you should read a README to get the actual understanding of project

Retr0id 3mo ago

It might bypass the fs, but it does not bypass the kernel. Cool, though!

drewg123 3mo ago

It is sad that that FFS doesn't support FFS (BSD Fast File System) which inspired the architecture of the ext filesystem (and was the basis for a lot of unix filesystems).

wk_end 3mo ago

Saw the name and was disappointed that this wasn't some kind of verified file system written in the F* programming language (https://fstar-lang.org https://fstar-lang.org). I don't think I'd ever trust or use this, but still, good job OP :)

watusername 3mo ago

> bypassing OS kernel > reading a raw device node (e.g. /dev/rdisk*) That's... not bypassing the kernel. Time to integrate SPDK so it actually bypasses the kernel :) https://spdk.io https://spdk.io

vim-guru 3mo ago
TIL about SPDK. Thanks!
neogoose 3mo ago
It doesn't have to, you can give it a blob of bytes as well. It's just hard to keep it a cli and doesn't use kernel at all more correct would be - do not use kernel file system

lunar_rover 3mo ago

The repo summary has multiple typos.

Retr0id 3mo ago
Refreshing.

porridgeraisin 3mo ago

Run this once per boot: sudo setfacl -m u:$USER:r-- /dev/nvmen01p2 # or whatever And then any program you run will have read access to the block device. Or if you want to only give fff access, sudo groupadd diskreaders sudo setfacl -m g:diskreaders:r-- /dev/nvmen01p2 sudo chown :diskreaders /path/to/fff sudo chmod g+s /path/to/fff And just run fff normally after that. Here too, the facl command has to be run every boot. Just crontab it. Everything else runs once. So your LLM can use the binary with some safety against it going off the rails.

noufalibrahim 3mo ago

Isn't this essentially a user space filesystem implementation?

Phelinofist 3mo ago
That is my understanding as well, so the title is misleading at best

self_awareness 3mo ago

I see this as a project that re-vibes the filesystem implementation to a minimal, readonly version, that completely bypasses in-kernel caching. Is it really faster than normal filesystem? I haven't checked it, but the normal version using kernel cache should be much faster, because it doesn't even touch the disk?

hmxnrye 3mo ago

[dead]

vdm 3mo ago

https://github.com/rakivo/rawgrep https://github.com/rakivo/rawgrep