4 ms·
For this use case, it might be better to just store blobs in SQLite: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
by RcouF1uZ4gsC 3y ago
For this use case, it might be better to just store blobs in SQLite: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- rollcat 3y agoAgree, also sqlite might turn out faster than a filesystem even for much more practical cases: https://sqlite.org/fasterthanfs.html https://sqlite.org/fasterthanfs.html However it still tickles my curiosity and exercises an interesting pathological case. As a kid I've had this idea, what if I could store the data bytes in the file names of empty files, did I just create space out of nowhere? It's always beautiful whenever you rediscover that computers are not magic.
- anacrolix 3y agoI built https://github.com/anacrolix/squirrel https://github.com/anacrolix/squirrel for just this purpose. However you might find that batch inserting empty files might only be 10x faster or so than creating empty files directly on the filesystem if we are to believe the 10k/s the OP achieved.
- actionfromafar 3y agoThanks!
- jerf 3y agoGiven that this just creates empty files named "file-#.txt", you can "just" create a Fuse file system that simply exposes a billion files named that way for basically no space on disk and no time spent creating it. In fact given that the author said this took over a day to run I could probably write this code faster than the author was able to run the script once on a concrete file system. This isn't really a problem about optimizing the storage of a billion empty files, it's about what the file systems do with them.