5 ms·
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
by wayneftw 7y ago
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
- bob1029 7y agoThis sounds like a fantastic idea. You would go from 4000 directories and 50000 files to a single file with b-tree indices, hashes, etc. You could probably get a 100x speedup over the current approach.
- demosito666 7y agoEven better would be not to use it entirely - it'd solve far bigger problems with node.
- dymk 7y agoHere you go: https://github.com/guardianproject/libsqlfs https://github.com/guardianproject/libsqlfs
- Fnoord 7y agoThank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your <asterisk>nix [1] tools keep working. [1] Calling it Nix confuses with NixOS...
- mkl 7y agoSo call it *nix? It takes two asterisks two turn on italics between them.
- Fnoord 7y agoCheers. Yeah, I guess it triggered cause I used two asterisks in my text since originally I used the term twice.
- deleted 7y ago[deleted]
- ferzul 7y agowell, it was *nix before nix (the package manager) existed. So I think Nix, as good as it is, stuffed up with that name.
- mceachen 7y agoThere's already support for accessing zip and tar archives as filesystems (also see ASAR), which doesn't incur the overhead of SQL evaluation. At least on Windows, this would result in a huge speedup (just because you're avoiding antivirus latency). I don't expect it to help Mac or Linux that much (as their filesystems are comparatively much more efficient with larger numbers of small files).
- bpierre 7y agoIt’s kind of the idea behind Yarn’s Plug’n’Play: https://yarnpkg.com/en/docs/pnp https://yarnpkg.com/en/docs/pnp
- GordonS 7y agoThat would be amazing! To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal. Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)
- Fnoord 7y agoWSL1 is terrible for I/O (which I suppose this is). WSL2 should be adequate. Perhaps Dokan? Dokan has a FUSE wrapper [1] [1] https://dokan-dev.github.io https://dokan-dev.github.io
- majewsky 7y agoNot sure if /s :) With a FUSE filesystem, you're still incurring the syscall overhead that's causing this performance disparity in the first case, plus now the kernel has to context-switch into the FUSEing process. So what would be 2 context switches for each syscall (program -> kernel -> program) now is 3 or 4 context switches (program -> kernel -> FUSE -> (kernel ->) program). FUSE has some amazing usecases, but performance is none of them.
- GordonS 7y agoErm, no, no sarcasm, but in truth I know very little about FUSE, beyond it being an API for userland file systems. I'm also approaching this as primarily a Windows user - a lot of the overhead on Windows is access checks, so my thinking is these would be implemented as noops.