3 ms·

I wondered the other day how things would behave in node if dependencies were in node_modules.sqlite3 (with posibility to eject to edit if needed).

Already__Taken 5y ago
Shouldn't it be possible to make an sql structure and mount it like an FS to find out?
- nightfly 5y ago
  That would be the worst of both worlds: The overhead of storing your data in SQL, plus the overhead of filesystem search and access. The better way to do this would be to have node access the modules in SQLite directly.
  - naikrovek 5y ago
    filesystem search and access is different when you're talking a single larger file versus multiple smaller files. All filesystems have a per-file overhead that would effectively be eliminated if you could pack all of your files into a database structure. Indices would also speed up access to individual rows of the table significantly. There is overhead in small file storage anyway, if the files are not the exact size, or a multiple, of the sector size. storing 1kb files on a disk where the sector size is 16kb is far more of an overhead expense than storing those files in an SQL database.
- mirekrusin 5y ago
  I was thinking more about using ESM loader hooks [0] directly. [0] https://nodejs.org/api/esm.html#esm_loaders https://nodejs.org/api/esm.html#esm_loaders
lukevp 5y ago
Well pnpm centralizes it so that you’re only referencing a single location via symlinks and that is a major speed up, I think moving dependencies out of the file system altogether would be nice. I’ve explored the possibility of this, but I think the way snowpack does streaming imports[1] in version 3.0 may be the best solution overall. I haven’t spent much time testing it but it appears to be a major process improvement. [1] https://michalkuncio.com/how-to-use-snowpack-without-node-modules/ https://michalkuncio.com/how-to-use-snowpack-without-node-mo...
- mirekrusin 5y ago
  Snowpack is great for f/e, what I was thinking more about is millions of files after npm i everywhere on b/e - on your local dev machine, on ci, in docker image etc.

Already__Taken 5y ago

Shouldn't it be possible to make an sql structure and mount it like an FS to find out?

nightfly 5y ago
That would be the worst of both worlds: The overhead of storing your data in SQL, plus the overhead of filesystem search and access. The better way to do this would be to have node access the modules in SQLite directly.
- naikrovek 5y ago
  filesystem search and access is different when you're talking a single larger file versus multiple smaller files. All filesystems have a per-file overhead that would effectively be eliminated if you could pack all of your files into a database structure. Indices would also speed up access to individual rows of the table significantly. There is overhead in small file storage anyway, if the files are not the exact size, or a multiple, of the sector size. storing 1kb files on a disk where the sector size is 16kb is far more of an overhead expense than storing those files in an SQL database.
mirekrusin 5y ago
I was thinking more about using ESM loader hooks [0] directly. [0] https://nodejs.org/api/esm.html#esm_loaders https://nodejs.org/api/esm.html#esm_loaders

lukevp 5y ago

Well pnpm centralizes it so that you’re only referencing a single location via symlinks and that is a major speed up, I think moving dependencies out of the file system altogether would be nice. I’ve explored the possibility of this, but I think the way snowpack does streaming imports[1] in version 3.0 may be the best solution overall. I haven’t spent much time testing it but it appears to be a major process improvement. [1] https://michalkuncio.com/how-to-use-snowpack-without-node-modules/ https://michalkuncio.com/how-to-use-snowpack-without-node-mo...

mirekrusin 5y ago
Snowpack is great for f/e, what I was thinking more about is millions of files after npm i everywhere on b/e - on your local dev machine, on ci, in docker image etc.