11 ms·

Node.js needs a virtual file system

275 points by voctor 7mo ago

buttsack 7mo ago
[flagged]
andrewmcwatters 7mo ago
[dead]
petcat 7mo ago
Are people still building new projects on Node.js? I would have thought the ecosystem was moving to deno or bun now
- rrr_oh_man 7mo ago
  Why?
- kitsune1 7mo ago
  The delusion in this comment is insane.
- jitl 7mo ago
  loud people on twitter are always switching to the new hotness. i personally can't see myself using bun until its reputation for segfaults goes away after a few more years of stabilizing. deno seems neat and has been around for longer, but its node compatibility story is still evolving; i'm also giving it another year before i try it.
  - _flux 7mo ago
    Wow, I thought you were exaggerating, but no: https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3Aopen%20segfault https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3... Open 80, closed 492.
    - petcat 7mo ago
      That's basically just Zig, right? Re-invented C but only fixed the syntax, not the problems.
      - jitl 7mo ago
        it’s hard to be convinced that zig is an acceptable idea for security critical runtime for web applications.
        _flux 7mo ago
        And as I understand it, most (if not all) of these segfaults were in casual use, not when someone is trying to attack it..
- dzogchen 7mo ago
  I don't really understand what the value proposition of Bun and Deno is. And I see huge problems with their governance and long-term sustainability. Node.js on the other hand is not owned or controlled by one entity. It is not beholden to the whims of investors or a large corporation. I have contributed to Node.js in the past and I was really impressed by its rock-solid governance model and processes. I think this an under-appreciated feature when evaluating tech options.
  - packetlost 7mo ago
    Deno has some pretty nice unique features like sandboxing that, afaik, don't exist in other runtimes (yet). It's enough of a draw that it's the recommended runtime for projects like yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/14404 https://github.com/yt-dlp/yt-dlp/issues/14404
    - worksonmine 7mo ago
      Node has sandboxing these days: https://nodejs.org/api/permissions.html https://nodejs.org/api/permissions.html
      - dzogchen 7mo ago
        No it doesn't, unfortunately. > The permission model implements a "seat belt" approach, which prevents trusted code from unintentionally changing files or using resources that access has not explicitly been granted to. It does not provide security guarantees in the presence of malicious code. Malicious code can bypass the permission model and execute arbitrary code without the restrictions imposed by the permission model. Deno's permissions model is actually a very nice feature. But it is not very granular so I think you end up just allowing everything a lot of the time. I also think sandboxing is a responsibility of the OS. And lastly, a lot of use cases do not really benefit from it (e.g. server applications).
  - zamadatix 7mo ago
    If one gets nothing from them directly, they've at least been a good kick to get several features into Node. It's almost like neovim was to vim, perhaps to a lesser extent.
- zadikian 7mo ago
  Yes people are using Node.js, most likely the majority.
PaulHoule 7mo ago
Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.
- MarleTangible 7mo ago
  The number of files in the node modules folder is crazy, any amount of organization that can tame that chaos is welcomed.
  - koolba 7mo ago
    And if you thought malware hiding in a mess of files was bad, just wait till you see it in two layers of container files.
    - PaulHoule 7mo ago
      Or worse yet, the performance load of anti-malware software that has to look inside ZIP files. Look, most of us realized around 2004 or so that if you had a choice between Norton and the virus you would pick the virus. In the Windows world we standardized around Defender because there is some bound on how much Defender degrades the performance of your machine which was not the case with competitive antivirus software. I've done a few projects which involved getting container file formats like ZIP and PDF (e.g. you know it's a graph of resources in which some of those resources are containers that contain more resources, right?) and now that I think of it you ought to be able to virus scan ZIP files quickly and intelligently but the whole problem with the antivirus industry is that nobody ever considers the cost.
      - ronsor 7mo ago
        Now we'll have to encrypt the files to prevent the performance hit of antivirus peeking inside. Oh, wait...
- fmorel 7mo ago
  I remember when Firefox started putting everything into jars for similar reasons. https://web.archive.org/web/20161003115800/https://blog.mozilla.org/tglek/2010/09/14/firefox-4-jar-jar-jar/ https://web.archive.org/web/20161003115800/https://blog.mozi...
moralestapia 7mo ago
>Let me be honest: a PR that size would normally take months of full-time work. This one happened because I built it with Claude Code. The node.js codebase and standard library has a very high standard of quality, hope that doesn't get washed out by sloppy AI-generated code. OTOH, Matteo is an excellent engineer and the community owes a lot to him. So I guess the code is solid :).
austin-cheney 7mo ago
Most of the 4 justifications mentioned sound like mitigations of otherwise bad design decisions. JavaScript in the browser went down this path for the longest time where new standards were introduced only to solve for stupid people instead of actually introducing new capabilities that were otherwise unachievable. I do see some original benefits to a VFS though, bad application decisions aside, but they are exceedingly minor. As an aside I think JavaScript would benefit from an in-memory database. This would be more of language enhancement than a Node.js enhancement. Imagine the extended application capabilities of an object/array store native to the language that takes queries using JS logic to return one or more objects/records. No SQL language and no third party databases for stuff that you don't want to keep in offline storage on a disk.
- iainmerrick 7mo ago
  Why would you want a language enhancement for that, rather than just writing it in JS code? (or perhaps WASM)
- duped 7mo ago
  > As an aside I think JavaScript would benefit from an in-memory database. isn't that just global state, or do you mean you want that to be persistent?
- dotancohen 7mo ago
  > I think JavaScript would benefit from an in-memory database. That database would probably look a lot like a JSON object. What are you suggesting, that a global JSON object does not solve?
  - austin-cheney 7mo ago
    Whether it is an object, array, something else, or a combination thereof is a design decision. It is not so much about the design of the structure, which should be determined by execution performance considerations, but how information is added, removed and retrieved. Gathering one or more records from a JSON object, or array index, by value of some child property somewhere in a descendant structure of the instance index always feels like a one-off based upon the shape of the data. That could just be a query which is more elegant to read and yet still achieves superior execution performance compared to a bunch of nested loops or string of function array methods. The more structures you have in a given application and the larger those structures become in their schemas the more valuable a uniform storage and retrieval solution becomes.
wccrawford 7mo ago
I'm not convinced that allowing Node to import "code generated at runtime" is actually a good thing. I think it should have to go through the hoops to get loaded, for security reasons. I like the idea of it mocking the file system for tests, but I feel like that should probably be part of the test suite, not Node. The example towards the end that stores data in a sqlite provider and then saves it as a JSON file is mind-boggling to me. Especially for a system that's supposed to be about not saving to the disk. Perhaps it's just a bad example, but I'm really trying to figure out how this isn't just adding complexity.
- TheRealPomax 7mo ago
  But then you go "hang on, doesn't ESM exist?" and you realize that argument 4 isn't even true. You can literally do what this argument says you can't, by creating a blob instead of "writing a temp file" and then importing that using the same dynamic import we've had available since <checks his watch> 2020.
  - notnullorvoid 7mo ago
    There's also a module expression proposal, that would remove the need to use blob imports. https://github.com/tc39/proposal-module-expressions https://github.com/tc39/proposal-module-expressions
  - dfabulich 7mo ago
    A virtual filesystem makes it possible for the ESM you import to statically import other files in the virtual filesystem, which isn't possible by just dynamically importing a blob. Anything your blob module imports has to be updated to dynamically import its dependencies via blobs.
    - apatheticonion 7mo ago
      Correct. Especially painful if you use Worker threads or .node files
- Normal_gaussian 7mo ago
westurner 7mo ago
Is node::vfs the new solution for JupyterLite filesystems? From https://github.com/jupyterlite/jupyterlite/issues/949#issuecomment-1425660192 https://github.com/jupyterlite/jupyterlite/issues/949#issuec... : > Ideally, the virtual filesystem of JupyterLite would be shared with the one from the virtual terminal. emscripten-core/emscripten > "New File System Implementation": https://github.com/emscripten-core/emscripten/issues/15041#issuecomment-1330177957 https://github.com/emscripten-core/emscripten/issues/15041#i... : > [ BrowserFS, isomorphic-git/lightningfs, ] pyodide/pyodide: "Native file system API" #738: https://github.com/pyodide/pyodide/issues/738 https://github.com/pyodide/pyodide/issues/738 re: [Chrome,] Filesystem API : > jupyterlab-git [should work with the same VFS as Jupyter kernels and Terminals] pyodide/pyodide: "ENH Add API for mounting native file system" #2987: https://github.com/pyodide/pyodide/pull/2987 https://github.com/pyodide/pyodide/pull/2987
mg 7mo ago
You can’t import or require() a module that only exists in memory. You can convert it into a data url and import that, can't you?
- doctorpangloss 7mo ago
  [flagged]
- afavour 7mo ago
  What happens to relative imports?
ozlikethewizard 7mo ago
I'm not convinced this needs to be in core Node, but being able to have serverless functions access a file system without providing storage would definitely have some use cases. Had some fun with video processing recently that this would be perfect for.
indutny 7mo ago
Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md
- epolanski 7mo ago
  Do as I say, not as I do. On a more serious note, I think that this will be thoroughly reviewed before it gets merged and Node has an entire security team that overviews these.
  - indutny 7mo ago
    As someone who was a part of the aforementioned security team I'm not sure I'd be interested in reviewing such volume of machine generated code, expecting trap at every corner. The implicit assumption that I observed at many OSS projects I've been involved with is that first time contributions are rarely accepted if they are too large in volume, and "core contributor" designation exists to signal "I put effort into this code, stand by it, and respect everyone's time in reviewing it". The PR in the post violates this social contract.
    - lemagedurage 7mo ago
      [dead]
    - epolanski 7mo ago
      For free, you can decide to do what you want, if it's your job, it's a bit different and you may have to do so, especially considering Collina, is one of the largest contributors of the project and member of the technical committee.
      - exe34 7mo ago
        > if it's your job, it's a bit different and you may have to do so Oh I'd use an llm to generate large amounts of feedback and request changes!
leontloveless 7mo ago
[dead]
Normal_gaussian 7mo ago
yarn pnp is currently broken on Node v25.7+; - https://github.com/yarnpkg/berry/issues/7065 https://github.com/yarnpkg/berry/issues/7065 - https://github.com/nodejs/node/issues/62012 https://github.com/nodejs/node/issues/62012 This is because yarn patches fs in order to introduce virtual file path resolution of modules in the yarn cache (which are zips), which is quite brittle and was broken by a seemingly unrelated change in 25.7. The discussion in issue 62012 is notable - it was suggested yarn just wait for vfs to land. This is interesting to me in two ways: firstly, the node team seems quite happy for non-trivial amounts of the ecosystem to just be broken, and suggests relying on what I'm assuming will be an experimental API when it does land; secondly, it implies a lot of confidence that this feature will land before LTS.
- chrisweekly 7mo ago
  Strong rec to choose PNPM over yarn. I just posted this in a peer comment: https://news.ycombinator.com/item?id=47415173 https://news.ycombinator.com/item?id=47415173 Not spamming, not affiliated, just trying to help others avoid so much needless suffering.
  - zadikian 7mo ago
    I just use npm because I like to stay as vanilla as possible. Glad that alternatives exist though.
    - Normal_gaussian 7mo ago
      This can't be overstated. The main benefit with yarn berry (v4+) is being able to commit the dependencies to the repo - I have yarn based tools that I wrote years ago that just work wheras I frequently find npm and python tools are broken due to version changes. However this benefit comes at a setup cost and a lot more on disk complexity - one off tools are just npm and done.
  - Normal_gaussian 7mo ago
    This is quite spammy; you could mitigate it by explaining what you think the "needless suffering" is. Having been using npm, pnpm, and yarn for many years the only benefit I find with pnpm is a little bit of speed when using the cli, but not enough that I notice; I've outlined the major yarn benefit to me 'in a peer comment' (which I didn't realise was you when I answered) https://news.ycombinator.com/item?id=47415660 https://news.ycombinator.com/item?id=47415660 I expect yarn to have a real competitor sooner rather than later that will replace it; and I do wonder if it is this vfs module that will enable it.
notnullorvoid 7mo ago
I could see something like this being useful if it could be passed to workers to replace any fs access inside the worker.
mohsen1 7mo ago
Yarn, pnpm, webpack all have solutions for this. Great to see this becoming a standard. I have a project that is severely handicapped due to FS. Running 13k tests takes 40 minutes where a virtual file system that Node would just work with it would cut the run time to 3 minutes. I experimented with some hacks and decided to stay with slow but native FS solution. What I really want is a way of swapping FS with VFS in a Node.js program harness. Something like node --use-vfs --vfs-cache=BIG_JSON_FILE So basically Node never touches the disk and load everything from the memory
- Normal_gaussian 7mo ago
  The way to do this today is to do it outside of node. Using an overlay fs with the overlay being a ramfs. You can even chroot into it if you can't scope the paths you need to be just downstream from some directory. Or, just use docker.
  - mohsen1 7mo ago
    making that work cross platform is pure pain
    - Normal_gaussian 7mo ago
      yes and no. Waiting 40mins for every test run is pure pain, platform specific ramfs type mounting is quite scriptable. Yes some devs might need to install a dependency, but its not a complex script.
    - skydhash 7mo ago
      What are the other OS? There's a bunch of solutions described on Wikipedia https://en.wikipedia.org/wiki/List_of_RAM_drive_software https://en.wikipedia.org/wiki/List_of_RAM_drive_software
pier25 7mo ago
The Node team has lost the plot IMO. By far the most critical issue is the over reliance on third party NPM packages for even fundamental needs like connecting to a database.
- afavour 7mo ago
  What would a Node-native database connection layer look like? What other platforms have that? Databases are third party tech, I don’t think it’s unreasonable to use a third party NPM module to connect to them.
  - ksherlock 7mo ago
    Perl has DBI. PHP has PDO.
    - Spivak 7mo ago
      Python has DB-API.
  - mike_hearn 7mo ago
    Most obviously, Java has JDBC. I think .NET has an equivalent. Drivers are needed but they're often first party, coming directly from the DB vendor itself. Java also has a JIT compiling JS engine that can be sandboxed and given a VFS: https://www.graalvm.org/latest/security-guide/sandboxing/ https://www.graalvm.org/latest/security-guide/sandboxing/ N.B. there's a NodeJS compatible mode, but you can't use VFS+sandboxing and NodeJS compatibility together because the NodeJS mode actually uses the real NodeJS codebase, just swapping out V8. For combining it all together you'd want something like https://elide.dev https://elide.dev which reimplemented some of the Node APIs on top of the JVM, so it's sandboxable and virtualizable.
    - LunaSea 7mo ago
      > Most obviously, Java has JDBC. I think .NET has an equivalent. Drivers are needed but they're often first party, coming directly from the DB vendor itself. So it's an external dependency that is not part of Java. It doesn't really matter if the code comes from the vendor or not. Especially for OpenSource databases.
rigorclaw 7mo ago
[flagged]
openinstaclaw 7mo ago
[dead]
adzm 7mo ago
How does electron do this with its packaged files? I suppose it does not work with module resolution?
lacoolj 7mo ago
Using Claude for code you use yourself or at your own company internally is one thing, but when you start injecting it into widely-shared projects like this (or, the linux kernel, or Debian, etc) there will always be a lingering feeling of the project being tainted. Just my opinion, probably not a popular one. But I will be avoiding an upgrade to Node.js after 24.14 for a while if this is becoming an acceptable precedent.
- atomicnumber3 7mo ago
  I still think everyone is trying to run away from the copyright problems with AI, and suspect it's going to come back to bite them. Eventually. (No I'm not willing to bet on exactly when because I'm sure it'll be a lot longer than I'd like).
aplomb1026 7mo ago
[dead]
bronlund 7mo ago
Yeah. That’s what we need. More Node.
- devnotes77 7mo ago
  [flagged]
gwbas1c 7mo ago
Can you dynamically load code via eval? (I know, I know, it's ugly and has its own set of problems)
syrusakbary 7mo ago
[flagged]
- szmarczak 7mo ago
  HN comments isn't a place to advertise your product.
  - szszrk 7mo ago
    I am not so sure about that. I recall multiple posts that start with most upvoted comments from founders... Wonder what Dang says about that.
torginus 7mo ago
Why do people keep reinventing OS features? There's Docker, OverlayFS, FUSE, ZFS or Btrfs snapshots? Do you not trust your OS to do this correctly, or do you think you can do better? A lot of this stuff existed 5, 10, 15 years ago... Somehow there's been a trend for every effing program to grow and absorb the features and responsibilities of every other program. Actually, I have a brilliant idea, what if we used nodejs, and added html display capabilities, and browser features? After all Cursor has already proven you can vibecode a browser, why not just do it? I'm just tired at this point
- ryandrake 7mo ago
  One of the worst is media players that all insist on grafting their own "library" on top of my already-working OS filesystem. So I can't just run the media player and play files. No, that would be too simple. I have to first "import" my media into a "library" abstraction and then store that library somewhere else on my filesystem. Terrible!
  - SAI_Peregrinus 7mo ago
    There's a legitimate problem they're trying to solve there: there are several ways to sort media that don't match up well with a hierarchical filesystem¹. They solve it badly. Good players maintain a database for efficient queries of media metadata, and periodically rescan the folders to update it. Shitty media players try to manage the files themselves, and still end up needing to maintain a database. The worst of these use the database to manage the contents of their storage files (or store the files themselves in the database), if something isn't in the database they delete the files. Adobe Lightroom Classic does this, if your database gets corrupted it deletes all your RAW files! ¹E.g. if you've got music, and it's sorted `artist/album/track<n>.extension`, and two artists collaborate on an album, which one gets the album in their folder? What if you want to sort all songs in the display by publication date? Even if they use the files on your filesystem without moving them, some sort of metadata database will be needed for efficient display & search.
    - eviks 7mo ago
themafia 7mo ago
> You can’t import or require() a module that only exists in memory. Sure you can. Function() exists and require.cache exists. This is _intentionally_ exploitable.
verdverm 7mo ago
Separate the valid critiques on other comments, Go's io.FS interface is really nice for making these sorts of things. Is there something like this in Node already? (with base implementations like host and in memory)
- dabbz 7mo ago
  Another thread about Edge.js has some similar concepts that might be applicable? They're not 1:1 the same but it involves Sandboxing https://news.ycombinator.com/item?id=47417398 https://news.ycombinator.com/item?id=47417398
  - verdverm 7mo ago
    I would put virtual or filesystem abstractions in a different category than sandboxing, which puts restrictions over the virtual or native implementations.
    - dabbz 7mo ago
      I mean that's a fair distinction. There's definitely some overlap depending on needs though.
sidewndr46 7mo ago
Don't all projects eventually grow to encompass service discovery?
gnarbarian 7mo ago
one of the reasons I prefer deno is the availability of indexeddb (and all the other great stuff that comes with it out of the box)
butz 7mo ago
How about trying to reduce dependencies? 11ty is going in correct direction, dropping significant chunk of various dependencies or replacing them with packages with no dependencies or using platform features, that becomes readily available.
- pier25 7mo ago
  Not a priority for the Node team, unfortunately.
deleted 7mo ago
[deleted]
giancarlostoro 7mo ago
> I pointed the AI at the tedious parts, the stuff that makes a 14k-line PR possible but no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage, and generating docs. This is the biggest takeaway for me for AI. It's not even that nobody wants to do these things, its that by the time you finish your tasks, you have no time to do these things, because your manage / scrum master / powers that be want you to work on the next task.
- Lerc 7mo ago
  I think the insight there is that the increased productivity of AI could be used to add features where the end results are weighing the ability of the AI against the ability of an individual implementing the same thing. The alternative is that you work on the same number of features and utilize the ability to make those features as robust as you know they could be, but you have other pressing matters to attend to. That's weighing the ability of AI against the ability of neglect.
- Culonavirus 7mo ago
  That's perfectly understandable. But has no business being in a large open source project, let alone world class one like Node or (god forbid) the Linux kernel. Get that shit the fuck out.
  - potsandpans 7mo ago
    > Get that shit the fuck out. No.
minraws 7mo ago
Why is this not a library what is this insanity??
wei03288 7mo ago
[flagged]
socalgal2 7mo ago
What's special about node.js here? Does golang, C#, python, ruby, java, etc... have a virtual file system? I get it, I've implemented things for tests, I'm just wondering if this shouldn't be solved at an OS level. --- update Let's put this another way, my code does effectively, child_process.spawn('something-that-reads-and-write-a-file') now I'm back to the same issue. To test I need a virtual file system. Node providing one won't help.
- sauercrowd 7mo ago
  Go actually does https://pkg.go.dev/embed https://pkg.go.dev/embed I do think it's more painful to distribute files when you're a distributed as a single binary vs scripts, since the latter has to figure out bundling of files anyway. But still - it does exist
  - benatkin 7mo ago
    Embed is read-only at runtime. This proposed vfs module for Node.js is a full virtual file system.
    - sauercrowd 7mo ago
      true
      - benatkin 7mo ago
        It's cool that it fits into golang's readable file system interface so it can be used polymorphically. I don't know if golang has very complete interfaces for a read and write file system that could be used for a full vfs. If it does, that's nice, and a starting point for a similar vfs! I'm also not sure whether it should go into the standard library or not.
  - nine_k 7mo ago
    Zip files are created in such a way that they can be a part of an executable file. (This is how self-extracting archives used to work.) Support for reading zip files is lightweight, and is present almost everywhere. A ZIP fork embedded into the executable should be an obvious read-only VFS implementation. Bring your assets with you, even maybe build them with the standard zip utility. It should take relatively few LOCs, provided that libzip is already linked into the executable anyway.
AgentMarket 7mo ago
[flagged]
cmrx64 7mo ago
this is a pretty bad vfs. there are pure “cap manifest” approaches that don’t pull in decades of cruft semantics. don’t build systems that aren’t objectstore native in 2025 (since this work was initiated in december).
il-b 7mo ago
> no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage That’s so dehumanizing, I would happily write such code.
iam_circuit 7mo ago
[dead]
chmod775 7mo ago
Yes, but no. Node itself merely needs a standardized, pluggable layer of indirection in its file APIs. If someone wants to implement a VFS using that, that's cool. Basically an "fs-core" that everything ultimately goes through, and which can be switched out/layered with another implementation. Think express-style routing but for the filesystem. That'll keep things simple in node's codebase while handing more power to users.
keepamovin 7mo ago
The way I bundle into SEA is modules that need to be imported from disk (that can't be bundled due to node or wasm modules), is just include them in the assets, and do a "write to tmp, import, delete" flow. It works. Not saying vfs is bad, just it's not impossible in a few lines of code to set up that. My idea for a simple version of a vfs in node is to use a RAM disk/RAMfs - would that work?
philo23 7mo ago
Little bit saddened the sqlite provider doesn't use the SQLite archive format under the hood. Seems like it'd be a good fit for what they're trying to achieve + give you an easy way to create/extract the files out of the virtual file system. The sqlar schema is missing some of the info thats being stored atm, but there's nothing stopping you from adding your own fields/tables on top of the format, if anything the docs encourage it. It is just a sqlite database at the end of the day. https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
AndyKelley 7mo ago
I used Node.js extensively for about a decade, both professionally and as a hobby. I don't buy the problem statement. These arguments don't even make sense, they look LLM generated. I can't even formulate a disagreement against this nonsense.
est 7mo ago
This brings back memories back in the days I mess around Resource Hacker for Win32 EXEs. I miss those days where you can tweak all kinds of software GUI by your self. Change icons, menus, shortcut keys, etc.
AntonCTO 7mo ago
It's not Node.js that needs a virtual file system. It's JavaScript.
AgentNode 7mo ago
[flagged]
huksley 7mo ago
There is already memfs package which implements virtual fs, with other packages as well. What we need is to support import/require working with that vfs.
ThomIves 7mo ago
I loved this notion from just the title, and then eagerly read the post. BRAVO! I am investigating if I can, and how to, exploit this ASAP.

275 points by voctor 7mo ago

buttsack 7mo ago

[flagged]

andrewmcwatters 7mo ago

[dead]

petcat 7mo ago

Are people still building new projects on Node.js? I would have thought the ecosystem was moving to deno or bun now

rrr_oh_man 7mo ago
Why?
kitsune1 7mo ago
The delusion in this comment is insane.
jitl 7mo ago
loud people on twitter are always switching to the new hotness. i personally can't see myself using bun until its reputation for segfaults goes away after a few more years of stabilizing. deno seems neat and has been around for longer, but its node compatibility story is still evolving; i'm also giving it another year before i try it.
- _flux 7mo ago
  Wow, I thought you were exaggerating, but no: https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3Aopen%20segfault https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3... Open 80, closed 492.
  - petcat 7mo ago
    That's basically just Zig, right? Re-invented C but only fixed the syntax, not the problems.
    - jitl 7mo ago
      it’s hard to be convinced that zig is an acceptable idea for security critical runtime for web applications.
      - _flux 7mo ago
        And as I understand it, most (if not all) of these segfaults were in casual use, not when someone is trying to attack it..
dzogchen 7mo ago
I don't really understand what the value proposition of Bun and Deno is. And I see huge problems with their governance and long-term sustainability. Node.js on the other hand is not owned or controlled by one entity. It is not beholden to the whims of investors or a large corporation. I have contributed to Node.js in the past and I was really impressed by its rock-solid governance model and processes. I think this an under-appreciated feature when evaluating tech options.
- packetlost 7mo ago
  Deno has some pretty nice unique features like sandboxing that, afaik, don't exist in other runtimes (yet). It's enough of a draw that it's the recommended runtime for projects like yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/14404 https://github.com/yt-dlp/yt-dlp/issues/14404
  - worksonmine 7mo ago
    Node has sandboxing these days: https://nodejs.org/api/permissions.html https://nodejs.org/api/permissions.html
    - dzogchen 7mo ago
      No it doesn't, unfortunately. > The permission model implements a "seat belt" approach, which prevents trusted code from unintentionally changing files or using resources that access has not explicitly been granted to. It does not provide security guarantees in the presence of malicious code. Malicious code can bypass the permission model and execute arbitrary code without the restrictions imposed by the permission model. Deno's permissions model is actually a very nice feature. But it is not very granular so I think you end up just allowing everything a lot of the time. I also think sandboxing is a responsibility of the OS. And lastly, a lot of use cases do not really benefit from it (e.g. server applications).
- zamadatix 7mo ago
  If one gets nothing from them directly, they've at least been a good kick to get several features into Node. It's almost like neovim was to vim, perhaps to a lesser extent.
zadikian 7mo ago
Yes people are using Node.js, most likely the majority.

PaulHoule 7mo ago

Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.

MarleTangible 7mo ago
The number of files in the node modules folder is crazy, any amount of organization that can tame that chaos is welcomed.
- koolba 7mo ago
  And if you thought malware hiding in a mess of files was bad, just wait till you see it in two layers of container files.
  - PaulHoule 7mo ago
    Or worse yet, the performance load of anti-malware software that has to look inside ZIP files. Look, most of us realized around 2004 or so that if you had a choice between Norton and the virus you would pick the virus. In the Windows world we standardized around Defender because there is some bound on how much Defender degrades the performance of your machine which was not the case with competitive antivirus software. I've done a few projects which involved getting container file formats like ZIP and PDF (e.g. you know it's a graph of resources in which some of those resources are containers that contain more resources, right?) and now that I think of it you ought to be able to virus scan ZIP files quickly and intelligently but the whole problem with the antivirus industry is that nobody ever considers the cost.
    - ronsor 7mo ago
      Now we'll have to encrypt the files to prevent the performance hit of antivirus peeking inside. Oh, wait...
fmorel 7mo ago
I remember when Firefox started putting everything into jars for similar reasons. https://web.archive.org/web/20161003115800/https://blog.mozilla.org/tglek/2010/09/14/firefox-4-jar-jar-jar/ https://web.archive.org/web/20161003115800/https://blog.mozi...

moralestapia 7mo ago

>Let me be honest: a PR that size would normally take months of full-time work. This one happened because I built it with Claude Code. The node.js codebase and standard library has a very high standard of quality, hope that doesn't get washed out by sloppy AI-generated code. OTOH, Matteo is an excellent engineer and the community owes a lot to him. So I guess the code is solid :).

austin-cheney 7mo ago

Most of the 4 justifications mentioned sound like mitigations of otherwise bad design decisions. JavaScript in the browser went down this path for the longest time where new standards were introduced only to solve for stupid people instead of actually introducing new capabilities that were otherwise unachievable. I do see some original benefits to a VFS though, bad application decisions aside, but they are exceedingly minor. As an aside I think JavaScript would benefit from an in-memory database. This would be more of language enhancement than a Node.js enhancement. Imagine the extended application capabilities of an object/array store native to the language that takes queries using JS logic to return one or more objects/records. No SQL language and no third party databases for stuff that you don't want to keep in offline storage on a disk.

iainmerrick 7mo ago
Why would you want a language enhancement for that, rather than just writing it in JS code? (or perhaps WASM)
duped 7mo ago
> As an aside I think JavaScript would benefit from an in-memory database. isn't that just global state, or do you mean you want that to be persistent?
dotancohen 7mo ago
> I think JavaScript would benefit from an in-memory database. That database would probably look a lot like a JSON object. What are you suggesting, that a global JSON object does not solve?
- austin-cheney 7mo ago
  Whether it is an object, array, something else, or a combination thereof is a design decision. It is not so much about the design of the structure, which should be determined by execution performance considerations, but how information is added, removed and retrieved. Gathering one or more records from a JSON object, or array index, by value of some child property somewhere in a descendant structure of the instance index always feels like a one-off based upon the shape of the data. That could just be a query which is more elegant to read and yet still achieves superior execution performance compared to a bunch of nested loops or string of function array methods. The more structures you have in a given application and the larger those structures become in their schemas the more valuable a uniform storage and retrieval solution becomes.

wccrawford 7mo ago

I'm not convinced that allowing Node to import "code generated at runtime" is actually a good thing. I think it should have to go through the hoops to get loaded, for security reasons. I like the idea of it mocking the file system for tests, but I feel like that should probably be part of the test suite, not Node. The example towards the end that stores data in a sqlite provider and then saves it as a JSON file is mind-boggling to me. Especially for a system that's supposed to be about not saving to the disk. Perhaps it's just a bad example, but I'm really trying to figure out how this isn't just adding complexity.

TheRealPomax 7mo ago
But then you go "hang on, doesn't ESM exist?" and you realize that argument 4 isn't even true. You can literally do what this argument says you can't, by creating a blob instead of "writing a temp file" and then importing that using the same dynamic import we've had available since <checks his watch> 2020.
- notnullorvoid 7mo ago
  There's also a module expression proposal, that would remove the need to use blob imports. https://github.com/tc39/proposal-module-expressions https://github.com/tc39/proposal-module-expressions
- dfabulich 7mo ago
  A virtual filesystem makes it possible for the ESM you import to statically import other files in the virtual filesystem, which isn't possible by just dynamically importing a blob. Anything your blob module imports has to be updated to dynamically import its dependencies via blobs.
  - apatheticonion 7mo ago
    Correct. Especially painful if you use Worker threads or .node files
Normal_gaussian 7mo ago

westurner 7mo ago

Is node::vfs the new solution for JupyterLite filesystems? From https://github.com/jupyterlite/jupyterlite/issues/949#issuecomment-1425660192 https://github.com/jupyterlite/jupyterlite/issues/949#issuec... : > Ideally, the virtual filesystem of JupyterLite would be shared with the one from the virtual terminal. emscripten-core/emscripten > "New File System Implementation": https://github.com/emscripten-core/emscripten/issues/15041#issuecomment-1330177957 https://github.com/emscripten-core/emscripten/issues/15041#i... : > [ BrowserFS, isomorphic-git/lightningfs, ] pyodide/pyodide: "Native file system API" #738: https://github.com/pyodide/pyodide/issues/738 https://github.com/pyodide/pyodide/issues/738 re: [Chrome,] Filesystem API : > jupyterlab-git [should work with the same VFS as Jupyter kernels and Terminals] pyodide/pyodide: "ENH Add API for mounting native file system" #2987: https://github.com/pyodide/pyodide/pull/2987 https://github.com/pyodide/pyodide/pull/2987

mg 7mo ago

You can’t import or require() a module that only exists in memory. You can convert it into a data url and import that, can't you?

doctorpangloss 7mo ago
[flagged]
afavour 7mo ago
What happens to relative imports?

ozlikethewizard 7mo ago

I'm not convinced this needs to be in core Node, but being able to have serverless functions access a file system without providing storage would definitely have some use cases. Had some fun with video processing recently that this would be perfect for.

indutny 7mo ago

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

epolanski 7mo ago
Do as I say, not as I do. On a more serious note, I think that this will be thoroughly reviewed before it gets merged and Node has an entire security team that overviews these.
- indutny 7mo ago
  As someone who was a part of the aforementioned security team I'm not sure I'd be interested in reviewing such volume of machine generated code, expecting trap at every corner. The implicit assumption that I observed at many OSS projects I've been involved with is that first time contributions are rarely accepted if they are too large in volume, and "core contributor" designation exists to signal "I put effort into this code, stand by it, and respect everyone's time in reviewing it". The PR in the post violates this social contract.
  - lemagedurage 7mo ago
    [dead]
  - epolanski 7mo ago
    For free, you can decide to do what you want, if it's your job, it's a bit different and you may have to do so, especially considering Collina, is one of the largest contributors of the project and member of the technical committee.
    - exe34 7mo ago
      > if it's your job, it's a bit different and you may have to do so Oh I'd use an llm to generate large amounts of feedback and request changes!

leontloveless 7mo ago

[dead]

Normal_gaussian 7mo ago

yarn pnp is currently broken on Node v25.7+; - https://github.com/yarnpkg/berry/issues/7065 https://github.com/yarnpkg/berry/issues/7065 - https://github.com/nodejs/node/issues/62012 https://github.com/nodejs/node/issues/62012 This is because yarn patches fs in order to introduce virtual file path resolution of modules in the yarn cache (which are zips), which is quite brittle and was broken by a seemingly unrelated change in 25.7. The discussion in issue 62012 is notable - it was suggested yarn just wait for vfs to land. This is interesting to me in two ways: firstly, the node team seems quite happy for non-trivial amounts of the ecosystem to just be broken, and suggests relying on what I'm assuming will be an experimental API when it does land; secondly, it implies a lot of confidence that this feature will land before LTS.

chrisweekly 7mo ago
Strong rec to choose PNPM over yarn. I just posted this in a peer comment: https://news.ycombinator.com/item?id=47415173 https://news.ycombinator.com/item?id=47415173 Not spamming, not affiliated, just trying to help others avoid so much needless suffering.
- zadikian 7mo ago
  I just use npm because I like to stay as vanilla as possible. Glad that alternatives exist though.
  - Normal_gaussian 7mo ago
    This can't be overstated. The main benefit with yarn berry (v4+) is being able to commit the dependencies to the repo - I have yarn based tools that I wrote years ago that just work wheras I frequently find npm and python tools are broken due to version changes. However this benefit comes at a setup cost and a lot more on disk complexity - one off tools are just npm and done.
- Normal_gaussian 7mo ago
  This is quite spammy; you could mitigate it by explaining what you think the "needless suffering" is. Having been using npm, pnpm, and yarn for many years the only benefit I find with pnpm is a little bit of speed when using the cli, but not enough that I notice; I've outlined the major yarn benefit to me 'in a peer comment' (which I didn't realise was you when I answered) https://news.ycombinator.com/item?id=47415660 https://news.ycombinator.com/item?id=47415660 I expect yarn to have a real competitor sooner rather than later that will replace it; and I do wonder if it is this vfs module that will enable it.

notnullorvoid 7mo ago

I could see something like this being useful if it could be passed to workers to replace any fs access inside the worker.

mohsen1 7mo ago

Yarn, pnpm, webpack all have solutions for this. Great to see this becoming a standard. I have a project that is severely handicapped due to FS. Running 13k tests takes 40 minutes where a virtual file system that Node would just work with it would cut the run time to 3 minutes. I experimented with some hacks and decided to stay with slow but native FS solution. What I really want is a way of swapping FS with VFS in a Node.js program harness. Something like node --use-vfs --vfs-cache=BIG_JSON_FILE So basically Node never touches the disk and load everything from the memory

Normal_gaussian 7mo ago
The way to do this today is to do it outside of node. Using an overlay fs with the overlay being a ramfs. You can even chroot into it if you can't scope the paths you need to be just downstream from some directory. Or, just use docker.
- mohsen1 7mo ago
  making that work cross platform is pure pain
  - Normal_gaussian 7mo ago
    yes and no. Waiting 40mins for every test run is pure pain, platform specific ramfs type mounting is quite scriptable. Yes some devs might need to install a dependency, but its not a complex script.
  - skydhash 7mo ago
    What are the other OS? There's a bunch of solutions described on Wikipedia https://en.wikipedia.org/wiki/List_of_RAM_drive_software https://en.wikipedia.org/wiki/List_of_RAM_drive_software

pier25 7mo ago

The Node team has lost the plot IMO. By far the most critical issue is the over reliance on third party NPM packages for even fundamental needs like connecting to a database.

afavour 7mo ago
What would a Node-native database connection layer look like? What other platforms have that? Databases are third party tech, I don’t think it’s unreasonable to use a third party NPM module to connect to them.
- ksherlock 7mo ago
  Perl has DBI. PHP has PDO.
  - Spivak 7mo ago
    Python has DB-API.
- mike_hearn 7mo ago
  Most obviously, Java has JDBC. I think .NET has an equivalent. Drivers are needed but they're often first party, coming directly from the DB vendor itself. Java also has a JIT compiling JS engine that can be sandboxed and given a VFS: https://www.graalvm.org/latest/security-guide/sandboxing/ https://www.graalvm.org/latest/security-guide/sandboxing/ N.B. there's a NodeJS compatible mode, but you can't use VFS+sandboxing and NodeJS compatibility together because the NodeJS mode actually uses the real NodeJS codebase, just swapping out V8. For combining it all together you'd want something like https://elide.dev https://elide.dev which reimplemented some of the Node APIs on top of the JVM, so it's sandboxable and virtualizable.
  - LunaSea 7mo ago
    > Most obviously, Java has JDBC. I think .NET has an equivalent. Drivers are needed but they're often first party, coming directly from the DB vendor itself. So it's an external dependency that is not part of Java. It doesn't really matter if the code comes from the vendor or not. Especially for OpenSource databases.

rigorclaw 7mo ago

[flagged]

openinstaclaw 7mo ago

[dead]

adzm 7mo ago

How does electron do this with its packaged files? I suppose it does not work with module resolution?

lacoolj 7mo ago

Using Claude for code you use yourself or at your own company internally is one thing, but when you start injecting it into widely-shared projects like this (or, the linux kernel, or Debian, etc) there will always be a lingering feeling of the project being tainted. Just my opinion, probably not a popular one. But I will be avoiding an upgrade to Node.js after 24.14 for a while if this is becoming an acceptable precedent.

atomicnumber3 7mo ago
I still think everyone is trying to run away from the copyright problems with AI, and suspect it's going to come back to bite them. Eventually. (No I'm not willing to bet on exactly when because I'm sure it'll be a lot longer than I'd like).

aplomb1026 7mo ago

[dead]

bronlund 7mo ago

Yeah. That’s what we need. More Node.

devnotes77 7mo ago
[flagged]

gwbas1c 7mo ago

Can you dynamically load code via eval? (I know, I know, it's ugly and has its own set of problems)

syrusakbary 7mo ago

[flagged]

szmarczak 7mo ago
HN comments isn't a place to advertise your product.
- szszrk 7mo ago
  I am not so sure about that. I recall multiple posts that start with most upvoted comments from founders... Wonder what Dang says about that.

torginus 7mo ago

Why do people keep reinventing OS features? There's Docker, OverlayFS, FUSE, ZFS or Btrfs snapshots? Do you not trust your OS to do this correctly, or do you think you can do better? A lot of this stuff existed 5, 10, 15 years ago... Somehow there's been a trend for every effing program to grow and absorb the features and responsibilities of every other program. Actually, I have a brilliant idea, what if we used nodejs, and added html display capabilities, and browser features? After all Cursor has already proven you can vibecode a browser, why not just do it? I'm just tired at this point

ryandrake 7mo ago
One of the worst is media players that all insist on grafting their own "library" on top of my already-working OS filesystem. So I can't just run the media player and play files. No, that would be too simple. I have to first "import" my media into a "library" abstraction and then store that library somewhere else on my filesystem. Terrible!
- SAI_Peregrinus 7mo ago
  There's a legitimate problem they're trying to solve there: there are several ways to sort media that don't match up well with a hierarchical filesystem¹. They solve it badly. Good players maintain a database for efficient queries of media metadata, and periodically rescan the folders to update it. Shitty media players try to manage the files themselves, and still end up needing to maintain a database. The worst of these use the database to manage the contents of their storage files (or store the files themselves in the database), if something isn't in the database they delete the files. Adobe Lightroom Classic does this, if your database gets corrupted it deletes all your RAW files! ¹E.g. if you've got music, and it's sorted `artist/album/track<n>.extension`, and two artists collaborate on an album, which one gets the album in their folder? What if you want to sort all songs in the display by publication date? Even if they use the files on your filesystem without moving them, some sort of metadata database will be needed for efficient display & search.
  - eviks 7mo ago

themafia 7mo ago

> You can’t import or require() a module that only exists in memory. Sure you can. Function() exists and require.cache exists. This is _intentionally_ exploitable.

verdverm 7mo ago

Separate the valid critiques on other comments, Go's io.FS interface is really nice for making these sorts of things. Is there something like this in Node already? (with base implementations like host and in memory)

dabbz 7mo ago
Another thread about Edge.js has some similar concepts that might be applicable? They're not 1:1 the same but it involves Sandboxing https://news.ycombinator.com/item?id=47417398 https://news.ycombinator.com/item?id=47417398
- verdverm 7mo ago
  I would put virtual or filesystem abstractions in a different category than sandboxing, which puts restrictions over the virtual or native implementations.
  - dabbz 7mo ago
    I mean that's a fair distinction. There's definitely some overlap depending on needs though.

sidewndr46 7mo ago

Don't all projects eventually grow to encompass service discovery?

gnarbarian 7mo ago

one of the reasons I prefer deno is the availability of indexeddb (and all the other great stuff that comes with it out of the box)

butz 7mo ago

How about trying to reduce dependencies? 11ty is going in correct direction, dropping significant chunk of various dependencies or replacing them with packages with no dependencies or using platform features, that becomes readily available.

pier25 7mo ago
Not a priority for the Node team, unfortunately.

deleted 7mo ago

[deleted]

giancarlostoro 7mo ago

> I pointed the AI at the tedious parts, the stuff that makes a 14k-line PR possible but no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage, and generating docs. This is the biggest takeaway for me for AI. It's not even that nobody wants to do these things, its that by the time you finish your tasks, you have no time to do these things, because your manage / scrum master / powers that be want you to work on the next task.

Lerc 7mo ago
I think the insight there is that the increased productivity of AI could be used to add features where the end results are weighing the ability of the AI against the ability of an individual implementing the same thing. The alternative is that you work on the same number of features and utilize the ability to make those features as robust as you know they could be, but you have other pressing matters to attend to. That's weighing the ability of AI against the ability of neglect.
Culonavirus 7mo ago
That's perfectly understandable. But has no business being in a large open source project, let alone world class one like Node or (god forbid) the Linux kernel. Get that shit the fuck out.
- potsandpans 7mo ago
  > Get that shit the fuck out. No.

minraws 7mo ago

Why is this not a library what is this insanity??

wei03288 7mo ago

[flagged]

socalgal2 7mo ago

What's special about node.js here? Does golang, C#, python, ruby, java, etc... have a virtual file system? I get it, I've implemented things for tests, I'm just wondering if this shouldn't be solved at an OS level. --- update Let's put this another way, my code does effectively, child_process.spawn('something-that-reads-and-write-a-file') now I'm back to the same issue. To test I need a virtual file system. Node providing one won't help.

sauercrowd 7mo ago
Go actually does https://pkg.go.dev/embed https://pkg.go.dev/embed I do think it's more painful to distribute files when you're a distributed as a single binary vs scripts, since the latter has to figure out bundling of files anyway. But still - it does exist
- benatkin 7mo ago
  Embed is read-only at runtime. This proposed vfs module for Node.js is a full virtual file system.
  - sauercrowd 7mo ago
    true
    - benatkin 7mo ago
      It's cool that it fits into golang's readable file system interface so it can be used polymorphically. I don't know if golang has very complete interfaces for a read and write file system that could be used for a full vfs. If it does, that's nice, and a starting point for a similar vfs! I'm also not sure whether it should go into the standard library or not.
- nine_k 7mo ago
  Zip files are created in such a way that they can be a part of an executable file. (This is how self-extracting archives used to work.) Support for reading zip files is lightweight, and is present almost everywhere. A ZIP fork embedded into the executable should be an obvious read-only VFS implementation. Bring your assets with you, even maybe build them with the standard zip utility. It should take relatively few LOCs, provided that libzip is already linked into the executable anyway.

AgentMarket 7mo ago

[flagged]

cmrx64 7mo ago

this is a pretty bad vfs. there are pure “cap manifest” approaches that don’t pull in decades of cruft semantics. don’t build systems that aren’t objectstore native in 2025 (since this work was initiated in december).

il-b 7mo ago

> no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage That’s so dehumanizing, I would happily write such code.

iam_circuit 7mo ago

[dead]

chmod775 7mo ago

Yes, but no. Node itself merely needs a standardized, pluggable layer of indirection in its file APIs. If someone wants to implement a VFS using that, that's cool. Basically an "fs-core" that everything ultimately goes through, and which can be switched out/layered with another implementation. Think express-style routing but for the filesystem. That'll keep things simple in node's codebase while handing more power to users.

keepamovin 7mo ago

The way I bundle into SEA is modules that need to be imported from disk (that can't be bundled due to node or wasm modules), is just include them in the assets, and do a "write to tmp, import, delete" flow. It works. Not saying vfs is bad, just it's not impossible in a few lines of code to set up that. My idea for a simple version of a vfs in node is to use a RAM disk/RAMfs - would that work?

philo23 7mo ago

Little bit saddened the sqlite provider doesn't use the SQLite archive format under the hood. Seems like it'd be a good fit for what they're trying to achieve + give you an easy way to create/extract the files out of the virtual file system. The sqlar schema is missing some of the info thats being stored atm, but there's nothing stopping you from adding your own fields/tables on top of the format, if anything the docs encourage it. It is just a sqlite database at the end of the day. https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html

AndyKelley 7mo ago

I used Node.js extensively for about a decade, both professionally and as a hobby. I don't buy the problem statement. These arguments don't even make sense, they look LLM generated. I can't even formulate a disagreement against this nonsense.

est 7mo ago

This brings back memories back in the days I mess around Resource Hacker for Win32 EXEs. I miss those days where you can tweak all kinds of software GUI by your self. Change icons, menus, shortcut keys, etc.

AntonCTO 7mo ago

It's not Node.js that needs a virtual file system. It's JavaScript.

AgentNode 7mo ago

[flagged]

huksley 7mo ago

There is already memfs package which implements virtual fs, with other packages as well. What we need is to support import/require working with that vfs.

ThomIves 7mo ago

I loved this notion from just the title, and then eagerly read the post. BRAVO! I am investigating if I can, and how to, exploit this ASAP.