8 ms·

Behind the scenes of Bun Install

431 points by Bogdanp 1y ago

blizdiddy 1y ago
I used bun for the first time last week. It was awesome! The built-in server and SQLite meant i didn’t need any dependencies besides bun itself, which is certainly my favorite way to develop. I do almost all of my development in vanilla js despite loathing the node ecosystem, so i really should have checked it out sooner.
- k__ 1y ago
  I tried using Bun a few times, and I really liked working with it. Much better than Node. However...! I always managed to hit a road block with Bun and had to go back to Node. First it was the crypto module that wasn't compatible with Nodejs signatures (now fixed), next Playwright refused working with Bun (via Crawlee).
  - Cthulhu_ 1y ago
    I think this is the big one that slows adoption of "better" / "faster" tooling down, that is, backwards compatibility and drop-in-replacement-ability. Probably a lot of Hyrum's Law.
  - koakuma-chan 1y ago
    You can use Bun as package manager only. You don't have to use Bun as runtime.
    - winterrdog 1y ago
      Sure? Does it work if I have packages that have nodejs c++ addons?
      - abejfehr 1y ago
        Why wouldn’t it? The end result of a npm install or a bun install is that the node_modules folder is structured in the way it needs to be, and I think it can run node-gyp for the packages that need it.
    - iansinnott 1y ago
      Indeed! also as a test runner/lib if you're not doing browser automation. bun definitely has benefits even if not used as a runtime.
      - koakuma-chan 1y ago
        I believe Playwright worked for me with the latest Bun though
  - jherdman 1y ago
    Storybook is another for me.
  - drewbitt 1y ago
    Deno doesn't work with crawlee either unfortunately
  - petralithic 1y ago
    You should try Deno, they have good Node compatibility
    - erikpukinskis 1y ago
      Does it? Last I tried, several years ago, coverage of the Node APIs was not good. I wanted to send data over UDP and a lot of Node basics there were missing.
      - spartanatreyu 1y ago
        Deno's node compat is way better now. They're still missing niche things and they tend to target the things that most devs (and their dependencies) are actually using. But I can see they have it in their compat stuff now and it looks like it's working in the repl locally: https://docs.deno.com/api/node/dgram/ https://docs.deno.com/api/node/dgram/
        rererereferred 1y ago
        This page tracks Deno's compatibility with Node by running its tests: https://ffmathy.github.io/is-deno-compatible-yet/ https://ffmathy.github.io/is-deno-compatible-yet/
  - epolanski 1y ago
    Playwright has been fixed one year ago I think.
  - Jarred 1y ago
    Playwright support will improve soon. We are rewriting node:http’s client implementation to pass node’s test suite. Expecting that to land next week.
    - ukblewis 1y ago
      That may be just the best way to drop that you are introducing a rewrite which fixes a set of bugs that affect users
- simantel 1y ago
  Node also has a built-in server and SQLite these days though? Or if you want a lot more functionality with just one dependency, Hono is great.
  - blizdiddy 1y ago
    And how many dependencies does Hono have? Looks like about 26. And how many dependencies do those have? A single static zig executable isn’t the same as a a pipeline of package management dependencies susceptible to supply chain attacks and the worst bitrot we’ve had since the DOS era.
    - bakkoting 1y ago
      > And how many dependencies does Hono have? Zero. I'm guessing you're looking at the `devDependencies` in its package.json, but those are only used by the people building the project, not by people merely consuming it.
      - PxldLtd 1y ago
        That doesn't prevent supply chain attacks. Dev dependencies are still software dependencies and add a certain level of risk.
        bakkoting 1y ago
        Only in the sense that any other software on the developers' machines adds a certain level of risk.
        arcfour 1y ago
        This is needlessly pedantic unless you are writing from an OS, browser, etc. that you wrote entirely by yourself, without using an editor or linter or compiler not written by you, in which case I tip my cap to you.
jcmartinezdev 1y ago
[dead]
rs_rs_rs_rs_rs 1y ago
Python has uv, JS has bun, what does Ruby or PHP have? Are the devs using those languages happy with how fast the current popular dependency managers are?
- tommasoamici 1y ago
  it's pretty new, but in Ruby there's `rv` which is clearly inspired by `uv`: https://github.com/spinel-coop/rv https://github.com/spinel-coop/rv. >Brought to you by Spinel >Spinel.coop is a collective of Ruby open source maintainers building next-generation developer tooling, like rv, and offering flat-rate, unlimited access to maintainers who come from the core teams of Rails, Hotwire, Bundler, RubyGems, rbenv, and more.
- deleted 1y ago
  [deleted]
- JamesSwift 1y ago
  Youre looking at it wrong. Python has nix, JS has nix, ruby and php have nix : D Thats closer to how pnpm achieves speed up though. I know there is 'rv' recently, but havent tried it.
  - koakuma-chan 1y ago
    You mean nix the package manager? I used to use NixOS and I had to switch off because of endless mess with environment variables.
    - JamesSwift 1y ago
      Yes, nix package manager. Or devenv for a more streamlined version of what I'm describing, similar to mise but powered by nix.
- hu3 1y ago
  PHP is getting Mago (written in Rust). Repo: https://github.com/carthage-software/mago https://github.com/carthage-software/mago Announcement 9 months ago: https://www.reddit.com/r/PHP/comments/1h9zh83/announcing_mago_an_oxidized_toolchain_for_php/ https://www.reddit.com/r/PHP/comments/1h9zh83/announcing_mag... For now its main features are 3: formatting, linting and fixing lint issues. I hope they add package management to do what composer does.
aleyan 1y ago
I have been excited about bun for about a year, and I thought that 2025 is going to be its breakout year. It is really surprising to me that it is not more popular. I scanned top 100k repos on GitHub, and for new repos in 2025, npm is 35 times more popular and pnpm is 11 time more popular than bun [0][1]. The other up and coming javascript runtime, deno is not so popular either. I wonder why that is? Is it because it is a runtime, and getting compatibility there is harder than just for a straight package manager? Can someone who tried bun and didn't adopt it personally or at work chime in and say why? [0] https://aleyan.com/blog/2025-task-runners-census/#javascript-ecosystem---pnpm-rising https://aleyan.com/blog/2025-task-runners-census/#javascript... [1] https://news.ycombinator.com/item?id=44559375 https://news.ycombinator.com/item?id=44559375
- koakuma-chan 1y ago
  > I wonder why that is? LLMs default to npm
  - fkyoureadthedoc 1y ago
    You sure it's not just because npm has been around for 15 years as the default package manager for node?
    - koakuma-chan 1y ago
      Didn't prevent me from switching to Bun as the cost is 0.
- madeofpalk 1y ago
  Honestly, it doesn't really solve a big problem I have, and introduces all the problem with being "new" and less used.
- williamstein 1y ago
  I am also very curious what people think about this. To me, as a project, Node gives off a vibe of being mature, democratic and community driven, especially after successfully navigating then io.js fork drama etc a few years ago. Clearly neither bun nor deno are community driven democratic projects, since they are both VC funded.
wink 1y ago
> Node.js uses libuv, a C library that abstracts platform differences and manages async I/O through a thread pool. > Bun does it differently. Bun is written in Zig, a programming language that compiles to native code with direct system call access: Guess what, C/C++ also compiles to native code. I mean, I get what they're saying and it's good, and nodejs could have probably done that as well, but didn't. But don't phrase it like it's inherently not capable. No one forced npm to be using this abstraction, and npm probably should have been a nodejs addon in C/C++ in the first place. (If anything of this sounds like a defense of npm or node, it is not.)
- k__ 1y ago
  To me, the reasoning seems to be: Npm, pnpm, and yarn are written in JS, so they have to use Node.js facilities, which are based on libuv, which isn't optimal in this case. Bun is written in Zig, so it doesn't need libuv, and can so it's own thing. Obviously, someone could write a Node.js package manager in C/C++ as a native module to do the same, but that's not what npm, pnpm, and yarn did.
- lkbm 1y ago
  Isn't the issue not that libuv is C, but that the thing calling it (Node.js) is Javascript, so you have to switch modes each time you have libuv make a system call?
djfobbz 1y ago
I really like Bun too, but I had a hard time getting it to play nicely with WSL1 on Windows 10 (which I prefer over WSL2). For example: ~/: bun install error: An unknown error occurred (Unexpected)
- lfx 1y ago
  Why you prefer WSL1 over WSL2?
  - tracker1 1y ago
    FS calls across the OS boundary are significantly faster in WSL1, as the biggest example from the top of my head. I prefer WSL2 myself, but I avoid using the /mnt/c/ paths as much as possible, and never, ever run a database (like sqlite) across that boundary, you will regret it.
  - djfobbz 1y ago
    WSL1's just faster, no weird networking issues, and I can edit the Linux files from both Windows and Linux without headaches.
k__ 1y ago
"... the last 4 bytes of the gzip format. These bytes are special since store the uncompressed size of the file!" What's the reason for this? I could imagine, many tools could profit from knowing the decompressed file size in advance.
- 8cvor6j844qw_d6 1y ago
  gzip.py [1] --- def _read_eof(self): # We've read to the end of the file, so we have to rewind in order # to reread the 8 bytes containing the CRC and the file size. # We check the that the computed CRC and size of the # uncompressed data matches the stored values. Note that the size # stored is the true file size mod 2*32. --- [1]: https://stackoverflow.com/a/1704576 https://stackoverflow.com/a/1704576
- philipwhiuk 1y ago
  It's straight from the GZIP spec if you assume there's a single GZIP "member": https://www.ietf.org/rfc/rfc1952.txt https://www.ietf.org/rfc/rfc1952.txt > ISIZE (Input SIZE) > This contains the size of the original (uncompressed) input data modulo 2^32. So there's two big caveats: 1. Your data is a single GIZP member (I guess this means everything in a folder) 2. Your data is < 2^32 bytes.
  - k__ 1y ago
    Yeah, I understood that. I was just wondering why GZIP specified it that way.
    - ncruces 1y ago
      Because it allows streaming compression.
      - k__ 1y ago
        Ah, makes sense. Thanks!
  - jerf 1y ago
thornewolf 1y ago
I think they forgot to include the benchmark time for "npm (cached)" inside the Binary Manifest Caching section. We have bun, bun (cached), npm. I think the summary statistics are also incorrect.
- sapper123 1y ago
  They don’t seem to clear the cache in between fresh runs. This is evident from the lower bound range time being the same time as the cached mean. This leads them to the incorrect conclusion that bun fresh runs are faster than npm cached, which doesn’t seem to be the case.
tracker1 1y ago
I'm somewhat curious how Deno stands up with this... also, not sure what packages are being installed. I'd probably start a vite template project for react+ts+mui as a baseline, since that's a relatively typical application combo for tooling. Maybe hono+zod+openapi as well.
- steve_adams_86 1y ago
  I think Deno isn't included in the benchmark because it's a harder comparison to make than it might seem. Deno's dependency architecture isn't built around npm; that compatibility layer is a retrofit on top of the core (which is evident in the source code, if you ever want to see). Deno's core architecture around dependency management uses a different, URL-based paradigm. It's not as fast, but... It's different. It also allows for improved security and cool features like the ability to easily host your own secure registry. You don't have to use npm or jsr. It's very cool, but different from what is being benchmarked here.
  - tracker1 1y ago
    All the same, you can run deno install in a directory with a package.json file an it will resolve and install to node_modules. The process is also written in compiled code, like bun... so I was just curious. edit: replied to my own post... looks like `deno install --allow-scripts` is about 1s slower than bun once deno.lock exists.
- tracker1 1y ago
  For my own curiousity on a React app on my work desktop. - Clean `bun install`, 48s - converted package-lock.json - With bun.lock, no node_modules, 19s - Clean with `deno install --allow-scripts`, 1m20s - with deno.lock, no node_modules, 20s - Clean `npm i`, 26s - `npm ci` (package-lock.json), no node_modules, 1m,2s (wild) So, looks like if Deno added a package-lock.json conversion similar to bun the installs would be very similar all around. I have no control over the security software used on this machine, was just convenience as I was in front of it. Hopefully someone can put eyes on this issue: https://github.com/denoland/deno/issues/25815 https://github.com/denoland/deno/issues/25815
paularmstrong 1y ago
This is all well and good, but the time it takes to install node modules is not a critical blocker for any project that I've ever been a part of. It's a drop in the bucket compared to human (ability and time to complete changes) and infrastructure (CI/deploy/costs). Cutting 20 seconds off the dependency install time is just not a make or break issue.
- tracker1 1y ago
  It's more than enough to lose your focus. If you can make a process take a couple seconds or less vs over 15, you should do that.
  - paularmstrong 1y ago
    How often are you doing a full install of dependencies? Re-runs for me using npm/pnpm/yarn are 1-2 seconds at worst in very large monorepos. I can't imagine needing to do full installs on any sort frequency.
    - paularmstrong 1y ago
      I also just tried bun install in my main work monorepo vs yarn. bun took 12s and yarn took 15s. This is hardly a difference worth noting.
      - tracker1 1y ago
        Yeah, I find HDD speed can make more of a difference too. Gen 5 PCIe is amazing.. the difference in Rust builds was pretty phenomenal over even a good gen 4 drive.
    - tracker1 1y ago
      I find that it's heavily dependent on the drive speed, so I've leaned into getting current generation, very fast drives as much as possible when I put together new computers and sometimes a mid-generation upgrade. Considering I often do consulting work across random projects, I pretty often am having to figure out and install things in one mono repo managed with pnpm, another with yarn, etc... so the pain is relatively real, that said, fastest drive matters as much or more, especially with build steps. When handling merge/pull requests, I'll often do a clean step (removing node_modules, and temp files) before a full install and build to test everything works. I know not everyone else is this diligent, but this can happen several times a day... Automation (usually via docker) can help a lot with many things tested through a CI/CD environment, that said, I'm also not a fan of having to wait for too long for that process... it's too easy to get side-tracked and off-task. I tend to set alarms/timers throughout the day just so I don't miss meetings. I don't want to take a moment to look at HN, and next I know it's a few hours later. Yeah, that's my problem... but others share it. So, again, if you can make something take less than 15s that typically takes much more, I'm in favor... I went from eslint to Rome/Biome for similar reasons... I will switch to faster tooling to reduce the risk of going off-task and not getting back.
manuhabitela 1y ago
I'm impressed how pleasant and easy to read this pretty technical explanation was. Good job on the writing.
- winterrdog 1y ago
  Truth! Lydia is very good at presenting complex ideas simply and well. I've read and watched most of her work or videos. She really goes to great lengths in her work to make it come to life. Highly recommend her articles and YouTube videos. Though she's been writing less I think due to her current job
robinhood 1y ago
Complex subject, beautifully simple to read. Congrats to the author. Also: I love that super passionate people still exist, and are willing to challenge the statut quo by attacking really hard things - things I don't have the brain to even think about. It's not normal that we have better computers each month and slower softwares. If only everyone (myself included) were better at writing more efficient code.
- ljm 1y ago
  I didn’t know it was written in Zig. That’s a fascinating choice to me given how young the language is. Amazing to see it being used in a practical way in production.
  - robinhood 1y ago
    Zig was created in 2016 though - almost 10 years at this point. Perhaps the surprise here is that we are not as exposed to this language on well-known and established projects as other languages like Rust, Go and C.
    - pdpi 1y ago
      Zig is still at the 0.x stage, and there's still a bunch of churn going on on really basic stuff like IO and memory allocation. I really enjoy writing it, but it's by no means stable to the point many people would write production software in it.
    - ivanjermakov 1y ago
      To be fair, Zig 10 years ago is drastically different language from Zig today.
      - ojosilva 1y ago
        Which is unfortunately a problem for AI trained on Zig, it makes some AI-assisted Zig coding more challenging, like Q&A and code-completion. It's sad that this glass-ceiling has been enacted for new languages and frameworks, not a deal-breaker at all, just that suddenly there's this penalty on time-to-deliver on anything Zig. But then... the same issue exists when hiring good programmers for lesser-known tech. There'll probably be a strategy (AEO?) for this in the future for newcomers and the underrepresented, like endless examples posted by a sane AI to their docs and github for instance so it gets picked up by training sets or live, tool calling, web-searches.
swyx 1y ago
i'm curious why Yarn is that much slower than npm? whats the opposite of this article?
phildougherty 1y ago
Bun is FUN to say.
moffkalast 1y ago
Anyone else also having a first association to https://xkcd.com/1682 https://xkcd.com/1682 instead of, you know, bread?
alberth 1y ago
I really enjoyed the writing style of this post. A few things: - I feel like this post repurposed could be a great explanation on why io_uring is so important. - I wonder if Zig recently io updates in v0.15 make any perf improvement to Bun beyond its current fast perf.
randomsofr 1y ago
wow, crazy to see yarn being so slow, when it used to beat npm by a lot, at a company i was we went from npm, to yarn, to pnpm, back to npm. Nowadays i try to use Bun as much as possible, but Vercel still does not uses natively for Next.
- chrisweekly 1y ago
  why leave pnpm?
captn3m0 1y ago
> The M4 Max MacBook I'm using to write this would've ranked among the 50 fastest supercomputers on Earth in 2009. I attempted to validate this: You'd need >75 TFlop/s to get into the top50 in the TOP500[0] rankings in 2009. M4 Max review says 18.4 TFlop/s at FP32, but TOP500 uses LINPACK, which uses FP64 precision. An M2 benchmark gives a 1:4 ratio for double precision, so you'd get maybe 9 TFlop/s at FP64? That wouldn't make it to TOP500 in 2009. [0]: https://top500.org/lists/top500/list/2009/06/ https://top500.org/lists/top500/list/2009/06/
- nine_k 1y ago
  > Now multiply that by thousands of concurrent connections each doing multiple I/O operations. Servers spent ~95% of their time waiting for I/O operations. Well, no. The particular thread of execution might have been spending 95% of time waiting for I/O, but a server (the machine serving the thousands of connections) would easily run at 70%-80% of CPU utilization (because above that, tail latency starts to suffer badly). If your server had 5% CPU utilization under full load, you were not running enough parallel processes, or did not install enough RAM to do so. Well, it's a technicality, but the post is devoted to technicalities, and such small blunders erode the trust to the rest of the post. (I'm saying this as a fan of Bun.)
  - ukblewis 1y ago
    If you’ve never seen a machine stuck waiting for disk I/O, I don’t know what to tell you… but it is common, even with SSDs it can happen (as they point out due to handoff between the OS and user level process takes time)
- fleebee 1y ago
  I'm guessing that's an LLM hallucination. The conclusion section especially has some hints it was pulled out of an LLM: > The package managers we benchmarked weren't built wrong, they were solutions designed for the constraints of their time. > Buns approach wasn't revolutionary, it was just willing to look at what actually slows things down today. > Installing packages 25x faster isn't "magic": it's what happens when tools are built for the hardware we actually have.
atonse 1y ago
I absolutely loved reading this. It's such an excellent example of a situation where Computer Science principles are very important in day to day software development. So many of these concepts (Big O, temporal and spatial locality, algorithmic complexity, lower level user space/kernel space concepts, filesystems, copy on write), are ALL the kinds of things you cover in a good CS program. And in this and similar lower level packages, you use all of them to great effect.
- epolanski 1y ago
  This is about software engineering not computer science. CS is the study of computations and their theory (programming languages, algorithms, cryptography, machine learning, etc). SE is the application of engineering principles to building scalable and reliable software.
  - atonse 1y ago
    Without getting bogged down in rigid definitions of phrases, do we both agree that this is about the application of deeper technical concepts and algorithms (usually taught as part of a computer science curriculum) towards real world problems than the normal “build this login form” or “write these 5 queries to generate this report that shows up in an html table” that 75% of devs do daily?
LeicaLatte 1y ago
Liking the package management from first principles as a systems-level optimization problem rather than file scripting. resembling a database engine - dependency aware task scheduling, cache locality, sys call overhead - they are all there.
markasoftware 1y ago
I'm pretty confused about why it's beneficial to wait to read the whole compressed file before decompressing. Surely the benefit of beginning decompression before the download is complete outweigh having to copy the memory around a few extra times as the vector is resized?
- Jarred 1y ago
  Streaming prevents many optimizations because the code can’t assume it’s done when run once, so it has to suspend / resume, clone extra data for longer, and handle boundary cases more carefully. It’s usually only worth it after ~tens of megabytes, but vast majority of npm packages are much smaller than that. So if you can skip it, it’s better.
  - yencabulator 1y ago
    Streaming compression with a large buffer size handles everything in a single batch for small files.
WesolyKubeczek 1y ago
macOS has hardlinks too. Why not use them?
wojtek1942 1y ago
> However, this mode switching is expensive! Just this switch alone costs 1000-1500 CPU cycles in pure overhead, before any actual work happens. ... > On a 3GHz processor, 1000-1500 cycles is about 500 nanoseconds. This might sound negligibly fast, but modern SSDs can handle over 1 million operations per second. If each operation requires a system call, you're burning 1.5 billion cycles per second just on mode switching. > Package installation makes thousands of these system calls. Installing React and its dependencies might trigger 50,000+ system calls: that's seconds of CPU time lost to mode switching alone! Not even reading files or installing packages, just switching between user and kernel mode. Am I missing something or is this incorrect. They claim 500ns per syscall with 50k syscalls. 500ns * 50000 = 25 milliseconds. So that is very far from "seconds of CPU time lost to mode switching alone!" right?
- Bolwin 1y ago
  Read further. In one of the later benchmarks, yarn makes 4 million syscalls. Still only about 2 secs, but still.
RestartKernel 1y ago
This is very nicely written, but I don't quite get how Linux's hardlinks are equivalent to MacOS's clonefile. If I understand correctly, wouldn't the former unexpectedly update files across all your projects if you modify just one "copy"?
yahoozoo 1y ago
Good article but it sounds a lot like AI wrote it.
Jarred 1y ago
I work on Bun and also spent a lot of time optimizing bun install. Happy to answer any questions
- nzoschke 1y ago
  I just want to say thanks to you and the team and community. Bun is a treat to use.
- norman784 1y ago
  AFAIK `bun install` is similar to `npm install` in the sense that it installs everything in a flat structure inside node modules, why didn't you choose something like pnpm that I believe is better, because you cannot by mistake import a transitive dependency. Maybe that's a non issue for most, but I care about those things.
  - Jarred 1y ago
    You can use `bun install --linker=isolated`, which we might make the default in Bun v1.3. The main downside is it makes your app load slightly slower since now every file path is a symlink. https://bun.sh/docs/install/isolated https://bun.sh/docs/install/isolated
    - norman784 1y ago
      I didn't knew about it, thanks!
- hu3 1y ago
  Kinda offtopic but how to you manage to stay so productive for so long? Vitamins/supplements? Sleep? Exercise? Vacations? I have sprints of great productivity but it's hard to keep it for long.
valtism 1y ago
I had no idea Lydia was working for Bun now. Her technical writing is absolutely top notch
rtpg 1y ago
bun installs are fast, but I think they might be so fast and concurrent they cause npm to really get confused sometimes. I end up hitting 500s from npm from time to time installing by bun and I just don't know why. Really wish the norm was that companies hosted their own registries for their own usage, so I could justify the expense and effort instead of dealing with registries being half busted kinda randomly.
- mnahkies 1y ago
  > Really wish the norm was that companies hosted their own registries for their own usage Is this not the norm? I've never worked anywhere that didn't use/host their own registry - both for hosting private packages, but also as a caching proxy to the public registry (and therefore more control over availability, security policy) https://verdaccio.org/ https://verdaccio.org/ is my go to self hosted solution, but the cloud providers have managed solutions and there's also jFrog Artifactory. One corollary of this is that many commercial usages of packages don't contribute much to download stats, as often they download each version at most once.
mrcarrot 1y ago
The "Optimized Tarball Extraction" confuses me a bit. It begins by illustrating how other package managers have to repeatedly copy the received, compressed data into larger and larger buffers (not mentioning anything about the buffer where the decompressed data goes), and then says that: > Bun takes a different approach by buffering the entire tarball before decompressing. But seems to sidestep _how_ it does this any differently than the "bad" snippet the section opened with (presumably it checks the Content-Length header when it's fetching the tarball or something, and can assume the size it gets from there is correct). All it says about this is: > Once Bun has the complete tarball in memory it can read the last 4 bytes of the gzip format. Then it explains how it can pre-allocate a buffer for the decompressed data, but we never saw how this buffer allocation happens in the "bad" example! > These bytes are special since store the uncompressed size of the file! Instead of having to guess how large the uncompressed file will be, Bun can pre-allocate memory to eliminate buffer resizing entirely Presumably the saving is in the slow package managers having to expand _both_ of the buffers involved, while bun preallocates at least one of them?
- Jarred 1y ago
  Here is the code: https://github.com/oven-sh/bun/blob/7d5f5ad7728b4ede521906a41f53ce5b7951ba4e/src/install/extract_tarball.zig#L174-L190 https://github.com/oven-sh/bun/blob/7d5f5ad7728b4ede521906a4... We trust the self-reported size by gzip up to 64 MB, try to allocate enough space for all the output, then run it through libdeflate. This is instead of a loop that decompresses it chunk-by-chunk and then extracts it chunk-by-chunk and resizing a big tarball many times over.
  - mrcarrot 1y ago
    Thanks - this does make sense in isolation. I think my actual issue is that the "most package managers do something like this" example code snippet at the start of [1] doesn't seem to quite make sense - or doesn't match what I guess would actually happen in the decompress-in-a-loop scenario? As in, it appears to illustrate building up a buffer holding the compressed data that's being received (since the "// ... decompress from buffer ..." comment at the end suggests what we're receiving in `chunk` is compressed), but I guess the problem with the decompress-as-the-data-arrives approach in reality is having to re-allocate the buffer for the decompressed data? [1] https://bun.com/blog/behind-the-scenes-of-bun-install#optimized-tarball-extraction https://bun.com/blog/behind-the-scenes-of-bun-install#optimi...
azangru 1y ago
I am probably being stupid; but aren't install commands run relatively rarely by developers (less than once a day perhaps)? Is it such an important issue how long it takes for `x install` to finish? Or is the concern about the time spent in CI/CD?
- tuetuopay 1y ago
  CICD is a major usage. But dependencies version bumps are also a big part of it. In the python ecosystem I’ve had poetry take minutes to resolve the ansible dependencies after bumping the version. And then you see uv take milliseconds to do a full install from scratch.
PiraticSomate 1y ago
[dead]
1vuio0pswjnm7 1y ago
https://github.com/oven-sh/bun/releases/expanded_assets/bun-v1.2.21 https://github.com/oven-sh/bun/releases/expanded_assets/bun-... progress: dynamically-linked musl binaries (tnx) next: statically-linked musl binaries

431 points by Bogdanp 1y ago

I used bun for the first time last week. It was awesome! The built-in server and SQLite meant i didn’t need any dependencies besides bun itself, which is certainly my favorite way to develop. I do almost all of my development in vanilla js despite loathing the node ecosystem, so i really should have checked it out sooner.

k__ 1y ago
I tried using Bun a few times, and I really liked working with it. Much better than Node. However...! I always managed to hit a road block with Bun and had to go back to Node. First it was the crypto module that wasn't compatible with Nodejs signatures (now fixed), next Playwright refused working with Bun (via Crawlee).
- Cthulhu_ 1y ago
  I think this is the big one that slows adoption of "better" / "faster" tooling down, that is, backwards compatibility and drop-in-replacement-ability. Probably a lot of Hyrum's Law.
- koakuma-chan 1y ago
  You can use Bun as package manager only. You don't have to use Bun as runtime.
  - winterrdog 1y ago
    Sure? Does it work if I have packages that have nodejs c++ addons?
    - abejfehr 1y ago
      Why wouldn’t it? The end result of a npm install or a bun install is that the node_modules folder is structured in the way it needs to be, and I think it can run node-gyp for the packages that need it.
  - iansinnott 1y ago
    Indeed! also as a test runner/lib if you're not doing browser automation. bun definitely has benefits even if not used as a runtime.
    - koakuma-chan 1y ago
      I believe Playwright worked for me with the latest Bun though
- jherdman 1y ago
  Storybook is another for me.
- drewbitt 1y ago
  Deno doesn't work with crawlee either unfortunately
- petralithic 1y ago
  You should try Deno, they have good Node compatibility
  - erikpukinskis 1y ago
    Does it? Last I tried, several years ago, coverage of the Node APIs was not good. I wanted to send data over UDP and a lot of Node basics there were missing.
    - spartanatreyu 1y ago
      Deno's node compat is way better now. They're still missing niche things and they tend to target the things that most devs (and their dependencies) are actually using. But I can see they have it in their compat stuff now and it looks like it's working in the repl locally: https://docs.deno.com/api/node/dgram/ https://docs.deno.com/api/node/dgram/
      - rererereferred 1y ago
        This page tracks Deno's compatibility with Node by running its tests: https://ffmathy.github.io/is-deno-compatible-yet/ https://ffmathy.github.io/is-deno-compatible-yet/
- epolanski 1y ago
  Playwright has been fixed one year ago I think.
- Jarred 1y ago
  Playwright support will improve soon. We are rewriting node:http’s client implementation to pass node’s test suite. Expecting that to land next week.
  - ukblewis 1y ago
    That may be just the best way to drop that you are introducing a rewrite which fixes a set of bugs that affect users
simantel 1y ago
Node also has a built-in server and SQLite these days though? Or if you want a lot more functionality with just one dependency, Hono is great.
- blizdiddy 1y ago
  And how many dependencies does Hono have? Looks like about 26. And how many dependencies do those have? A single static zig executable isn’t the same as a a pipeline of package management dependencies susceptible to supply chain attacks and the worst bitrot we’ve had since the DOS era.
  - bakkoting 1y ago
    > And how many dependencies does Hono have? Zero. I'm guessing you're looking at the `devDependencies` in its package.json, but those are only used by the people building the project, not by people merely consuming it.
    - PxldLtd 1y ago
      That doesn't prevent supply chain attacks. Dev dependencies are still software dependencies and add a certain level of risk.
      - bakkoting 1y ago
        Only in the sense that any other software on the developers' machines adds a certain level of risk.
      - arcfour 1y ago
        This is needlessly pedantic unless you are writing from an OS, browser, etc. that you wrote entirely by yourself, without using an editor or linter or compiler not written by you, in which case I tip my cap to you.

jcmartinezdev 1y ago

[dead]

rs_rs_rs_rs_rs 1y ago

Python has uv, JS has bun, what does Ruby or PHP have? Are the devs using those languages happy with how fast the current popular dependency managers are?

tommasoamici 1y ago
it's pretty new, but in Ruby there's `rv` which is clearly inspired by `uv`: https://github.com/spinel-coop/rv https://github.com/spinel-coop/rv. >Brought to you by Spinel >Spinel.coop is a collective of Ruby open source maintainers building next-generation developer tooling, like rv, and offering flat-rate, unlimited access to maintainers who come from the core teams of Rails, Hotwire, Bundler, RubyGems, rbenv, and more.
deleted 1y ago
[deleted]
JamesSwift 1y ago
Youre looking at it wrong. Python has nix, JS has nix, ruby and php have nix : D Thats closer to how pnpm achieves speed up though. I know there is 'rv' recently, but havent tried it.
- koakuma-chan 1y ago
  You mean nix the package manager? I used to use NixOS and I had to switch off because of endless mess with environment variables.
  - JamesSwift 1y ago
    Yes, nix package manager. Or devenv for a more streamlined version of what I'm describing, similar to mise but powered by nix.
hu3 1y ago
PHP is getting Mago (written in Rust). Repo: https://github.com/carthage-software/mago https://github.com/carthage-software/mago Announcement 9 months ago: https://www.reddit.com/r/PHP/comments/1h9zh83/announcing_mago_an_oxidized_toolchain_for_php/ https://www.reddit.com/r/PHP/comments/1h9zh83/announcing_mag... For now its main features are 3: formatting, linting and fixing lint issues. I hope they add package management to do what composer does.

aleyan 1y ago

I have been excited about bun for about a year, and I thought that 2025 is going to be its breakout year. It is really surprising to me that it is not more popular. I scanned top 100k repos on GitHub, and for new repos in 2025, npm is 35 times more popular and pnpm is 11 time more popular than bun [0][1]. The other up and coming javascript runtime, deno is not so popular either. I wonder why that is? Is it because it is a runtime, and getting compatibility there is harder than just for a straight package manager? Can someone who tried bun and didn't adopt it personally or at work chime in and say why? [0] https://aleyan.com/blog/2025-task-runners-census/#javascript-ecosystem---pnpm-rising https://aleyan.com/blog/2025-task-runners-census/#javascript... [1] https://news.ycombinator.com/item?id=44559375 https://news.ycombinator.com/item?id=44559375

koakuma-chan 1y ago
> I wonder why that is? LLMs default to npm
- fkyoureadthedoc 1y ago
  You sure it's not just because npm has been around for 15 years as the default package manager for node?
  - koakuma-chan 1y ago
    Didn't prevent me from switching to Bun as the cost is 0.
madeofpalk 1y ago
Honestly, it doesn't really solve a big problem I have, and introduces all the problem with being "new" and less used.
williamstein 1y ago
I am also very curious what people think about this. To me, as a project, Node gives off a vibe of being mature, democratic and community driven, especially after successfully navigating then io.js fork drama etc a few years ago. Clearly neither bun nor deno are community driven democratic projects, since they are both VC funded.

wink 1y ago

> Node.js uses libuv, a C library that abstracts platform differences and manages async I/O through a thread pool. > Bun does it differently. Bun is written in Zig, a programming language that compiles to native code with direct system call access: Guess what, C/C++ also compiles to native code. I mean, I get what they're saying and it's good, and nodejs could have probably done that as well, but didn't. But don't phrase it like it's inherently not capable. No one forced npm to be using this abstraction, and npm probably should have been a nodejs addon in C/C++ in the first place. (If anything of this sounds like a defense of npm or node, it is not.)

k__ 1y ago
To me, the reasoning seems to be: Npm, pnpm, and yarn are written in JS, so they have to use Node.js facilities, which are based on libuv, which isn't optimal in this case. Bun is written in Zig, so it doesn't need libuv, and can so it's own thing. Obviously, someone could write a Node.js package manager in C/C++ as a native module to do the same, but that's not what npm, pnpm, and yarn did.
lkbm 1y ago
Isn't the issue not that libuv is C, but that the thing calling it (Node.js) is Javascript, so you have to switch modes each time you have libuv make a system call?

djfobbz 1y ago

I really like Bun too, but I had a hard time getting it to play nicely with WSL1 on Windows 10 (which I prefer over WSL2). For example: ~/: bun install error: An unknown error occurred (Unexpected)

lfx 1y ago
Why you prefer WSL1 over WSL2?
- tracker1 1y ago
  FS calls across the OS boundary are significantly faster in WSL1, as the biggest example from the top of my head. I prefer WSL2 myself, but I avoid using the /mnt/c/ paths as much as possible, and never, ever run a database (like sqlite) across that boundary, you will regret it.
- djfobbz 1y ago
  WSL1's just faster, no weird networking issues, and I can edit the Linux files from both Windows and Linux without headaches.

k__ 1y ago

"... the last 4 bytes of the gzip format. These bytes are special since store the uncompressed size of the file!" What's the reason for this? I could imagine, many tools could profit from knowing the decompressed file size in advance.

8cvor6j844qw_d6 1y ago
gzip.py [1] --- def _read_eof(self): # We've read to the end of the file, so we have to rewind in order # to reread the 8 bytes containing the CRC and the file size. # We check the that the computed CRC and size of the # uncompressed data matches the stored values. Note that the size # stored is the true file size mod 2*32. --- [1]: https://stackoverflow.com/a/1704576 https://stackoverflow.com/a/1704576
philipwhiuk 1y ago
It's straight from the GZIP spec if you assume there's a single GZIP "member": https://www.ietf.org/rfc/rfc1952.txt https://www.ietf.org/rfc/rfc1952.txt > ISIZE (Input SIZE) > This contains the size of the original (uncompressed) input data modulo 2^32. So there's two big caveats: 1. Your data is a single GIZP member (I guess this means everything in a folder) 2. Your data is < 2^32 bytes.
- k__ 1y ago
  Yeah, I understood that. I was just wondering why GZIP specified it that way.
  - ncruces 1y ago
    Because it allows streaming compression.
    - k__ 1y ago
      Ah, makes sense. Thanks!
- jerf 1y ago

thornewolf 1y ago

I think they forgot to include the benchmark time for "npm (cached)" inside the Binary Manifest Caching section. We have bun, bun (cached), npm. I think the summary statistics are also incorrect.

sapper123 1y ago
They don’t seem to clear the cache in between fresh runs. This is evident from the lower bound range time being the same time as the cached mean. This leads them to the incorrect conclusion that bun fresh runs are faster than npm cached, which doesn’t seem to be the case.

tracker1 1y ago

I'm somewhat curious how Deno stands up with this... also, not sure what packages are being installed. I'd probably start a vite template project for react+ts+mui as a baseline, since that's a relatively typical application combo for tooling. Maybe hono+zod+openapi as well.

steve_adams_86 1y ago
I think Deno isn't included in the benchmark because it's a harder comparison to make than it might seem. Deno's dependency architecture isn't built around npm; that compatibility layer is a retrofit on top of the core (which is evident in the source code, if you ever want to see). Deno's core architecture around dependency management uses a different, URL-based paradigm. It's not as fast, but... It's different. It also allows for improved security and cool features like the ability to easily host your own secure registry. You don't have to use npm or jsr. It's very cool, but different from what is being benchmarked here.
- tracker1 1y ago
  All the same, you can run deno install in a directory with a package.json file an it will resolve and install to node_modules. The process is also written in compiled code, like bun... so I was just curious. edit: replied to my own post... looks like `deno install --allow-scripts` is about 1s slower than bun once deno.lock exists.
tracker1 1y ago
For my own curiousity on a React app on my work desktop. - Clean `bun install`, 48s - converted package-lock.json - With bun.lock, no node_modules, 19s - Clean with `deno install --allow-scripts`, 1m20s - with deno.lock, no node_modules, 20s - Clean `npm i`, 26s - `npm ci` (package-lock.json), no node_modules, 1m,2s (wild) So, looks like if Deno added a package-lock.json conversion similar to bun the installs would be very similar all around. I have no control over the security software used on this machine, was just convenience as I was in front of it. Hopefully someone can put eyes on this issue: https://github.com/denoland/deno/issues/25815 https://github.com/denoland/deno/issues/25815

paularmstrong 1y ago

This is all well and good, but the time it takes to install node modules is not a critical blocker for any project that I've ever been a part of. It's a drop in the bucket compared to human (ability and time to complete changes) and infrastructure (CI/deploy/costs). Cutting 20 seconds off the dependency install time is just not a make or break issue.

tracker1 1y ago
It's more than enough to lose your focus. If you can make a process take a couple seconds or less vs over 15, you should do that.
- paularmstrong 1y ago
  How often are you doing a full install of dependencies? Re-runs for me using npm/pnpm/yarn are 1-2 seconds at worst in very large monorepos. I can't imagine needing to do full installs on any sort frequency.
  - paularmstrong 1y ago
    I also just tried bun install in my main work monorepo vs yarn. bun took 12s and yarn took 15s. This is hardly a difference worth noting.
    - tracker1 1y ago
      Yeah, I find HDD speed can make more of a difference too. Gen 5 PCIe is amazing.. the difference in Rust builds was pretty phenomenal over even a good gen 4 drive.
  - tracker1 1y ago
    I find that it's heavily dependent on the drive speed, so I've leaned into getting current generation, very fast drives as much as possible when I put together new computers and sometimes a mid-generation upgrade. Considering I often do consulting work across random projects, I pretty often am having to figure out and install things in one mono repo managed with pnpm, another with yarn, etc... so the pain is relatively real, that said, fastest drive matters as much or more, especially with build steps. When handling merge/pull requests, I'll often do a clean step (removing node_modules, and temp files) before a full install and build to test everything works. I know not everyone else is this diligent, but this can happen several times a day... Automation (usually via docker) can help a lot with many things tested through a CI/CD environment, that said, I'm also not a fan of having to wait for too long for that process... it's too easy to get side-tracked and off-task. I tend to set alarms/timers throughout the day just so I don't miss meetings. I don't want to take a moment to look at HN, and next I know it's a few hours later. Yeah, that's my problem... but others share it. So, again, if you can make something take less than 15s that typically takes much more, I'm in favor... I went from eslint to Rome/Biome for similar reasons... I will switch to faster tooling to reduce the risk of going off-task and not getting back.

manuhabitela 1y ago

I'm impressed how pleasant and easy to read this pretty technical explanation was. Good job on the writing.

winterrdog 1y ago
Truth! Lydia is very good at presenting complex ideas simply and well. I've read and watched most of her work or videos. She really goes to great lengths in her work to make it come to life. Highly recommend her articles and YouTube videos. Though she's been writing less I think due to her current job

robinhood 1y ago

Complex subject, beautifully simple to read. Congrats to the author. Also: I love that super passionate people still exist, and are willing to challenge the statut quo by attacking really hard things - things I don't have the brain to even think about. It's not normal that we have better computers each month and slower softwares. If only everyone (myself included) were better at writing more efficient code.

ljm 1y ago
I didn’t know it was written in Zig. That’s a fascinating choice to me given how young the language is. Amazing to see it being used in a practical way in production.
- robinhood 1y ago
  Zig was created in 2016 though - almost 10 years at this point. Perhaps the surprise here is that we are not as exposed to this language on well-known and established projects as other languages like Rust, Go and C.
  - pdpi 1y ago
    Zig is still at the 0.x stage, and there's still a bunch of churn going on on really basic stuff like IO and memory allocation. I really enjoy writing it, but it's by no means stable to the point many people would write production software in it.
  - ivanjermakov 1y ago
    To be fair, Zig 10 years ago is drastically different language from Zig today.
    - ojosilva 1y ago
      Which is unfortunately a problem for AI trained on Zig, it makes some AI-assisted Zig coding more challenging, like Q&A and code-completion. It's sad that this glass-ceiling has been enacted for new languages and frameworks, not a deal-breaker at all, just that suddenly there's this penalty on time-to-deliver on anything Zig. But then... the same issue exists when hiring good programmers for lesser-known tech. There'll probably be a strategy (AEO?) for this in the future for newcomers and the underrepresented, like endless examples posted by a sane AI to their docs and github for instance so it gets picked up by training sets or live, tool calling, web-searches.

swyx 1y ago

i'm curious why Yarn is that much slower than npm? whats the opposite of this article?

phildougherty 1y ago

Bun is FUN to say.

moffkalast 1y ago

Anyone else also having a first association to https://xkcd.com/1682 https://xkcd.com/1682 instead of, you know, bread?

alberth 1y ago

I really enjoyed the writing style of this post. A few things: - I feel like this post repurposed could be a great explanation on why io_uring is so important. - I wonder if Zig recently io updates in v0.15 make any perf improvement to Bun beyond its current fast perf.

randomsofr 1y ago

wow, crazy to see yarn being so slow, when it used to beat npm by a lot, at a company i was we went from npm, to yarn, to pnpm, back to npm. Nowadays i try to use Bun as much as possible, but Vercel still does not uses natively for Next.

chrisweekly 1y ago
why leave pnpm?

captn3m0 1y ago

> The M4 Max MacBook I'm using to write this would've ranked among the 50 fastest supercomputers on Earth in 2009. I attempted to validate this: You'd need >75 TFlop/s to get into the top50 in the TOP500[0] rankings in 2009. M4 Max review says 18.4 TFlop/s at FP32, but TOP500 uses LINPACK, which uses FP64 precision. An M2 benchmark gives a 1:4 ratio for double precision, so you'd get maybe 9 TFlop/s at FP64? That wouldn't make it to TOP500 in 2009. [0]: https://top500.org/lists/top500/list/2009/06/ https://top500.org/lists/top500/list/2009/06/

nine_k 1y ago
> Now multiply that by thousands of concurrent connections each doing multiple I/O operations. Servers spent ~95% of their time waiting for I/O operations. Well, no. The particular thread of execution might have been spending 95% of time waiting for I/O, but a server (the machine serving the thousands of connections) would easily run at 70%-80% of CPU utilization (because above that, tail latency starts to suffer badly). If your server had 5% CPU utilization under full load, you were not running enough parallel processes, or did not install enough RAM to do so. Well, it's a technicality, but the post is devoted to technicalities, and such small blunders erode the trust to the rest of the post. (I'm saying this as a fan of Bun.)
- ukblewis 1y ago
  If you’ve never seen a machine stuck waiting for disk I/O, I don’t know what to tell you… but it is common, even with SSDs it can happen (as they point out due to handoff between the OS and user level process takes time)
fleebee 1y ago
I'm guessing that's an LLM hallucination. The conclusion section especially has some hints it was pulled out of an LLM: > The package managers we benchmarked weren't built wrong, they were solutions designed for the constraints of their time. > Buns approach wasn't revolutionary, it was just willing to look at what actually slows things down today. > Installing packages 25x faster isn't "magic": it's what happens when tools are built for the hardware we actually have.

atonse 1y ago

I absolutely loved reading this. It's such an excellent example of a situation where Computer Science principles are very important in day to day software development. So many of these concepts (Big O, temporal and spatial locality, algorithmic complexity, lower level user space/kernel space concepts, filesystems, copy on write), are ALL the kinds of things you cover in a good CS program. And in this and similar lower level packages, you use all of them to great effect.

epolanski 1y ago
This is about software engineering not computer science. CS is the study of computations and their theory (programming languages, algorithms, cryptography, machine learning, etc). SE is the application of engineering principles to building scalable and reliable software.
- atonse 1y ago
  Without getting bogged down in rigid definitions of phrases, do we both agree that this is about the application of deeper technical concepts and algorithms (usually taught as part of a computer science curriculum) towards real world problems than the normal “build this login form” or “write these 5 queries to generate this report that shows up in an html table” that 75% of devs do daily?

LeicaLatte 1y ago

Liking the package management from first principles as a systems-level optimization problem rather than file scripting. resembling a database engine - dependency aware task scheduling, cache locality, sys call overhead - they are all there.

markasoftware 1y ago

I'm pretty confused about why it's beneficial to wait to read the whole compressed file before decompressing. Surely the benefit of beginning decompression before the download is complete outweigh having to copy the memory around a few extra times as the vector is resized?

Jarred 1y ago
Streaming prevents many optimizations because the code can’t assume it’s done when run once, so it has to suspend / resume, clone extra data for longer, and handle boundary cases more carefully. It’s usually only worth it after ~tens of megabytes, but vast majority of npm packages are much smaller than that. So if you can skip it, it’s better.
- yencabulator 1y ago
  Streaming compression with a large buffer size handles everything in a single batch for small files.

WesolyKubeczek 1y ago

macOS has hardlinks too. Why not use them?

wojtek1942 1y ago

> However, this mode switching is expensive! Just this switch alone costs 1000-1500 CPU cycles in pure overhead, before any actual work happens. ... > On a 3GHz processor, 1000-1500 cycles is about 500 nanoseconds. This might sound negligibly fast, but modern SSDs can handle over 1 million operations per second. If each operation requires a system call, you're burning 1.5 billion cycles per second just on mode switching. > Package installation makes thousands of these system calls. Installing React and its dependencies might trigger 50,000+ system calls: that's seconds of CPU time lost to mode switching alone! Not even reading files or installing packages, just switching between user and kernel mode. Am I missing something or is this incorrect. They claim 500ns per syscall with 50k syscalls. 500ns * 50000 = 25 milliseconds. So that is very far from "seconds of CPU time lost to mode switching alone!" right?

Bolwin 1y ago
Read further. In one of the later benchmarks, yarn makes 4 million syscalls. Still only about 2 secs, but still.

RestartKernel 1y ago

This is very nicely written, but I don't quite get how Linux's hardlinks are equivalent to MacOS's clonefile. If I understand correctly, wouldn't the former unexpectedly update files across all your projects if you modify just one "copy"?

yahoozoo 1y ago

Good article but it sounds a lot like AI wrote it.

Jarred 1y ago

I work on Bun and also spent a lot of time optimizing bun install. Happy to answer any questions

nzoschke 1y ago
I just want to say thanks to you and the team and community. Bun is a treat to use.
norman784 1y ago
AFAIK `bun install` is similar to `npm install` in the sense that it installs everything in a flat structure inside node modules, why didn't you choose something like pnpm that I believe is better, because you cannot by mistake import a transitive dependency. Maybe that's a non issue for most, but I care about those things.
- Jarred 1y ago
  You can use `bun install --linker=isolated`, which we might make the default in Bun v1.3. The main downside is it makes your app load slightly slower since now every file path is a symlink. https://bun.sh/docs/install/isolated https://bun.sh/docs/install/isolated
  - norman784 1y ago
    I didn't knew about it, thanks!
hu3 1y ago
Kinda offtopic but how to you manage to stay so productive for so long? Vitamins/supplements? Sleep? Exercise? Vacations? I have sprints of great productivity but it's hard to keep it for long.

valtism 1y ago

I had no idea Lydia was working for Bun now. Her technical writing is absolutely top notch

rtpg 1y ago

bun installs are fast, but I think they might be so fast and concurrent they cause npm to really get confused sometimes. I end up hitting 500s from npm from time to time installing by bun and I just don't know why. Really wish the norm was that companies hosted their own registries for their own usage, so I could justify the expense and effort instead of dealing with registries being half busted kinda randomly.

mnahkies 1y ago
> Really wish the norm was that companies hosted their own registries for their own usage Is this not the norm? I've never worked anywhere that didn't use/host their own registry - both for hosting private packages, but also as a caching proxy to the public registry (and therefore more control over availability, security policy) https://verdaccio.org/ https://verdaccio.org/ is my go to self hosted solution, but the cloud providers have managed solutions and there's also jFrog Artifactory. One corollary of this is that many commercial usages of packages don't contribute much to download stats, as often they download each version at most once.

mrcarrot 1y ago

The "Optimized Tarball Extraction" confuses me a bit. It begins by illustrating how other package managers have to repeatedly copy the received, compressed data into larger and larger buffers (not mentioning anything about the buffer where the decompressed data goes), and then says that: > Bun takes a different approach by buffering the entire tarball before decompressing. But seems to sidestep _how_ it does this any differently than the "bad" snippet the section opened with (presumably it checks the Content-Length header when it's fetching the tarball or something, and can assume the size it gets from there is correct). All it says about this is: > Once Bun has the complete tarball in memory it can read the last 4 bytes of the gzip format. Then it explains how it can pre-allocate a buffer for the decompressed data, but we never saw how this buffer allocation happens in the "bad" example! > These bytes are special since store the uncompressed size of the file! Instead of having to guess how large the uncompressed file will be, Bun can pre-allocate memory to eliminate buffer resizing entirely Presumably the saving is in the slow package managers having to expand _both_ of the buffers involved, while bun preallocates at least one of them?

Jarred 1y ago
Here is the code: https://github.com/oven-sh/bun/blob/7d5f5ad7728b4ede521906a41f53ce5b7951ba4e/src/install/extract_tarball.zig#L174-L190 https://github.com/oven-sh/bun/blob/7d5f5ad7728b4ede521906a4... We trust the self-reported size by gzip up to 64 MB, try to allocate enough space for all the output, then run it through libdeflate. This is instead of a loop that decompresses it chunk-by-chunk and then extracts it chunk-by-chunk and resizing a big tarball many times over.
- mrcarrot 1y ago
  Thanks - this does make sense in isolation. I think my actual issue is that the "most package managers do something like this" example code snippet at the start of [1] doesn't seem to quite make sense - or doesn't match what I guess would actually happen in the decompress-in-a-loop scenario? As in, it appears to illustrate building up a buffer holding the compressed data that's being received (since the "// ... decompress from buffer ..." comment at the end suggests what we're receiving in `chunk` is compressed), but I guess the problem with the decompress-as-the-data-arrives approach in reality is having to re-allocate the buffer for the decompressed data? [1] https://bun.com/blog/behind-the-scenes-of-bun-install#optimized-tarball-extraction https://bun.com/blog/behind-the-scenes-of-bun-install#optimi...

azangru 1y ago

I am probably being stupid; but aren't install commands run relatively rarely by developers (less than once a day perhaps)? Is it such an important issue how long it takes for `x install` to finish? Or is the concern about the time spent in CI/CD?

tuetuopay 1y ago
CICD is a major usage. But dependencies version bumps are also a big part of it. In the python ecosystem I’ve had poetry take minutes to resolve the ansible dependencies after bumping the version. And then you see uv take milliseconds to do a full install from scratch.

PiraticSomate 1y ago

[dead]

1vuio0pswjnm7 1y ago

https://github.com/oven-sh/bun/releases/expanded_assets/bun-v1.2.21 https://github.com/oven-sh/bun/releases/expanded_assets/bun-... progress: dynamically-linked musl binaries (tnx) next: statically-linked musl binaries