5 ms·

It seems like bun caches the manifest responses. PNPM, for example, resolves all package versions when installing (without a lockfile), which is slower. The reg

It seems like bun caches the manifest responses. PNPM, for example, resolves all package versions when installing (without a lockfile), which is slower. The registry does have a 300 second cache time, so not faulting you there, but it means your benchmark is on the fully cached path, which you'd only hit when installing something for the first time. Subsequent installs would use the lockfile and bun and PNPM seem fast* in that case. If I install a simple nextjs app, then remove node_modules, the lockfile, and the ~/.bun/install/cache/*.npm files (i.e. keep the contents, remove the manifests) and then install, bun takes around ~3-4s. PNPM is consistently faster for me at around ~2-3s. I'm not familiar with bun's internals so I may be doing something wrong. One piece of feedback, having the lockfile be binary is a HUGE turn off for me. Impossible to diff. Is there another format? * I will mention that even in the best case scenario with PNPM (i.e. lockfile and node_modules) it still takes 400ms to start up, which, yes, is quite slow. So every action APART from the initial install is much MUCH faster with bun. I still feel 400ms is good enough for a package manager which is invoked sporadically. Compare that to esbuild which is something you invoke constantly, and having that be fast is such a godsend.

Jarred 4y ago
> It seems like the main thing that bun does to stay ahead is cache the manifest responses. PNPM, for example, resolves all package versions when installing (without a lockfile), which is slower. This isn't the main optimization. The main optimization is the system calls used to copy/link files. To see the difference, compare `bun install --backend=copyfile` with `bun install --backend=hardlink` (hardlink should be the default). The other big optimization is the binary formats for both the lockfile and the manifest. npm clients waste a lot of time parsing JSON. The more minor optimizations have to do with reducing memory usage. The binary lockfile format interns the strings (very repetitive strings). However, many of these strings are tiny, so it's actually more expensive to store a hash and a length separately from the string itself. Instead, Bun stores the string as 8 bytes and one bit bit says whether the entire string is contained inside those 8 bytes or if it's a memory offset into the lockfile's string buffer (since 64-bit pointers can't use the full memory address and bun currently only targets 64-bit CPUs, this works) yarn also caches the manifest responses. > If I install a simple nextjs app, then remove node_modules, the lockfile, and the ~/.bun/install/cache/.npm files (i.e. keep the contents, remove the manifests) and then install, bun takes around ~3-4s. PNPM is consistently faster for me at around ~2-3s. This sounds like a concurrency bug with scheduling tasks from the main thread to the HTTP thread. I would love someone to help review the code for the thread pool & async io. > One piece of feedback, having the lockfile be binary is a HUGE turn off for me. Impossible to diff. Is there another format? If you do `bun install -y`, it will output as a yarn v1 lockfile. If you add this to your .gitattributes: *.lockb binary diff=lockb It will print the diff as a yarn lockfile.
- zebracanevra 4y ago
  > If you add this to your .gitattributes: Not applicable to GitHub etc though. I'm also not seeing any speed differences when using -y/yarn lockfile. Why not make it the default?
  - brasic 4y ago
    > Not applicable to GitHub etc though. GitHub (disclosure: where I work) does respect some directives in a repo’s .gitattributes file. For example, you can use them to override language detection or mark files as generated or vendored to change diff presentation. You can also improve the diff hunk headers we generate by default by specifying e.g. `*.rb diff=ruby` (although come to think of it I don’t know why that’s necessary since we already know the filetype — I’ll look into it) In principal there’s no reason we couldn’t extend our existing rich diff support used for diffing things like images to enhance the presentation of lockfile diffs. There’s not a huge benefit for text-based lock files but for binary ones (if such a scheme were to take off) it would be a lot more useful.
    - hoten 4y ago
      Any way to use `.gitattributes` to specify a file is _not_ generated? I work on a repo with a build/ directory with build scripts, which is unfortunately excluded by default from GitHub's file search or quick-file selection (T).
      - brasic 4y ago
        Yes! Use `<pattern> -linguist-generated` (the minus sets a negative override for any gitattribute). Here's a test demonstrating that this usage works: https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/test/test_repository.rb#L63-L117 https://github.com/github/linguist/blob/32ec19c013a7f81ffaee...
        jakub_g 4y ago
        Does this really work for jump to file? (we're not talking language statistics or supressing diffs on PRs, which is mostly what linguist readme is talking about). Quoting the docs on finding files: https://docs.github.com/en/search-github/searching-on-github/finding-files-on-github https://docs.github.com/en/search-github/searching-on-github... > File finder results exclude some directories like build, log, tmp, and vendor. To search for files within these directories, use the filename code search qualifier. (The inability of quick jumping to files from /build/ folder with `T` has been driving me crazy for YEARS!) Correct me if I'm wrong, but checking those two files: - https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml https://github.com/github/linguist/blob/master/lib/linguist/... - https://github.com/github/linguist/blob/master/lib/linguist/generated.rb https://github.com/github/linguist/blob/master/lib/linguist/... I don't see `/build` matching anything there. So to me this `/build` suppression from search results seems like controlled by some other piece of software at GitHub :/ Also, files from `/build` are not hidden in diffs, so per this table: https://github.com/github/linguist/blob/HEAD/docs/overrides.md#summary https://github.com/github/linguist/blob/HEAD/docs/overrides.... they are not "linguist-generated".

Jarred 4y ago

> It seems like the main thing that bun does to stay ahead is cache the manifest responses. PNPM, for example, resolves all package versions when installing (without a lockfile), which is slower. This isn't the main optimization. The main optimization is the system calls used to copy/link files. To see the difference, compare `bun install --backend=copyfile` with `bun install --backend=hardlink` (hardlink should be the default). The other big optimization is the binary formats for both the lockfile and the manifest. npm clients waste a lot of time parsing JSON. The more minor optimizations have to do with reducing memory usage. The binary lockfile format interns the strings (very repetitive strings). However, many of these strings are tiny, so it's actually more expensive to store a hash and a length separately from the string itself. Instead, Bun stores the string as 8 bytes and one bit bit says whether the entire string is contained inside those 8 bytes or if it's a memory offset into the lockfile's string buffer (since 64-bit pointers can't use the full memory address and bun currently only targets 64-bit CPUs, this works) yarn also caches the manifest responses. > If I install a simple nextjs app, then remove node_modules, the lockfile, and the ~/.bun/install/cache/.npm files (i.e. keep the contents, remove the manifests) and then install, bun takes around ~3-4s. PNPM is consistently faster for me at around ~2-3s. This sounds like a concurrency bug with scheduling tasks from the main thread to the HTTP thread. I would love someone to help review the code for the thread pool & async io. > One piece of feedback, having the lockfile be binary is a HUGE turn off for me. Impossible to diff. Is there another format? If you do `bun install -y`, it will output as a yarn v1 lockfile. If you add this to your .gitattributes: *.lockb binary diff=lockb It will print the diff as a yarn lockfile.

zebracanevra 4y ago
> If you add this to your .gitattributes: Not applicable to GitHub etc though. I'm also not seeing any speed differences when using -y/yarn lockfile. Why not make it the default?
- brasic 4y ago
  > Not applicable to GitHub etc though. GitHub (disclosure: where I work) does respect some directives in a repo’s .gitattributes file. For example, you can use them to override language detection or mark files as generated or vendored to change diff presentation. You can also improve the diff hunk headers we generate by default by specifying e.g. `*.rb diff=ruby` (although come to think of it I don’t know why that’s necessary since we already know the filetype — I’ll look into it) In principal there’s no reason we couldn’t extend our existing rich diff support used for diffing things like images to enhance the presentation of lockfile diffs. There’s not a huge benefit for text-based lock files but for binary ones (if such a scheme were to take off) it would be a lot more useful.
  - hoten 4y ago
    Any way to use `.gitattributes` to specify a file is _not_ generated? I work on a repo with a build/ directory with build scripts, which is unfortunately excluded by default from GitHub's file search or quick-file selection (T).
    - brasic 4y ago
      Yes! Use `<pattern> -linguist-generated` (the minus sets a negative override for any gitattribute). Here's a test demonstrating that this usage works: https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/test/test_repository.rb#L63-L117 https://github.com/github/linguist/blob/32ec19c013a7f81ffaee...
      - jakub_g 4y ago
        Does this really work for jump to file? (we're not talking language statistics or supressing diffs on PRs, which is mostly what linguist readme is talking about). Quoting the docs on finding files: https://docs.github.com/en/search-github/searching-on-github/finding-files-on-github https://docs.github.com/en/search-github/searching-on-github... > File finder results exclude some directories like build, log, tmp, and vendor. To search for files within these directories, use the filename code search qualifier. (The inability of quick jumping to files from /build/ folder with `T` has been driving me crazy for YEARS!) Correct me if I'm wrong, but checking those two files: - https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml https://github.com/github/linguist/blob/master/lib/linguist/... - https://github.com/github/linguist/blob/master/lib/linguist/generated.rb https://github.com/github/linguist/blob/master/lib/linguist/... I don't see `/build` matching anything there. So to me this `/build` suppression from search results seems like controlled by some other piece of software at GitHub :/ Also, files from `/build` are not hidden in diffs, so per this table: https://github.com/github/linguist/blob/HEAD/docs/overrides.md#summary https://github.com/github/linguist/blob/HEAD/docs/overrides.... they are not "linguist-generated".