4 ms·

Did you benchmark if a the binary lockfile actually makes any appreciable difference for execution time? Considering the speed with which a fast parser can gob

Did you benchmark if a the binary lockfile actually makes any appreciable difference for execution time? Considering the speed with which a fast parser can gobble up JSON I'm somewhat skeptical that this would be relevant for common operations.

joeblubaugh 4y ago
> The other big optimization is the binary formats for both the lockfile and the manifest. npm clients waste a lot of time parsing JSON. Yes he did
- Ygg2 4y ago
  Where exactly? I don't see it either. Perf data that shows that was the issue.
  - rattray 4y ago
    Take a look at Jarred's twitter, and you'll see he spends a lot of time profiling things: https://news.ycombinator.com/item?id=31993429 https://news.ycombinator.com/item?id=31993429 Of course, I can't say for sure that he looked at the fastest possible way to parse json here, but my intuition would be that if he didn't, it's because he had an educated guess that it'd still be slower.
    - Ygg2 4y ago
      That's not logically connected to statement "parsing JSON is major bottle neck". It's just comparison of execution times of several different package manager. Better would be parsing JSON vs binary in Bun.
jahewson 4y ago
Fast JSON parsers use many exotic tricks. Picking one optimisation and baking it into the file format isn’t so bad.
- the_duke 4y ago
  You don't need to go straight to simdjson et al, something like Rust serde which desierializes to typed structs with data bllike strings borrowed from the input can be very fast.
  - IshKebab 4y ago
    It's still very slow compared to binary formats. Especially indexed ones like SQLite.
    - laumars 4y ago
      Nobody is arguing that JSON is equally as performant as binary formats. What the others are saying is that the amount of JSON in your average lock file should be small enough that parsing it is negligible. If you were dealing with a multi-gigabyte lock file then it would be a different matter but frankly I agree with their point that parsing a lock file which is only a few KB shouldn’t be a differentiator (and if it is, then the JSON parser is the issue, and fixing that should be the priority rather than changing to a binary format). Moreover the earlier comment about lock files needing to be human readable is correct. Being able to read, diff and edit them is absolutely a feature worth preserving even if it costs you a fraction of a second in execution time.
      - IshKebab 4y ago
        > I agree with their point that parsing a lock file which is only a few KB You mean a few MB? NPM projects typically have thousands of dependencies. A 10MB lock file wouldn't be atypical and parse time for a 10MB JSON file can absolutely be significant. Especially if you have to do it multiple times. > Being able to read, diff and edit them is absolutely a feature worth preserving even if it costs you a fraction of a second in execution time. You can read and edit a SQLite file way easier than a huge JSON file.

joeblubaugh 4y ago

> The other big optimization is the binary formats for both the lockfile and the manifest. npm clients waste a lot of time parsing JSON. Yes he did

Ygg2 4y ago
Where exactly? I don't see it either. Perf data that shows that was the issue.
- rattray 4y ago
  Take a look at Jarred's twitter, and you'll see he spends a lot of time profiling things: https://news.ycombinator.com/item?id=31993429 https://news.ycombinator.com/item?id=31993429 Of course, I can't say for sure that he looked at the fastest possible way to parse json here, but my intuition would be that if he didn't, it's because he had an educated guess that it'd still be slower.
  - Ygg2 4y ago
    That's not logically connected to statement "parsing JSON is major bottle neck". It's just comparison of execution times of several different package manager. Better would be parsing JSON vs binary in Bun.

jahewson 4y ago

Fast JSON parsers use many exotic tricks. Picking one optimisation and baking it into the file format isn’t so bad.

the_duke 4y ago
You don't need to go straight to simdjson et al, something like Rust serde which desierializes to typed structs with data bllike strings borrowed from the input can be very fast.
- IshKebab 4y ago
  It's still very slow compared to binary formats. Especially indexed ones like SQLite.
  - laumars 4y ago
    Nobody is arguing that JSON is equally as performant as binary formats. What the others are saying is that the amount of JSON in your average lock file should be small enough that parsing it is negligible. If you were dealing with a multi-gigabyte lock file then it would be a different matter but frankly I agree with their point that parsing a lock file which is only a few KB shouldn’t be a differentiator (and if it is, then the JSON parser is the issue, and fixing that should be the priority rather than changing to a binary format). Moreover the earlier comment about lock files needing to be human readable is correct. Being able to read, diff and edit them is absolutely a feature worth preserving even if it costs you a fraction of a second in execution time.
    - IshKebab 4y ago
      > I agree with their point that parsing a lock file which is only a few KB You mean a few MB? NPM projects typically have thousands of dependencies. A 10MB lock file wouldn't be atypical and parse time for a 10MB JSON file can absolutely be significant. Especially if you have to do it multiple times. > Being able to read, diff and edit them is absolutely a feature worth preserving even if it costs you a fraction of a second in execution time. You can read and edit a SQLite file way easier than a huge JSON file.