4 ms·

I made this Happy to answer any questions or feedback

by Jarred 5y ago

ectopod 5y ago
Don't use 32-bit file times! Change it quick while you have the chance.
throwaway375 5y ago
What do you think about Valve VPK?
brandmeyer 5y ago
If you manage the index using a B-tree, then you can perform partial updates of the B-tree by appending the minimum number of pages needed to represent the changes. At that point, you can append N additional new files to the tail of the archive, and add a new index that re-uses some of the original index. Just an idea to check the "append" box. See also B-trees, Shadowing, and Clones https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf
- Jarred 5y ago
  Good idea, worried a little about impact to serialization/deserialization time though. Maybe could still store it as a flat array
ksec 5y ago
How far are we from public beta or Bun 1.0? Not related to bun or hop is the use of Zig. I am wondering if you will do a write up on it someday.
- Jarred 5y ago
  Something like two weeks before a public beta
Const-me 5y ago
Consider variable-length integers for file sizes and string lengths. When I need them, I usually implementing what’s written in MKV spec: https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-size-integer https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-si... That’s a good way to bypass that 4GB size limit, and instead of wasting 8 bytes per number this will even save a few bytes. However, I’m not sure how easy it is to implement in the language you’re using. I only did that in C++ and modern C#. They both have intrinsics to emit BSWAP instruction (or an ARM equivalent) to flip byte order of an integer, helps with performance of that code.
iainmerrick 5y ago
Have you looked at using LZ4 compression? Supposedly it has a very fast decompressor, so it could be better than uncompressed files for your purposes.
- Jarred 5y ago
  I don't think builtin compression is necessary, but it would be better if the CLI supported it similarly to tar. Compressing a 57.5 MB .hop archive with `gzip -c` returns an 11.7 MB file. This is a similar compression ratio as tar -> tar.gz Compression is great for sending files over the network, but for fast reads, it might be undesirable
  - iainmerrick 5y ago
    It’s worth benchmarking -- LZ4 can decompress faster than SSD read speeds, so it could well be a net time saving.
  - eby 5y ago
    How about compressing the files while building the archive instead of compressing the entire archive itself? That ought to preserve reasonably fast reads. It won't give optimal compression but in practice should do alright (and of course will be smaller than no compression).

I made this Happy to answer any questions or feedback

by Jarred 5y ago

I made this Happy to answer any questions or feedback

ectopod 5y ago

Don't use 32-bit file times! Change it quick while you have the chance.

throwaway375 5y ago

What do you think about Valve VPK?

brandmeyer 5y ago

If you manage the index using a B-tree, then you can perform partial updates of the B-tree by appending the minimum number of pages needed to represent the changes. At that point, you can append N additional new files to the tail of the archive, and add a new index that re-uses some of the original index. Just an idea to check the "append" box. See also B-trees, Shadowing, and Clones https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf

Jarred 5y ago
Good idea, worried a little about impact to serialization/deserialization time though. Maybe could still store it as a flat array

ksec 5y ago

How far are we from public beta or Bun 1.0? Not related to bun or hop is the use of Zig. I am wondering if you will do a write up on it someday.

Jarred 5y ago
Something like two weeks before a public beta

Const-me 5y ago

Consider variable-length integers for file sizes and string lengths. When I need them, I usually implementing what’s written in MKV spec: https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-size-integer https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-si... That’s a good way to bypass that 4GB size limit, and instead of wasting 8 bytes per number this will even save a few bytes. However, I’m not sure how easy it is to implement in the language you’re using. I only did that in C++ and modern C#. They both have intrinsics to emit BSWAP instruction (or an ARM equivalent) to flip byte order of an integer, helps with performance of that code.

iainmerrick 5y ago

Have you looked at using LZ4 compression? Supposedly it has a very fast decompressor, so it could be better than uncompressed files for your purposes.

Jarred 5y ago
I don't think builtin compression is necessary, but it would be better if the CLI supported it similarly to tar. Compressing a 57.5 MB .hop archive with `gzip -c` returns an 11.7 MB file. This is a similar compression ratio as tar -> tar.gz Compression is great for sending files over the network, but for fast reads, it might be undesirable
- iainmerrick 5y ago
  It’s worth benchmarking -- LZ4 can decompress faster than SSD read speeds, so it could well be a net time saving.
- eby 5y ago
  How about compressing the files while building the archive instead of compressing the entire archive itself? That ought to preserve reasonably fast reads. It won't give optimal compression but in practice should do alright (and of course will be smaller than no compression).