peterwaller-arm

6 ms·

1.
by peterwaller-arm 5y ago
Author here. This is correct, we set out to do binary diffing but we soon discovered that if you put similar enough object files together in a stream, and then compress the stream, zstandard does a fantastic job at compressing and decompres
2.
by peterwaller-arm 5y ago
Yes, this is less great than I would like! :( :)
3.
by peterwaller-arm 5y ago
Author here, I'd like to see such a comparison too actually, but I'm not in the position to do the work at the moment. We did some preliminary experiments at the beginning, but a lot changed over the course of the project and I do
4.
by peterwaller-arm 5y ago
Yes you could do that. On the other hand, all revisions for a month is 100MiB, and all revisions we've built spanning 2019-now are a total of 2.8GiB, so we opted to forego implementing any object negotiation and just say 'you have
5.
by peterwaller-arm 5y ago
Nice, I bet dwarfs would do well at our use case too. Thanks for sharing.
6.
by peterwaller-arm 5y ago
ThinLTO can be pretty quick if you have enough cores, it might work. Not sure how well the LTO objects compress against each other when you have small changes to them. It might work reasonably. manyclangs is optimized to provide you with a
7.
by peterwaller-arm 5y ago
> maybe mention this as well in the README? We've tweaked the readme, I hope it's clearer. It would be great to provide this for gcc too. The project is new and we've just started out. I know less about gcc's build s
8.
by peterwaller-arm 5y ago
(Copying from Q&A) Before starting out some time ago, I did some experiments with bup. I had a good experience with bup and high expectations for it. However, I found that quite a lot of performance was left on the table, so I was motiv
9.
by peterwaller-arm 5y ago
<3
10.
by peterwaller-arm 5y ago
We've just added an applicability section, which explains a bit more what we do. We don't have any ELF specific heuristics [0]. https://github.com/elfshaker/elfshaker#applicability In summary, for manyclangs,
11.
by peterwaller-arm 5y ago
Author here. Compressed data is unlikely to work well in general, unless it never changes.
12.
by peterwaller-arm 5y ago
Author here. No architecture specific processing currently. Most of the magic happens in zstandard (hat tip to this amazing project). Please see our new applicability section which explains the result in a bit more detail: https:/&#x2
13.
by peterwaller-arm 5y ago
Done, hopefully this is clearer. Please let us know if you see a way to improve it further: https://github.com/elfshaker/elfshaker/pull/60
14.
by peterwaller-arm 5y ago
(Disclosure: I work for Arm, opinions are my own) Author here. elfshaker itself does not have a dependency on any architecture to our knowledge. We support the architectures we have use of. Contributions to add missing support are welcome.
15.
by peterwaller-arm 5y ago
Author here. The executables shipped in manyclangs are release builds! The catch is that manyclangs stores object files pre-link. Executables are materialized by relinking after they are extracted with elfshaker. The stored object files are
16.
by peterwaller-arm 5y ago
Author here. I've used bup, and elfshaker was partially inspired by it! It's great. However, during initial experiments on this project I found bup to be slow, taking quite a long time to snapshot and extract. I think this could i
17.
by peterwaller-arm 5y ago
Author here, I agree with xdfgh1112, please take care before using brand new software to store your backups!
18.
by peterwaller-arm 5y ago
Author here, this software is young, please don't use it for backups! But also, in general, it might not work well for your use case, and our use case is niche. Please give it a try before making assumptions about any suitability for u
19.
by peterwaller-arm 5y ago
Performing poorly with non-textual data happens for a a number of reasons. Binary data, when changed, often have a lot of 'non-local' changes in them. For example, a PSD file might well have a compression algorithm already applied
20.
by peterwaller-arm 5y ago
Author here, I don't think it would apply well to that scenario. elfshaker is good for manyclangs where we ship 2,000 revisions in one file (pack), so the cost of individual revision is amortized. If one build of llvm+clang costs you s
21.
by peterwaller-arm 5y ago
Author here, it works particularly well for our presented use case because it has these properties: * There are many files, * Most of them don't change very often, * When they do change, the deltas of the binaries are not huge. So, if
22.
by peterwaller-arm 5y ago
Author here, I don't currently know how this compares to git-lfs. It it is possible git-lfs would perform quite well on the same inputs as elfshaker works on. If git-lfs does already work well for your use case I'd recommend using
23.
by peterwaller-arm 5y ago
Author here. I'm unsure whether this would apply very well to cargo or not. If it has lots of pre-link object files, then maybe.
24.
by peterwaller-arm 5y ago
Author here. Maybe, it's a fun idea. I have toyed with providing a fuse filesystem for access to a pack but my time for completing this is limited at the moment.
25.
by peterwaller-arm 5y ago
Author here. We'd love this to be a thing, but this is young software, so we don't recommend relying on this as a single way of doing a backup for now. Bear in mind that our main use case is for things that you can reproduce in pr
26.
by peterwaller-arm 5y ago
One of the authors here, thanks for the feedback. We've tried to improve it here: https://github.com/elfshaker/elfshaker/pull/59
27.
by peterwaller-arm 5y ago
An author here, I agree! The packfile format is heavily inspired by git, and git may also do quite well at this. We did some preliminary experiments with git a while back but found we were able to do the packing and extraction much faster a