4 ms·

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems includ

by agwa 12y ago

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function by taking the memory address of the XML node struct, which makes documentation generated with XSLT non-reproducible (memory addresses are non-deterministic because of address space randomization and randomized hash tables). (I'm the author of strip-nondeterminism, a tool in our custom toolchain that attempts to normalize files after they're built.)

contingencies 12y ago
strip-nondeterminism "is a Perl module for stripping bits of non-deterministic information, such as timestamps and file system order, from files such as gzipped files, ZIP archives, and Jar files. It can be used as a post-processing step to make a build reproducible, when the build process itself cannot be made deterministic. It is used as part of the Reproducible Builds project." Browse source @ https://anonscm.debian.org/cgit/reproducible/strip-nondeterminism.git/tree/ https://anonscm.debian.org/cgit/reproducible/strip-nondeterm... PS. non-deterministic filesystem order .. doesn't this go away if you use tmpfs?
jzwinck 12y ago
Maybe you could fix the libxslt problem by using a pool allocator for the nodes. The pool would contain only the nodes, and you could use the offset in the pool as their ID, rather than the full virtual address. Just some food for thought.
- csmeu 12y ago
  That probably isn't architecture independent.
- agwa 12y ago
  That's a very interesting idea. It's possible to tell libxml2 to use a custom allocator, but unfortunately it will use that allocator for all objects, not just nodes, and thus would probably be affected by hash table randomization. The architecture dependence is also a problem. My current solution is a patch that uses a hash table to map the memory address of a node to a counter that increments in a deterministic order. It's a massive hack and I hate it. I'm currently trying to assess what its performance impact is. The proper solution would be for libxml2 to maintain a counter and assign every node a deterministic ID, but that would require extending the _xmlNode struct, which would break ABI compatibility because the struct is not opaque.
dochtman 12y ago
Thanks for your work! I've been wondering -- why do compiled binaries actually include timestamps? For human-readable stuff, I sort of understand, but for binaries I don't see we'd want to include timestamps.
- revelation 12y ago
  I guess it's related to debugging purposes. If you know when a binary was built, you know that a source file X with a changetime later than the build is different to the source file used for building the binary.

by agwa 12y ago

contingencies 12y ago

strip-nondeterminism "is a Perl module for stripping bits of non-deterministic information, such as timestamps and file system order, from files such as gzipped files, ZIP archives, and Jar files. It can be used as a post-processing step to make a build reproducible, when the build process itself cannot be made deterministic. It is used as part of the Reproducible Builds project." Browse source @ https://anonscm.debian.org/cgit/reproducible/strip-nondeterminism.git/tree/ https://anonscm.debian.org/cgit/reproducible/strip-nondeterm... PS. non-deterministic filesystem order .. doesn't this go away if you use tmpfs?

jzwinck 12y ago

Maybe you could fix the libxslt problem by using a pool allocator for the nodes. The pool would contain only the nodes, and you could use the offset in the pool as their ID, rather than the full virtual address. Just some food for thought.

csmeu 12y ago
That probably isn't architecture independent.
agwa 12y ago
That's a very interesting idea. It's possible to tell libxml2 to use a custom allocator, but unfortunately it will use that allocator for all objects, not just nodes, and thus would probably be affected by hash table randomization. The architecture dependence is also a problem. My current solution is a patch that uses a hash table to map the memory address of a node to a counter that increments in a deterministic order. It's a massive hack and I hate it. I'm currently trying to assess what its performance impact is. The proper solution would be for libxml2 to maintain a counter and assign every node a deterministic ID, but that would require extending the _xmlNode struct, which would break ABI compatibility because the struct is not opaque.

dochtman 12y ago

Thanks for your work! I've been wondering -- why do compiled binaries actually include timestamps? For human-readable stuff, I sort of understand, but for binaries I don't see we'd want to include timestamps.

revelation 12y ago
I guess it's related to debugging purposes. If you know when a binary was built, you know that a source file X with a changetime later than the build is different to the source file used for building the binary.