4 ms·

Some time ago I made a fun hack where I abuse the Go transparency log to store immutable url->fingerprint mappings of arbitrary urls: https://github.com/mkmik/

Some time ago I made a fun hack where I abuse the Go transparency log to store immutable url->fingerprint mappings of arbitrary urls: https://github.com/mkmik/getsum https://github.com/mkmik/getsum Example: $ getsum https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh 9808adad931ac652e8ff5022a74507c532250c2091d21d6aebc7064573669cc5 now, if somebody ever replaces that file on github, getsum will fail with an error. (I was too lazy to make getsum actually download the file if the checksum matches, but that should be trivial; for now just do "getsum $url && wget $url"). This works by creating synthetic Go modules that embed "facts" about URLs inside generated Go code, which gets then downloaded by getsum and parsed using the stdlib "go/parser" package. Since all Go modules are recorded in the Go "sumdb" public transparency log, once this module has been generated it cannot ever change. (Getsum uses a constant "version" for the synthetic packages, thus facts are immutable). I'm confident this works because it piggy-backs on the stable production transparent logs infrastructure of a high-profile project managed by a high-profile company. On the other hand the Go transparency logs admins could ban my "getsum.pub" domain if the generated traffic bothers them. This was for me just a quick way to show to some colleagues the potentials of transparency logs etc, and of course to have fun.

dependenttypes 6y ago
> (I was too lazy to make getsum actually download the file if the checksum matches, but that should be trivial; for now just do "getsum $url && wget $url"). Please don't. Instead check the hash of the local file. A malicious server could serve a different file for getsum and for wget (bonus point: you won't have to download it twice)
- ithkuil 6y ago
  yes that's indeed a problem that needs to be dealt with. That said, technically the getsum client doesn't download the file; it only fetches "$url.sha256" if present or a "SHA256SUMS" in the same directory and compares that with the one stored in the transparency log. It follows a common pattern on release sites (such as major projects on github). In other words getsum only ensures that the checksum file itself is immutable, and the checksum file then is used to ensure that bigger file also hasn't changed. The reason for that lies in the fact that getsum has a server side component (hosted on getsum.pub) that serves the Go modules fetched by the Go sumdb. Thus, the correct instructions are: $ (echo -n $(getsum https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh); echo " bazel-3.4.1-installer-linux-x86_64.sh") > bazel-3.4.1-installer-linux-x86_64.sh.sha256 && wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh && sha256sum -c bazel-3.4.1-installer-linux-x86_64.sh.sha256 Big file downloaders tend to be more complicated than expected; progress bars, resuming downloads etc etc. Perhaps getsum instead of downloading the file it could verify the file? E.g.: $ wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh && getsum -c bazel-3.4.1-installer-linux-x86_64.sh https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh EDIT: just implemented the "-c" flag described above

dependenttypes 6y ago

> (I was too lazy to make getsum actually download the file if the checksum matches, but that should be trivial; for now just do "getsum $url && wget $url"). Please don't. Instead check the hash of the local file. A malicious server could serve a different file for getsum and for wget (bonus point: you won't have to download it twice)

ithkuil 6y ago
yes that's indeed a problem that needs to be dealt with. That said, technically the getsum client doesn't download the file; it only fetches "$url.sha256" if present or a "SHA256SUMS" in the same directory and compares that with the one stored in the transparency log. It follows a common pattern on release sites (such as major projects on github). In other words getsum only ensures that the checksum file itself is immutable, and the checksum file then is used to ensure that bigger file also hasn't changed. The reason for that lies in the fact that getsum has a server side component (hosted on getsum.pub) that serves the Go modules fetched by the Go sumdb. Thus, the correct instructions are: $ (echo -n $(getsum https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh); echo " bazel-3.4.1-installer-linux-x86_64.sh") > bazel-3.4.1-installer-linux-x86_64.sh.sha256 && wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh && sha256sum -c bazel-3.4.1-installer-linux-x86_64.sh.sha256 Big file downloaders tend to be more complicated than expected; progress bars, resuming downloads etc etc. Perhaps getsum instead of downloading the file it could verify the file? E.g.: $ wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh && getsum -c bazel-3.4.1-installer-linux-x86_64.sh https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh EDIT: just implemented the "-c" flag described above