4 ms·

I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most dif

by slaymaker1907 3y ago

I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most difficult step). Make sure logging into or changing any of the signing stuff requires at least 2 of the devs, one to initiate the JIT and one to monitor. Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine. Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.

bawolff 3y ago
I do not think any of that would have prevented this situation.
- slaymaker1907 3y ago
  Upon further reading, I think you might be correct. I initially thought a good signing process would be sufficient since it sounded like this malicious blob was secretly being included in the tarball by the build server, but it instead seems to be the case that the malicious binary was included in the repo as a test file. You could probably still protect against this sort of attack using signing, but it would be much more laborious and annoying to get working. The idea is that you would somehow declare that OpenSSH binaries must be signed by a *particular* key/authority, that VS Code is signed by Microsoft, Chrome signed by Google, etc. Additionally, the config declaring all of this obviously needs to be secured so you'd need to lock down access to those files (changing key associations would need to require more permissions than just updating old software or installing new software to be useful).
  - Aloisius 3y ago
    It was both. The binary object file was obfuscated in two "test" xz files in the repo itself. The code to extract the object file and inject it into the build process was only in the GitHub release tarball. The code in the tarball could have been prevented if only automated tarballs were permitted (for instance, GitHub's branch/tag source tarballs) or caught after the fact by verifying the file hashes in the tarball against those in the repo.
    - bawolff 3y ago
      > The code to extract the object file and inject it into the build process was only in the GitHub release tarball. I thought it was in both, however the backdoor wasn't inserted with the default build time options, so it ended up in the tarball, but not if you just did make unless you set things up the same as the build server.
      - Aloisius 3y ago
        The release tarball contained code that can't be generated from the repo, no matter what settings you use. JiaT75 created the release on GitHub which involves creating a description and uploading tarballs. Their PGP key has been used to sign them since May 2023. 1. The repo contained two test xz files which contained the obfuscated exploit .o file 2. JiaT75 created the release on GitHub which allows you to upload your own tarball for the release. 3. The release tarball contained a modified build-to-host.m4 that extracted the exploit .o so it would be linked into the library if building on a Linux x86-64 machine. 4. Fedora/Debian/whoever pulled the release source tarball from GitHub and ran their builds for x86-64 Linux which caused the exploit .o file to be linked into the binary they then packaged up for release
selfmodruntime 3y ago
All this riffraff and OSS maintainers basically work for free.

bawolff 3y ago

I do not think any of that would have prevented this situation.

slaymaker1907 3y ago
Upon further reading, I think you might be correct. I initially thought a good signing process would be sufficient since it sounded like this malicious blob was secretly being included in the tarball by the build server, but it instead seems to be the case that the malicious binary was included in the repo as a test file. You could probably still protect against this sort of attack using signing, but it would be much more laborious and annoying to get working. The idea is that you would somehow declare that OpenSSH binaries must be signed by a *particular* key/authority, that VS Code is signed by Microsoft, Chrome signed by Google, etc. Additionally, the config declaring all of this obviously needs to be secured so you'd need to lock down access to those files (changing key associations would need to require more permissions than just updating old software or installing new software to be useful).
- Aloisius 3y ago
  It was both. The binary object file was obfuscated in two "test" xz files in the repo itself. The code to extract the object file and inject it into the build process was only in the GitHub release tarball. The code in the tarball could have been prevented if only automated tarballs were permitted (for instance, GitHub's branch/tag source tarballs) or caught after the fact by verifying the file hashes in the tarball against those in the repo.
  - bawolff 3y ago
    > The code to extract the object file and inject it into the build process was only in the GitHub release tarball. I thought it was in both, however the backdoor wasn't inserted with the default build time options, so it ended up in the tarball, but not if you just did make unless you set things up the same as the build server.
    - Aloisius 3y ago
      The release tarball contained code that can't be generated from the repo, no matter what settings you use. JiaT75 created the release on GitHub which involves creating a description and uploading tarballs. Their PGP key has been used to sign them since May 2023. 1. The repo contained two test xz files which contained the obfuscated exploit .o file 2. JiaT75 created the release on GitHub which allows you to upload your own tarball for the release. 3. The release tarball contained a modified build-to-host.m4 that extracted the exploit .o so it would be linked into the library if building on a Linux x86-64 machine. 4. Fedora/Debian/whoever pulled the release source tarball from GitHub and ran their builds for x86-64 Linux which caused the exploit .o file to be linked into the binary they then packaged up for release

selfmodruntime 3y ago

All this riffraff and OSS maintainers basically work for free.