3 ms·

One quite annoying element is that as a third party you cannot access the attestations of the deleted releases any more. I really wanted to see if the attestati

One quite annoying element is that as a third party you cannot access the attestations of the deleted releases any more. I really wanted to see if the attestations would help here to figure out what happened. But maybe I’m just not informed enough about where to look. Another element here is that the releases seemingly were deleted and re-created? I thought that was prevented by PyPI?

woodruffw 2y ago
The attestations are checked into the public transparency log, so they’re still accessible — that’s how I did a decent amount of the triage in the write up. You can find them in the write up by searching for “Sigstore” (I would direct link them, but I’m on mobile). > Another element here is that the releases seemingly were deleted and re-created? I thought that was prevented by PyPI? Hmm, where do you see this? The release history on PyPI doesn’t show any recreations[1]. [1]: https://pypi.org/project/ultralytics/ https://pypi.org/project/ultralytics/
- the_mitsuhiko 2y ago
  > You can find them in the write up by searching for “Sigstore” (I would direct link them, but I’m on mobile). Yeah, I know they are in sigstore, I just did not know how to find them. Is there an interface for this I missed? > Hmm, where do you see this? The release history on PyPI doesn’t show any recreations[1]. Then I completely misunderstood what happened. Was this in fact completely made up releases that were not even intended to be triggered? Eg: a bot released .41 without there being an intent of being an actual .41 release? I thought that UltralyticsAssistant was the developer, not the attacker. Do they also control that thing?
  - woodruffw 2y ago
    > Is there an interface for this I missed? That would be search.sigstore.dev, unless I'm misunderstanding what you mean. > Was this in fact completely made up releases that were not even intended to be triggered? Eg: a bot released .41 without there being an intent of being an actual .41 release? I thought that UltralyticsAssistant was the developer, not the attacker. Do they also control that thing? .41 and .42 were triggered directly from the repository. One was triggered by the UltralyticsAssistant account and included a human bypass, which strongly suggests that the attacker controlled (and maybe still controls) that bot account. The last two compromised releases were published directly via API token, not via the source repo, which strongly suggests that the attacker either exfil’d an old API token from CI/CD or that they’re in control of the developer’s account on PyPI. Those ones don’t have attestations, while the first two releases do (two each, one per dist per release).
    - the_mitsuhiko 2y ago
      > .41 and .42 were triggered directly from the repository. One was triggered by the UltralyticsAssistant account and included a human bypass, which strongly suggests that the attacker controlled (and maybe still controls) that bot account. Ah, but if they controlled the bot then didn't they have other problems too? If that is the case, then disregard my comment. I was under the impression that this was not the attacker. > That would be search.sigstore.dev, unless I'm misunderstanding what you mean. No, that's it in theory I suppose. I did try this but when I used the commit I thought triggered the release (cb260c243ffa3e0cc84820095cd88be2f5db86ca) I did not see it show up.
      - woodruffw 2y ago
        > Ah, but if they controlled the bot then didn't they have other problems too? Yep — my theory is that this all starts with the insecure trigger + template injection, and that the attacker exfil’d the bot’s PAT and stale PyPI API token at that point. The first round of attacks used the bot PAT and cache poisoning, and then the attacker pivoted to the PyPI token once the first vector was closed off. > I used the commit I thought triggered the release (cb260c243ffa3e0cc84820095cd88be2f5db86ca) I did not see it show up I think I know what’s happening there: that search UI only indexes by commit for attestations produced by gitsign, not every attestation containing a commit. I used it by finding the entry IDs from the release action logs on GitHub Actions, but if/when those are flushed someone who doesn’t already know them will need to seek the log in order to find the attestations. That’s not ideal; the search.sigstore.dev service would ideally have more indices like crt.sh does.

woodruffw 2y ago

The attestations are checked into the public transparency log, so they’re still accessible — that’s how I did a decent amount of the triage in the write up. You can find them in the write up by searching for “Sigstore” (I would direct link them, but I’m on mobile). > Another element here is that the releases seemingly were deleted and re-created? I thought that was prevented by PyPI? Hmm, where do you see this? The release history on PyPI doesn’t show any recreations[1]. [1]: https://pypi.org/project/ultralytics/ https://pypi.org/project/ultralytics/

the_mitsuhiko 2y ago
> You can find them in the write up by searching for “Sigstore” (I would direct link them, but I’m on mobile). Yeah, I know they are in sigstore, I just did not know how to find them. Is there an interface for this I missed? > Hmm, where do you see this? The release history on PyPI doesn’t show any recreations[1]. Then I completely misunderstood what happened. Was this in fact completely made up releases that were not even intended to be triggered? Eg: a bot released .41 without there being an intent of being an actual .41 release? I thought that UltralyticsAssistant was the developer, not the attacker. Do they also control that thing?
- woodruffw 2y ago
  > Is there an interface for this I missed? That would be search.sigstore.dev, unless I'm misunderstanding what you mean. > Was this in fact completely made up releases that were not even intended to be triggered? Eg: a bot released .41 without there being an intent of being an actual .41 release? I thought that UltralyticsAssistant was the developer, not the attacker. Do they also control that thing? .41 and .42 were triggered directly from the repository. One was triggered by the UltralyticsAssistant account and included a human bypass, which strongly suggests that the attacker controlled (and maybe still controls) that bot account. The last two compromised releases were published directly via API token, not via the source repo, which strongly suggests that the attacker either exfil’d an old API token from CI/CD or that they’re in control of the developer’s account on PyPI. Those ones don’t have attestations, while the first two releases do (two each, one per dist per release).
  - the_mitsuhiko 2y ago
    > .41 and .42 were triggered directly from the repository. One was triggered by the UltralyticsAssistant account and included a human bypass, which strongly suggests that the attacker controlled (and maybe still controls) that bot account. Ah, but if they controlled the bot then didn't they have other problems too? If that is the case, then disregard my comment. I was under the impression that this was not the attacker. > That would be search.sigstore.dev, unless I'm misunderstanding what you mean. No, that's it in theory I suppose. I did try this but when I used the commit I thought triggered the release (cb260c243ffa3e0cc84820095cd88be2f5db86ca) I did not see it show up.
    - woodruffw 2y ago
      > Ah, but if they controlled the bot then didn't they have other problems too? Yep — my theory is that this all starts with the insecure trigger + template injection, and that the attacker exfil’d the bot’s PAT and stale PyPI API token at that point. The first round of attacks used the bot PAT and cache poisoning, and then the attacker pivoted to the PyPI token once the first vector was closed off. > I used the commit I thought triggered the release (cb260c243ffa3e0cc84820095cd88be2f5db86ca) I did not see it show up I think I know what’s happening there: that search UI only indexes by commit for attestations produced by gitsign, not every attestation containing a commit. I used it by finding the entry IDs from the release action logs on GitHub Actions, but if/when those are flushed someone who doesn’t already know them will need to seek the log in order to find the attestations. That’s not ideal; the search.sigstore.dev service would ideally have more indices like crt.sh does.