5 ms·

Ultralytics AI model hijacked to infect thousands with cryptominer

aucisson_masque 2y ago
> Ultralytics tools are open-source and are used by numerous projects spanning a wide range of industries and applications. Open source and popular doesn't necessarily mean safe. Technically you can read the code source but no one does that and especially for each update.
- smarx007 2y ago
  The vulnerability was not in the source code of the repo. See https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-injection https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj... for a deeper analysis - a great read!
  - lostmsu 2y ago
    It was. The vulnerability is the use of `pull_request_target` on a workflow that has access to secrets.
    - smarx007 2y ago
      You are right, my choice of words was poor. The vuln is exactly as you describe, and it's the malicious payload that was not in the codebase (cache poisoning which is not detectable by reviewing the code of the repo).
- ashishbijlani 2y ago
  This is exactly why I'm building Packj audit [1]. It detects malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj
anakaine 2y ago
Google Collab banned affected users. I wonder how that's going to be resolved, of if Google will just do their usual and make it close to impossible to appeal and get unbanned.
- CatWChainsaw 2y ago
  I think they consult some arcane equation when they need to make a decision. Two of the most important variables are virality of the incident and whether or not anyone affected has an in at Google. Mercury in retrograde messes this equation up.
- macrolime 2y ago
  This is why you should never ever use Google Collab.
  - sva_ 2y ago
    Should be easy to migrate?
- ro_bit 2y ago
  It looks like they did, but the comments after seem to indicate there are still affected users https://github.com/googlecolab/colabtools/issues/4979#issuecomment-2521059512 https://github.com/googlecolab/colabtools/issues/4979#issuec...
Imnimo 2y ago
Somewhat tangentially, I really dislike that Ultralytics (and others) started slapping higher version numbers of their YOLO variants. Redmon used the numbering scheme v2 and v3 for his improvements on his original model. But Ultralytics' 11 is it's own thing with no connection to Redmon. I just think it gives a misleading impression of what the history is.
- kookamamie 2y ago
  Agreed. YOLO, the midel name, was essentially hijacked.
- daemonologist 2y ago
  Ultralytics also had, for at least ~a year, a language model replying to GitHub issues using their CEO's account (without any kind of disclosure). It was frequently confidently incorrect and probably wasted thousands of developer hours (because when the CEO replies to your issue with advice why wouldn't you take it at face value?!) Looks like they've since given the bot its own account but that experience definitely soured me on the company. (Also, there's an MIT licensed implementation of "yolov9" here: https://github.com/WongKinYiu/YOLO https://github.com/WongKinYiu/YOLO . Affiliated with neither Redmond nor Ultralytics as far as I know.)
  - Loughla 2y ago
    >a language model replying to GitHub issues using their CEO's account (without any kind of disclosure) What is the possible justification for this? And did they just not do any oversight at all? Did no one notice the CEO was suddenly full of shit?
    - tensorturtle 2y ago
      The ultralytics/ultralytics repo is pretty beginner friendly (kudos to them for that) but I surmise that it therefore draws a lot of beginner level coders who can't immediately tell that the AI generated "solutions" are bs.
zb3 2y ago
This appears to be a code injection in a PR branch name, not an AI model compromise..
- geraldcombs 2y ago
  Yeah, I think the story that's being missed here is that GitHub allows branches named "$(curl...|bash)".
  - justinclift 2y ago
    Sounds like something GitHub should be easily able to detect too. Hopefully that's an exploit path they'll close soon, if they've not done so already.
  - 3eb7988a1663 2y ago
    What/where does code get evaluated from a branch name?
    - cookiengineer 2y ago
      In Github Actions, because their runners use echo commands to print out environment variables and variables that have been declared via inputs, which in return is parsed in the UI. So technically, all environment variables are unsanitized and this was only the first problem in a list of bugs. This bug specifically used the "pull_request" event/action because it is automatically executed without any chance of stopping it, and was using details exposed via the pull requests head.ref. Next up: git usernames and emails that use shellcode injection names, because github probably won't introduce sanitization to all variables/inputs now. This is a prime example why you should never ever use a shell to log arbitrary data.
  - SahAssar 2y ago
    Github should not disallow/filter branch names. Developers should write (and be be given the tools to) workflows that do not allow command injection.
quuxplusone 2y ago
Serendipitously comes one day after this story[1] was on the front page: at least one Debian maintainer failing to realize the risks of non-alphanumeric usernames. "What could go wrong?" Well, here's Git allowing branch names to contain dollar signs, backticks, etc., because "what could go wrong?"... and... well, this could. Names are identifiers. Allowing identifiers to contain anything besides identifier characters merely opens new and weird attack vectors. [1] https://news.ycombinator.com/item?id=42338134 https://news.ycombinator.com/item?id=42338134

> Ultralytics tools are open-source and are used by numerous projects spanning a wide range of industries and applications. Open source and popular doesn't necessarily mean safe. Technically you can read the code source but no one does that and especially for each update.

smarx007 2y ago
The vulnerability was not in the source code of the repo. See https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-injection https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj... for a deeper analysis - a great read!
- lostmsu 2y ago
  It was. The vulnerability is the use of `pull_request_target` on a workflow that has access to secrets.
  - smarx007 2y ago
    You are right, my choice of words was poor. The vuln is exactly as you describe, and it's the malicious payload that was not in the codebase (cache poisoning which is not detectable by reviewing the code of the repo).
ashishbijlani 2y ago
This is exactly why I'm building Packj audit [1]. It detects malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj

anakaine 2y ago

Google Collab banned affected users. I wonder how that's going to be resolved, of if Google will just do their usual and make it close to impossible to appeal and get unbanned.

CatWChainsaw 2y ago
I think they consult some arcane equation when they need to make a decision. Two of the most important variables are virality of the incident and whether or not anyone affected has an in at Google. Mercury in retrograde messes this equation up.
macrolime 2y ago
This is why you should never ever use Google Collab.
- sva_ 2y ago
  Should be easy to migrate?
ro_bit 2y ago
It looks like they did, but the comments after seem to indicate there are still affected users https://github.com/googlecolab/colabtools/issues/4979#issuecomment-2521059512 https://github.com/googlecolab/colabtools/issues/4979#issuec...

Imnimo 2y ago

Somewhat tangentially, I really dislike that Ultralytics (and others) started slapping higher version numbers of their YOLO variants. Redmon used the numbering scheme v2 and v3 for his improvements on his original model. But Ultralytics' 11 is it's own thing with no connection to Redmon. I just think it gives a misleading impression of what the history is.

kookamamie 2y ago
Agreed. YOLO, the midel name, was essentially hijacked.
daemonologist 2y ago
Ultralytics also had, for at least ~a year, a language model replying to GitHub issues using their CEO's account (without any kind of disclosure). It was frequently confidently incorrect and probably wasted thousands of developer hours (because when the CEO replies to your issue with advice why wouldn't you take it at face value?!) Looks like they've since given the bot its own account but that experience definitely soured me on the company. (Also, there's an MIT licensed implementation of "yolov9" here: https://github.com/WongKinYiu/YOLO https://github.com/WongKinYiu/YOLO . Affiliated with neither Redmond nor Ultralytics as far as I know.)
- Loughla 2y ago
  >a language model replying to GitHub issues using their CEO's account (without any kind of disclosure) What is the possible justification for this? And did they just not do any oversight at all? Did no one notice the CEO was suddenly full of shit?
  - tensorturtle 2y ago
    The ultralytics/ultralytics repo is pretty beginner friendly (kudos to them for that) but I surmise that it therefore draws a lot of beginner level coders who can't immediately tell that the AI generated "solutions" are bs.

zb3 2y ago

This appears to be a code injection in a PR branch name, not an AI model compromise..

geraldcombs 2y ago
Yeah, I think the story that's being missed here is that GitHub allows branches named "$(curl...|bash)".
- justinclift 2y ago
  Sounds like something GitHub should be easily able to detect too. Hopefully that's an exploit path they'll close soon, if they've not done so already.
- 3eb7988a1663 2y ago
  What/where does code get evaluated from a branch name?
  - cookiengineer 2y ago
    In Github Actions, because their runners use echo commands to print out environment variables and variables that have been declared via inputs, which in return is parsed in the UI. So technically, all environment variables are unsanitized and this was only the first problem in a list of bugs. This bug specifically used the "pull_request" event/action because it is automatically executed without any chance of stopping it, and was using details exposed via the pull requests head.ref. Next up: git usernames and emails that use shellcode injection names, because github probably won't introduce sanitization to all variables/inputs now. This is a prime example why you should never ever use a shell to log arbitrary data.
- SahAssar 2y ago
  Github should not disallow/filter branch names. Developers should write (and be be given the tools to) workflows that do not allow command injection.

quuxplusone 2y ago

Serendipitously comes one day after this story[1] was on the front page: at least one Debian maintainer failing to realize the risks of non-alphanumeric usernames. "What could go wrong?" Well, here's Git allowing branch names to contain dollar signs, backticks, etc., because "what could go wrong?"... and... well, this could. Names are identifiers. Allowing identifiers to contain anything besides identifier characters merely opens new and weird attack vectors. [1] https://news.ycombinator.com/item?id=42338134 https://news.ycombinator.com/item?id=42338134