6 ms·

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far: 1. Looks like this originated from the trivvy used in our ci/cd -

• redrove 6mo ago
  >1. Looks like this originated from the trivvy used in our ci/cd Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?
  • wheelerwj 6mo ago
    The latest trivy attack was announced just yesterday. If you go out to dinner or take a night off its totally plausible to have not seen it.
    • franktankbank 6mo ago
      [flagged]
      • embedding-shape 6mo ago
        Probably more "serious human" than "serious over-capitalist" or "seriously overworked". Good for them.
    • anishgupta 6mo ago
      afaik the trivy attack was first in the news on March 19th for the github actions and for docker images it was on March 23rd
• Imustaskforhelp 6mo ago
  > - Krrish Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts) Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not) I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.
  • redrove 6mo ago
    >I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well. Token in CI could've been way too broad.
  • franktankbank 6mo ago
    He would have to state he didn't in fact make all those commits and close the issue.
  • detente18 6mo ago
    It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy. We have deleted all our pypi publishing tokens. Our accounts had 2fa, so it's a bad token here. We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).
    • redrove 6mo ago
      How did PYPI_PUBLISH lead to a full GH account takeover?
      • franktankbank 6mo ago
        Don't hold your breath for an answer.
      • ezekg 6mo ago
        I'd imagine the attacker published a new compromised version of their package, which the author eventually downloaded, which pwned everything else.
• outside2344 6mo ago
  Is it just in 1.82.8 or are previous versions impacted?
  • Imustaskforhelp 6mo ago
    1.82.7 is also impacted if I remember correctly.
    • GrayShade 6mo ago
      1.82.7 doesn't have litellm_init.pth in the archive. You can download them from pypi to check. EDIT: no, it's compromised, see proxy/proxy_server.py.
      • cpburns2009 6mo ago
        1.82.7 has the payload in `litellm/proxy/proxy_server.py` which executes on import.
  • deleted 6mo ago
    [deleted]
• bognition 6mo ago
  The decision to block all downloads is pretty disruptive, especially for people on pinned known good versions. Its breaking a bunch of my systems that are all launched with `uv run`
  • cpburns2009 6mo ago
    That's PyPI's behavior when they quarantine a package.
  • Shank 6mo ago
    > Its breaking a bunch of my systems that are all launched with `uv run` From a security standpoint, you would rather pull in a library that is compromised and run a credential stealer? It seems like this is the exact intended and best behavior.
  • MeetingsBrowser 6mo ago
    Are you sure you are pinned to a “known good” version? No one initially knows how much is compromised
  • tedivm 6mo ago
    You should be using build artifacts, not relying on `uv run` to install packages on the fly. Besides the massive security risk, it also means that you're dependent on a bunch of external infrastructure every time you launch. PyPI going down should not bring down your systems.
    • zbentley 6mo ago
      This is the right answer. Unfortunately, this is very rarely practiced. More strangely (to me), this is often addressed by adding loads of fallible/partial caching (in e.g. CICD or deployment infrastructure) for package managers rather than building and publishing temporary/per-user/per-feature ephemeral packages for dev/testing to an internal registry. Since the latter's usually less complex and more reliable, it's odd that it's so rarely practiced.
    • lanstin 6mo ago
• kleton 6mo ago
  There are hundreds of PRs fixing valid issues to your github repo seemingly in limbo for weeks. What is the maintainer state over there?
  • zparky 6mo ago
    Not really the time for that. There's also PRs being merged every hour of the day.
  • michh 6mo ago
    increasing the (social) pressure on maintainers to get PRs merged seems like the last thing you should be doing in light of preventing malicious code ending up in dependencies like this i'd much rather see a million open PRs than a single malicious PR sneak through due to lack of thorough review.
• detente18 6mo ago
  Update: - Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted We are still scanning our project to see if there's any more gaps. If you're a security expert and want to help, email me - krrish@berri.ai
  • cosmicweather 6mo ago
    > All maintainer accounts have been changed What about the compromised accounts(as in your main account)? Are they completely unrecoverable?
    • detente18 6mo ago
      I deleted it, to be safe.
  • MadsRC 6mo ago
    Dropped you a mail from mads.havmand@nansen.ai
    • ting0 6mo ago
      [flagged]
      • kvdveer 6mo ago
        > If you're a security expert and want to help, email me ... And > Dropped you a mail from [email] I don't think there is any indication of a compromise, they are just offering help.
        • ij23 6mo ago
          Hi all, Ishaan from LiteLLM here (LiteLLM maintainer) The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI. We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow. All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri. Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe. From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages. The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately. We are actively investigating full scope and blast radius. Our immediate next steps include: reviewing all BerriAI repositories for impact, scanning CircleCI builds to understand blast radius and mitigate it, hardening release and publishing controls, including maintainership and credential governance, and strengthening our incident communication process for enterprise customers. We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.
• ozozozd 6mo ago
  Kudos for this update. Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.
• harekrishnarai 6mo ago
  > it seems your personal account is also compromised. I just checked for the github search here https://github.com/search?q=%22teampcp+owns%22 https://github.com/search?q=%22teampcp+owns%22
• vintagedave 6mo ago
  This must be super stressful for you, but I do want to note your "I'm sorry for this." It's really human. It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward." Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.
  • cyanydeez 6mo ago
    Lawyers are slowly eating humanity.
    • singleshot_ 6mo ago
      Allegedly*
    • bmurphy1976 6mo ago
      For now. They're about to get hit by the AI wave as bad as us software devs. Who knows what's on the other side of this.
      • blueone 6mo ago
        Sorry that I have to be the one to tell you this, but lawyers are fine. Sure, AI will have an impact, but nothing like the once hyped idea that it would replace lawyers. It has actually been amusing to watch the hype cycle play out around AI when it comes to lawyers.
        • throwawaytea 6mo ago
          My parents had a weird green card and paperwork issue that was becoming a big problem. Everyone in their social circle recommended an immigration type lawyer. Everyone. My dad was confident he could figure it out based on his perplexity Pro account. He attacked the problem from several angles and used it for help with what to do, how to do it, what to ask for when visiting offices, how to press them to move forward, and tons of other things. Got the problem resolved. So it definitely can reduce hiring lawyers even.
• Imustaskforhelp 6mo ago
  I just want to share an update the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518 https://github.com/BerriAI/litellm/issues/24518
• mrexcess 6mo ago
  You're making great software and I'm sorry this happened to you. Don't get discouraged, keep bringing the open source disruption!
• kingreflex 6mo ago
  we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe? also how are we sure that docker images aren't affected?
  • saltyoldman 6mo ago
    Docker deployments are more safe even if affected because there is a lower chance (but not zero) that you didn't mount all your credentials into the image. It would have access to LLM keys of course, but that's not really what the hacker is after. He's after private SSH keys. That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.
    • anishgupta 6mo ago
      yep joining here late but docker images are not affected as we saw on twitter
• rao-v 6mo ago
  I put together a little script to search for and list installed litellm versions on my systems here: https://github.com/kinchahoy/uvpowered-tools/blob/main/inventory_litellm.sh https://github.com/kinchahoy/uvpowered-tools/blob/main/inven... It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters. Obviously read it before running it etc.
• daprichard 6mo ago
  [flagged]
• mikert89 6mo ago
  Similar to delve, this guy has almost no work experience. You have to wonder if YC and the cult of extremely young founders is causing instability issues in society at large?
  • zdragnar 6mo ago
    Welcome to the new era, where programming is neither a skill nor a trade, but a task to be automated away by anyone with a paid subscription.
    • mikert89 6mo ago
      alot of software isnt that important so its fine, but some actually is important. especially with a branding name slapped on it that people will trust
      • whattheheckheck 6mo ago
        The industry needs to step up and plant a flag for professionalization certifications for proper software engineering. Real hard exams etc
        • jacamera 6mo ago
          I can't even imagine what these exams would look like. The entire profession seems to boil down to making the appropriate tradeoffs for your specific application in your specific domain using your specific tech stack. There's almost nothing that you always should or shouldn't do.
          • xenophonf 6mo ago
            All engineering professions are like that. NCEES has been licensing Professional Engineers for over a hundred years. The only thing stopping CS/SE is an unwillingness to submit to anything resembling oversight.
      • deleted 6mo ago
• pojzon 6mo ago
  This is just one of many projects that was a victim of Trivy hack. There are millions of those projects and this issue will be exploited in next months if not years.
• driftnode 6mo ago
  the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential
  • kreelman 6mo ago
    I wonder if there are a few things here.... It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does. It seems like software tests need to have a class of test that checks whether any of the components of an application have been compromised in some way. This in itself may be somewhat complex... We are in a world where we can't assume secure operation of components anymore. This is kinda sad, but here we are....
    • driftnode 6mo ago
      The sad part is you're right that we can't assume secure operation of components anymore, but the tooling hasn't caught up to that reality. Chroot jails help with runtime isolation but the attack here happened at build time, the malicious code was already in the package before any test could run. And the supply chain is deep. Trivy gets compromised, which gives CI access, which gives PyPI access. Even if you jail your own builds you're trusting that every tool in your pipeline wasn't the entry point. 97 million monthly downloads means a lot of people's "secure" pipelines just ran attacker code with full access.
  • sobellian 6mo ago
    If the payload is a credential stealer then they can use that to escalate into basically anything right?
    • driftnode 6mo ago
      Yes and the scary part is you might never know the full extent. A credential stealer grabs whatever is in memory or env during the build, ships it out, and the attacker uses those creds weeks later from a completely different IP. The compromised package gets caught and reverted, everyone thinks the incident is over, meanwhile the stolen tokens are still valid. I wonder how many teams who installed 1.82.7 actually rotated all their CI secrets after this, not just uninstalled the bad version.
• N_Lens 6mo ago
  Good work! Sorry to hear you're in this situation, good luck and godspeed!
• edf13 6mo ago
  [dead]