5 ms·
We got admin access to Baseten's production GitHub in 25 minutes
- philipkiely 18d agoHey all Philip from Baseten here. Posting this on behalf of our security team. I wanted to confirm that we collaborated with Strix on the remediation of the reported vulnerability. We thank Strix for their responsible disclosure. We took immediate steps to invalidate the leaked key and remove the public container image. Our logs confirm the vulnerability was never exploited and no customer data was exposed.
- bearsyankees 18d ago+1 -- kudos to the Baseten team for their super professional response to all of this, it is clear why they are a generational company (-- Alex from Strix)
- ej_campbell 18d agoWhat distinguishes their response from non-generational companies? Do others fail to rotate their exposed github secrets that have admin access?
- agos 18d agosee, non-generational companies often miss the chance to turn penetration testing into a marketing opportunity
- usewik 18d agoOh, the positive externalities of unsolicited penetration (testing).
- VoidWhisperer 18d agoI think that many other companies (especially larger ones, I suppose) don't respond as promptly to security issues.
- justinclift 18d ago> Our logs confirm You retain all logs back through to (at least) March 2023?
- ErroneousBosh 18d agoYou don't? For some stuff, I've got logs going back to 1993...
- bdcravens 18d agoMany companies only keep logs as long as they're legally required to. It can't be discoverable if it doesn't exist ...
- indymike 18d ago> It can't be discoverable if it doesn't exist ... This cuts both ways. I've seen plenty of litigation go south because one side had evidence and the other side had nothing because they deleted/shredded/lost the proof.
- ErroneousBosh 18d ago> It can't be discoverable if it doesn't exist ... That's great, and for some things the court can ask you "Well *why* haven't you got it?" and then you're fucked. Now you're explaining in front of a parliamentary committee why you destroyed what would turn out to be evidence.
- AdamJacobMuller 18d ago"Our standard process is to only retain logs when legally required to, either due to being notified about a litigation or through legally mandated periods" is a fully complete sentence. Unless you're required to retain logs for some reason like a litigation hold or legally or contractually mandated retention period and you violate those, while the adversarial party might be annoyed at you for not retaining logs there isn't much they can or will do beyond being annoyed. Of course if you destroy logs after being notified of litigation or inquiry, you're gonna have a bad day.
- ActionHank 18d ago“Vulnerability” isn’t really the right term for “we left something explicitly vulnerable and exposed to the internet”
- philipwhiuk 18d agoWas this part of a planned penetration test or did they just compromise your infrastructure first?
- bearsyankees 18d agoWe were (and still are) considering them as an inference provider and did a quick check first... but kudos to their team for the fast patch
- vatsachak 18d agoWe really are entering the AI economy. Now if only we knew if the stonks would go up or down (due to global turmoil) before I throw my savings at the SPY
- swyx 18d ago> Baseten handled this well. The timeline was: > July 13, 11:10 PM: I reported the live basetenbot token, the public Harbor project, and the repository permissions. > July 14, morning: Baseten made the Harbor project private. I flagged that the token itself still worked. > July 14, 4:34 PM: Anton from Baseten Security confirmed the issue as critical and said they had made the Harbor project private and rotated the token. He also asked us to securely delete the images we'd pulled. > July 14, 5:05 PM: We confirmed deletion and sent over two lower-severity findings from the same scan. > July 17: Baseten closed out the remaining findings. > September: We let Baseten know we planned to disclose the finding publicly and sent them a draft of this post. They also sent us some T-shirts and sweatshirts as a thank-you for finding this critical bug. well done all around. i think my only open question is what default security boundaries should all vibecoded internal agents follow as a learning we can take from this
- deleted 18d ago[deleted]
- mtlynch 18d agoGood in terms of prompt communication and fix. Absurdly bad in terms of reward. Earlier in the article, it mentions that Baseten is valued at $13B. They can't dig into their couch cushions to give a few thousand dollars to the researcher privately disclosing a bug that let an attacker escalate to admin in their GitHub org? This sends the message that honest researchers should not waste their time looking for vulnerabilities in Baseten, but it's a good target for criminals who want to monetize these vulnerabilities.
- deleted 18d ago[deleted]
- sheepscreek 18d agoYeah companies need to quickly understand that having good actors try and hack you is a good thing - those hacks get reported and another door gets sealed shut for bad actors. This is more true today than ever before as the bar for a successful attack has never been lower. We’ll see a resurgence of the script-kiddie, or shall I say, vibe-kiddie :-/
- ramon156 18d agoi quite liked using strix. last time i tried it, deepseek was a mess and bloated the context with nonsense. that was ~5 months ago, i wonder how it performs now
- deleted 18d ago[deleted]
- bearsyankees 18d agoWe've made a lot of awesome changes recently, would love any feedback on the latest version :)
- sandeepkd 18d agoThis sounds interesting and twisted in some sense 1. A start up is validating a service provider to ensure that they are secure enough so that they can trust them before signing up for their service 2. The service provider is already trusted by so many big name companies who handed over their data, the customers data to them Should it not be other way around? On a different note, the finding is not just one off absolute, rather its a symptom which points to certain experience and expertise level for security practices. To be fair its hard to blame the start up folks, they are running against time and cutting corners is somewhat critical for survival for their business
- bearsyankees 18d agoYeah... interesting paradigm
- aatd86 18d agoThat is great marketing for strix, pretty bad for baseten. I don't think someone can have a better story to advertise their own security product. Did not know about strix but I am going to look it up now. Might add it to my stack.
- Sytten 18d agoThey are a great team! You will defacto also use Caido if you add it to your stack
- bearsyankees 18d agoLet us know if you have any feedback!
- lukeify 18d agoIf I enter an address to "Start testing", I expect at least a preview of the report rather than being dumped on the signup page.
- bearsyankees 18d agoYeah understood — we need to make sure you own the domain first though
- jamesreadsnews 18d agoIs there a sample report somewhere to get a feel for the output?
- bearsyankees 18d agoyes! app.strix.ai/demo
- lukeify 18d agoMight be too nerdy for some, but a TXT DNS rule could work. That or a sample report. It looks like a cool product though, thanks for sharing.
- brewmarche 18d agoYeah I have seen this issue a few times. If you use Docker build arguments that way add `--provenance=false` to get rid of all that build metadata. Build secrets are still better since they allow you to scope the secret inside of the Dockerfile. Also, the metadata can be useful to inspect images.
- bearsyankees 18d agoYeah honestly I wasn't too familiar with this beforehand but now have a sense of the best practices going forward
- thrownaway22 18d agoBaseten carries the Soc 2 Type II and HIPAA Compliant logos on its front page. They also have logos for customers including: OpenEvidence (medical related, used by almost 2/3 of physicians in the US, claim HIPAA compliance), Harvey (legal related, claim "binding terms on data protection, data access, incident response SLAs, and other controls aligned with SOC2, ISO, GDPR and other standards.") From TFA: > That token had admin and push access to Baseten's main product repo, the GitOps repo that drives their clusters, and their Homebrew tap, plus read/write access to other private repositories including specific repos per customers. > The image build dated to March 2023, and the token still worked when we found it in July 2026. What are the legal implications here?
- conception 18d agoUnless github had regulated data, unlikely, the legal implications are few. Document the issue, remediate and no findings on the next audit. Done.
- hmokiguess 18d agoGiven the build is from 2023 one would expect that at least the token would have been rotated, and I suspect some of these compliance checks do require rotation of tokens/passwords. That said, the whole compliance industry is a joke.
- icedchai 18d agoBox checking is an important business!
- antonvs 18d agoI can’t help noticing that an LLM can check boxes.
- fragmede 18d agotokens yes, password rotation, no. In 2017: > NIST changed the guidance with SP 800-63B, published June 2017. It explicitly said: "Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically)." Instead, passwords should be changed when there is evidence they have been compromised, not every 30/60/90 days.
- athrowaway3z 18d agoA Markdown-as-a-Service where the interface is a Docker container. I get how these choices might be the local optimum for a desired UX, but damn is it depressing to extrapolate where software as a whole is going.
- kibac 18d agoI wonder what model was used for this. Also as far as I know Baseten does not have any abliterated models in their repertoire.
- guessmyname 18d agoEither Mythos 5.1 or GPT 5.6 Cyber (aka. GPT Daybreak Red)
- bearsyankees 18d agoNeither, actually :)
- kibac 18d agoChinese? I can not imagine how a western state of the art model would follow through with such a task and not require some major trickery.
- NyxWulf 18d agoThe writing sounds like Claude to me
- PaoloBarbolini 18d agoEven something like Qwen 3.8 27b could do this. Lookup certificate transparency and continue from there https://crt.sh/?Identity=baseten.co&exclude=expired&match=ILIKE https://crt.sh/?Identity=baseten.co&exclude=expired&match=IL...
- smallnix 18d agoThe bot snippets talk claudish. I'd say Opus 5. But they must be Cyber Verification Program approved by Anthropic I suppose for the LLM not to block them.
- bearsyankees 18d agoHate to burst your bubble but wasn't Claude......
- hmokiguess 18d agoAn easily preventable issue with proper engineering culture around defense in depth and principle of least privilege, awful look on Baseten here. Kudos for Strix to find it, and especially with how it chose to disclose and report it.
- catidegla 18d ago[flagged]
- wxw 18d ago> [pen-testing agent] came back with an active GitHub personal access token for basetenbot. That token had admin and push access to Baseten's main product repo, the GitOps repo that drives their clusters, and their Homebrew tap, plus read/write access to other private repositories including specific repos per customers. And the agent found the token in Docker build history after finding a Baseten image repository. I wonder how many of these kinds of agent-driven security exploits we're not hearing about these days (i.e. driven by bad actors), worrying.
- codemog 18d agoIs this legal? I know I can’t try and break into my neighbors house even if I have no intent of going inside and stealing once I break the lock.
- deleted 18d ago[deleted]
- kadoban 18d ago> I know I can’t try and break into my neighbors house even if I have no intent of going inside and stealing once I break the lock. They didn't break in. They found a key that their neighbor dropped and returned it. > Is this legal? Generally, yes (though ask a lawyer if you're going to do security work). Security researchers do occasionally get legal flak though, depending on which idiot they annoy by pointing out issues.
- otterley 18d agoIAAL (not legal advice, consult a lawyer in your jurisdiction). You really do not want to pen-test a target without their permission. If you're identified as a culprit, the Feds will shove the CFAA so far up your ass you'll need a proctologist.
- wpasc 18d agoas a lawyer, can you speculate as to why anthropic/openai aren't facing many or any consequences for their agents? I'm not asking in a "grab the pitchforks" way. more out of genuine curiosity as my uninformed recollection of the CFAA is as you describe it.
- otterley 18d agoThe 9th Circuit Court of appeals recently published this that is somewhat related (Amazon v. Perplexity): https://cases.justia.com/federal/appellate-courts/ca9/26-1444/26-1444-2026-08-04.pdf?ts=1785861090 https://cases.justia.com/federal/appellate-courts/ca9/26-144... Look at pages 10-17 to see how the law is evolving here.
- calvinmorrison 18d ago> We build Strix, an autonomous hacking again. > But... we're a security company. > So... we pointed Strix at *.baseten.co and let it run without credentials or source code. lawyers wet dream. and a perfect case. A security company who KNOWS the law unleashed an AI agent to violate the laws
- DaSHacka 18d agoA lawyers wet dream is when a security company.... Finds an issue, does not abuse it, and reports it to the affected party for it to be patched? I feel like people like you are more of a lawyers wet dream, in that they'll happily litigate a frivolous case for you while billing you hourly.
- usewik 18d agoFeelings don't really matter in the legal world. Statements and actions do.
- DaSHacka 18d agoAnd, most notably, intent. https://www.justice.gov/archives/opa/pr/department-justice-announces-new-policy-charging-cases-under-computer-fraud-and-abuse-act https://www.justice.gov/archives/opa/pr/department-justice-a...
- calvinmorrison 18d agoif i find someones key on the ground and take the key and walk into their house and poke around, and make sure to leave a letter, this is a good thing?
- iJohnDoe 18d agoThey did abuse if you read the article. They crossed a few lines.
- usewik 18d agoAgreed. Pretty sure you are supposed to ask for permission before pentesting someone. Hopefully they, being a security company, know that.
- hunterbrooks 18d agoJust signed up for strix, is it common for these type of products to want access to my Github repo's? Shouldn't the attack surface be outside them?
- bearsyankees 18d agoYou can also just do an external pentest -- Github is for continuous CI/CD coverage
- mschuster91 18d ago> So Strix enumerated hosts, looked through certificate logs, mapped the full surface. If there is anything that you should do while setting up infrastructure... it is getting rid of single-host SSL certificates. If you're on Amazon... just let it issue wildcard certificates and place an ALB in front of hosts that terminates the SSL connection. The very second a subdomain appears in any of the CT logs directly, you've lost, it will get hammered. And keep your public and private Git, Docker, npm and whatnot registries separate infrastructure, with the private stuff only reachable from within the corporate network, preferably just servers. Too many a company got hacked and lost significant data because of someone exploiting a GitLab RCE on an instance that hosted both private and intentionally-public repositories. (Yes, I have been there.) > It is their GitOps: the repository contains the desired state of the clusters, and it applies that state to the infrastructure. That's another thing I frankly do not get why people are still doing it. It's fine to have a Git pipeline do a lint, even a terraform plan using a read-only token (although that token needs access to the statefile aka s3 bucket... and there will be relevant secrets there). But, IMHO, a terraform apply should always, always be run on a machine of a sysadmin manually doing the apply. A human, you can hold accountable, and you can keep them at a good security posture with short-lived session tokens. But a Git pipeline where there is a cloud provider token with full admin permissions? That is one Gitlab RCE patch or Github issue away from being compromised. Besides, one repository holding all the IaC stuff? That just sounds like hour long `terraform refresh` sessions.
- lantry 18d ago> Besides, one repository holding all the IaC stuff? That just sounds like hour long `terraform refresh` sessions. If it is terraform, then typically it's split up into multiple "root modules" which get planned and applied separately, even though it's all in one repo.
- gz09 18d ago> A human, you can hold accountable, and you can keep them at a good security posture with short-lived session tokens. You can do this too (and better) with a repo: OIDC/Workload identity trust relationship between github and aws for short lived tokens + a github environment setup that requires manual approval. Bonus: It also gives you an audit trail with a github action log as opposed to a sysadmin running something on a laptop. The problem here was mostly that they (for some reason) happened to use (and leak) a PAT.
- stopthe 18d agoSo often recent breaches involve Github in one way or another. How is it still considered a sane choice to host anything proprietary there? If your business is built around open source, ok, put a mirror on Github. But CI/CD, gitops, FDEs' stuff have no place on a public cloud. C-level execs may not know bits from bytes, but by now they should've understood that this is akin to storing ammonium nitrate in the open air.
- afdbcreid 18d agoThis wasn't GitHub's fault in any way.
- fulafel 18d agoGithub personal access tokens are security footguns. This is an apparently old and forgotten image containing a token from 2023 and it the token gets you admin acces to their repos.
- stevage 18d agoSo many security breaches involve Linux in one way or another. Your argument doesn't really work.
- askl 18d agoSo many security breaches involve the internet in some way. Maybe we should just turn it off.
- stopthe 18d agoThat was not an attack on Github itself. GH made enormous impact in the open source movement and is still beneficial for every software engineer by providing a free and very useful service. Meanwhile, for a software business, Github is a wide and deep attack vector and nobody seems to be concerned about it. Of course the same can be said about any other public git hosting, especially if it combines CI/CD, artifact distribution, identity and trust management.
- throwitaway222 18d agoIt may make sense to change security practices so nothing has long term access. Everything should be rotated monthly, and maybe within a few years, hourly.
- Watchtrail 18d ago[flagged]
- throwitaway222 17d agoOne man operations that are in production - this is the perfect use case for that same person to run an agent to look for credentials that are not used, may have been exposed, etc... And obviously spend some cash on code review agents.
- grey-area 18d ago…by finding an admin token in logs.
- vikas123456789 18d agocannot delete the card from billing.
- eleumik 18d agoAlways check. I used a disk wiper , I checked the disk sectors, bang files under 1kb not really wiped...my "secrets" case..
- nrmitchi 18d agoWhether it's valid or not, there is something that rubs me the wrong way about a security tools company using a real customer/vendor as a marketing campaign. This "story" could have been told without naming, bluntly, their "victim". It would be different if it was some complex, multi-step exploit, but the tone is closer to "look how much Baseten fucked up!". Strix also crossed the line at this point: > Strix decided to pull an image and see what was inside. You're going past the white-hat point here when you start active exfiltrating data and looking at it. Once you start using credentials from the exfiltrated data and start listing and poking around internal systems, you are way past it. Listen, I get it, their product is "meant for" self-testing, so it assumes it's safe to go digging. After all, it's a self check. That is exactly why it's irresponsible, and borderline illegal, for them to point it at a third party. Even if they had "permission", I dobut that permission extended to "and also search and/or download our repos if you can". The overall tone is less than professional. Statements like (in bold) "This is an insane amount of access to leave in a publicly downloadable image." Everyone is aware of this, and it's phrase like it was a purposeful decision. Security tools from teams that actively shit on the people they're designed to "help" feels wrong. Edit: For clarity on my point about "pulling repos", this post includes descriptions of the purpose and functionality of multiple repos (which is past what a name gives them), and they explicitly state: "A listing of that private repo showed a top-level customers/ directory, with subdirectory after subdirectory named after Baseten customers". Strix explicitly took action that they knew they were not permitted, and extracted confidential customer information. Claiming "We didn't clone the customer repo" when you, instead, just listed the contents of the repo, is not a valid defense.
- FL410 18d agoAgreed. I suppose they'd have slightly less credibility by saying "we hacked <unnamed company>" but it strikes me as far classier than naming & shaming.
- ivraatiems 18d agoI'm not sure this is "naming and shaming" because I don't seen an intent to shame. They disclosed the vulnerability privately, waited months for patches, and were commended by the organization with the vulnerabilities. There's no shame here, this was a mistake, probably made by a human, and ultimately corrected. Nobody seems upset by the outcome!
- 0xbadcafebee 18d agoIt should be illegal to produce software products that people will depend on and are this blatantly insecure
- aantoon 18d ago[dead]
- ivraatiems 18d agoThis fits neatly into the category of "not something an unmotivated huamn would bother to look for, but absolutely something a human could find if they were interested." It increasingly feels like the power of these agents is less that they find things humans COULDN'T find, and more that they find many things much more quickly than humans would bother to do. I don't know if this is a great advert for Strix over other agents - what did their agent do that Claude or Codex couldn't? It didn't do anything that I couldn't do, if I wanted to.
- iJohnDoe 18d agoThe article reads like it was written by a child. I imagine their company is run the same way.
- SaucyWrong 18d agoAs a security software engineer I value and have a lot of experience with disclosures like this. At the last two B2Bs I worked at, I would also work personally with prospect security teams that wanted to run their red team at us (with approval and rules of engagement) This is a valuable disclosure but I wonder about two things: a) was the decision to run Strix against a prospective vendor domain negotiated in advance? b) if the answer to a) is “no” then it is apparent that while Strix want to ensure their customers only run it against domains they own (totally fair) they have a double standard for their own use. I don’t know, I’m accustomed to getting disclosures from any Jane or Joe via bug bounties etc., but it feels like a courtesy notice would be nice before a prospective customer lets their agentic hacker off the leash. EDIT: for typos.
- smurda 18d ago[dead]
- fhn 18d agoI'd call Docker out on this. Why the hell is it recording
- adithyassekhar 18d agoThis is impressive and something a human will never bother to find. But please stop saying Strix then did, Strix then went, Srix then this, then that. My monkey brain just can’t accept an LLM being referred to as if it is a living being with autonomy. It’s not. I’ll accept when we actually see AI models.
- marysol5 18d ago>something a human will never find Eh, yes it is. And something that humans find all the time. They even call out a non-AI tool that helped.
- adithyassekhar 18d agoCorrected
- wiredbox 18d agoThis did not need an AI agent at all...could have been discovered with deterministic pen-testing tools that have been around forever...
- rdwrrr 18d agoGot admin because someone left the front door key on the street. Thats some impressive hacking.
- ekorondy 18d ago[flagged]
- saiyamshah1496 17d ago[flagged]
- spncai 17d ago[flagged]
- beyondscaletech 17d ago[flagged]