4 ms·

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. Thanks for checking this out and for flagging

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly. If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here. CLI docs: https://learn.chatgpt.com/docs/security/cli https://learn.chatgpt.com/docs/security/cli EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybersecurity-products-san-francisco/ https://openai.com/careers/full-stack-software-engineer-cybe...

robotswantdata 2mo ago
Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!
- dangelosaurus 2mo ago
  Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us. Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!
gizmodo59 2mo ago
When would I use this over the plugin in codex? Which I think can be invoked from cli as well
- dangelosaurus 2mo ago
  The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration. We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build. Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
strictnein 2mo ago
Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?
- dangelosaurus 2mo ago
  We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
  - ignoramous 2mo ago
    Exciting! Is there any open GitHub issue we can track?
    - dangelosaurus 2mo ago
      Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support. The issue tracker is here: https://github.com/openai/codex-security/issues https://github.com/openai/codex-security/issues If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.
      - ignoramous 2mo ago
        Thanks! https://github.com/openai/codex-security/issues/52 https://github.com/openai/codex-security/issues/52
troupo 2mo ago
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. > Thanks for checking this out and for flagging the auth issues. Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed". You just have to look at what these labs routinely produce, and their own products. Edit to respond to @simonw whose comment I saw before he retracted it ;) This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
- deleted 2mo ago
  [deleted]
- imrozim 2mo ago
  [dead]
vladoh 2mo ago
This looks great, thanks for open-sourcing it! How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
- dangelosaurus 2mo ago
  Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that. For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass. If you're an open-source maintainer, you can apply for conditional Codex Security access here: https://openai.com/form/codex-for-oss/ https://openai.com/form/codex-for-oss/ For enterprise teams, the public Daybreak onboarding guide is here: https://help.openai.com/en/articles/20001261-enterprise-daybreak-onboarding https://help.openai.com/en/articles/20001261-enterprise-dayb... If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).
  - theplumber 2mo ago
    >> For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals Or perhaps a better option is to use something like Kimi K3 and cancel the GPT subscription altogether.
    - Culonavirus 2mo ago
      Or try Grok, 4.5 seems pretty capable, should be close to K3 in many coding tasks. I use it for code review of what other "stronger" models shit out (like Sol) and it constantly finds even pretty big bugs or just not robust enough solutions (Sol tends to overengineer, yes, but I'm not so sure it overengineers the right parts, so far my experience woth it has been mid. Except it understanding my drawings and collages and it being capable of far better frontend/design dev than 5.4 or even 5.5 was).
deleted 2mo ago
[deleted]
6thbit 2mo ago
How does it fare against its own codebase?
Quai 2mo ago
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P). It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
- dangelosaurus 2mo ago
  Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.
orangelimesoda 2mo ago
> Have experience shipping production full-stack products across modern web frontends and backend services. I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these. Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions? All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen? I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.) I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist). Anyway end rant. Cool job, hope you fill it.
- jameshart 2mo ago
  This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.
  - solid_fuel 2mo ago
    Professional app development requires an understanding of security.
    - corndoge 2mo ago
      I think the evidence contradicts you at this point
      - orangelimesoda 2mo ago
        So you don't need to know anything about credit cards for someone to enter it into an app? I don't get the take (is it bad sarcasm?)
        devmor 2mo ago
        To be fair, for the vast majority of cases, no you don't. It is extremely rare for companies to roll their own payment processing anymore, or even handle PCI scope at all.
waterTanuki 2mo ago
Why does this need an entirely separate repo instead of being a feature in the existing Codex project?
noname120 2mo ago
Hi! Any chance you may have tangential positions opening in Zürich?
sudo_cowsay 2mo ago
Hi! Any remote internship for a high schooler? lol
_the_inflator 2mo ago
Off topic: Just some thx and kudos to you guys. I used Promptpoo at the beginning of the year - it was exactly, what I needed, very much still a niche thing hardly anyone was using. I totally missed the acquisition - but well deserved. I am currently re-evaluating PF again for my upcoming project, and happy to see that it is more than simply thriving.
- MattRix 2mo ago
  unfortunate typo lol
jonas_kgomo 2mo ago
Can we build a different app experience on this open-source tool? Does the license permit

robotswantdata 2mo ago

Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!

dangelosaurus 2mo ago
Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us. Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!

gizmodo59 2mo ago

When would I use this over the plugin in codex? Which I think can be invoked from cli as well

dangelosaurus 2mo ago
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration. We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build. Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.

strictnein 2mo ago

Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?

dangelosaurus 2mo ago
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
- ignoramous 2mo ago
  Exciting! Is there any open GitHub issue we can track?
  - dangelosaurus 2mo ago
    Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support. The issue tracker is here: https://github.com/openai/codex-security/issues https://github.com/openai/codex-security/issues If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.
    - ignoramous 2mo ago
      Thanks! https://github.com/openai/codex-security/issues/52 https://github.com/openai/codex-security/issues/52

troupo 2mo ago

> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. > Thanks for checking this out and for flagging the auth issues. Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed". You just have to look at what these labs routinely produce, and their own products. Edit to respond to @simonw whose comment I saw before he retracted it ;) This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)

deleted 2mo ago
[deleted]
imrozim 2mo ago
[dead]

vladoh 2mo ago

This looks great, thanks for open-sourcing it! How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.

dangelosaurus 2mo ago
Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that. For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass. If you're an open-source maintainer, you can apply for conditional Codex Security access here: https://openai.com/form/codex-for-oss/ https://openai.com/form/codex-for-oss/ For enterprise teams, the public Daybreak onboarding guide is here: https://help.openai.com/en/articles/20001261-enterprise-daybreak-onboarding https://help.openai.com/en/articles/20001261-enterprise-dayb... If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).
- theplumber 2mo ago
  >> For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals Or perhaps a better option is to use something like Kimi K3 and cancel the GPT subscription altogether.
  - Culonavirus 2mo ago
    Or try Grok, 4.5 seems pretty capable, should be close to K3 in many coding tasks. I use it for code review of what other "stronger" models shit out (like Sol) and it constantly finds even pretty big bugs or just not robust enough solutions (Sol tends to overengineer, yes, but I'm not so sure it overengineers the right parts, so far my experience woth it has been mid. Except it understanding my drawings and collages and it being capable of far better frontend/design dev than 5.4 or even 5.5 was).

deleted 2mo ago

[deleted]

6thbit 2mo ago

How does it fare against its own codebase?

Quai 2mo ago

I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P). It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)

dangelosaurus 2mo ago
Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.

orangelimesoda 2mo ago

> Have experience shipping production full-stack products across modern web frontends and backend services. I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these. Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions? All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen? I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.) I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist). Anyway end rant. Cool job, hope you fill it.

jameshart 2mo ago
This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.
- solid_fuel 2mo ago
  Professional app development requires an understanding of security.
  - corndoge 2mo ago
    I think the evidence contradicts you at this point
    - orangelimesoda 2mo ago
      So you don't need to know anything about credit cards for someone to enter it into an app? I don't get the take (is it bad sarcasm?)
      - devmor 2mo ago
        To be fair, for the vast majority of cases, no you don't. It is extremely rare for companies to roll their own payment processing anymore, or even handle PCI scope at all.

waterTanuki 2mo ago

Why does this need an entirely separate repo instead of being a feature in the existing Codex project?

noname120 2mo ago

Hi! Any chance you may have tangential positions opening in Zürich?

sudo_cowsay 2mo ago

Hi! Any remote internship for a high schooler? lol

_the_inflator 2mo ago

Off topic: Just some thx and kudos to you guys. I used Promptpoo at the beginning of the year - it was exactly, what I needed, very much still a niche thing hardly anyone was using. I totally missed the acquisition - but well deserved. I am currently re-evaluating PF again for my upcoming project, and happy to see that it is more than simply thriving.

MattRix 2mo ago
unfortunate typo lol

jonas_kgomo 2mo ago

Can we build a different app experience on this open-source tool? Does the license permit