6 ms·

I wonder how much this thing costs to run. https://github.com/anthropics/defending-code-reference-harness/blob/main/docs/troubleshooting.md#rate-limits https:/

by simonw 5mo ago

I wonder how much this thing costs to run. https://github.com/anthropics/defending-code-reference-harness/blob/main/docs/troubleshooting.md#rate-limits https://github.com/anthropics/defending-code-reference-harne... says: > As a rough guideline, expect ~10K uncached input tokens/min and ~2K output tokens/min per agent. You can scale parallelism up to your account's ITPM limit (roughly 10 agents per 100K ITPM). My guess would be hundreds of dollars with Opus and thousands of dollars with Mythos.

Analemma_ 5mo ago
I mean, you don't need to run it all the time, right? You do it once over your entire existing codebase to start and then once over the diff in your CI/CD pipeline when you make a new change. I'm sure it's not literally that simple but I doubt these need to churn 24/7/365 either.
- xerxes249 5mo ago
  In the Mythos blogpost they revealed to run the model like a 1000 times on the same code-base maybe with slightly different prompt or temperature. That suggests it will just be pay to win. If the 'attacker' spends more money/tokens than the 'defender' you will eventually be outclassed.
  - sofixa 5mo ago
    It's even worse, it's loot box style. Not pay to win, but pay to have the chance to win. The result will always be non-deterministic, so for some cases it can give you what you're looking for from the first time, or it can take 1000 tries.
    - beering 5mo ago
      It’s never not been “loot box style”. None of your past hired security audits were guaranteed to catch all issues?
- jazz9k 5mo ago
  Companies don't make production pushes yearly. For many, it's two week sprints..and that's one project. This doesn't make any sense cost-wise. It would be cheaper to just hire a security engineer.
  - vessenes 5mo ago
    I agree the cost curve has shifted. But if we take the Mozilla team's Mythos report as a broad baseline, you need to hire something like 10 security engineers to equal the Mythos productivity. Put another way, everyone's under hiring security by a LOT right now, we just have been lucky enough to see similar under hiring on hackers.
- vb-8448 5mo ago
  You are supposed to run it on full codebase before any single PR gets merge.
nikcub 5mo ago
It's becoming apparent that it requires more tokens to secure code than it does to write it May even be an order of magnitude more
- Mtinie 5mo ago
  In all seriousness, wasn’t that always the case? Writing bad code is relatively cheap. Ensuring code isn’t bad is the expensive part.
  - chrisweekly 5mo ago
    Sort of? The definition of "bad" from a security PoV is rapidly expanding, in light of relatively new capabilities and increasingly cheap access to exploitable vulnerabilities.
    - fny 5mo ago
      I don't think the definition of "bad" is expanding. Rather the ability to detect and exploit "bad" is.
      - chrisweekly 5mo ago
        fair point. another way of putting it might be to say that, for all extant software, much more of it is "bad" than we realized even a month or two ago -- and the cost to create and maintain "good" software is increasing (even as the naive / surface-level / apparent cost is plummeting)
        kenjackson 5mo ago
        Same thing happened with the growth of the internet. There was a time when there was basically no consideration of buffer overflow.
- bflesch 5mo ago
  It's weird because why can't they train the AI to simply output secure code? The basic security flaws with regards to input validation and overflows should never ever be output by an AI. For "security flaws due to bad design" I'll cut them slack until AGI is achieved.
binyu 5mo ago
Claude workflows in ultra code mode works in a very similar fashion and it consumes a moderate amount of the session usage limit, depending on the complexity of the task. With the API it would probably get expensive quickly though
Terretta 5mo ago
If you compare to their managed service, that estimate is likely 1/10th expectation, depending on codebase. But even this larger number, in turn, can be about 1/10th the cost of a formal engagement to discover the type of findings it seems to be going for: things that do not show up from PR reviews or even /security-review without the pre-work steps in the open-source framework guided by an expert. That's not counting the time and delay to figure out how to do that engagement. Bluntly: if it matters, while this is a month's vibing budget for a single scan, it is also "pennies on the dollar" dirt cheap. At the same time, its findings still need an expert. Its suggestions may be helpful, they may be actively harmful, depends on the prework quality. Recommendation to IT department heads: spend a couple grand on this, use the scare page to rustle up the budget to build a relationship with a red team that can find, triage, help remediate if needed, and train your in-house team to be "security minded".
eranation 5mo ago
We actually created a calculator to estimate scanning costs (including whether you do this continuously or not) https://ai-cost-calculator.arnica.io https://ai-cost-calculator.arnica.io It's an estimate, so it might be wrong, but it gives the ballpark based on our experience. Happy to hear everyone's feedback.
niros_valtos 5mo ago
I think that the cost of Opus is already prohibitively expensive, so not sure how that would compare to Mythos. Check this calculator- it shows that a company with 100 devs can hit ~2.5M cost on tokens annually, which is wild! https://ai-cost-calculator.arnica.io https://ai-cost-calculator.arnica.io
- pixl97 5mo ago
  It's wild, but how many FLOPs in computation is occurring in those 2.5M in tokens doing? Might not sound quite as wild using that metric.
- Quinner 5mo ago
  A 100 dev team is going to cost on the order of $25m a year (keep in mind cost is not just their salary, but also the HR/Management org to support a team of that size, benefits, office space, hardware/software). So if you think you get a 10% boost in productivity out of Opus, its not prohibitive at all.
mmaney13 5mo ago
Just another example of an overextension of technology in a scenario where applying a proper harness would suffice. Reminiscent of the early days of tax automation where importing a W2 cost hundreds of dollars until people realized typing in 6 boxes worth of data was easy and paying the automation fee ate up their entire tax return.
kolesnikov-arch 5mo ago
[flagged]

by simonw 5mo ago

Analemma_ 5mo ago

I mean, you don't need to run it all the time, right? You do it once over your entire existing codebase to start and then once over the diff in your CI/CD pipeline when you make a new change. I'm sure it's not literally that simple but I doubt these need to churn 24/7/365 either.

xerxes249 5mo ago
In the Mythos blogpost they revealed to run the model like a 1000 times on the same code-base maybe with slightly different prompt or temperature. That suggests it will just be pay to win. If the 'attacker' spends more money/tokens than the 'defender' you will eventually be outclassed.
- sofixa 5mo ago
  It's even worse, it's loot box style. Not pay to win, but pay to have the chance to win. The result will always be non-deterministic, so for some cases it can give you what you're looking for from the first time, or it can take 1000 tries.
  - beering 5mo ago
    It’s never not been “loot box style”. None of your past hired security audits were guaranteed to catch all issues?
jazz9k 5mo ago
Companies don't make production pushes yearly. For many, it's two week sprints..and that's one project. This doesn't make any sense cost-wise. It would be cheaper to just hire a security engineer.
- vessenes 5mo ago
  I agree the cost curve has shifted. But if we take the Mozilla team's Mythos report as a broad baseline, you need to hire something like 10 security engineers to equal the Mythos productivity. Put another way, everyone's under hiring security by a LOT right now, we just have been lucky enough to see similar under hiring on hackers.
vb-8448 5mo ago
You are supposed to run it on full codebase before any single PR gets merge.

nikcub 5mo ago

It's becoming apparent that it requires more tokens to secure code than it does to write it May even be an order of magnitude more

Mtinie 5mo ago
In all seriousness, wasn’t that always the case? Writing bad code is relatively cheap. Ensuring code isn’t bad is the expensive part.
- chrisweekly 5mo ago
  Sort of? The definition of "bad" from a security PoV is rapidly expanding, in light of relatively new capabilities and increasingly cheap access to exploitable vulnerabilities.
  - fny 5mo ago
    I don't think the definition of "bad" is expanding. Rather the ability to detect and exploit "bad" is.
    - chrisweekly 5mo ago
      fair point. another way of putting it might be to say that, for all extant software, much more of it is "bad" than we realized even a month or two ago -- and the cost to create and maintain "good" software is increasing (even as the naive / surface-level / apparent cost is plummeting)
      - kenjackson 5mo ago
        Same thing happened with the growth of the internet. There was a time when there was basically no consideration of buffer overflow.
bflesch 5mo ago
It's weird because why can't they train the AI to simply output secure code? The basic security flaws with regards to input validation and overflows should never ever be output by an AI. For "security flaws due to bad design" I'll cut them slack until AGI is achieved.

binyu 5mo ago

Claude workflows in ultra code mode works in a very similar fashion and it consumes a moderate amount of the session usage limit, depending on the complexity of the task. With the API it would probably get expensive quickly though

Terretta 5mo ago

If you compare to their managed service, that estimate is likely 1/10th expectation, depending on codebase. But even this larger number, in turn, can be about 1/10th the cost of a formal engagement to discover the type of findings it seems to be going for: things that do not show up from PR reviews or even /security-review without the pre-work steps in the open-source framework guided by an expert. That's not counting the time and delay to figure out how to do that engagement. Bluntly: if it matters, while this is a month's vibing budget for a single scan, it is also "pennies on the dollar" dirt cheap. At the same time, its findings still need an expert. Its suggestions may be helpful, they may be actively harmful, depends on the prework quality. Recommendation to IT department heads: spend a couple grand on this, use the scare page to rustle up the budget to build a relationship with a red team that can find, triage, help remediate if needed, and train your in-house team to be "security minded".

eranation 5mo ago

We actually created a calculator to estimate scanning costs (including whether you do this continuously or not) https://ai-cost-calculator.arnica.io https://ai-cost-calculator.arnica.io It's an estimate, so it might be wrong, but it gives the ballpark based on our experience. Happy to hear everyone's feedback.

niros_valtos 5mo ago

I think that the cost of Opus is already prohibitively expensive, so not sure how that would compare to Mythos. Check this calculator- it shows that a company with 100 devs can hit ~2.5M cost on tokens annually, which is wild! https://ai-cost-calculator.arnica.io https://ai-cost-calculator.arnica.io

pixl97 5mo ago
It's wild, but how many FLOPs in computation is occurring in those 2.5M in tokens doing? Might not sound quite as wild using that metric.
Quinner 5mo ago
A 100 dev team is going to cost on the order of $25m a year (keep in mind cost is not just their salary, but also the HR/Management org to support a team of that size, benefits, office space, hardware/software). So if you think you get a 10% boost in productivity out of Opus, its not prohibitive at all.

mmaney13 5mo ago

Just another example of an overextension of technology in a scenario where applying a proper harness would suffice. Reminiscent of the early days of tax automation where importing a W2 cost hundreds of dollars until people realized typing in 6 boxes worth of data was easy and paying the automation fee ate up their entire tax return.

kolesnikov-arch 5mo ago

[flagged]