4 ms·

This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattend

This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work.

HarHarVeryFunny 1mo ago
Sounds like OpenAI are in the token-maxxing camp, so who knows what individual employees are doing to work their way up the leaderboard? If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need? Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?
- bigcat12345678 1mo ago
  End of day, output and results are top target of measurements, token consumption is the obvious number that they would like to disclose for their own business benefits and a simple metrics that correlate with the output. Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit.
  - taurath 1mo ago
    Taking a profit means you have to show numbers and the sooner you show numbers the harder it is to take people’s money.
- auggierose 1mo ago
  That was Anthropic.
otherme123 1mo ago
That would be the mother of all circular accounting: the main clients of OpenAI are OpenAI employees.
continuitykit 1mo ago
[flagged]
carlgreene 1mo ago
I suspect the $8000/day figure is the equivalent in API costs. But I also suspect gross margin on their API rates are 80-90%
paxys 1mo ago
These researchers are paid millions of dollars for their work. I doubt trust is really an issue at that level.
- nozzlegear 1mo ago
  Imagine if one of the humans at OpenAI was misaligned! We should get the AI to research this possibility once they've been aligned.
- queuebert 1mo ago
  Yes, because no employee with million-dollar comp has ever been untrustworthy in the history of business.
andai 1mo ago
Can you elaborate on this? Especially the tooling. I tried something similar and I remember it was still pretty dodgy in February.
- jaggederest 1mo ago
  my stack in a sentence: refine the docs/prompts/skills often, that's your biggest job, use both frontier labs models reviewing each other, don't solve individual problems only the systemic ones (set standards strategically, don't define tactics) If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think. Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)
  - andai 1mo ago
    Thanks! > If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. This part jumped out at me. There's something to watch out for here. I recently had a funny experience. I delegated a major feature to an agent. It turned out that it had implemented it precisely backwards, in a way which was pointless and which made things worse. But it had written countless tests for the feature and all the tests were green. I realised in that moment that even formal verification would not have helped, because it would simply have written a mathematical proof of the correctness of the incorrect feature...
    - jaggederest 1mo ago
nojs 1mo ago
> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware) How are you running jobs unattended 24/7 without hitting your token limits?
- dataplumb3r 1mo ago
  My only experience in >24h agents is with economically sane models (one of GLM5.2, 5.3-flash for orchestration, DSV4-flash for implementation, and glm5.3|sol|kimi3 agents + subagents reviewing at the end) Over 24h my token spend is <30$. Excluding tokens for review it's <10$. With the absurdly gigantic subscription subsidies and a reasonable workflow I suspect one could run parallel agents. I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
  - nsndjcjjdjd 1mo ago
    This sounds like more work than just writing the code yourself. You'll say it isn't. I don't believe you.
    - dataplumb3r 1mo ago
      That is an incorrect presumption - I think it's plausible this is more work; it's certainly far more taxing. I'm at a point in my career where a small minority of my time is coding. The AIs can do in a day what would have taken me a week uninterrupted with acceptable (in some cases inferior prior to human feedback--but in some cases superior!) quality. As I do not have 10 let alone 40 hours per week to devote to coding I think it increases the amount of high quality work product I can create with a given time investment. As I review it I merge small independent units and decompose the work. All that is to say I don't really like it - but I suspect for most *well defined* coding tasks human produced code from highly experienced engineers will largely cease to exist in the next year -- getting cheap/relatively horrible models to produce good code is now straightforward. OTOH I never use AI for any human facing communication outside of making my writing shorter. IMO AI slop "documents" are almost certainly a drag on organizational productivity.

HarHarVeryFunny 1mo ago

Sounds like OpenAI are in the token-maxxing camp, so who knows what individual employees are doing to work their way up the leaderboard? If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need? Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?

bigcat12345678 1mo ago
End of day, output and results are top target of measurements, token consumption is the obvious number that they would like to disclose for their own business benefits and a simple metrics that correlate with the output. Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit.
- taurath 1mo ago
  Taking a profit means you have to show numbers and the sooner you show numbers the harder it is to take people’s money.
auggierose 1mo ago
That was Anthropic.

otherme123 1mo ago

That would be the mother of all circular accounting: the main clients of OpenAI are OpenAI employees.

continuitykit 1mo ago

[flagged]

carlgreene 1mo ago

I suspect the $8000/day figure is the equivalent in API costs. But I also suspect gross margin on their API rates are 80-90%

paxys 1mo ago

These researchers are paid millions of dollars for their work. I doubt trust is really an issue at that level.

nozzlegear 1mo ago
Imagine if one of the humans at OpenAI was misaligned! We should get the AI to research this possibility once they've been aligned.
queuebert 1mo ago
Yes, because no employee with million-dollar comp has ever been untrustworthy in the history of business.

andai 1mo ago

Can you elaborate on this? Especially the tooling. I tried something similar and I remember it was still pretty dodgy in February.

jaggederest 1mo ago
my stack in a sentence: refine the docs/prompts/skills often, that's your biggest job, use both frontier labs models reviewing each other, don't solve individual problems only the systemic ones (set standards strategically, don't define tactics) If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think. Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)
- andai 1mo ago
  Thanks! > If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. This part jumped out at me. There's something to watch out for here. I recently had a funny experience. I delegated a major feature to an agent. It turned out that it had implemented it precisely backwards, in a way which was pointless and which made things worse. But it had written countless tests for the feature and all the tests were green. I realised in that moment that even formal verification would not have helped, because it would simply have written a mathematical proof of the correctness of the incorrect feature...
  - jaggederest 1mo ago

nojs 1mo ago

> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware) How are you running jobs unattended 24/7 without hitting your token limits?

dataplumb3r 1mo ago
My only experience in >24h agents is with economically sane models (one of GLM5.2, 5.3-flash for orchestration, DSV4-flash for implementation, and glm5.3|sol|kimi3 agents + subagents reviewing at the end) Over 24h my token spend is <30$. Excluding tokens for review it's <10$. With the absurdly gigantic subscription subsidies and a reasonable workflow I suspect one could run parallel agents. I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
- nsndjcjjdjd 1mo ago
  This sounds like more work than just writing the code yourself. You'll say it isn't. I don't believe you.
  - dataplumb3r 1mo ago
    That is an incorrect presumption - I think it's plausible this is more work; it's certainly far more taxing. I'm at a point in my career where a small minority of my time is coding. The AIs can do in a day what would have taken me a week uninterrupted with acceptable (in some cases inferior prior to human feedback--but in some cases superior!) quality. As I do not have 10 let alone 40 hours per week to devote to coding I think it increases the amount of high quality work product I can create with a given time investment. As I review it I merge small independent units and decompose the work. All that is to say I don't really like it - but I suspect for most *well defined* coding tasks human produced code from highly experienced engineers will largely cease to exist in the next year -- getting cheap/relatively horrible models to produce good code is now straightforward. OTOH I never use AI for any human facing communication outside of making my writing shorter. IMO AI slop "documents" are almost certainly a drag on organizational productivity.