4 ms·
I swear some days ago someone here claimed Openai succeeded cutting down their compute cost by half with a breakthrough optimization. So this is it?
by siva7 3mo ago
I swear some days ago someone here claimed Openai succeeded cutting down their compute cost by half with a breakthrough optimization. So this is it?
- simonw 3mo agoThat was an article in The Information but it didn't read very well to me, I didn't get the impression the author was enough of a technical expert on how LLMs work to credibly evaluate the claim, which came from an insider rumor: https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half?rc=cpi0u7 https://www.theinformation.com/newsletters/ai-agenda/openai-... > OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference, or running existing models, thanks to some newly-discovered optimizations, according to a person with knowledge of those discussions.
- jiggawatts 3mo agoI bet that since this bug has made headlines, there are some panicked engineers at OpenAI desperately trying to figure out how to fix it without undoing their “magic optimisation”.
- brookst 3mo agoIs there any indication that optimization shipped? Depending on whether it’s R&D or pragmatic, I’d expect it to take months at least.
- SyneRyder 3mo agoMy understanding of the rumor is that it wasn't OpenAI itself, but one of the post-blip OpenAI breakaway groups (rumoured to be Thinking Machines) who have made a breakthrough and seem to be shopping it to OpenAI. I don't think it has actually been implemented by OpenAI yet.