3 ms·
You don’t need the entire codebase in context in every moment to migrate it. Also AI being non deterministic does not prevent it from one-shotting perfect solu
by JV00 2mo ago
You don’t need the entire codebase in context in every moment to migrate it.
Also AI being non deterministic does not prevent it from one-shotting perfect solutions 100% of the time for simple enough problems.
And every model generation brings this bar higher. So that’s really not a fundamental problem.
And we can also implement llm inference deterministically if we want, it’s just that it’s not worth the loss in performance to do it.
- sandeepkd 2mo ago> And we can also implement llm inference deterministically if we want, it’s just that it’s not worth the loss in performance to do it. My understanding is thats not possible (different from being practical), wonder if you have any literature, research to back up that claim?
- JV00 2mo agoThis one from thinking machines: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in... Was discussed a lot also here on hn
- sandeepkd 2mo agoThanks for sharing, it was really good read. The authors in the article have done a good job for sure to validate their hypothesis and make it work for a particular subset of problem. 1. Local hardware, no networking or HTTP requests 2. No other parallel requests 3. 1000 runs bounded by 1000 tokens I have worked a little bit in academia and I am not a big fan of the way favorable samples for the hypothesis are kept and unfavorable ones are thrown away. I might be wrong here, however its highly unlikely that the team would have just worked with one query. Chances are that a lot of different prompts with varying number of runs would have been tried to see what works and supports the hypothesis.