4 ms·

To bring some data to a sour grapes fight: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-human

To bring some data to a sour grapes fight: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-humaneval For code generation, GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5. Given the recent release of MagicCoder / Instruct-OSS, that means a small prompt library + a small 7B model you can self-host beats the much fancier GPT4. Similar to when simple NNs destroyed a decade of Bayesian modeling theses & research programs, it's frustrating for folks going other paths. But it doesn't make the work 'wrong'.

harlanlewis 3y ago
> GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5 The link you shared doesn’t quite reflect this. Omitting other models… LATS (gpt-4): 94.4 Reflexion (gpt-4): 91.0 gpt-4: 86.6 … LATS (gpt-3.5): 83.8 … zero-shot (gpt-4): 67.0 zero-shot (gpt-3.5): 48.1 I’m not quite sure how to translate leaderboards like these into actual utility, but it certainly feels like “good enough” is only going to get more accessible and I agree with what I think is your broader point - more sophisticated techniques will make small, affordable, self-hostable models viable in their own right. I’m optimistic we’re on a path where further improvement isn’t totally dependent on just throwing money at more parameters.
- lmeyerov 3y ago
  Ah you're right, LATS GPT3.5 is 84 while standalone GPT4 is 87 Given standalone GPT3.5 is "just" 48.. it's less about beating and more about meeting RE:Good Enough & Feel... very much agreed. I find it very task dependent! For example, GPT4 is 'good enough' that developers are comfortable copy-pasting & trying, even vs stack overflow results. We haven't seen LATS+MagicCoder yet, but as MagicCoder 7b already meets+exceeds GPT3.5 for HumanEval, there's a plausible hope for agent-aided GPT4-grade tools being always-on for all coding tasks, and sooner vs later. We made that bet for Louie.AI's interactive analyst interface, and as each month passes, evidence mounts. We can go surprisingly far with GPT3.5 before wanting to switch to GPT4 for this kind of interaction scenario. Conversely... I've yet to see a true long-running autonomous coding autoGPT where the error rate doesn't kill it. We're experimenting with design partners on directions here -- think autonomous investigations etc -- but there's more on the advanced fringe and with special use cases, guard rails, etc. For most of our users and use cases... we're able to more reliably deliver -- today -- on the interactive scenarios with smaller snippets.
spaceywilly 3y ago
This right here. I feel like the focus on just throwing more GPU at the problem is a mistake many of these companies are making at the moment. The real breakthroughs will come when we figure out how to use the current models and compute power more efficiently. If it’s prompt engineering that leads to this breakthrough, so be it.

harlanlewis 3y ago

> GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5 The link you shared doesn’t quite reflect this. Omitting other models… LATS (gpt-4): 94.4 Reflexion (gpt-4): 91.0 gpt-4: 86.6 … LATS (gpt-3.5): 83.8 … zero-shot (gpt-4): 67.0 zero-shot (gpt-3.5): 48.1 I’m not quite sure how to translate leaderboards like these into actual utility, but it certainly feels like “good enough” is only going to get more accessible and I agree with what I think is your broader point - more sophisticated techniques will make small, affordable, self-hostable models viable in their own right. I’m optimistic we’re on a path where further improvement isn’t totally dependent on just throwing money at more parameters.

lmeyerov 3y ago
Ah you're right, LATS GPT3.5 is 84 while standalone GPT4 is 87 Given standalone GPT3.5 is "just" 48.. it's less about beating and more about meeting RE:Good Enough & Feel... very much agreed. I find it very task dependent! For example, GPT4 is 'good enough' that developers are comfortable copy-pasting & trying, even vs stack overflow results. We haven't seen LATS+MagicCoder yet, but as MagicCoder 7b already meets+exceeds GPT3.5 for HumanEval, there's a plausible hope for agent-aided GPT4-grade tools being always-on for all coding tasks, and sooner vs later. We made that bet for Louie.AI's interactive analyst interface, and as each month passes, evidence mounts. We can go surprisingly far with GPT3.5 before wanting to switch to GPT4 for this kind of interaction scenario. Conversely... I've yet to see a true long-running autonomous coding autoGPT where the error rate doesn't kill it. We're experimenting with design partners on directions here -- think autonomous investigations etc -- but there's more on the advanced fringe and with special use cases, guard rails, etc. For most of our users and use cases... we're able to more reliably deliver -- today -- on the interactive scenarios with smaller snippets.

spaceywilly 3y ago

This right here. I feel like the focus on just throwing more GPU at the problem is a mistake many of these companies are making at the moment. The real breakthroughs will come when we figure out how to use the current models and compute power more efficiently. If it’s prompt engineering that leads to this breakthrough, so be it.