3 ms·
Is Devin a new LLM? Perhaps equiped with code and deploy plug-ins? The comparisons against other LLMs would suggest so. The real world eval benchmark puts Clau
by jasfi 3y ago
Is Devin a new LLM? Perhaps equiped with code and deploy plug-ins? The comparisons against other LLMs would suggest so.
The real world eval benchmark puts Claude 2 way ahead of GPT-4, which doesn't sound right.
- nrub 3y agoI've seen a few suspect benchmarks for recent announcements of LLM releases. I'm sure they made an attempt at an honest benchmark, but until there's an independent assessment and benchmark (preferably multiple) you have to assume that there's bias in anyone's self published benchmarks like this. I'm guessing it's a fine-tuning of some existing LLM model or API, but this largely seems to be an "agent" and UI that includes some SWE like workflow coding to allow more complex requests to be asked than just an LLM could provide.