4 ms·

Coding LLMs really shine for greenfield projects in popular programming languages. They don’t do half as well in large codebases that use ad-hoc frameworks. Fo

by muglug 3y ago

Coding LLMs really shine for greenfield projects in popular programming languages. They don’t do half as well in large codebases that use ad-hoc frameworks. For example, it has no idea how to retrieve the currently-logged-in user object for a new endpoint you want it to build. The solution there is to fine-tune it on your codebase, but that’s likely a few years away for the average LLM user.

troupo 3y ago
Greenfield projects in less popular programming languages and frameworks is a miss, too. Currently working on an Elixir project with Ash Framework, and I wouldn't trust a single Chat GPT output for either of them.
drsim 3y ago
Google demonstrated codebase aware LLMs at Next last week. I believe it was with their Codey model. It knew the data model and methods that existed in code, and used these.
- muglug 3y ago
  Yeah, it looked like a demo of something they're still working on. In this video they say "Custom-tuned models on code" are coming soon, but aren't there yet: https://youtu.be/qbxj_JDQ4Lk?si=qSTzMBpmqqJD5o9u&t=1632 https://youtu.be/qbxj_JDQ4Lk?si=qSTzMBpmqqJD5o9u&t=1632
- ilaksh 3y ago
  Are you sure you're not thinking of Cody the tool? https://github.com/sourcegraph/cody https://github.com/sourcegraph/cody The names are very similar.
tomr75 3y ago
Check out cursor.so Just builds a local vectordb with your code base
runeks 3y ago
> Coding LLMs really shine for greenfield projects in popular programming languages. Greenfield? Unless we disagree on the definition of greenfield (no prior work), then a plane spotter app is anything but greenfield. GitHub returns 11 repos for JavaScript alone.
milofeynman 3y ago
Yet. The wider context window and better tooling are all being worked on.
swalsh 3y ago
Using fine-tuning to solve this problem would be like using a 500lb bomb to solve your mouse problem.
- Breza 3y ago
  You could say the same thing about using a cell phone and a massively complex backend infrastructure to call a cab instead of your hand. I think the next generation of devs will come up relying on these tools, just like how I'd have a hard time going back to writing code through telnet like I did back in the day.

by muglug 3y ago

troupo 3y ago

Greenfield projects in less popular programming languages and frameworks is a miss, too. Currently working on an Elixir project with Ash Framework, and I wouldn't trust a single Chat GPT output for either of them.

drsim 3y ago

Google demonstrated codebase aware LLMs at Next last week. I believe it was with their Codey model. It knew the data model and methods that existed in code, and used these.

muglug 3y ago
Yeah, it looked like a demo of something they're still working on. In this video they say "Custom-tuned models on code" are coming soon, but aren't there yet: https://youtu.be/qbxj_JDQ4Lk?si=qSTzMBpmqqJD5o9u&t=1632 https://youtu.be/qbxj_JDQ4Lk?si=qSTzMBpmqqJD5o9u&t=1632
ilaksh 3y ago
Are you sure you're not thinking of Cody the tool? https://github.com/sourcegraph/cody https://github.com/sourcegraph/cody The names are very similar.

tomr75 3y ago

Check out cursor.so Just builds a local vectordb with your code base

runeks 3y ago

> Coding LLMs really shine for greenfield projects in popular programming languages. Greenfield? Unless we disagree on the definition of greenfield (no prior work), then a plane spotter app is anything but greenfield. GitHub returns 11 repos for JavaScript alone.

milofeynman 3y ago

Yet. The wider context window and better tooling are all being worked on.

swalsh 3y ago

Using fine-tuning to solve this problem would be like using a 500lb bomb to solve your mouse problem.

Breza 3y ago
You could say the same thing about using a cell phone and a massively complex backend infrastructure to call a cab instead of your hand. I think the next generation of devs will come up relying on these tools, just like how I'd have a hard time going back to writing code through telnet like I did back in the day.