3 ms·
"I hope we can put to rest the argument that LLMs are only marginally useful in coding" I more often heard the argument, they are not useful for them. I agree.
by lukan 2y ago
"I hope we can put to rest the argument that LLMs are only marginally useful in coding"
I more often heard the argument, they are not useful for them. I agree.
If a LLM would be trained on my codebase and the exact libaries and APIs I use - I would use them daily I guess. But currently they still make too many misstake and mess up different APIs for example, so not useful to me, except for small experiments.
But if I could train deepseek on my codebase for a reasonable amount(and they seemed to have improved on the training?), running it locally on my workstation: then I am likely in as well.
- rane 2y agoThe idea is that you give the libraries and APIs as context with your prompt.
- lukan 2y agoThank you. But that doesn't work for me. If you mean just the name of the version in the prompt? No way. If you mean all the libary and my code in the contextwindow? Way too small.
- rane 2y agoNot _all_ the code. Just the relevant parts.
- kragen 2y agoMaybe the LLM could issue queries to fetch parts of your codebase as it needs to look at them, using something like GDB or cscope.
- smallerfish 2y agoPlay around with projects in Claude for an hour. You'll see.
- simonw 2y agoGive it examples of the library being used in the way you need. Here's an example transcript where I did that: https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d36eb https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d3...
- lukan 2y agoThank you, I experimented in that direction as well. But for my actual codebase, that is sadly not 100% clear code, it would require lots and lots of work, to give examples so it has enough of the right context, to work good enough. While working I am jumping a lot between context and files. Where a LLM hopefully one day will be helpful, will be refactoring it all. But currently I would need to spend more time setting up context, than solving it myself. With limited scope, like in your example - I do use LLMs regulary.
- r00fus 2y agoThere's a fairly low ceiling for max context tokens no matter the size of the model. Your hobby/small codebase may work, but for large codebases, you will need to do RAG and currently it's not perfect at absorbing the codebase and being able to answer questions on it.
- AlwaysRock 2y agoWe are getting closer and closer to that. For a while llm assistants were not all that useful on larger projects because they had limited context. That context has increased a lot over the last 6 months. Some tools will even analysis your entire codebase and use that in responses. It is frustrating that any smaller tool or api seem to stump llms currently but it seems like context is the main thing that is missing and that is increasing more and more.
- lukan 2y agoI have not kept up, can you recommend something?
- simonw 2y agoMy review of 2024 is a good place to catch up on what's changed in the past 12 months: https://simonwillison.net/2024/Dec/31/llms-in-2024/ https://simonwillison.net/2024/Dec/31/llms-in-2024/
- kragen 2y agoThat post is the best summary I've seen of what happened in LLMs last year, but what's crazy is that it feels like you wrote it so long ago, and it's only been four weeks! So much has changed since then!
- sureglymop 2y agoI am working on something even deeper. I have been working on a platform for personal data collection. Basically a server and an agent on your devices that records keystrokes, websites visited, active windows etc. The idea is that I gather this data now and it may become useful in the future. Imagine getting a "helper AI" that still keeps your essence, opinions and behavior. That's what I'm hoping for with this.
- lukan 2y agoI am not sure if this was sarcasm, but I believe big data was already yesterday?
- sureglymop 2y agoNot sarcasm. This is more a reaction to big data. Here's an analogy: Imagine cloud providers like iCloud, Google Drive, OneDrive etc. As a reaction to those, Owncloud and Nextcloud emerged for personal (well, also business) use. My idea with this is inspired by that. It's just for personal use and to address my own needs.
- svilen_dobrev 2y agoeh, a hint. i was digging around some thing in these veins long time ago - more like collecting one's notions, not exact low-level actions - but apart of it being impossible back then, i dropped it for this simple reason: if you build such thing, it will know about you much more than you know. And that, in somebody else's hands.. identity theft would seem like walk in the park.
- sureglymop 2y agoFor sure, thank you for that hint. One of the most important things to consider is that something like this can't be misused on someone else, e.g. as a surveillance tool. I should have clarified, I'm only building this for myself and my own use, there are no plans to take it further than that. Basically, I am trying to learn while building something that satisfies my own needs.