5 ms·

> the commits produced Maybe this is the problem ? I quite like using LLMs for coding, but I don't think we are in a position where a LLM is able to create a r

by nzach 2y ago

> the commits produced Maybe this is the problem ? I quite like using LLMs for coding, but I don't think we are in a position where a LLM is able to create a reasonable commit. For me using LLMs for coding is like a pair programming session where YOU are the co-pilot. The AI will happily fill you screen with a lot of text, but you have the responsibility to steer the session. Recently I've been using Supermaven in my editor. I like to think of it as 'LSP on steroids', it's not that smart but is pretty fast and for me this is important. Another way I use LLMs to help me is by asking open-ended questions to a more capable but slower LLM. Something like "What happens when I read a message from a deleted offset in a Kafka topic?" to o1. Most of the time it doesn't give great answers, but it generally gives good keywords to start a more focused Google search.

fovc 2y ago
The “pair programming” approach with good models is just slow enough that I lose focus on each step. The faster models I’ve tried are not good enough except for straightforward things where it’s faster to just use emacs/LSP refactoring and editing tools. Maybe supermaven manages to beat the “good enough, fast enough” bar; I’ll have to try it!
- nzach 2y ago
  One thing I've realized after using a really fast model is that the time it takes the model to generate a suggestion is proportional to the size of suggestion I'm willing to accept. And in my experience the quality of suggestions decreases when the suggestion size increases. If the model takes a couple seconds to generate a suggestion I get inclined to accept several lines of code. But if the suggestion takes just 300ms to generate I don't feel the "need" to accept the suggested code. I'm not really sure why that happens, maybe that's just the sunk cost fallacy happening right in my editor? If I wait 5 seconds for a suggestion and don't use the suggestion did I effectively just wasted 5 seconds of my life for no good reason?
- jitl 2y ago
  I think having supermaven or cursor-style IDE integration is really key to making LSP worth it, otherwise friction around the workflow overwhelms the gains for many tasks. Like, I would never use AI to generate code or tests if it involved copy-pasting code in and out of webpage text boxes. But with IDE integration, I often write a function signature and doc comment with no plans to ask AI for anything, but the AI happens to offer a correct tab completion of the whole function. That's great, much less typing for me.
godelski 2y ago
Do you think this is actually faster than reading the docs? Every attempt I've had with LLM pair programming (I do it at least once a month trying to measure performance) ends up taking more time than if I turned to Google out the docs, even with how bad Google has gotten. Though it _feels_ faster because it _feels_ like you're making continual progress where reading docs doesn't have the same feeling. I suspect this is a confounder but I'm open to just being bad at AI programming (though isn't it meant to be universal? I mean I'm a ML researcher and that's what the papers promise). I'm also curious if you think it helps you improve. Docs tend to give extra information that turns out to be useful now and many times later. I still like and use LLMs a lot though. I find them useful in a similar way to your last paragraph. My favorite usage is to ask it domain topics where I'm not a domain expert. I never trust the response (it's commonly at best oversimplified/misleading but often wrong), but since it will use similar language to those in the field I can pick out keywords to improve a google search, especially when caught in term collision hell (i.e. Google overfitting and ignoring words/quotes/etc). I do also find it helpful in validating what I think some code does. But same as above, low trust and use as a launching off point for deeper understanding. Basically, I'm using LLMs as a fuzzy database optimized towards the data median with a human language interface. That is, after all, what they are.
- nzach 2y ago
  > Do you think this is actually faster than reading the docs? It depends on your goals, I guess. Do I really need to read the whole D3.js docs just to transform a csv into a pretty map? I'm not arguing against the docs, I genuinely think that by reading the D3.js docs I would become a better professional. But what is the ROI for this effort ? Nowadays learning about a topic is a choice we can make, we can create things that solve real problems without the need to fully understand the solution. This wasn't feasible a couple years back. And choosing 'not to learn' too many times is a great recipe for disaster, so I understand why a lot of people are worried about giving this option to people. Besides that the "always read the docs" theory makes an assumption that isn't always true. This assumes you know what are you looking for and where you can find it. When I was younger I was assigned to a task that required me to put a new feature in a jenga tower of custom bash scripts and I've found a bug that completely stumped me, it took me an entire week to figure out I was missing some quotes around $@ when passing arguments from one script to the next. I've spent several hours trowing random combinations of keywords to try finding something relevant to my problem. Now I know this is a bash-related problem, but at the time this wasn't clear. It might have been something in my logic, or something to do with not being an TTY, or something to do with the version of the tools I was using.... Having a LLM would have saved a week of frustration because I could just vaguely describe my problem and ask questions about my issue to point me to right direction faster. > I never trust the response (it's commonly at best oversimplified/misleading but often wrong) This reminds me of my middle school. When I was there technology and especially Wikipedia was just starting to get popular and almost every week some teacher would lecture us about the dangers of these new technologies and how you should never trust anything on the internet. As time passed the quality of Wikipedia content increased, but this idea of never blindly trusting something you found on the internet really stuck with me. And now LLMs are just another thing on the internet you should never blindly trust for me. Maybe that is part of the reason why I don't get too angry when the LLM tell me something that is wrong. > I'm using LLMs as a fuzzy database That is a really good way of putting it. And I agree this is a great use-case for LMMs. But I think we still have a lot to learn about how to effectively use LLMs. I just hope people don't get too caught up in all the marketing surrounding this AI hype cycle we are living right now.

by nzach 2y ago

fovc 2y ago

The “pair programming” approach with good models is just slow enough that I lose focus on each step. The faster models I’ve tried are not good enough except for straightforward things where it’s faster to just use emacs/LSP refactoring and editing tools. Maybe supermaven manages to beat the “good enough, fast enough” bar; I’ll have to try it!

nzach 2y ago
One thing I've realized after using a really fast model is that the time it takes the model to generate a suggestion is proportional to the size of suggestion I'm willing to accept. And in my experience the quality of suggestions decreases when the suggestion size increases. If the model takes a couple seconds to generate a suggestion I get inclined to accept several lines of code. But if the suggestion takes just 300ms to generate I don't feel the "need" to accept the suggested code. I'm not really sure why that happens, maybe that's just the sunk cost fallacy happening right in my editor? If I wait 5 seconds for a suggestion and don't use the suggestion did I effectively just wasted 5 seconds of my life for no good reason?
jitl 2y ago
I think having supermaven or cursor-style IDE integration is really key to making LSP worth it, otherwise friction around the workflow overwhelms the gains for many tasks. Like, I would never use AI to generate code or tests if it involved copy-pasting code in and out of webpage text boxes. But with IDE integration, I often write a function signature and doc comment with no plans to ask AI for anything, but the AI happens to offer a correct tab completion of the whole function. That's great, much less typing for me.

godelski 2y ago

Do you think this is actually faster than reading the docs? Every attempt I've had with LLM pair programming (I do it at least once a month trying to measure performance) ends up taking more time than if I turned to Google out the docs, even with how bad Google has gotten. Though it _feels_ faster because it _feels_ like you're making continual progress where reading docs doesn't have the same feeling. I suspect this is a confounder but I'm open to just being bad at AI programming (though isn't it meant to be universal? I mean I'm a ML researcher and that's what the papers promise). I'm also curious if you think it helps you improve. Docs tend to give extra information that turns out to be useful now and many times later. I still like and use LLMs a lot though. I find them useful in a similar way to your last paragraph. My favorite usage is to ask it domain topics where I'm not a domain expert. I never trust the response (it's commonly at best oversimplified/misleading but often wrong), but since it will use similar language to those in the field I can pick out keywords to improve a google search, especially when caught in term collision hell (i.e. Google overfitting and ignoring words/quotes/etc). I do also find it helpful in validating what I think some code does. But same as above, low trust and use as a launching off point for deeper understanding. Basically, I'm using LLMs as a fuzzy database optimized towards the data median with a human language interface. That is, after all, what they are.

nzach 2y ago
> Do you think this is actually faster than reading the docs? It depends on your goals, I guess. Do I really need to read the whole D3.js docs just to transform a csv into a pretty map? I'm not arguing against the docs, I genuinely think that by reading the D3.js docs I would become a better professional. But what is the ROI for this effort ? Nowadays learning about a topic is a choice we can make, we can create things that solve real problems without the need to fully understand the solution. This wasn't feasible a couple years back. And choosing 'not to learn' too many times is a great recipe for disaster, so I understand why a lot of people are worried about giving this option to people. Besides that the "always read the docs" theory makes an assumption that isn't always true. This assumes you know what are you looking for and where you can find it. When I was younger I was assigned to a task that required me to put a new feature in a jenga tower of custom bash scripts and I've found a bug that completely stumped me, it took me an entire week to figure out I was missing some quotes around $@ when passing arguments from one script to the next. I've spent several hours trowing random combinations of keywords to try finding something relevant to my problem. Now I know this is a bash-related problem, but at the time this wasn't clear. It might have been something in my logic, or something to do with not being an TTY, or something to do with the version of the tools I was using.... Having a LLM would have saved a week of frustration because I could just vaguely describe my problem and ask questions about my issue to point me to right direction faster. > I never trust the response (it's commonly at best oversimplified/misleading but often wrong) This reminds me of my middle school. When I was there technology and especially Wikipedia was just starting to get popular and almost every week some teacher would lecture us about the dangers of these new technologies and how you should never trust anything on the internet. As time passed the quality of Wikipedia content increased, but this idea of never blindly trusting something you found on the internet really stuck with me. And now LLMs are just another thing on the internet you should never blindly trust for me. Maybe that is part of the reason why I don't get too angry when the LLM tell me something that is wrong. > I'm using LLMs as a fuzzy database That is a really good way of putting it. And I agree this is a great use-case for LMMs. But I think we still have a lot to learn about how to effectively use LLMs. I just hope people don't get too caught up in all the marketing surrounding this AI hype cycle we are living right now.