11 ms·

This is my personal experience, despite everyone swearing that it's a game changer. I've tried a fair few times now because people swear everything is revolutio

by jchw 2y ago

This is my personal experience, despite everyone swearing that it's a game changer. I've tried a fair few times now because people swear everything is revolutionary but I find them almost as annoying as helpful, and as many others have noticed you really have to be careful before accepting the code it outputs as correct, the subtly incorrect bits are extremely insidious. An example: in one case I was trying an AI tool to write some reasonably tricky logic for validation and at some point it got something very close to right but flipped part of a conditional. It took me probably 30 minutes to notice even though it should have been pretty obvious. The best I can say is that the implementation in Jetbrains IntelliJ IDEA is pretty good. It's basically only useful for some repetitive Java boilerplate, but actually that's perfect, it's mindless enough yet easy to validate. It makes me dislike programming in Java a little bit less.

nicholasjarnold 2y ago
While I agree with your sentiment regarding implementing complex bespoke logic, I think there are likely measurable gains when an experienced engineer is operating outside of their current comfort zone. Allow me an anecdote - I was recently doing some HTML/CSS to update the public website for a client. This is not where I typically operate, though I obviously have some exposure over the years. Using Supermaven I was able to very quickly perform website surgery that would have ultimately taken me much more time reading Tailwind documentation and loading my RAM with details that would be mostly superfluous to 99% of my professional workload. It's in contexts like this where I think these tools shine. 'Game changer' is clickbait phrasing for these tools in their current incarnations, but they can certainly save time while producing acceptable results when used appropriately.
neocron 2y ago
I somewhat agree, but I don't use tools like copilot integrated into my IDE at all (tried it in beta) and I just hate that the suggestions it made were even worse than what the IDE suggested before. It was kinda what I wanted but like you said with small errors that were hard to findy because mostly it was working. But what I reall like is using stuff like edge copilot and other chat-like ai tools to just ask questions and get some, mosty really well, summarized answers. Usually about tech I never used before, haven't used in ages or need to be migrated to much newee versions with breaking changes. It saves me a lot of reading the docs, fiddling with it, reading the docs again, reading quite a few posts about it e.g. on stack overflow and than finally getting to work on the actual project. I just ask my questions, try the suggested approach, fact check the docs and get to work
com2kid 2y ago
Cursor.ai is worlds better than copilot. Open up a new code base, ask it to analyze the code and summarize the different main parts of it. I was making a json schema for a typescript type and as soon as I typed the name of the json schmea, cursor offered to auto complete the schema. "I think there is a concurrency bug in this file, can you find it for me?" that alone saved me potentially hours of debugging. When working with APIs I don't often touch, such as Canvas, AI saves me hours to days of effort.
sanitycheck 2y ago
I've done the same, and my experiences also match those in the article. Using a LLM as an autocomplete (typescript, kotlin) I found it'd be right about 10% of the time, very obviously wrong ~30% of the time, but mostly (~60%) *plausible*. Plausible is the worst, because if I don't spot the error immediately it'll come back to bite me in the coming hours/weeks and it takes me much longer to fully comprehend existing code than it does to write new code myself. Asking questions fares better, but I never have easy questions and for difficult ones it'll usually be wrong. Like, if I'm asking an LLM a question I'm usually at a point where I've read the docs and there's something I don't understand - so it's not that surprising that the LLM won't understand it either. Occasionally pointing out the flaw will get a better answer, but more often it just leads to a circle of flawed answers. I feel like people who say this is some sort of revolution are perhaps people who weren't very good at finding information themselves. The only places where this stuff seems to be an unqualified success so far is for jobs where the quality of the output is not very important.

by jchw 2y ago

nicholasjarnold 2y ago

While I agree with your sentiment regarding implementing complex bespoke logic, I think there are likely measurable gains when an experienced engineer is operating outside of their current comfort zone. Allow me an anecdote - I was recently doing some HTML/CSS to update the public website for a client. This is not where I typically operate, though I obviously have some exposure over the years. Using Supermaven I was able to very quickly perform website surgery that would have ultimately taken me much more time reading Tailwind documentation and loading my RAM with details that would be mostly superfluous to 99% of my professional workload. It's in contexts like this where I think these tools shine. 'Game changer' is clickbait phrasing for these tools in their current incarnations, but they can certainly save time while producing acceptable results when used appropriately.

neocron 2y ago

I somewhat agree, but I don't use tools like copilot integrated into my IDE at all (tried it in beta) and I just hate that the suggestions it made were even worse than what the IDE suggested before. It was kinda what I wanted but like you said with small errors that were hard to findy because mostly it was working. But what I reall like is using stuff like edge copilot and other chat-like ai tools to just ask questions and get some, mosty really well, summarized answers. Usually about tech I never used before, haven't used in ages or need to be migrated to much newee versions with breaking changes. It saves me a lot of reading the docs, fiddling with it, reading the docs again, reading quite a few posts about it e.g. on stack overflow and than finally getting to work on the actual project. I just ask my questions, try the suggested approach, fact check the docs and get to work

com2kid 2y ago

Cursor.ai is worlds better than copilot. Open up a new code base, ask it to analyze the code and summarize the different main parts of it. I was making a json schema for a typescript type and as soon as I typed the name of the json schmea, cursor offered to auto complete the schema. "I think there is a concurrency bug in this file, can you find it for me?" that alone saved me potentially hours of debugging. When working with APIs I don't often touch, such as Canvas, AI saves me hours to days of effort.

sanitycheck 2y ago

I've done the same, and my experiences also match those in the article. Using a LLM as an autocomplete (typescript, kotlin) I found it'd be right about 10% of the time, very obviously wrong ~30% of the time, but mostly (~60%) *plausible*. Plausible is the worst, because if I don't spot the error immediately it'll come back to bite me in the coming hours/weeks and it takes me much longer to fully comprehend existing code than it does to write new code myself. Asking questions fares better, but I never have easy questions and for difficult ones it'll usually be wrong. Like, if I'm asking an LLM a question I'm usually at a point where I've read the docs and there's something I don't understand - so it's not that surprising that the LLM won't understand it either. Occasionally pointing out the flaw will get a better answer, but more often it just leads to a circle of flawed answers. I feel like people who say this is some sort of revolution are perhaps people who weren't very good at finding information themselves. The only places where this stuff seems to be an unqualified success so far is for jobs where the quality of the output is not very important.