11 ms·

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significa

by thih9 2y ago

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significantly with doing plumbing, but it has no problems with connecting the pipes wrong. I need to double and triple check the updated code - or fix the resulting errors when I don’t do that. So: boilerplate and outer app layers, yes; architecture and core libraries, no. Curious, is that a property of all ai assisted tools for now? Or would copilot, perhaps with its new models, offer a different experience?

MuffinFlavored 2y ago
> in practice I’m not noticing a productivity boost. How can this be possible if you literally admit its tab completion is mindblowing? Isn't really good tab completion good enough for at least a 5% producitvity boost? 10%? 20%? Select line of code, prompt it to refactor, verify they are good, accept the changes
- m3kw9 2y ago
  In experience, I always have to spend time to check and most times it doesn’t do what I need unless it’s very simple asks.
- thih9 2y ago
  > How can this be possible if you literally admit its tab completion is mindblowing? What about it makes it impossible? I’m impressed by what AI assistants can do - and in practice it doesn’t help me personally. > Select line of code, prompt it to refactor, verify they are good, accept the changes. It’s the “verify” part that I find tricky. Do it too fast and you spend more time debugging than you originally gained. Do it too slow and you don’t gain much time. There is a whole category of bugs that I’m unlikely to write myself but I’m likely to overlook when reading code. Mixing up variable types, mixing up variables with similar names, misusing functions I’m unfamiliar with and more.
  - beepbooptheory 2y ago
    I think the essential point around impressive vs helpful sums up so much of the discourse around this stuff. Its all just where you fall on the line between "impressive is necessarily good" and "no it isn't".
  - throttlebody 2y ago
    How does AI learn from it's mistakes ? Genuine question as i have only briefly used ChatGpt and found it interesting but not usefull.
- bigstrat2003 2y ago
  > How can this be possible if you literally admit its tab completion is mindblowing? If I had a knife of perfect sharpness which never dulled, that would be mind-blowing. It also would very likely not make me a better cook.
  - goosejuice 2y ago
    Having a sharp knife will make you far more efficient if you know how to use it in the first place and sharpening knives takes time away from doing the cooking. This is kind of a wonky analogy though.
- kortilla 2y ago
  If someone can eat 20 golf balls that’s impressive but it doesn’t improve my golf game
  - hackable_sand 2y ago
    That is actually very impressive.
- bradford 2y ago
  > How can this be possible if you literally admit its tab completion is mindblowing? I might suggest that coding doesn't take as much of our time as we might think it does. Hypothetically: Suppose coding takes 20% of your total clock time. If you improve your coding efficiency by 10%, you've only improved your total job efficiency by 2%. This is great, but probably not the mind-blowing gain that's hyped by the AI boom. (I used 20% as a sample here, but it's not far away from my anecdotal experience, where so much of my time is spent in spec gathering, communication, meeting security/compliance standards, etc).
  - MuffinFlavored 2y ago
    > that's hyped by the AI boom I always personally thought the AI boom was more about 'what can be done automation/improvement wise for non-tech industries'
MangoCoffee 2y ago
Time will tell. As a GitHub Copilot user, I still review the code. SpaceX's advancements are impressive, from rocket blow up to successfully catching the Starship booster. Who knows what AI will be capable of in 5-10 years? Perhaps it will revolutionize code assistance or even replace developers
- outworlder 2y ago
  > SpaceX's advancements are impressive, from rocket blow up to successfully catching the Starship booster. That felt like it was LLM generated since that doesn't have anything to do with the subject being discussed. Not only it's on a different industry but it's a completely different set of problems. We know what's involved in catching a rocket. It's a massive engineering challenge yes, but we all know it can be done(whether or not it makes sense or is economically viable are different issues). Even going to the Moon – which was a massive project and took massive focus from an entire country to do – was a matter of developing the equipment, procedures, calculations (and yes, some software). We knew back then it could be done, and roughly how. Artificial intelligence? We don't know enough about "intelligence". There isn't even a target to reach right now. If we said "resources aren't a problem, let's build AI", there isn't a single person on this planet that can tell you how to build such an AI or even which technologies need to be developed. More to the point, current LLMs are able to probabilistically generate data based on prompts. That's pretty much it. They don't "know" anything about what they are generating, they can't reason about it. In order for "AI" to replace developers entirely, we need other big advancements in the field, which may or may not come.
  - dspillett 2y ago
    > Artificial intelligence? We don't know enough about "intelligence". The problem I have with this objection is that it, like many discussions, conflates LLMs (glorified predictive text) and other technologies currently being referred to as AI, with AGI. Most of these technologies should still be called machine learning as they aren't really doing anything intelligent in the sense of general intelligence. As you say yourself: they don't know anything. And by inference, they aren't reasoning about anything. Boilerplate code for common problems, and some not so common ones, which is what LLMs are getting pretty OK at and might in the coming years be very good at, is a definable problem that we understand quite well. And much as we like to think of ourselves as "computer scientists", the vast majority of what we do boils down to boilerplate code using common primitives, that are remarkably similar across many problem domains that might on first look appear to be quite different, because many of the same primitives and compound structures are used. The bits that require actual intelligence are often quite small (this is how I survive as a dev!), or are away from the development coalface (for instance: discovering and defining the problems before we can solve them, or describing the problem & solution such that someone or an "AI" can do the legwork). > we need other big advancements in the field, which may or may not come. I'm waiting for an LLM being guided to create a better LLM, and eventually down that chain a real AGI popping into existence, much like the infinite improbability drive being created by clever use of a late version finite improbability generator. This is (hopefully) many years (in fact I'm hoping for at least a couple of decades so I can be safely retired or nearly there!) from happening, but it feels like such things are just over the next deep valley of disillusionment.
imafish 2y ago
I rarely use the tab completion. Instead I use the chat and manually select files I know should be in context. I am barely writing any code myself anymore. Just sanity checking that the output and “piping” is correct. My productivity (in frontend work at least) is significantly higher than before.
- big_jimmer 2y ago
  Out of curiosity, how long have you been working as a developer? Just that, in my experience, this is mostly true for juniors and mids (depending on the company, language, product etc. etc.). For example, I often find that copilot will hallucinate tailwind classes that don't exist in our design system library, or make simple logical errors when building charts (sometimes incorrect ranges, rarely hallucinated fields) and as soon as I start bringing in 3rd party services or poorly named legacy APIs all hope is lost and I'm better off going it alone with an LSP and a prayer.
  - imafish 2y ago
    Around 10 years. I also find that claude hallucinates once in a while, but I usually catch it. My main job becomes requesting and reviewing code instead of writing it. But I don't think it is fair to compare copilot and cursor. I have not been able to gain any significant productivity boost from using copilot (which I have in my Visual Studio proper).
SparkyMcUnicorn 2y ago
I haven't used Cursor, but I use Aider with Sonnet 3.5 and also use Copilot for "autocomplete". I'd highly recommend reading through Aider's docs[0], because I think it's relevant for any AI tool you use. A lot of people harp on prompting, and while a good prompt is important I often see developers making other mistakes like not providing context that's good, correct, or even too much[1]. When I find models are going on the wrong path with something, or "connecting the pipes wrong", I often add code comments that provide additional clarity. Not only does this help future me/devs, but the more I steer AI towards correct results, the fewer problems models seem to have going forward. Everybody seems to be having wildly different experiences using AI for coding assistance, but I've personally found it to be a big productivity boost. [0] https://aider.chat/docs/usage/tips.html https://aider.chat/docs/usage/tips.html [1] https://aider.chat/docs/troubleshooting/edit-errors.html#reduce-distractions https://aider.chat/docs/troubleshooting/edit-errors.html#red...
- realce 2y ago
  Totally agree that heavy commenting is the best convention for helping the assistant help you best. I try to comment in a way that makes a file or function into a "story" or kind of a single narrative.
  - jascha_eng 2y ago
    That's super interesting, I've been removing a lot of the redundant comments from the AI results. But adding new more explanatory ones that make it easier for both AI and humans to understand the code base makes a lot of sense in my head. I was big on writing code to be easy to read for humans, but it being easy to read for AI hasn't been a large concern of mine.
bob1029 2y ago
It's the subtle errors that are really difficult to navigate. I got burned for about 40 hours on a conditional being backward in the middle of an otherwise flawless method. The apparent speed up is mostly a deception. It definitely helps with rough outlines and approaches. But, the faster you go, the less you will notice the fine details, and the more assumptions you will accumulate before realizing the fundamental error. I'd rather find out I was wrong within the same day. I'd probably have written some unit tests and played around with that function a lot more if I had handcrafted it.
- enneff 2y ago
  That’s the thing, isn’t it? The craft of programming in the small is one of being intimate with the details, thinking things through conscientiously. LLMs don’t do that.
  - Nevermark 2y ago
    Perhaps it should be prompted to then? Ask it to review its own code for any problems? Also identify typical and corner cases and generate tests? Question marks here because I have not used the tool. The size & depth of each accepted code step is still up to the developer slash prompter
    - nrclark 2y ago
      I use Chatgpt for coding / API questions pretty frequently. It's bad at writing code with any kind of non-trivial design complexity. There have been a bunch of times where I've asked it to write me a snippet of code, and it cheerfully gave me back something that doesn't work for one reason or another. Hallucinated methods are common. Then I ask it to check its code, and it'll find the error and give me back code with a different error. I'll repeat the process a few times before it eventually gets back to code that resembles its first attempt. Then I'll give up and write it myself. As an example of a task that it failed to do: I asked it to write me an example Python function that runs a subprocess, prints its stdout transparently (so that I can use it for running interactive applications), but also records the process's stdout so that I can use it later. I wanted something that used non-blocking I/O methods, so that I didn't have to explicitly poll every N milliseconds or something.
knallfrosch 2y ago
I use it for an unfamiliar programming language and it's very nice. You can also ask it to explain badly documented code.
weitendorf 2y ago
I'm building a tool in this space and believe it's actually multiple separate problems. From most to least solvable: 1. AI coding tools benefit a lot from explicit instructions/specifications and context for how their output will be used. This is actually a very similar problem to when eg someone asks a programmer "build me a website to do X" and then being unhappy with the result because they actually wanted to do "something like X", and a payments portal, and yellow buttons, and to host it on their existing website. So models need to be given those particular instructions somehow (there are many ways to do it, I think my approach is one of the best so far) and context (eg RAG via find-references, other files in your codebase, etc) 2. AI makes coding errors, bad assumptions, and mistakes just like humans. It's rather difficult to implement auto-correction in a good way, and goes beyond mere code-writing into "agentic" territory. This is also what I'm working on. 3. AI tools don't have architecture/software/system design knowledge appropriate represented in their training data and all the other techniques used to refine the model before releasing it. More accurately, they might have knowledge in the form of eg all the blog posts and docs out there about it, but not skill. Actually, there is some improvement here, because I think o1 and 3.5 sonnet are doing some kind of reinforcement-learning/self-training to get better at this. But it's not easily addressable on your end. 4. There is ultimately a ton of context cached in your brain that you cannot realistically share with the AI model, either because it's not written anywhere or there is just too much of it. For example, you may want to structure your code in a certain way because your next feature will extend it or use it. Or your product is hosted on serving platform Y which has an implementation detail where it tries automatically setting Content-Type response headers by appending them to existing headers, so manually setting Content-Type in the response causes bugs on certain clients. You can't magically stuff all of this into the model context. My product tries to address all of these to varying extents. The largest gains in coding come from making it easier to specify requirements and self-correct, but architecture/design are much harder and not something we're working on much. You or anybody else can feel free to email me if you're interested in meeting for a product demo/feedback session - so far people really like our approach to setting output specs.
fullstackwife 2y ago
One of the reasons for that may be the price: large code changes with multi turn conversation can eat up a lot of tokens, while those tools charge you a flat price per month. Probably many hacks are done under the hood to keep *their* costs low, and the user experiences this as lower quality responses. Still the "architecture and core libraries" is rather corner case, something at the bottom of their current sales funnel. also: do you really want to get equivalent of 1 FTE work for 20 USD per month?:)
tomrod 2y ago
I'd love an autoselected LLM that is fine-tuned to the syntax I'm actively using -- Cursor has a bit of a head start, but where Github and others can take it could be mindblowing (Cursor's moat is a decent VS Code extension -- I'm not sure it's a deep moat though).
pnathan 2y ago
In general I do not find AI a net positive. Other tools seem to do at least as well in general. it can be used if you want the reliability of a random forum poster. which... sure. knock yourself out. sometimes there's gems in that dirt. I'm getting _very_ bearish on using LLMs for things that aren't pattern recognition.
bmitc 2y ago
That's my exact experience with GitHub Copilot. Even boilerplate stuff it sucks at as well. I have no idea why its autocomplete is so bad when it has access to my code, the function signatures, types, etc. It gets stuff wrong all the time. For example, it will just flat out suggest functions that don't exist, neither in the Python core libraries or in my own modules. It doesn't make sense. I have all but given up on using Copilot for code development. I still do use it for autocomplete and boilerplate stuff, but I still have to review that. So there's still quite a bit of overhead, as it introduces subtle errors, especially in languages like Python. Beyond that, it's failure rate at producing running, correct code is basically 100%.
miki123211 2y ago
For now, I mostly use AI as a "faster typist". If it wants to complete what I wanted to type anyway, or something extremely similar, I just press tab, otherwise I type my own code. I'd say about 70% of individual lines are obvious enough if you have the surrounding context that this works pretty well in practice. This number is somewhat lower in normal code and higher in unit tests. Another use case is writing one-off scripts that aren't connected to any codebase in particular. If you're doing a lot of work with data, this comes in very handy. Something like "here's the header of a CSV file", pass each row through model x, only pass these three fields, the model will give you annotations, put these back in the csv and save, show progress, save every n rows in case of crashes, when the output file exists, skip already processed rows." I'm not (yet) convinced by AI writing entire features, I tried that a few times and it was very inconsistent with the surrounding codebase. Managing which parts of the codebase to put in its context is definitely an art though. It's worth keeping in mind that this is the worst AI we'll ever have, so this will probably get better soon.
- zeroonetwothree 2y ago
  One off scripts do work very well.
- blitzar 2y ago
  Reminds me of how I use the satnav when driving. I don't close my eyes and do whatever it tells me to do. If I think I know better I don't "turn right at the next set of lights" I just drive on as I would have before GPS and eventually realise that I went the wrong way or the satnav realises there was a perfectly valid 2nd/3rd/4th path to get to where I wanted to go.
- DeathArrow 2y ago
  Yes, for simple boring tasks like converting a JSON to C# classes, it does wonders. For everything clever it kind of needs baby sitting which doesn't make it faster than writing the code myself. Even for the glorified use case of writing unit tests it sucks unless the code is very simple with few dependencies.
ianbutler 2y ago
I'm actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. At the end of the day it can belt out lines of code faster than I, or any human, can and I can review code very quickly so the overall productivity boost has been great. It does fundamentally alter my workflow. I'm very hands off keyboard when I'm working with AI in a way that is much more like working with someone or coaching someone to make something instead of doing the making myself. Which I'm fine with but recognize many developers aren't. I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching.
- __float 2y ago
  I'm not sure how many people are like me, but my attempts to use Copilot have largely been the context of writing code as usual, occasionally getting end-of-line or handful-of-lines completions from it. I suspect there's probably a bigger shift needed, but I haven't seen anyone (besides AI "influencers" I don't trust..?) showing what their day-to-day workflows look like. Is there a Vimcasts equivalent for learning the AI editor tips and tricks?
  - sbarre 2y ago
    Have you tried the chat mode? The autocomplete is somewhere between annoying and underwhelming for me, but the chat is super useful. Being able to just describe what you're thinking or what you're trying to do and having a bespoke code sample just show up (based on the code in your editor) that you can then either copy/paste in, cherry-pick from or just get inspired by, has been a great productivity booster.. Treat it like a pair programmer or a rubber duck and you might have a better experience. I did!
  - whimsicalism 2y ago
    Yeah using a chat interface
Aeolun 2y ago
> in practice I’m not noticing a productivity boost I am. Can suddenly do in a weekend what would have taken a week.
Yodel0914 2y ago
I find chatgpt incredibly useful for writing scripts against well-known APIs, or for a "better stackoverflow". Things like "how do I use a cursor in sql" or "in a devops yaml pipeline, I want to trigger another pipeline. How do I do that?". But working on our actual codebase with copilot in the IDE (Rider, in my case) is a net negative. It usually does OK when it's suggesting the completion of a single line, but when it decides to generate a whole block it invariably misunderstands the point of the code. I could imagine that getting better if I wrote more descriptive method names or comments, but the killer for me is that it just makes up methods and method signatures, even for objects that are part of publicly documented frameworks/APIs.
- jmpeax 2y ago
  Same here. If you need to lookup how to do something in an api I find it much faster to use chatgpt than to try to search through the janky official docs or in some github examples folder. Chatgpt is basically documentation search 2.0.
- deepfriedbits 2y ago
  I love your framing of it as a "better stackoverflow." That's so true. However, I feel like some of our complaints about accuracy and hidden bugs are temporary pain (12-36) months before the tools truly become mind-blowing productivity multipliers.
- DeathArrow 2y ago
  >"better stackoverflow" I had asked hard questions on SO that only a human with experience can answer. And I found answers on SO that only a human with experience can answer.
- vultour 2y ago
  The problem with this is that you lose the entire context around the answer you're given. There might be notes about limitations or edge cases in the documentation or on StackOverflow. I already see this with my coworkers who use LLMs, they often have no idea how or why things work and then they're absolutely perplexed when something goes wrong.
breckenedge 2y ago
Tab complete is just one of their proprietary models. I find chat-mode more helpful for refactoring and multi-file updates, even more when I specify the exact files to include.
- DeathArrow 2y ago
  I think SuperMaven has tab complete and let's you chose the model.
grensley 2y ago
I think there's a number of factors that make it work as well as it does for me: - Mostly writing React - Not using any obscure or new libraries - Naming things well - Keeping logic simple - Leaving a comment at the point where I'm about to make a shift from what the common logic would be - Getting a feel for when it's going to be able to correctly guess or not (and not even reading it if I think it's going to be wrong) - Trusting short blocks more than long ones
quotemstr 2y ago
AI will have the effect of shifting development effort from authorship to verification. As you note, we've come a long way towards making the writing of the code practically free, but we're going to need to beef up our tools for understanding code already written. I think we've only scratched the surface of AI-assisted program analysis.
dham 2y ago
With React, the guesses it makes around what props / types I want where, especially moving from file to file, is worth the price of admission. Everything else it does it icing on the cake. The new import suggestion is much quicker than the Typescript compiler lmao. And it's always the right one, instead of suggesting ones that aren't relevant. Composer can be hit or miss, but I've found it really good at game programming.
stuckkeys 2y ago
honestly. I hate it. I find myself banging my head on the table because of how much endless cycles it goes through and not give me the solution haha. I probably started projects over multiple times because the code it generated was so bad lol.
willtemperley 2y ago
I've found explaining boilerplate code to Claude or ChatGPT has been very worthwhile, the boring code gets written and the prompt is the documentation. OTOH tab completion in Intellij and Xcode has been useful occasionally, usually distracting and sometimes annoying. A way to fast toggle this would be good, but when I know what I want to code, good old code completion works nicely thanks.
faangguyindia 2y ago
Zed AI is much better than cursor I've found.
- goosejuice 2y ago
  In what way exactly? I wish Zed was better here but it's at about the same level as the current nvim plugins avail. As far as I can tell it's just a UI for adding context.
  - faangguyindia 2y ago
    It's fast and you can edit the context anywhere.
chilldsgn 2y ago
I used Jetbrains' AI Assistant for a while and I made the mistake of trusting some of its code, which introduced bugs and slowed me down to double-check a lot of the boilerplate I had it write for me. Really not worth it for me at this point. I must add that I found Anthropic's Claude quite useful (Sonnet 3.5) when I had to work on a legacy code base using Adobe ColdFusion (*vomit). I knew nothing of Coldfusion or the awful code base, and it helped me figure out a lot of things about Coldfusion without having to spend too much energy on learning and generating code for a framework I will never again use. I still had to make some updates to the code, but it was less cognitive effort that having to read docs and spend time Googling.
nsonha 2y ago
> cursor and its tab completion If this is how you use Cursor then you dont need Cursor. Autocomplete has existed even before AI, but Cursor's selling point is in multi-files editing and a sensible workflow that let users iterate through diffs in the UI. I have never used AI to generate code, only to edit. I can see it useful in both, and we should look at all usecases of AI instead of looking at it as glorified autocomplete.
- DeathArrow 2y ago
  >If this is how you use Cursor then you dont need Cursor. I've tried Cursor yesterday only to realise it can't run and debug C# projects.
  - bathwaterpizza 2y ago
    Cursor is just VS code, you can create a launch.json to do whatever you want, including using the DAP
  - nsonha 2y ago
    That's not a thing that AI can reliably do yet. If any code AI, including Cursor, can do that, it would be pretty ground breaking. There are some projects iterating on that front though. For example, one VSCode extension called Clide can use Anthropic Computer Use to control a headless browser and read console.log to debug in that limited way.
DeathArrow 2y ago
I didn't use it but I've seen videos where people used Cline with Claude to modify the source and afterwards, have Cline run the app collect the errors and fix them. The examples were small and trivial, though. I am not sure how that would work in large and complex code bases.
resonious 2y ago
I had the same experience with Copilot and stopped using it. Been somewhat tempted to try Cursor thinking maybe it's gotten better but this comment is suggesting that maybe it's not. I get the best use out of straight up ChatGPT. It's like a cheatsheet for everything.
chrisandchris 2y ago
Similar experience. I used Gitlab Duo and now the JetBrains AI assistant (the small one which does inly inline). What I notice is that line-completion is quite good if you read the produced code in prosa. But most of the times the internals are different, and so the completed line is still useless. E.g. assume an Enum Completion with values InlineCompletion, FullLine, NoCompletion. If i write > if (currentMode == Compl It will happily suggest > if (currentMode == Completion.FullLineCompletion) { while not realizing that this enum values does not exist.
onion2k 2y ago
boilerplate and outer app layers, yes; architecture and core libraries, no That's what you want though, isn't it? Especially the boilerplate bit. Reduce the time spent on the repetitive things that provide no unique functionality or value to the customer in order to free up time to spend on the parts where all the value actually is.
kamaal 2y ago
>>I need to double and triple check the updated code Well, one the most important skills in AI generated code era is the ability to read code, and quickly. Another thing is, writing smaller functions helps.
empath75 2y ago
I use copilot to write boilerplate code that I know how to write but I don't feel like writing it. When it gets it wrong, it's easy to tell and easy to fix.

by thih9 2y ago

MuffinFlavored 2y ago

> in practice I’m not noticing a productivity boost. How can this be possible if you literally admit its tab completion is mindblowing? Isn't really good tab completion good enough for at least a 5% producitvity boost? 10%? 20%? Select line of code, prompt it to refactor, verify they are good, accept the changes

m3kw9 2y ago
In experience, I always have to spend time to check and most times it doesn’t do what I need unless it’s very simple asks.
thih9 2y ago
> How can this be possible if you literally admit its tab completion is mindblowing? What about it makes it impossible? I’m impressed by what AI assistants can do - and in practice it doesn’t help me personally. > Select line of code, prompt it to refactor, verify they are good, accept the changes. It’s the “verify” part that I find tricky. Do it too fast and you spend more time debugging than you originally gained. Do it too slow and you don’t gain much time. There is a whole category of bugs that I’m unlikely to write myself but I’m likely to overlook when reading code. Mixing up variable types, mixing up variables with similar names, misusing functions I’m unfamiliar with and more.
- beepbooptheory 2y ago
  I think the essential point around impressive vs helpful sums up so much of the discourse around this stuff. Its all just where you fall on the line between "impressive is necessarily good" and "no it isn't".
- throttlebody 2y ago
  How does AI learn from it's mistakes ? Genuine question as i have only briefly used ChatGpt and found it interesting but not usefull.
bigstrat2003 2y ago
> How can this be possible if you literally admit its tab completion is mindblowing? If I had a knife of perfect sharpness which never dulled, that would be mind-blowing. It also would very likely not make me a better cook.
- goosejuice 2y ago
  Having a sharp knife will make you far more efficient if you know how to use it in the first place and sharpening knives takes time away from doing the cooking. This is kind of a wonky analogy though.
kortilla 2y ago
If someone can eat 20 golf balls that’s impressive but it doesn’t improve my golf game
- hackable_sand 2y ago
  That is actually very impressive.
bradford 2y ago
> How can this be possible if you literally admit its tab completion is mindblowing? I might suggest that coding doesn't take as much of our time as we might think it does. Hypothetically: Suppose coding takes 20% of your total clock time. If you improve your coding efficiency by 10%, you've only improved your total job efficiency by 2%. This is great, but probably not the mind-blowing gain that's hyped by the AI boom. (I used 20% as a sample here, but it's not far away from my anecdotal experience, where so much of my time is spent in spec gathering, communication, meeting security/compliance standards, etc).
- MuffinFlavored 2y ago
  > that's hyped by the AI boom I always personally thought the AI boom was more about 'what can be done automation/improvement wise for non-tech industries'

MangoCoffee 2y ago

Time will tell. As a GitHub Copilot user, I still review the code. SpaceX's advancements are impressive, from rocket blow up to successfully catching the Starship booster. Who knows what AI will be capable of in 5-10 years? Perhaps it will revolutionize code assistance or even replace developers

outworlder 2y ago
> SpaceX's advancements are impressive, from rocket blow up to successfully catching the Starship booster. That felt like it was LLM generated since that doesn't have anything to do with the subject being discussed. Not only it's on a different industry but it's a completely different set of problems. We know what's involved in catching a rocket. It's a massive engineering challenge yes, but we all know it can be done(whether or not it makes sense or is economically viable are different issues). Even going to the Moon – which was a massive project and took massive focus from an entire country to do – was a matter of developing the equipment, procedures, calculations (and yes, some software). We knew back then it could be done, and roughly how. Artificial intelligence? We don't know enough about "intelligence". There isn't even a target to reach right now. If we said "resources aren't a problem, let's build AI", there isn't a single person on this planet that can tell you how to build such an AI or even which technologies need to be developed. More to the point, current LLMs are able to probabilistically generate data based on prompts. That's pretty much it. They don't "know" anything about what they are generating, they can't reason about it. In order for "AI" to replace developers entirely, we need other big advancements in the field, which may or may not come.
- dspillett 2y ago
  > Artificial intelligence? We don't know enough about "intelligence". The problem I have with this objection is that it, like many discussions, conflates LLMs (glorified predictive text) and other technologies currently being referred to as AI, with AGI. Most of these technologies should still be called machine learning as they aren't really doing anything intelligent in the sense of general intelligence. As you say yourself: they don't know anything. And by inference, they aren't reasoning about anything. Boilerplate code for common problems, and some not so common ones, which is what LLMs are getting pretty OK at and might in the coming years be very good at, is a definable problem that we understand quite well. And much as we like to think of ourselves as "computer scientists", the vast majority of what we do boils down to boilerplate code using common primitives, that are remarkably similar across many problem domains that might on first look appear to be quite different, because many of the same primitives and compound structures are used. The bits that require actual intelligence are often quite small (this is how I survive as a dev!), or are away from the development coalface (for instance: discovering and defining the problems before we can solve them, or describing the problem & solution such that someone or an "AI" can do the legwork). > we need other big advancements in the field, which may or may not come. I'm waiting for an LLM being guided to create a better LLM, and eventually down that chain a real AGI popping into existence, much like the infinite improbability drive being created by clever use of a late version finite improbability generator. This is (hopefully) many years (in fact I'm hoping for at least a couple of decades so I can be safely retired or nearly there!) from happening, but it feels like such things are just over the next deep valley of disillusionment.

imafish 2y ago

I rarely use the tab completion. Instead I use the chat and manually select files I know should be in context. I am barely writing any code myself anymore. Just sanity checking that the output and “piping” is correct. My productivity (in frontend work at least) is significantly higher than before.

big_jimmer 2y ago
Out of curiosity, how long have you been working as a developer? Just that, in my experience, this is mostly true for juniors and mids (depending on the company, language, product etc. etc.). For example, I often find that copilot will hallucinate tailwind classes that don't exist in our design system library, or make simple logical errors when building charts (sometimes incorrect ranges, rarely hallucinated fields) and as soon as I start bringing in 3rd party services or poorly named legacy APIs all hope is lost and I'm better off going it alone with an LSP and a prayer.
- imafish 2y ago
  Around 10 years. I also find that claude hallucinates once in a while, but I usually catch it. My main job becomes requesting and reviewing code instead of writing it. But I don't think it is fair to compare copilot and cursor. I have not been able to gain any significant productivity boost from using copilot (which I have in my Visual Studio proper).

SparkyMcUnicorn 2y ago

I haven't used Cursor, but I use Aider with Sonnet 3.5 and also use Copilot for "autocomplete". I'd highly recommend reading through Aider's docs[0], because I think it's relevant for any AI tool you use. A lot of people harp on prompting, and while a good prompt is important I often see developers making other mistakes like not providing context that's good, correct, or even too much[1]. When I find models are going on the wrong path with something, or "connecting the pipes wrong", I often add code comments that provide additional clarity. Not only does this help future me/devs, but the more I steer AI towards correct results, the fewer problems models seem to have going forward. Everybody seems to be having wildly different experiences using AI for coding assistance, but I've personally found it to be a big productivity boost. [0] https://aider.chat/docs/usage/tips.html https://aider.chat/docs/usage/tips.html [1] https://aider.chat/docs/troubleshooting/edit-errors.html#reduce-distractions https://aider.chat/docs/troubleshooting/edit-errors.html#red...

realce 2y ago
Totally agree that heavy commenting is the best convention for helping the assistant help you best. I try to comment in a way that makes a file or function into a "story" or kind of a single narrative.
- jascha_eng 2y ago
  That's super interesting, I've been removing a lot of the redundant comments from the AI results. But adding new more explanatory ones that make it easier for both AI and humans to understand the code base makes a lot of sense in my head. I was big on writing code to be easy to read for humans, but it being easy to read for AI hasn't been a large concern of mine.

bob1029 2y ago

It's the subtle errors that are really difficult to navigate. I got burned for about 40 hours on a conditional being backward in the middle of an otherwise flawless method. The apparent speed up is mostly a deception. It definitely helps with rough outlines and approaches. But, the faster you go, the less you will notice the fine details, and the more assumptions you will accumulate before realizing the fundamental error. I'd rather find out I was wrong within the same day. I'd probably have written some unit tests and played around with that function a lot more if I had handcrafted it.

enneff 2y ago
That’s the thing, isn’t it? The craft of programming in the small is one of being intimate with the details, thinking things through conscientiously. LLMs don’t do that.
- Nevermark 2y ago
  Perhaps it should be prompted to then? Ask it to review its own code for any problems? Also identify typical and corner cases and generate tests? Question marks here because I have not used the tool. The size & depth of each accepted code step is still up to the developer slash prompter
  - nrclark 2y ago
    I use Chatgpt for coding / API questions pretty frequently. It's bad at writing code with any kind of non-trivial design complexity. There have been a bunch of times where I've asked it to write me a snippet of code, and it cheerfully gave me back something that doesn't work for one reason or another. Hallucinated methods are common. Then I ask it to check its code, and it'll find the error and give me back code with a different error. I'll repeat the process a few times before it eventually gets back to code that resembles its first attempt. Then I'll give up and write it myself. As an example of a task that it failed to do: I asked it to write me an example Python function that runs a subprocess, prints its stdout transparently (so that I can use it for running interactive applications), but also records the process's stdout so that I can use it later. I wanted something that used non-blocking I/O methods, so that I didn't have to explicitly poll every N milliseconds or something.

knallfrosch 2y ago

I use it for an unfamiliar programming language and it's very nice. You can also ask it to explain badly documented code.

weitendorf 2y ago

I'm building a tool in this space and believe it's actually multiple separate problems. From most to least solvable: 1. AI coding tools benefit a lot from explicit instructions/specifications and context for how their output will be used. This is actually a very similar problem to when eg someone asks a programmer "build me a website to do X" and then being unhappy with the result because they actually wanted to do "something like X", and a payments portal, and yellow buttons, and to host it on their existing website. So models need to be given those particular instructions somehow (there are many ways to do it, I think my approach is one of the best so far) and context (eg RAG via find-references, other files in your codebase, etc) 2. AI makes coding errors, bad assumptions, and mistakes just like humans. It's rather difficult to implement auto-correction in a good way, and goes beyond mere code-writing into "agentic" territory. This is also what I'm working on. 3. AI tools don't have architecture/software/system design knowledge appropriate represented in their training data and all the other techniques used to refine the model before releasing it. More accurately, they might have knowledge in the form of eg all the blog posts and docs out there about it, but not skill. Actually, there is some improvement here, because I think o1 and 3.5 sonnet are doing some kind of reinforcement-learning/self-training to get better at this. But it's not easily addressable on your end. 4. There is ultimately a ton of context cached in your brain that you cannot realistically share with the AI model, either because it's not written anywhere or there is just too much of it. For example, you may want to structure your code in a certain way because your next feature will extend it or use it. Or your product is hosted on serving platform Y which has an implementation detail where it tries automatically setting Content-Type response headers by appending them to existing headers, so manually setting Content-Type in the response causes bugs on certain clients. You can't magically stuff all of this into the model context. My product tries to address all of these to varying extents. The largest gains in coding come from making it easier to specify requirements and self-correct, but architecture/design are much harder and not something we're working on much. You or anybody else can feel free to email me if you're interested in meeting for a product demo/feedback session - so far people really like our approach to setting output specs.

fullstackwife 2y ago

One of the reasons for that may be the price: large code changes with multi turn conversation can eat up a lot of tokens, while those tools charge you a flat price per month. Probably many hacks are done under the hood to keep *their* costs low, and the user experiences this as lower quality responses. Still the "architecture and core libraries" is rather corner case, something at the bottom of their current sales funnel. also: do you really want to get equivalent of 1 FTE work for 20 USD per month?:)

tomrod 2y ago

I'd love an autoselected LLM that is fine-tuned to the syntax I'm actively using -- Cursor has a bit of a head start, but where Github and others can take it could be mindblowing (Cursor's moat is a decent VS Code extension -- I'm not sure it's a deep moat though).

pnathan 2y ago

In general I do not find AI a net positive. Other tools seem to do at least as well in general. it can be used if you want the reliability of a random forum poster. which... sure. knock yourself out. sometimes there's gems in that dirt. I'm getting _very_ bearish on using LLMs for things that aren't pattern recognition.

bmitc 2y ago

That's my exact experience with GitHub Copilot. Even boilerplate stuff it sucks at as well. I have no idea why its autocomplete is so bad when it has access to my code, the function signatures, types, etc. It gets stuff wrong all the time. For example, it will just flat out suggest functions that don't exist, neither in the Python core libraries or in my own modules. It doesn't make sense. I have all but given up on using Copilot for code development. I still do use it for autocomplete and boilerplate stuff, but I still have to review that. So there's still quite a bit of overhead, as it introduces subtle errors, especially in languages like Python. Beyond that, it's failure rate at producing running, correct code is basically 100%.

miki123211 2y ago

For now, I mostly use AI as a "faster typist". If it wants to complete what I wanted to type anyway, or something extremely similar, I just press tab, otherwise I type my own code. I'd say about 70% of individual lines are obvious enough if you have the surrounding context that this works pretty well in practice. This number is somewhat lower in normal code and higher in unit tests. Another use case is writing one-off scripts that aren't connected to any codebase in particular. If you're doing a lot of work with data, this comes in very handy. Something like "here's the header of a CSV file", pass each row through model x, only pass these three fields, the model will give you annotations, put these back in the csv and save, show progress, save every n rows in case of crashes, when the output file exists, skip already processed rows." I'm not (yet) convinced by AI writing entire features, I tried that a few times and it was very inconsistent with the surrounding codebase. Managing which parts of the codebase to put in its context is definitely an art though. It's worth keeping in mind that this is the worst AI we'll ever have, so this will probably get better soon.

zeroonetwothree 2y ago
One off scripts do work very well.
blitzar 2y ago
Reminds me of how I use the satnav when driving. I don't close my eyes and do whatever it tells me to do. If I think I know better I don't "turn right at the next set of lights" I just drive on as I would have before GPS and eventually realise that I went the wrong way or the satnav realises there was a perfectly valid 2nd/3rd/4th path to get to where I wanted to go.
DeathArrow 2y ago
Yes, for simple boring tasks like converting a JSON to C# classes, it does wonders. For everything clever it kind of needs baby sitting which doesn't make it faster than writing the code myself. Even for the glorified use case of writing unit tests it sucks unless the code is very simple with few dependencies.

ianbutler 2y ago

I'm actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. At the end of the day it can belt out lines of code faster than I, or any human, can and I can review code very quickly so the overall productivity boost has been great. It does fundamentally alter my workflow. I'm very hands off keyboard when I'm working with AI in a way that is much more like working with someone or coaching someone to make something instead of doing the making myself. Which I'm fine with but recognize many developers aren't. I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching.

__float 2y ago
I'm not sure how many people are like me, but my attempts to use Copilot have largely been the context of writing code as usual, occasionally getting end-of-line or handful-of-lines completions from it. I suspect there's probably a bigger shift needed, but I haven't seen anyone (besides AI "influencers" I don't trust..?) showing what their day-to-day workflows look like. Is there a Vimcasts equivalent for learning the AI editor tips and tricks?
- sbarre 2y ago
  Have you tried the chat mode? The autocomplete is somewhere between annoying and underwhelming for me, but the chat is super useful. Being able to just describe what you're thinking or what you're trying to do and having a bespoke code sample just show up (based on the code in your editor) that you can then either copy/paste in, cherry-pick from or just get inspired by, has been a great productivity booster.. Treat it like a pair programmer or a rubber duck and you might have a better experience. I did!
- whimsicalism 2y ago
  Yeah using a chat interface

Aeolun 2y ago

> in practice I’m not noticing a productivity boost I am. Can suddenly do in a weekend what would have taken a week.

Yodel0914 2y ago

I find chatgpt incredibly useful for writing scripts against well-known APIs, or for a "better stackoverflow". Things like "how do I use a cursor in sql" or "in a devops yaml pipeline, I want to trigger another pipeline. How do I do that?". But working on our actual codebase with copilot in the IDE (Rider, in my case) is a net negative. It usually does OK when it's suggesting the completion of a single line, but when it decides to generate a whole block it invariably misunderstands the point of the code. I could imagine that getting better if I wrote more descriptive method names or comments, but the killer for me is that it just makes up methods and method signatures, even for objects that are part of publicly documented frameworks/APIs.

jmpeax 2y ago
Same here. If you need to lookup how to do something in an api I find it much faster to use chatgpt than to try to search through the janky official docs or in some github examples folder. Chatgpt is basically documentation search 2.0.
deepfriedbits 2y ago
I love your framing of it as a "better stackoverflow." That's so true. However, I feel like some of our complaints about accuracy and hidden bugs are temporary pain (12-36) months before the tools truly become mind-blowing productivity multipliers.
DeathArrow 2y ago
>"better stackoverflow" I had asked hard questions on SO that only a human with experience can answer. And I found answers on SO that only a human with experience can answer.
vultour 2y ago
The problem with this is that you lose the entire context around the answer you're given. There might be notes about limitations or edge cases in the documentation or on StackOverflow. I already see this with my coworkers who use LLMs, they often have no idea how or why things work and then they're absolutely perplexed when something goes wrong.

breckenedge 2y ago

Tab complete is just one of their proprietary models. I find chat-mode more helpful for refactoring and multi-file updates, even more when I specify the exact files to include.

DeathArrow 2y ago
I think SuperMaven has tab complete and let's you chose the model.

grensley 2y ago

I think there's a number of factors that make it work as well as it does for me: - Mostly writing React - Not using any obscure or new libraries - Naming things well - Keeping logic simple - Leaving a comment at the point where I'm about to make a shift from what the common logic would be - Getting a feel for when it's going to be able to correctly guess or not (and not even reading it if I think it's going to be wrong) - Trusting short blocks more than long ones

quotemstr 2y ago

AI will have the effect of shifting development effort from authorship to verification. As you note, we've come a long way towards making the writing of the code practically free, but we're going to need to beef up our tools for understanding code already written. I think we've only scratched the surface of AI-assisted program analysis.

dham 2y ago

With React, the guesses it makes around what props / types I want where, especially moving from file to file, is worth the price of admission. Everything else it does it icing on the cake. The new import suggestion is much quicker than the Typescript compiler lmao. And it's always the right one, instead of suggesting ones that aren't relevant. Composer can be hit or miss, but I've found it really good at game programming.

stuckkeys 2y ago

honestly. I hate it. I find myself banging my head on the table because of how much endless cycles it goes through and not give me the solution haha. I probably started projects over multiple times because the code it generated was so bad lol.

willtemperley 2y ago

I've found explaining boilerplate code to Claude or ChatGPT has been very worthwhile, the boring code gets written and the prompt is the documentation. OTOH tab completion in Intellij and Xcode has been useful occasionally, usually distracting and sometimes annoying. A way to fast toggle this would be good, but when I know what I want to code, good old code completion works nicely thanks.

faangguyindia 2y ago

Zed AI is much better than cursor I've found.

goosejuice 2y ago
In what way exactly? I wish Zed was better here but it's at about the same level as the current nvim plugins avail. As far as I can tell it's just a UI for adding context.
- faangguyindia 2y ago
  It's fast and you can edit the context anywhere.

chilldsgn 2y ago

I used Jetbrains' AI Assistant for a while and I made the mistake of trusting some of its code, which introduced bugs and slowed me down to double-check a lot of the boilerplate I had it write for me. Really not worth it for me at this point. I must add that I found Anthropic's Claude quite useful (Sonnet 3.5) when I had to work on a legacy code base using Adobe ColdFusion (*vomit). I knew nothing of Coldfusion or the awful code base, and it helped me figure out a lot of things about Coldfusion without having to spend too much energy on learning and generating code for a framework I will never again use. I still had to make some updates to the code, but it was less cognitive effort that having to read docs and spend time Googling.

nsonha 2y ago

> cursor and its tab completion If this is how you use Cursor then you dont need Cursor. Autocomplete has existed even before AI, but Cursor's selling point is in multi-files editing and a sensible workflow that let users iterate through diffs in the UI. I have never used AI to generate code, only to edit. I can see it useful in both, and we should look at all usecases of AI instead of looking at it as glorified autocomplete.

DeathArrow 2y ago
>If this is how you use Cursor then you dont need Cursor. I've tried Cursor yesterday only to realise it can't run and debug C# projects.
- bathwaterpizza 2y ago
  Cursor is just VS code, you can create a launch.json to do whatever you want, including using the DAP
- nsonha 2y ago
  That's not a thing that AI can reliably do yet. If any code AI, including Cursor, can do that, it would be pretty ground breaking. There are some projects iterating on that front though. For example, one VSCode extension called Clide can use Anthropic Computer Use to control a headless browser and read console.log to debug in that limited way.

DeathArrow 2y ago

I didn't use it but I've seen videos where people used Cline with Claude to modify the source and afterwards, have Cline run the app collect the errors and fix them. The examples were small and trivial, though. I am not sure how that would work in large and complex code bases.

resonious 2y ago

I had the same experience with Copilot and stopped using it. Been somewhat tempted to try Cursor thinking maybe it's gotten better but this comment is suggesting that maybe it's not. I get the best use out of straight up ChatGPT. It's like a cheatsheet for everything.

chrisandchris 2y ago

Similar experience. I used Gitlab Duo and now the JetBrains AI assistant (the small one which does inly inline). What I notice is that line-completion is quite good if you read the produced code in prosa. But most of the times the internals are different, and so the completed line is still useless. E.g. assume an Enum Completion with values InlineCompletion, FullLine, NoCompletion. If i write > if (currentMode == Compl It will happily suggest > if (currentMode == Completion.FullLineCompletion) { while not realizing that this enum values does not exist.

onion2k 2y ago

boilerplate and outer app layers, yes; architecture and core libraries, no That's what you want though, isn't it? Especially the boilerplate bit. Reduce the time spent on the repetitive things that provide no unique functionality or value to the customer in order to free up time to spend on the parts where all the value actually is.

kamaal 2y ago

>>I need to double and triple check the updated code Well, one the most important skills in AI generated code era is the ability to read code, and quickly. Another thing is, writing smaller functions helps.

empath75 2y ago

I use copilot to write boilerplate code that I know how to write but I don't feel like writing it. When it gets it wrong, it's easy to tell and easy to fix.