9 ms·

I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage bo

I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage boost in "productivity". This is not life-changing for me. And for some tasks, it's actually worse than nothing. As in, I spend time feeding it a task, and it just completely fails to do anything useful.

jimmaswell 2y ago
I've been using it for over a year I think. I don't often feed it tasks with comments so much as go about things the same as usual and let it autocomplete. The time and cognitive load saved adds up massively. I've had to go without it for a bit while my workplace gets its license in order for the corporate version and the personal version has an issue with the proxy, and it's been agonizing going without it again. I almost forgot how much it sucks having to jump to google every other minute, and it was easy to start to take for granted how much context copilot was letting me not have to hold onto in my head. It really lets me work on the problem as opposed to being mired in immaterial details. It feels like I'm at least 2x slower overall without it.
- rockskon 2y ago
  I don't know about you but LLMs spit out garbage nonsense frequent enough that I can't trust their output in any context I cannot personally verify the validity of.
  - ab5tract 2y ago
    Same here. The Cody autocomplete is so off base all the time that it’s deactivated. I serve Cody a direct question about 1-3 times a week. Of that, it gets maybe 50% correct on the first try. I don’t bother with a second try because by then I’ve already spent the equivalent amount of time looking at the relevant library source code and/or docs would have taken.
- atq2119 2y ago
  > I almost forgot how much it sucks having to jump to google every other minute Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule. I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.
  - namaria 2y ago
    Claiming LLMs are a massive boost for coding productivity is becoming a red flag that the claimant has a tenuous grasp on the skills necessary. Yeah if you have to look up everything all the time and you can't tell the AI slop isn't very good, you can put out code quite fast.
    - soulofmischief 2y ago
      Comments like this are a great example of the Dunning-Kruger effect. Your comment is actually an indication that you don't have the mastery required to get useful, productive output from a high quality LLM. Maybe you don't push your boundaries as an engineer and thus rarely need to know new things or at least learn new API surfaces. Maybe you don't know how to effectively prompt an LLM. Maybe you lack the mastery to analyze and refine the results. Maybe you just like doing things the slow way. I too remember a time as an early programmer where I eschewed even Intellisense and basic auto complete... I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers just because you don't understand their workflow. Just because you've had subpar experiences with a new tool doesn't mean it's not a useful tool in another engineer's toolkit.
      - namaria 2y ago
        Funny you should make claims about my skills when you have exactly zero data about my abilities or performance. Evaluating my skills based on how I evaluated someone else's skills when they tell me about their abilities with and without a crutch, and throwing big academic sounding expressions with 'effect' in them might be intimidating to some but to me it just transparently sounds pretentious and way off mark, since, like I said, you have zero data about my abilities or output. > I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers You're clearly coming from an emotional place because you feel slighted. There is no 'class of engineers' in my evaluation. I recommend reading comments more closely, thinking about their content, and not getting offended when someone points out signs of lacking skills, because you might just be advertising your own limitations.
framapotari 2y ago
If you're already a competent developer, I think that's a reasonable expectation of impact on productivity. I think the 'life-changing' part comes in helping someone get to the point of building things with code where before they couldn't (or believed they couldn't). It does a lot better job of turning the enthusiasts and code-curious into amateurs vs. empowering professionals.
- jimmaswell 2y ago
  > turning the enthusiasts and code-curious into amateurs vs. empowering professionals. I'm firmly in #2. My other comment goes over how. I'm intrigued to see how devs in #1 grow. One might be wary those devs would grow into bad habits and not thinking for themselves, but it might be a case of the ancient Greek rant against written books hindering memorization. Could be that they'll actually grow to be even better devs unburdened by time wasted on trivial details.
datavirtue 2y ago
If you are in maintenance mode your visits to Copilot will be rare. If you are building greenfield, use goes through the roof. All those test cases, nevermind all the POC and framework scaffolding and other boilerplate that is now completely unacceptable as a use of developer time.
- recursive 2y ago
  I'm building "greenfield". I still use it at least daily, but the benefit just struggles to outweigh the cost of invoking it. Maybe I don't understand how to use it.
  - datavirtue 2y ago
    It really depends on what you are doing and what tech you are using. I use it to teach me or build out ideas quickly or solve for complex issues. Mostly these days I use it as a memory aid or to bounce ideas off it. In my job I have to move quickly and stay focused as I'm driving improvements to a tech stack that reaches across four verticals, each having their own quirks and tech stacks. It's great for jogging my memory and helping to flush out ideas and approaches that I then bounce off the dev teams. Super helpful.

jimmaswell 2y ago

I've been using it for over a year I think. I don't often feed it tasks with comments so much as go about things the same as usual and let it autocomplete. The time and cognitive load saved adds up massively. I've had to go without it for a bit while my workplace gets its license in order for the corporate version and the personal version has an issue with the proxy, and it's been agonizing going without it again. I almost forgot how much it sucks having to jump to google every other minute, and it was easy to start to take for granted how much context copilot was letting me not have to hold onto in my head. It really lets me work on the problem as opposed to being mired in immaterial details. It feels like I'm at least 2x slower overall without it.

rockskon 2y ago
I don't know about you but LLMs spit out garbage nonsense frequent enough that I can't trust their output in any context I cannot personally verify the validity of.
- ab5tract 2y ago
  Same here. The Cody autocomplete is so off base all the time that it’s deactivated. I serve Cody a direct question about 1-3 times a week. Of that, it gets maybe 50% correct on the first try. I don’t bother with a second try because by then I’ve already spent the equivalent amount of time looking at the relevant library source code and/or docs would have taken.
atq2119 2y ago
> I almost forgot how much it sucks having to jump to google every other minute Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule. I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.
- namaria 2y ago
  Claiming LLMs are a massive boost for coding productivity is becoming a red flag that the claimant has a tenuous grasp on the skills necessary. Yeah if you have to look up everything all the time and you can't tell the AI slop isn't very good, you can put out code quite fast.
  - soulofmischief 2y ago
    Comments like this are a great example of the Dunning-Kruger effect. Your comment is actually an indication that you don't have the mastery required to get useful, productive output from a high quality LLM. Maybe you don't push your boundaries as an engineer and thus rarely need to know new things or at least learn new API surfaces. Maybe you don't know how to effectively prompt an LLM. Maybe you lack the mastery to analyze and refine the results. Maybe you just like doing things the slow way. I too remember a time as an early programmer where I eschewed even Intellisense and basic auto complete... I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers just because you don't understand their workflow. Just because you've had subpar experiences with a new tool doesn't mean it's not a useful tool in another engineer's toolkit.
    - namaria 2y ago
      Funny you should make claims about my skills when you have exactly zero data about my abilities or performance. Evaluating my skills based on how I evaluated someone else's skills when they tell me about their abilities with and without a crutch, and throwing big academic sounding expressions with 'effect' in them might be intimidating to some but to me it just transparently sounds pretentious and way off mark, since, like I said, you have zero data about my abilities or output. > I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers You're clearly coming from an emotional place because you feel slighted. There is no 'class of engineers' in my evaluation. I recommend reading comments more closely, thinking about their content, and not getting offended when someone points out signs of lacking skills, because you might just be advertising your own limitations.

framapotari 2y ago

If you're already a competent developer, I think that's a reasonable expectation of impact on productivity. I think the 'life-changing' part comes in helping someone get to the point of building things with code where before they couldn't (or believed they couldn't). It does a lot better job of turning the enthusiasts and code-curious into amateurs vs. empowering professionals.

jimmaswell 2y ago
> turning the enthusiasts and code-curious into amateurs vs. empowering professionals. I'm firmly in #2. My other comment goes over how. I'm intrigued to see how devs in #1 grow. One might be wary those devs would grow into bad habits and not thinking for themselves, but it might be a case of the ancient Greek rant against written books hindering memorization. Could be that they'll actually grow to be even better devs unburdened by time wasted on trivial details.

datavirtue 2y ago

If you are in maintenance mode your visits to Copilot will be rare. If you are building greenfield, use goes through the roof. All those test cases, nevermind all the POC and framework scaffolding and other boilerplate that is now completely unacceptable as a use of developer time.

recursive 2y ago
I'm building "greenfield". I still use it at least daily, but the benefit just struggles to outweigh the cost of invoking it. Maybe I don't understand how to use it.
- datavirtue 2y ago
  It really depends on what you are doing and what tech you are using. I use it to teach me or build out ideas quickly or solve for complex issues. Mostly these days I use it as a memory aid or to bounce ideas off it. In my job I have to move quickly and stay focused as I'm driving improvements to a tech stack that reaches across four verticals, each having their own quirks and tech stacks. It's great for jogging my memory and helping to flush out ideas and approaches that I then bounce off the dev teams. Super helpful.