15 ms·

Did GitHub Copilot increase my productivity?

• AnarchismIsCool 2y ago
  Local maxima go BRRRRRRR AI will get there eventually, but this current paradigm seems increasingly only useful for spam and shitty clip art. Even so, everyone is throwing absurd amounts of investment capital at it in the hopes that something useful will happen. It's a pretty clear depiction of the investor class being so detached from the technical reality of what they're investing in that they just sit around lighting billions on fire thinking that they're getting richer instead of poorer. Go ahead and buy more H100s though...
  • simonw 2y ago
    I've been finding this stuff genuinely useful for two years now, across Copilot and ChatGPT and Claude 3 Opus and similar tools. Either I'm a dimwit, easily conned by hype and shiny tools to the point that I can imagine benefits for two years that simply aren't there... or there's something to them.
    • aimazon 2y ago
      I don’t think you’re a dimwit but I read your post[1] with an example and I am curious to whether you feel you’re losing something by telling the LLM it’s wrong and to try again, rather than going through the exploratory/iterative learning process yourself. For example, would you have known to ask about GeoJSON if you had not come across and learned about it pre-LLM? More succinctly: do you feel you’re learning more or less or an equal amount when using LLMs? [1] https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-case-study/ https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-cas...
      • noduerme 2y ago
        You seem to be writing from a lost world when people enjoyed coding because they enjoyed learning new things with every project. I think that's sadly something only enjoyed at work by a very small minority these days.
        • simonw 2y ago
          I very much aim to learn something new with every project. The thing I like most about working with LLMs is that they let me take on MORE projects!
        • eru 2y ago
          I don't think those 'good old days' ever existed. Or rather, the situation hasn't really gotten worse.
          • noduerme 2y ago
            I probably wasn't clear I still live in that world, both professionally and in personal projects. But my perception is that the vast majority of devs and engineers do not.
      • simonw 2y ago
        Absolutely I could have learned more from that particular project if I'd spent more time with it rather than getting the LLM to do the work... but that's why I like it as an example: since it was effectively a distraction (a "side quest") the alternative wasn't learning more, it was not doing it at all (and learning nothing). I'm able to get really great results out of LLMs because I have 20+ years of experience helping me know what questions to ask of them. I do feel like my rate of learning has increased though, because I'm much more likely to try out a completely new technology when I know an LLM can flatten the learning curve for me a bit.
        • gtirloni 2y ago
          > I do feel like my rate of learning has increased It's the same for me. Lots of experience knowing what to ask for. It does a better job in summarizing knowledge and getting me a relatively coherent explanation. Much faster than using Google Search to find and connect the dots from dozens of pages. I just don't see much benefit in its reasoning and code assistance features besides basic stuff.
        • yawaramin 2y ago
          > I have 20+ years of experience helping me know what questions to ask of them. That seems rather unlikely given that LLMs didn't exist more than seven years ago, much less 20+.
          • Loocid 2y ago
            They mean they have 20 years experience in programming, which helps them ask good questions.
            • simonw 2y ago
              Yeah, that's what I meant. With LLMs having deep domain knowledge in a field helps enormously in terms of prompting them as productively as possible.
              • 2y ago
      • erhaetherth 2y ago
        I was trying to fetch key-value pairs out of a database using PHP+PDO the other day and I knew there was a nice easy to do it but I couldn't remember how. Something about fetchAll, maybe PDO::FETCH_GROUP|PDO::FETCH_COLUMN.... what was it? So I asked a couple LLMs. They wrote out loops for me to format the data how I wanted. I could have copy-and-pasted that in and it would probably have worked. But I felt there was something better yet, so back to Google I go. It's `PDO::FETCH_KEY_PAIR`. It's built-in. But oddly kind of hard to find unless you know the right thing to search for, and "key pair" was not springing to my mind. Point is, if you just let the LLMs do your work you won't even find these better ways of doing things. And I'm quite afraid of what the LLMs are going to do to Google, Stackoverflow and documentation in general. In a couple users it'll be ungoogleable too.
        • fzeroracer 2y ago
          As noted LLMs can only give you what you ask for, but for a lot of problems what you ask for isn't what you need; it's two or three steps removed. And LLMs can't tell you that you're doing something wrong; unlike curmudgeonly users on SO or in various forums/channels. My gut feeling is that we're going to enter into a 'dark age' of coding where a lot of previously available resources are going to be ransacked and made hard to find in favor of big corporation owned LLMs. It's already having an extremely bad effect on search in general; we're potentially only a few fights away from sites like SO having users leave en masse. That's why I think having a strong network of engineers to talk with will become more important than ever, almost a return to the IRC days.
        • DrSiemer 2y ago
          Strong disagree. I have learned so many new things since I started using LLM's. Some of them I'm actually embarrassed to admit, because I should have known about them a decade ago. If you work in a small company and you are the most experienced developer, you don't often get feedback on how you can improve things. The trick is, quite simply: just ask. I regularly dump some code I wrote in a language model and then ask what can be done better. I would never do that in any online space, because first, I don't wait an answer maybe some day, I need an answer NOW. And second, I prefer to avoid being called a fool.
    • anonzzzies 2y ago
      Nah, it seems a lot of people are using them wrong? OP seems to also fall in that category. I sit next to many people in pair programming situations and it’s quite weird to see very smart programmers using gpt or copilot; they are the ones shouting ‘stochastic parrot’ on forums and yet expect some kind of mind reading magic when using these tools. When I show that you can put a comment for your function, like you should do anyway, and it writes the function, it usually clicks. Many still find it ‘faster and easier’ to write the function themselves and that’s fine, but, like OP, doing things like entering ???? and generally expecting a useful result will not be optimal.
      • ern 2y ago
        You make an interesting point. I once worked with someone who was brilliant, but had gotten into the field via a math degree, and we were building an enterprise-software product. We tried pair-programming, and he cracked within a few minutes. He couldn't articulate his thought processes verbally. Sounds similar to the scenario you're describing.
    • blibble 2y ago
      I wouldn't feel too bad, there's orders more magnitude more money put into this con vs. the last one (NFTs)
      • eru 2y ago
        Your counting of 'cons' seems pretty idiosyncratic. You know there have been lots of other technologies people have been working on in the meantime and concurrently? Some of them more questionable, some of them less.
    • selcuka 2y ago
      I basically say the same when people ask me if the AI bubble is going to burst. While I agree that the current push for LLMs should be taken with a grain of salt, I don't think it's a bubble either. I have been finding it useful too, and I don't think I will stop using it because it will look less shiny in the future.
      • BlueTemplar 2y ago
        The dot com was a bubble that burst spectacularly enough, yet we still use the World Wide Web, and even some of those websites.
    • tymscar 2y ago
      You are clearly not a dimwit, and you don’t only have way more experience than most people here, you also have some amazing projects under your belt. However I can’t help but notice that the vast majority of blogposts, talks, tweets, and basically everything else you do now is around LLMs. Do you not think that’s indicative of this being “hyped” and “shiny tools”?
      • ToValueFunfetti 2y ago
        If hype just means people are excited about it and talking about it, it should be a positive signal about the merit of the thing. I think people get the wrong idea by measuring P(X is good | people are excited about X) and finding it to be low. But P(X is good | people are not excited about X) is vastly lower, and 'hype' as here used is not really discrediting.
        • thaumasiotes 2y ago
          > I think people get the wrong idea by measuring P(X is good | people are excited about X) and finding it to be low. But P(X is good | people are not excited about X) is vastly lower Only if you're including things that don't exist. If you restrict yourself to things that people might conceivably talk about, that probability is actually very high, for the simple reason that bad things are exciting specifically because they are bad. Here are some boring things: - bread - sunlight - air - water - parents Note that starvation, darkness, suffocation, dehydration, and being orphaned are all much more exciting than their opposites.
          • ToValueFunfetti 2y ago
            Poor phrasing on my part; 'good' should have been 'worth talking about'.
      • tucnak 2y ago
        Deep learning is the most important thing to happen to philosophy since Socrates / Wittgenstein.
    • thefaux 2y ago
      You are creating a false dichotomy. Very few people are saying there are no benefits, but many have reasonable concerns both about the current efficacy of these tools as well as the expectation of continued exponential growth which is driving much of the current hype. What if we have already passed the inflection point where the exponential growth transitions to an s-curve? That would mean that this technology on its own would only get marginally better than is today. Maybe 2, 4 or even 10x better than it is today, but not 100x or 1000x. To break those barriers, we would need further innovations beyond just throwing more gpus and training data at the problem. I personally am short on LLMs because I believe it is much more likely that we have already crossed the inflection point or will soon. Again, they are impressive but ultimately I think that LLMs will be at best a footnote in history if they are even remembered at all in a few hundred years. But of course I could be wrong.
      • simonw 2y ago
        I've been generally staying away from the whole "imagine what this stuff could do next!" side of the discourse. My personal opinion there is that if all research froze today it would still take us years to figure out all of the potential use-cases and applications for the models we have access to right now, and how best to apply them.
    • vertis 2y ago
      No I'm with you. I don't use copilot though for undirected autocomplete, I use tools that allow me to give instructions and diff the results, Cursor and Aider are my current defaults, both with Claude 3 now[0]. Always looking for alternatives though. I think it's always going to be a YMMV experience with LLMs. I'm an extreme generalist with 23 years of experience so it compliments my strengths and weaknesses. I know how to program across 20 odd languages but most of them I need to look stuff up if I'm not using it frequently enough. Now though, don't really need to look stuff up. For me there are two groups, those that want to use LLMs and are pushing it forward and those that are would prefer that LLMs not exist and want it to be hype that goes away. Having followed the hype cycles of VR/AR and crypto I can feel a difference. Both of those felt like a solution in search of a problem. The true believers wanting it to be something like Ready Player One with VR (fun story, Oculus/Facebook actually handed out copies of the novel at Oculus Connect 2 in 2015). The hype around LLMs seems different; like applying a new solution to all the existing problems to see where it helps, and that's just with the current iteration. [0]: I'd prefer to be using local/open equivalents but the capabilities are still lacking.
  • mhh__ 2y ago
    > spam The first 100 lines of any project is spam (usually 0 entropy), so AI is a nice win.
    • eru 2y ago
      That depends on your language, and project.
    • YurgenJurgensen 2y ago
      If the entropy is 0, you didn't need an LLM. If it's the same every time you can just copy and paste those 100 lines from GitHub or StackOverflow.
      • mhh__ 2y ago
        Why bother copying and pasting though, that's my whole point. Google is terrible in comparison, just let it write the code for you.
  • noduerme 2y ago
    I think the nomenclature of calling generative AI an "AI" is just hype and that leads to disappointment. After all, it's much more exciting for investors than calling it the world's most expensive textual/audio/visual autocomplete. But that's really what this generatoon currently is. And it can feel like magic, because we're pattern-seeking and pattern-matching creatures, so something that seems to intuit the pattern we're looking for (even if imperfectly) can feel quite a bit like reasoning with another human. To be a bit less generous to humans, it's often the same as talking with another human, because we spend much of our lives writing boilerplate or making small talk or otherwise just kind of on autopilot and answering patterns with expected responses without seriously engaging our complex reasoning. (Do you have to really think about how to map-filter-reduce a dataset anymore? Or think seriously about how you really are when someone says "how are you"? We don't usually expect other people to, and most of the time we're satisfied by exchanging recognizable / somewhat coherent language patterns which we can count on each other to fill in the gaps of. But this is not the intelligence in human intelligence). The corporate hype machine works overtime to hype it as a solution, but that's just what they do. That's all spin. We see through it now, and we're getting to the point where people are asking the only important question about this tech, which is, how much will you pay for a nerfed autocomplete to do X?
    • tylerchilds 2y ago
      “how much will you pay for a needed autocomplete to do X?” i think this is a super important for anyone playing in the LLM space to calculate. currently “AI” providers are selling electricity at a loss to demonstrate product “value”, so even while asking questions is “free” today, there’s actually a finite resource under the hood that needs to have the bill paid in the end. an estimate, but from historic trends, free bills come due in around 7 years
      • eru 2y ago
        Even _if_ we don't see our current 'AIs' become more intelligent, I am very certain that we will see the amount of electricity needed to produce 2024-state-of-the-art results drop dramatically over the next few years. People are only just figuring out how any of this works, and they are happy to get any reasonable results at all. They aren't really competing on costs like energy efficiency, yet. --- However I don't really think that 'if' will come to pass: I expect that we will still see lots of advances in the 'intelligence' of these models, so people will compete on these, and worry about electricity usage afterwards.
  • echelon 2y ago
    > shitty clip art AI art is better than most humans, including most artists. You can take the outputs and immediately turn them into animation. Suddenly I, as a single individual, can easily make film content without figuring out set and lighting logistics or roping in dozens of people. This stuff is magic.
    • ghaff 2y ago
      I was somewhat with you through the first sentence then it got a bit breathless. As someone with some graphic design experience but who isn’t very good and can’t really draw, I used Photoshop and it’s genAI capability recently to put together a book cover for an ebook. Would it win any awards? I’m pretty sure not but it’s more than adequate for its purpose and is almost certainly better than anything I could have come up with even if I used some CC artwork. ADDED: I also had something of a vision for what I was looking for and (I think) enough of an eye to fiddle with things and get to the point where I went "This isn't half bad."
      • echelon 2y ago
        Exactly! Editing and controllability are getting better almost every week.
    • YurgenJurgensen 2y ago
      I've said it before and I'll say it again: The fact that you didn't think to include an example is telling. If you think AI 'art' is good, show me, and tell me why you think it's good.
      • zo1 2y ago
        Not OP. Conflating "art" (the "snobbish" definition if you will) with realistic depictions of imagined worlds, stories and settings is not helpful here. There are so many talented, imaginative people out there that have previously had Zero ability to manifest their thoughts into a visual form. AI has enabled that on so many levels and we should be grateful for it.
  • 015a 2y ago
    Yeah; its concerning how depressingly similar Sam's recent language is to, say, SBF's in his prime. Waxing poetically with faux-intellectualism about the sheer addressable market his business will capture (all of humanity) (and funded by the government) [1]. You can just listen to key players in the space, how their language has changed over the past year, and recognize that we're probably very near a local maxima / plateau / AI winter. Reality is: AI is startlingly expensive. Stupid, stupid expensive. Microsoft is building "Stargate", their $100B AI-focused supercomputer [2]. The money to build that isn't coming from selling AI services. Its coming from their old, boring, tried and true businesses; Windows, Office, Azure, and M365. Xbox is dying before our eyes. Their tried and true businesses make great money but aren't growing. Its up to AI to prove to investors that their P/E multiple is justified. [1] https://twitter.com/tsarnick/status/1789107043825262706 https://twitter.com/tsarnick/status/1789107043825262706 [2] https://www.tomsguide.com/ai/meet-stargate-the-dollar100-billion-ai-supercomputer-being-built-by-microsoft-and-openai https://www.tomsguide.com/ai/meet-stargate-the-dollar100-bil...
    • viking123 2y ago
      I always mix SBF and Altman in my mind, and have to spend few seconds thinking which was which.. But IMO the current AI hype has some almost religious elements to it
  • dogcomplex 2y ago
    It is honestly shocking how different people can be, even within programming. Been finding LLMs very useful for pair programming and low-effort spin-ups of projects I would just never do without the tools. Frankly, I'm waiting any day for an architecture to arise that takes lossy LLMs and has them error correct enough to produce 99.9999% reliable progress on simple tasks cheaply. This feels incredibly close, and if the unit of intelligence of each step of those is around GPT4 level? Jesus christ. Programming massively automated overnight. Any programmer that is this dismissive of these tools frankly... hasn't been using them right, and has no imagination - or a massive ego. This stuff is still far too early to make such judgements.
  • dartos 2y ago
    I don’t really think AI will get there the way we’re going about it. Statistical models will always be limited by their data and how edge case filled reality is.
  • noobermin 2y ago
    People are either misusing this meme or it makes no sense.
    • CaptainFever 2y ago
      It's likely an ironic usage.
  • mondrian 2y ago
    It's definitely useful, but in higher level chat use cases, not so much high-precision generation (yet). Rubber ducking, brainstorming, and search-like use cases are definitely a level above what we had.
  • n_ary 2y ago
    > It's a pretty clear depiction of the investor class being so detached from the technical reality I think, their ROI model is much different and subtle. Few decades back, everything ran on-premises and costed ok ... Then a huge investment had been shovelled into fancy frameworks and pushing strange paradigms (microservices, distributed systems, monoliths whatever flavour works for the tech giants) so now people just put together whatever and needs massive _cloud_ to run basic things that used to run on few MHz devices. In the end, the ROI is NOT from these models, those are just gambles in case one actually becomes a king maker. The actual ROI is from the _cloud_ where innocent people will rent expensive hardware to try and utilise these models and funneling wealth to the actual investments(cloud operators, hardware vendors) which are already part of portfolio of these investors. Basically, $1MM invested in random shiny startup will inspire other shiny startups to also spend similar in race to become unicorn, all the while the cloud operators and GPU vendors are laughing to the bank preparing their shareholders(usually same investors) dividends.
• mostertoaster 2y ago
  I don’t know if Copilot has made me more productive, I think I get the same stuff done, but I think it has made getting the stuff done more enjoyable. At times with better quality at times with worse probably.
  • mostertoaster 2y ago
    This made me think: the thing Microsoft has been good at for a very long time is developer tools. Think about how much better visual studio is was than Xcode. If Microsoft focuses copilot on simply making things more enjoyable rather than do the work for you, I think it will be an amazing thing.
• babyshake 2y ago
  Writing fairly standard typescript APIs and React code, Copilot is immensely helpful and almost always correct. It definitely hits a wall when I am working on something sufficiently esoteric. As does ChatGPT, etc.
  • anonzzzies 2y ago
    It is incredible that you can replace 90% of upwork/fiverr/juniors(and many seniors for that matter) with a $20/mo tool for doing the most common work like api integrations and frontend work (both of which I find the most boring work outside the non programming like devops and, worse, fighting tools and versions because of useless updates). > It definitely hits a wall when I am working on something sufficiently esoteric. So do humans. Hiring humans to do that work well is hard and expensive, so it’s not strange that is harder for LLMs as well. I am not sure that finding things LLMs are shite at makes them less productive for most things programmers are doing during the day, which is quite boring stuff. In my reality, which, before mid last year, had 100s of people (across clients I work with, not my company) doing integrations and frontend work, now fired most of these and focused more on optimising the pipeline for copilot, chatgpt and opus. My friend who runs an outsourcing operation in India almost went out of business mid last year when he saw his EU clients cancelling all his contracts because his people could be replaced by openai; he pivoted by having his people use ai, do ai projects and becoming more (everyone already was that but it was more heavy on the coding) business analyst heavy. He fired no one and is doing better than before now, but he doesn’t hire ‘programmers’ anymore.
    • azinman2 2y ago
      I’d love to know the specifics on the ppl hiring the consultants and the work output. In my experience you need someone with experience to chain together any genAI output, as well as validate that it’s even correct. It’s not like it magically does everything for you.
      • anonzzzies 2y ago
        Correct, but that was also the point; all these ‘consultants’ are senior programmers. Nothing is magically done; they use the ai to do the work they were mostly doing before with more people/themselves without ai; now they just focus more on business side. Higher pay and harder to replace (for now). My company does the same but in a niche; our specific tooling is easier to automate (which we did/are still doing) and we notice it is getting easier with the llms getting better. A year ago we still needed a human to evaluate even the basic scope of work, now llama3 + some tooling we built can automatically get us a locally generated report.
• anonzzzies 2y ago
  https://news.ycombinator.com/item?id=40293461 https://news.ycombinator.com/item?id=40293461
• jdlyga 2y ago
  From my experience, Github Copilot is useful for its seamless autocompletes. But Github Copilot chat, a new feature, noticeably lags behind GPT-4. I get far better performance copying and pasting code into ChatGPT (GPT-4) than using Github Copilot. Both definitely help productivity, especially when generating code using an unfamiliar framework or library. It's great for generating skeleton code for small programs using new libraries, but it in its current form can only get you around 80% of the way there. You will spend hours fighting with the prompt to try and get the remaining 20% complete when it's much easier to just finish it yourself.
  • simonw 2y ago
    Weird, because Copilot chat /is/ GPT-4 as far as I know. Or maybe they run it on 3.5?
    • heethjain21 2y ago
      It runs both. Had seen this on the discussions on Github. https://github.com/community/community/discussions/58059#discussioncomment-7900899 https://github.com/community/community/discussions/58059#dis...
      • simonw 2y ago
        That original note about it using both 4 and 3.5 was from this comment on November 2nd: https://github.com/microsoft/vscode-copilot-release/issues/6#issuecomment-1791319670 https://github.com/microsoft/vscode-copilot-release/issues/6... GPT-4 Turbo - the much cheaper version of 4 - was announced on November 6th. So there's a good change they've switched to that at some point in the last 6 months.
    • anonzzzies 2y ago
      They mix it to save on compute.
  • fendy3002 2y ago
    And for those who say that it doesn't speed up development, then maybe they're right, because the main benefit isn't at speed. It reduce the devs mental / cognitive load and pass it to the ai instead, we just need to check and review the result. If it doesn't monumentally increase the speed, it helps devs to stay productive longer
• Barrin92 2y ago
  I used it for a few months and I think it made me less productive to be honest. I developed this strange "autocomplete disease" where I'd wait for autocomplete when I could've literally typed stuff out myself, it's like some weird tool dependent mental stutter. And given that I've worked in the languages I write for a living, mostly C and Python for years it's not like it ever suggested anything I wouldn't have known how to write and especially in C it can produce some catastrophically shoddy code. The one case where it's kind of nice is if you do a side project with a library that doesn't have good documentation and you don't know the API and it spits out some nice example, but honestly I don't consider that worth paying for.
• KTibow 2y ago
  Shout out to Codeium, it's a free and fast code completion extension, the business model is that it's supported by corporations who pay to selfhost it
• matrix87 2y ago
  I don't really get the value proposition, for every second you spend writing something you spend another 10 verifying that it was done properly. Speeding up the process of writing is sort of a waste because of Amdahl's law How does copilot et al. speed up verification and review? For example, a C program is harder to verify than a Go program doing the same thing. Suppose you use it to verify something in an enterprise context. Are you confident in the result? Personally I'm too paranoid for that and I think more people should be
  • fzeroracer 2y ago
    This is where I stand. I think people fall into the trap of believing that it is reliable and useful because it can occasionally spit out boilerplate code that matches what boilerplate code is supposed to look like. Which is a very dangerous place to be in, because the more you assume that the code generated is reliable the more likely hallucinations are to sneak into your codebase. And now you don't have a good mental model of what your code is doing because you didn't actually write any of it. I have a feeling that codebases using and relying on copilot are, essentially, pushing disasters later down the line when bugs pop up and no one has the ability to actually debug the system.
  • mucle6 2y ago
    I am so confused. When I write code, I think "Okay I want to write a for loop that does xyz." How is it hard to know if copilot did or didn't do what you wanted?
    • serpix 2y ago
      This is how I've used it: "I want to write a function that reduces a map of customer data to a list of their phone numbers from their primary addresses only or contact address if there is no primary address" and then you look at the resulting flatmap, filter, reduce blob of AI generated code and figure out if what it does is correct for about a minute.
      • alecthomas 2y ago
• toddmorey 2y ago
  I don’t know if this is possible to reconfigure or turn off, but I HATE copilot guessing at file paths & file names (as I write an import for example). Usually I find the file reference broken because copilot hallucinated 3 directories. Autocomplete for files & paths worked great before copilot got involved.
• netule 2y ago
  I’ve found that early on in a brand new project, its output is very useful, but as the project grows more complex and interdependent, the suggestions become less so.
• ilrwbwrkhv 2y ago
  Even auto complete and language servers are bad for you. It causes api spread. No lsp and no copilot makes you write more maintainable, simpler code and you can "keep the program in your head."
  • thiht 2y ago
    Hot take
• muglug 2y ago
  GitHub touts a 55% improvement in coding speed based on a study conducted in-house that tested the participants’ ability to paste a pre-written prompt and then check the output: https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ https://github.blog/2022-09-07-research-quantifying-github-c... The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about coding velocity: https://arxiv.org/abs/2305.12050 https://arxiv.org/abs/2305.12050. In my own experience the time taken to review machine-generated suggestions often eats into developer time. This is not a critique of LLMs in general — I’ve found ChatGPT really great for kicking off greenfield projects in well-known languages and frameworks.
  • 12907835202 2y ago
    This last sentence is the most important. You can describe your dB schema in words and get laravel migration, models, controllers and if you wanted policies, form requests etc all near perfect. You can get a V1 in 10-20 minutes then go about handling all the actual logic
    • rm_-rf_slash 2y ago
      Yeah the copilot doesn’t need to be integrated into every keystroke, just able to analyze the context and kickstart the code. Getting up to speed with new libraries is so much easier with AI instead of the bad old days of trial-and-error-and-marked-as-duplicate
      • gtirloni 2y ago
        Can Copilot analyze my whole codebase these days or just the file I've opened? Honest question as I've stopped to use it a long time ago.
        • selcuka 2y ago
          It seems to be hit and miss, at least with the current PyCharm integration. In some cases it can infer information using other files, in some cases it can't (even if they are open in other tabs).
• 3523582908 2y ago
  > This is because, Artificial Intelligence is very much unlike Human Intelligence. What an interesting, nuanced take. Thanks for sharing, you've given me a lot to think about.
• b33j0r 2y ago
  I disable it when I actually need to recall what I know. It’s like a phone number. If you never dial your mom anymore, you will forget it.
  • eesmith 2y ago
    Or like GPS? While GPS gets me where I want to go, I find that using a map or no map at all gives me a better sense of place, so I need less time to get to the point where I don't need a GPS to get around.
    • b33j0r 2y ago
      100%. Retain the things that you need, don’t memorize wikipedia. See your landscape.
      • skydhash 2y ago
        100% to both. I have a hoard of PDFs. I use Dash as an offline doc browser. I save interesting links in a bookmark app and I still can find libraries documentation and source codes on web search (duckduckgo). So I just freshen up as I go (or on my free time). Memorize what I need/like and capture what I found interesting. I was doing Advent of Code in Common Lisp and the loop keyword page was always open (It's a mini language on its own). Later challenges became easier, as I learned new ways to iteratively express the solution. I could be faster with generated code, but having a bird view of the map will lead you to more interesting routes (more often speed is not the issue, correctness and maintainability is)
• jpollock 2y ago
  I enjoy working with interns, you can see them learn and they are always making new mistakes. I get a return on the effort of training them. They might even convert into full time employees and take some of my workload. I don't get that feeling from the LLMs. They have about the same skill level of an intern, but they don't _learn_. I can't offload any work to them, and they take the same level of effort to manage. I'm not in this job to manage interns. I'm in this job to solve problems. Training the intern is a payment by current me for future me. This isn't like going from books to Internet search or StackOverflow. It doesn't provide an immediate benefit to me, and I don't benefit in the future from my contributions. I'm not seeing the share-alike vibe necessary for scale. I want tools that make me faster and more efficient. That's how I keep increasing my wage. Maybe if I could _see_ the AI learn from my training? Maybe if I saw benefit from the effort? However, right now, I'm paying for the benefit of training the LLM.
  • n4r9 2y ago
    I have a similar take. As yet I'm unconvinced by claims that using an LLM and correcting the output is faster than just writing the code. What happened to "reading code is harder than writing it"? With an intern or very junior developer you have hope that in a few months/years time they'll be reducing rather than adding to your workload. With an LLM, either you're using it for really rote or boilerplate tasks (fair enough) or you spend so much time looking for subtle errors that it's not worth it. Plus, much less fun than actually crafting something yourself.
  • anguspmitchell 2y ago
    I think the intern analogy is right. The problem is that there’s always a communication cost to outsourcing something. And you can’t outsource a whole project to an LLM, like a human intern. You’re just outsourcing one micro-, sub-task after another. To use the human analogy, it’s more like you’re standing over their shoulder and telling them what function to write, one after the other. And they’re SUPER fast with small, pure functions, but they get confused with anything else. Is that a faster way to program? Maybe? If you can fluidly decompose things into small functions in your head and the problem can be solved that way? I don’t know though, I find myself using chatGPT for bigger meta questions much more frequently than the Copilot autocomplete
• amanzi 2y ago
  I feel like this bit is important for this discussion: "...I do not use Copilot for my day job. I use it for my own projects only..." My guess is the people that don't use Copilot because they don't trust it, or don't like the code it produces, are probably trying to use it in a professional scenario where they have strict coding standards. For hobbyist use or side-projects, Copilot is an incredible time saver. I use it with my Python side-projects, and it's truly amazing how much time it saves me. Some of the most common use-cases where it saves me time: - Adding docstrings. Just type `"""` after a method name, pause for a second, and Copilot will generate a pretty decent docstring. And usually it will mimic the style you've used to write other docstrings in the file. - Writing tests. This one can be a bit hit-and-miss, but I mostly get good a good starting point, and in the best case it will suggest test scenarios that I might not have thought about. - Creating basic functions/methods. I have terrible memory and usually need to refer to the docs for even fundamental Python code, e.g. reading or writing CSV files, looping over and modifying dictionaries, etc... But with Copilot, I get great results, often just by typing `def open_csv_file` and then pausing for a second to get the method code. If that doesn't work, adding a comment with what you want the function to do before writing the function name will help a lot. - Adding type hints to existing code. Just type ":" and pause for a second, and Copilot will almost always provide the correct type hint. And pressing a space after the closing parentheses in a function, will almost always return the correct return type hint.
  • leptons 2y ago
    I only use Copilot for my day job, because they paid for it. When I'm coding my personal projects on nights and weekends I find myself missing Copilot and I'm contemplating paying for it myself. I don't really trust it for everything, but sometimes it surprises me with how much it gets right. I just can't justify paying $19/month personally for the "Business" plan so they don't use my proprietary code to train their AI.
    • eru 2y ago
      > I just can't justify paying $19/month personally [...] Why not? Is it a significant fraction of your income? I find that the cost of hardware deprecation, electricity and internet access probably amounts to a lot more than that per month for me already. (Not to mention the opportunity cost of my time.) It's fine to be frugal, but it's also useful to compare your actual costs. (Though that's not always appreciated. I got into some minor trouble at an employer when I pointed out that the 15 GiB of storage space they 'graciously' gave us should cost them less than the company toilet paper we use every day. And I said 'should' because that's internal storage space, and I used numbers from AWS. But if they are paying significantly more for storage than what AWS charges (with enormous margins for Amazon), we also have a big problem.)
• maytc 2y ago
  Github Copilot definitely improved my coding productivity. However, most of my time is spent thinking about the problem to be solved than the actual coding.
• Seb-C 2y ago
  I still haven't found any interest in this wave of AIs for coding, mostly because writing code hasn't been a bottleneck for me since I learned vim and started using it as my only tool. Knowing what to write and how to organize the code is the really difficult part of our job, and I am unable to do this correctly when reviewing stuff written by others (or by an AI). I have been wondering if people are actually using those to save time writing boilerplate code, in which case it would make sense. Vim has greatly increased my productivity by allowing me to save time for all repetitive tasks, and it's consistent and reliable (but also super hard to learn). Maybe it's just a completely different approach in resolving the same problems?
  • jarsin 2y ago
    I don't understand the boilerplate argument because every major framework has had boilerplate generators for over 10+ years.
  • skydhash 2y ago
    Same. Most of my time is spent thinking about the domain space, constraints and processing flows. Refreshing my memory about a method, a class or a specific how to is fast based on a combination of Dash (offline docs), PDF Manuals, and opening the libraries pages in advance. And with Vim, I can copy-paste code while having complete control over the code. And thanks to the buffer model, I can have everything I'm thinking about in front of me instead of one single file (VSCode Tab model). Boilerplate has never been a bottleneck. And I like to have an idea of everything I have to manipulate, variables and property, functions and classes, modules and files. If something has not been written by me, I have some good reasons to trust it, either because I reviewed its source code or the library reputation (I'd probably read the source code anyway because some documentations are not good). IDEs have been able to provide the kind of completion that I like (auto-importation, symbols) and information I want (function signature and class properties). I'd rather take a moment to read a library's documentation and code examples than see ChatGPT generates something that has a non-zero chance of being an hallucination.
    • ukuina 2y ago
• Sytten 2y ago
  I mainly use it as a smart rust auto-complete and I would say it saves me probably 30% of coding time, everything boilerplate/tests it gets super well. I am a slowish typer though. It also makes it more enjoyable, just my experience.
• neilv 2y ago
  > If you put your cursor at the position indicated by ????, you can pretty reliably expect Copilot to write the rest of the code for you. Does the author mean that Copilot filled in the other `switch` cases? (Because that looks like "Here, let me plagiarize that for you. We'll call it 'AI' and 'boilerplate'.")
• black_13 2y ago
  [dead]
• marcus_holmes 2y ago
  Years ago, over a decade ago now, I was a .Net developer. Microsoft introduced Entity Framework, their new way of handling data in .Net applications. Promises made, promises believed, we all used it. I was especially glad of Lazy Loading, where I didn't have to load data from the database into my memory structures; the system would do that automatically. I could write my code as if all my memory structures were populated and not worry about it. Except, it didn't work consistently. Every now and again a memory structure would not be populated, for no apparent reason. Digging deep into technet, I found a small note saying "if this happens, then you can check whether the data has been loaded by checking the value of this flag and manually loading it if necessary" [0]. So, in other words, I have to manually load all my data because I can't trust EF to do it for me. [1] Long analogy short, this is where I think AI for coding is now. It gets things wrong enough that I have to manually check everything it does and correct it, to the point where I might as well just do it myself in the first place. This might not always be the case, but that's where I feel it is right now. [0] Entity Framework has moved on a lot since then, and apparently now can be trusted to lazily load data. I don't know because... [1] I spat the dummy, replaced Windows with Linux, and started learning Go. Which does exactly what it says it does, with no magic. Exactly what I needed, and I still love Go for this.
  • nercury 2y ago
    Pardon me for the tangent (just a general comment not directed to OP). What I have learned over the years is that the only way to properly use ORM is as a fancy query tool. Build the query, fetch/update data, MOVE THE DATA to separate business objects. Don't leave ORM entities shared across the sea of objects! Phew, thanks, I got that off my chest.
    • rafaelmn 2y ago
      That doesn't really help you with EF because there's plenty of stuff shared at context level. So depending on the order of queries in the context the same query can return different data. I hate EF and everything it stands for. :)
      • 2y ago
• thiht 2y ago
  I’ve configured Copilot to generate suggestions only when I explicitly use a keyboard shortcut and it’s been a game changer for me. I’ve developed a good intuition of what it’s good at and now I much prefer invoking it manually. When it automatically suggests garbage on every keypress, it gets frustrating pretty fast.
• esskay 2y ago
  Anecdotally I can say it's improved my productivity. Mainly because it's just a slightly better autocomplete than PHPStorm/Jetbrains IDE provides. Although they are now pushing their own AI tools for an extra fee pretty heavily but unlikely to pay for it given copilot pricing is pretty competitive. There are occasions where I'm just not in the mood to work something complex out so sticking a temporary comment in telling it what to do can sometimes yield a good enough result, all be it with some occasional minor tweaking. Where it's really shined is with repetitive tasks. It's nothing close to perfect but it's certainly a welcomed little helper even if it does get things wrong sometimes. In terms of the original question, I think its the cognative mental load thats being helped here more than anything, not having to waste brain power on the mundane bits is the big selling point for me.
• piperly 2y ago
  This part of the final sentence encapsulates everything: "...a lot of time is wasted in the back-and-forth."
• arrowsmith 2y ago
  > For reasons you can probably guess, I do not use Copilot for my day job. Maybe I'm dumb but these reasons aren't obvious to me. Why doesn't the author use copilot at his day job?
  • Tomte 2y ago
    Probably because the employer doesn't allow sending proprietary source code to random internet services on an employee's whim.
    • storafrid 2y ago
      Silly. Many big corporations trust Microsoft touching their code. There are enterprise policy controls for Copilot, which includes data privacy. If you want to make this argument, then the realistic reason would be that Copilot hasn't yet completed the SOC 2 compliance process. That would be a valid reason to wait, for corporations that are listed on Nasdaq or work with very sensitive data. But that's far off from the comment I'm replying to.
      • Tomte 2y ago
        At my employer this would be a firable offense. And we‘re under German worker protection laws.
• ogou 2y ago
  It is surprisingly helpful on old legacy projects with multiple contributors. It will imitate their code style and variable naming, even if outdated or poorly constructed. Saves time looking through spaghetti code to figure out why some function wasn't abstracted or named in a consistent way.
• klauserc 2y ago
  I find GitHub Copilot close to useless for production code. The worst, most obscure bugs I've had to debug in the last year were all in Copilot-written code. It _looks_ plausible, but it makes extremely subtle mistakes. Occasionally, you have repetitive sections of code where it can copy&adapt lines from the context, but that's about it. It's a different story for test code. Test code is often formulaic and "standardized" (given/when/then). For instance, I find myself writing the first test case and Copilot can come up with additional test cases. Or I might write the method name ( FeatureUnderTest_Scenario_ExpectedOutcome) and Copilot provides the implementation. I have not found any value in Copilot chat.
  • yuppiepuppie 2y ago
    Assuming you work on a team with pull requests and code review, how much do you also put blame on that process?
  • shaky-carrousel 2y ago
    Copilot for me is very useful with all the scaffold boring code. It sometimes helps with problems, but I have to guide it, and be very precise with my request.
    • zahrc 2y ago
      And even then it happens to ignore context or queries from start or halfway through. I'd rather spend the time coding then trying to bruteforce it to give me the answer I need.
  • devjab 2y ago
    I think automated tests are the one area that LLMs will truly improve productivity (and overall code quality). It’ll likely also lead to a lot of tests that actually tests nothing, but as a whole, it’ll hopefully be capable of both generating and updating tests if you give it some good inputs to do it on. Documentation is another area where I have high hopes. In the ideal world people update it as they change things. In reality, however, well… Then there is the design side of things. I really feel bad for designers of Icons now that you can get some really good one really fast by tasking one of the image generating AIs. I’m not sure LLMs will ever really be capable helpers as far as programming goes. Well I guess it’s two part, they can help with trivial tasks, but they can’t help with anything related to the actual work of generating business value with code. It’s two-sided of course. They certainly allow a lot of people write functioning, though really shitty, code. Which is a huge benefit for a lot of programming tasks where it doesn’t really matter that it’s inefficient and well terrible. We’ve already seen our more digitally inclined employees make great things with power apps, most of which are eventually replaced by more robust software as they scale. But we also see small Python programs helping out with tiny personal tasks around our offices, and while IT operations aren’t too happy it’s generating a lot of individual value that wasn’t there before.
• Havoc 2y ago
  Phind seems like a much better deal. Also does some code completion but is useful as a daily use chatbot too
• deergomoo 2y ago
  My job subscribed to Copilot but I ended up disabling it after a few days, I just found it enormously distracting and providing little value. From what I saw the predictions were very accurate, but constantly having something pop up as I type just kept nudging me out of my concentration state. Oddly I don’t feel the same about typical IDE autocomplete, so I’m not sure where the cutoff is there. For me it felt like having someone trying to finish every sentence I’m speaking: even if they’re right 100% of the time, it’s still very distracting and more than a little irritating.
  • cmehdy 2y ago
    In a way that seems a bit like delayed auditory feedback: https://en.m.wikipedia.org/wiki/Delayed_auditory_feedback https://en.m.wikipedia.org/wiki/Delayed_auditory_feedback And I wonder if different types of people react to it as differently as the stutter/non-stutter dichotomy.
  • cmehdy 2y ago
    In a way that seems a bit like delayed auditory feedback: https://en.m.wikipedia.org/wiki/Delayed_auditory_feedback https://en.m.wikipedia.org/wiki/Delayed_auditory_feedback And I wonder if different types of people react to it as differently as the stutter/non-stutter dichotomy.
• p0w3n3d 2y ago
  For my 2 month experience copilot is showing sometimes very good ideas, but usually it suggests rubbish. Copilot chat is trying to analyse code and answer but it also needs to be controlled. Overall feeling is to have a very fast and intelligent but rather foolish Junior Developer by your side, who reads a lot of documentation, but needs to be fully controlled
• chx 2y ago
  Every time... https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.
  • wzdd 2y ago
    This is exactly how I use it. It's interesting to see what the "expected" thing would be. Even if I don't use the generated code, the time spent reviewing it is not lost. I might learn about a new library, or convention, or way of expressing something syntactically which I wouldn't have thought of. Even when it gets things wrong, it can get things wrong in a way which lets me know that I've made an incorrect assumption elsewhere.
• mellosouls 2y ago
  I really wish writers of and commenters on these articles (especially those less impressed) would clarify the version they have used and whether they have experimented significantly with state of the art. Overwhelmingly in my experience of reading these back-and-forths, they are referring to 3.5 at best (as in this article). Yes, of course I accept there are users of the paid version (4+) and class competitors like Opus that don't find they help their productivity but unfortunately any comments referring to significantly less capable engines than those - without at least clarifying that - risk just adding noise to the conversation.
• deleted 2y ago
  [deleted]
• anshumankmr 2y ago
  Though I have never used CoPIlot myself, I have used Chat GPT for writing emails, adding doc strings, helping with basic boilerplate part like reading a file, generating JSON in a certain structure that the business needs, I can say it has helped but I still don't really trust these models.
  • spaceship__sun 2y ago
    Average hackernews user.
    • anshumankmr 2y ago
      here's the kicker: I work on creating solutions using Gen AI, mostly some something RAG, Chat with Databases and some other actually cool stuff
• jraph 2y ago
  > Without it, I find myself getting grumpy a lot more often when I need to write boilerplate code - "Ugh, Copilot would have done it for me!", and now I have to type it all out myself It seems to me a good IDE generates most of the boilerplate code I would need to write. For Java I use the community edition of IDEA, that's open source, it runs entirely locally and there's no lag. No open source code gets eaten and no forest get burnt. I spend a vanishingly small time writing boilerplate code. Or so I think. Am I just not noticing it, or is it rare and I work with high quality code?
• hippich 2y ago
  Copilot certainly increased my productivity. At least in a very specific context - C programming and working on some small subset of llvm internals. I am not very experienced C or C++ programmer, nor I know anything about LLVM API. Plus - copilot is very good at autocompletions. Yes, the code it generates come out often incorrect, but even my low level expertise allows me to spot issues. And even if the code is incorrect - it gives me clues where to dig. Even such a small detail as autocompleting debug printfs is a really big time-saver for me. I also think that co-pilot vs code integration extension is big part of it. I don't think I would be as productive with chat UI alone. I wonder if there is alternative I can use with local hosted llm that can give me full block autocompletions similar to co-pilot extension. Everything I've seen so far was at best line at a time autocompletion.
  • bogwog 2y ago
    > At least in a very specific context - C programming and working on some small subset of llvm internals. I am not very experienced C or C++ programmer, nor I know anything about LLVM API. I'm not sure what you're working on, but as someone dependent on LLVM-based tools, reading this is kind of terrifying. I'm sure your intentions are good, but I hope you're at least making it clear that you're heavily reliant on the output of Copilot when making contributions to something. C and C++ in particular are not the kind of language where someone inexperienced can be trusted to review. > Yes, the code it generates come out often incorrect, but even my low level expertise allows me to spot issues. Can you give an example of some issues? C and C++ are compiled, so there will be obvious compilation errors, but there are also a lot of "foot guns" that are not obvious and will bite you at runtime.
    • hippich 2y ago
      There is a very small impact area, and yes, folks who use it, aware of co-pilot being involved :)) The thing is - I don't think I would dare to do that I've done without co-pilot in the first place. So it is a win anyway
• doubloon 2y ago
  The answer is yes for most people. You can type in pseudocode for simple things and out pops something that basically works for a significant number of languages. Its absolutely terrible for really complex problems but most code is not solving complex problems.
  • TillE 2y ago
    > but most code is not solving complex problems I see this claim a lot, and I have no idea whether or not it's true. I just struggle to imagine that there are millions of people employed, well compensated for doing pretty easy jobs. In my professional work and my various unrelated personal projects and open source contributions, if you want to do anything useful or interesting, you get into the weeds very very quickly. The territory where LLMs are nearly useless. I'd say maybe 5% of my time is spent writing "easy" code without major complexities.
    • doubloon 2y ago
      Its opaque because its operations. Where vba and excel come in
• bogwog 2y ago
  I've felt pretty productive using ChatGPT to generate snippets for either an API/SDK/language I'm not too familiar with, or moderately simple things that I'm too lazy to do myself. Sometimes that can be a huge time saver especially if the documentation is poorly written, other times it might actually be slower as I wrestle with getting a good prompt (but with lower cognitive load than were I to do it myself). I still don't do it frequently enough to say it has changed the way I work though. I'm still not sure if this is a transformative way to code or not. I think in the future it might be able to do to programming languages what programming languages did to assembly... but it probably won't look like a chat UI or an autocomplete plugin.
• bdcravens 2y ago
  I don't use it to build a lot of "features" but it really is great for eliminating a lot of tedium: stuff like boilerplate, building up a hash using values already in my code, etc. That's the stuff that is soul-draining. A couple of days ago I was building up a Dockerfile for a new Rails project and I literally just let Copilot build it for me instead of look it up. It was probably 95% correct.
• mring33621 2y ago
  I am a 20+ year polyglot developer. I just cancelled my Github Copilot sub. Sometimes, it was magical, but more often it was mediocre and just got in the way. I generally have better luck with a back-and-forth chat with an LLM on the web and then cutting/pasting the snippets into my IDE. I can do those for free. Will try again in a year, probably.
  • sneak 2y ago
    I find ChatGPT4 to be substantially better than Copilot at writing Go code. I wish it easily integrated into an editor and I didn’t have to copy paste so much.
  • causal 2y ago
    +1 on being more productive with chat than Copilot. I find copilot mostly to be a useful Google replacement, e.g. if I forget the syntax for a for loop in JavaScript I can just write a comment like `// For loop over X` and it will spit out the boilerplate I need without breaking my flow. In either case it's really important not to get stuck "trying to make it do what I want". If an LLM can't solve your problem in 3 tries, you're probably asking too much of it.
• 999900000999 2y ago
  Chat GPT and friends are best when you tell it to write code from scratch, even then it's only good for straight forward tasks. Give it existing code to work with, particularly a larger code base and things get weird. Still, I love it for hacking out small prototypes. If you don't really care about performance or best practice, it's ok.
  • collyw 2y ago
    ChatGPT4 It won't accept moderately large files that I have given it. It was actually just an HTML file, I wanted to ask about css in it. How have you managed to give it a "larger codebase"?
    • 999900000999 2y ago
      By just giving it code files one by one. By larger I'm talking about maybe 5 or 6 files. Or to be honest Chat GPT isn't good when your fixing code that already exists.
• guluarte 2y ago
  I found copilot and other AI slowed my productivity, distracting me with irrelevant suggestions.
• banish-m4 2y ago
  It's good for creating similar boilerplate but otherwise generates crap that will waste your time.
• SOF3 2y ago
  Not targeting anyone, but after clicking into the link, I (1) noticed it is mdbook, (2) looked at the author GitHub and noticed C and Rust, and (3) predicted that Copilot is gonna be unpredictable with random serious unnoticeable mistakes. Looks like I got it right.
• TheRealDunkirk 2y ago
  It increased mine yesterday, by quite a bit. I was writing some JSON-to-array-of-arrays conversion code, and when I started to type it out -- dreading the inevitable off-by-one hunt -- CoPilot spat out the code, and it was correct! It helped that I've done this many times, and knew it was correct, but it saved me needing to think about it, or track down any bugs. This conversion code was in service to using Plotly (the JS graphing library) for the first time. I was trying to get the axes of my surface plot to be labelled with the correct values (instead of just their index), and spent probably a half hour looking at the demos and the docs, and not finding what I needed. I asked CoPilot in the VSCode chat window about it, and it gave me exactly what I hadn't been able to find. As soon as I saw it, I understood the natural extension of the API call I was seeing in the demo, but I still hadn't found a second data point to even start extrapolating, if you take my meaning. Anyway, I'm paying the $10/mo out of my own pocket, and yesterday -- at least to me -- it paid for itself for the month in just those two examples. I find it delightfully surprising. I just need to find a way to make it stop suggesting comments. I can type my own comments, thank you very much.
• shamano 2y ago
  I guess it does when you use it parsimoniously. Copilot helps me a lot when I have to write repetitive code, or when I have to change code slightly the same way in a lot of lines. I never add large amounts of code at one go though, I think it would certainly add subtle bugs then.