7 ms·

Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, eit

by diggan 1y ago

Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative. > This seems like it's fixing the symptom rather than the underlying issue? This is also my experience when you haven't setup a proper system prompt to address this for everything an LLM does. Funniest PRs are the ones that "resolves" test failures by removing/commenting out the test cases, or change the assertions. Googles and Microsofts models seems more likely to do this than OpenAIs and Anthropics models, I wonder if there is some difference in their internal processes that are leaking through here? The same PR as the quote above continues with 3 more messages before the human seemingly gives up: > please take a look > Your new tests aren't being run because the new file wasn't added to the csproj > Your added tests are failing. I can't imagine how the people who have to deal with this are feeling. It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. Another PR: https://github.com/dotnet/runtime/pull/115732/files https://github.com/dotnet/runtime/pull/115732/files How are people reviewing that? 90% of the page height is taken up by "Check failure", can hardly see the code/diff at all. And as a cherry on top, the unit test has a comment that say "Test expressions mentioned in the issue". This whole thing would be fucking hilarious if I didn't feel so bad for the humans who are on the other side of this.

worldsayshi 1y ago
> How are people reviewing that? I agree that not auto-collapsing repeated annotations is an annoying bug in the github interface. But just pointing out that annotations can be hidden in the ... menu to the right (which I just learned).
- codyvoda 1y ago
  or press “a”
- jon-wood 1y ago
  I'm not entirely sure why they're running linters on every available platform to begin with, it seems like a massive waste of compute to me when surely the output will be identical because it's analysing source code, not behaviour.
xnorswap 1y ago
> How are people reviewing that? 90% of the page height is taken up by "Check failure", Typically, you wouldn't bother manually reviewing something until the automated checks have passed.
- diggan 1y ago
  I dunno, when I review code, I don't review what's automatically checked anyways, but thinking about the change/diff in a broader context, and whatever isn't automatically checked. And the earlier you can steer people in the right direction, the better. But maybe this isn't the typical workflow.
  - xnorswap 1y ago
    The reality is more nuanced, there are situations where you'd want to glance over it anyway, such as looking for an opportunity to coach a junior dev. I'd rather hop in and get them on the right path rather than letting them struggle alone, particularly if they're struggling. If it's another senior developer though I'd happily leave them to it to get the unit tests all passing before I take a proper look at their work. But as a general principle, please at least get a PR through formatting checks before assigning it to a person.
  - Cthulhu_ 1y ago
    It's a waste of time tbh; fixing the checks may require the author to rethink or rewrite their entire solution, which means your review no longer applies. Let them finish a pull request before spending time reviewing it. That said, a merge request needs to have an issue written before it's picked up, so that the author does not spend time on a solution before the problem is understood. That's idealism though.
  - phkahler 1y ago
    >> And the earlier you can steer people in the right direction, the better. The earliest feedback you can get comes from the compiler. If it won't build successfully don't submit the PR.
spacecadet 1y ago
"I wonder if there is some difference in their internal processes that are leaking through here?" Maybe, but likely it is reality and their true company culture leaking through. Eventually some higher eq execs might come to the very late realization that they cant actually lead or build a worthwhile and productive company culture and all that remains is an insane reflection of that.
vasco 1y ago
> improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative Why do they even need it? Success is code getting merged 1st shot, failure gets worse the more requests for changes the agent gets. Asking for manual feedback seems like a waste of time. Measure cycle time and rate of approvals and change failure rate like you would for any developer.
surgical_fire 1y ago
> I can't imagine how the people who have to deal with this are feeling. It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. That comparison is awful. I work with quite a few Junior developers and they can be competent. Certainly don't make the silly mistakes that LLMs do, don't need nearly as much handholding, and tend to learn pretty quickly so I don't have to keep repeating myself. LLMs are decent code assistants when used with care, and can do a lot of heavy lifting, they certainly speed me up when I have a clear picture of what I want to do, and they are good to bounce off ideas when I am planning for something. That said, I really don't see how it could meaningfully replace an intern however, much less an actual developer.
- sbarre 1y ago
  I think that was the point of the comparison.. It's not like a regular junior developer, it's much worse.
  - spacemadness 1y ago
    And yet it got the job and lots of would be juniors didn’t, and it seems to be costing the company more in compute and senior dev handholding. Nice work silicon valley.
- safety1st 1y ago
  These GH interactions remind me of one of those offshore software outsourcing firms on Upwork or Freelancer.com that bid $3/hr on every project that gets posted. There's a PM who takes your task and gives it to a "developer" who potentially has never actually written a line of code, but maybe they've built a WordPress site by pointing and clicking in Elementor or something. After dozens of hours billed you will, in fact, get code where the new file wasn't added to the csproj or something like that, and when you point it out, they will bill another 20 hours, and send you a new copy of the project, where the test always fails. It's exactly like this. Nice to see that Microsoft has automated that, failure will be cheaper now.
ta1243 1y ago
> @copilot please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
yubblegum 1y ago
This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic, from the interview process, to imitating "big tech" songs and dances by small fry companies, and now this. Is there any joy left in being a professional software developer?
- bluefirebrand 1y ago
  Making quite a bit of money brings me a lot of joy compared to other industries But the actual software part? I'm not sure anymore
- diggan 1y ago
  > This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic I feel the same way today, but I got started around 2012 professionally. I wonder how much of this is just our fading optimism after seeing how shit really works behind the scenes, and how much the industry itself is responsible for it. I know we're not the only two people feeling this way either, but it seems all of us have different timescales from when it turned from "enjoyable" to "get me out of here".
  - salawat 1y ago
    My issue stems from the attitudes of the people we're doing it for. I started out doing it for humanity. To bring the bicycle for the mind to everyone. Then one day I woke up and realized the ones paying me were also the ones using it to run over or do circles around everyone else not equipped with a bicycle yet; and were colluding to make crippled bicycles that'd never liberate the masses as much as they themselves had been previously liberated; bicycles designed to monitor, or to undermine their owner, or more disgustingly, their "licensee". So I'm not doing it anymore. I'm not going to continue making deliberately crippled, overly complex, legally encumbered bicycles for the mind, purely intended as subjects for ARR extraction.
    - ecocentrik 1y ago
      It's hard to find anything wrong with your conclusions except that you're leaving out the part where they're trying to automate our contributions to devalue our skills. I'm surprised there isn't a movement to halt the use of AI for certain tasks in software development on the same level as the active resistance from doctors against socialized medicine in the US. These expensive toys will inevitably introduce catastrophic level bugs and security vulnerabilities into critical infrastructure software. Right now, most of Microsoft's product offerings, like GitHub and Office, are critical infrastructure software.
mrweasel 1y ago
At least we can tell the junior developers to not submit a pull-request before they have the tests running locally. At what point does the human developers just give up and close the PRs as "AI garbage". Keep the ones that works, then just junk the rest. I feel that at some point entertaining the machine becomes unbearable and people just stops doing it or rage close the PRs.
- pydry 1y ago
  When their performance reviews stop depending upon them not doing that. Microsoft's stock price is dependent on them proving that this is a success.
  - mrweasel 1y ago
    What happens when they can't prove that and development efficiency starts falling, because developers spend 50% of their time battling copilot?
    - latentsea 1y ago
      All that's required is enough mental gymnastics for someone to feel like they can call it a success. At no point is it actually required to be one.
      - mrweasel 1y ago
        You're probably correct: "Our developers a very happy with CoPilot. They now spend 50% of their time interacting with our AI offerings, either via VSCode, Github or Clippy." No need to specify why they are interact with it, all engagement is good engagement.
        ryandrake 1y ago
        By this measurement, a slower compiler is better than a faster one, because developers are using it for more of their time. Totally bonkers, Microsoft!
marmakoide 1y ago
Hot take : the whole LLM craze is fed by a delusion. LLM are good at mimicking human language, capturing some semantics on the way. With a large enough training set, the amount of semantic captured covers a large fraction of what the average human knows. This gives the illusion of intelligence, and the humans extrapolates on LLM capabilities, like actual coding. Because large amounts of code from textbooks and what not is on the training set, the illusion is convincing for people with shallow coding abilities. And then, while the tech is not mature, running on delusion and sunken costs, it's actually used for production stuffs. Butlerian Jihad when
- otabdeveloper4 1y ago
  > Butlerian Jihad when I estimate two more years for the bubble to pop.
- deleted 1y ago
  [deleted]
- nyarlathotep_ 1y ago
  I think the bubble is already a bit past peak. My sophisticated sentiment analysis (talking to co-workers other professional programmers and IT workers, HN and Reddit comments) seems to indicate a shift--there's a lot less storybook "Ay Eye is gonna take over the world" talk and a lot more distrust and even disdain than you'd see even 6 months ago. Moves like this will not go over well.
  - rasz 1y ago
    AI proponents would say you are witnessing third stage of 'First they ignore you, then they laugh at you, then they fight you, then you win'
prossercj 1y ago
This comment on that PR is pure gold. The bots are talking to each other: https://github.com/dotnet/runtime/pull/115732#issuecomment-2891990223 https://github.com/dotnet/runtime/pull/115732#issuecomment-2...
deleted 1y ago
[deleted]
dfxm12 1y ago
It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. Anyone who has dealt with Microsoft support knows this feeling well. Even talking to the higher level customer success folks feels like talking to a brick wall. After dozens of support cases, I can count on zero hands the number of issues that were closed satisfactorily. I appreciate Microsoft eating their dogfood here, but please don't make me eat it too! If anyone from MS is reading this, please release finished products that you are prepared to support!
cruffle_duffle 1y ago
It’s probably the junior devs that get to review these PRs. That and interns.
throwup238 1y ago
> Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative. The feedback buttons open a feedback form modal, they don’t reflect the number of feedback given like the emoji button. If you leave feedback, it will reflect your thumbs up/down (hiding the other button), it doesn’t say anything about whether anyone else has left feedback (I’ve tried it on my own repos).
belter 1y ago
This whole thread from yesterday take a whole different meaning: https://news.ycombinator.com/item?id=44031432 https://news.ycombinator.com/item?id=44031432 Comment in the GitHub discussion: "...You and I and every programmer who hasn't been living under a rock knows that AI isn't ready to be adopted at this scale yet, on the premier; 100M-user code-hosting platform. It doesn't make any sense except in brain-washed corporate-talk like "we are testing today what it can do tomorrow". I'm not saying that this couldn't be an adequate change some day, perhaps even in a few years but we all know this isn't it today. It's 100% financial-driven hype with a pinch of we're too big to fail mentality..."
- namaria 1y ago
  "Big data" -> "Cloud" -> "LLM-as-A(G)I" It's all just recycled rent seeking corporate hype for enterprise compute. The moment I had decided to learn Kubernetes years ago, got a book and saw microservices compared to 'object-oriented' programming I realized that. The 'big ball of mud' paper and the 'worse is better' rant frame it all pretty well in my view. Prioritize velocity, get slop in production, cope with the accidental complexity, rinse repeat. Eventually you get to a point where GPU farms seem like a reasonable way to auto-complete code. When you find yourself in a hole, stop digging. Any bigger excavator you send down there will only get buried when the mud crashes down.
deleted 1y ago
[deleted]
TheNewsIsHere 1y ago
> This whole thing would be fucking hilarious if I didn't feel so bad for the humans who are on the other side of this. Which will soon be anyone who directly or indirectly relies on Microsoft technologies. Some of these PRs, including at least one that I saw reworked certificate validation logic with not much more than a perfunctory “LGTM”, have been merged into main. Coincidentally, I wonder if issues orthogonal to this slop is why I’ve been getting so many HTTP 500 errors when using GitHub lately.

by diggan 1y ago

worldsayshi 1y ago

> How are people reviewing that? I agree that not auto-collapsing repeated annotations is an annoying bug in the github interface. But just pointing out that annotations can be hidden in the ... menu to the right (which I just learned).

codyvoda 1y ago
or press “a”
jon-wood 1y ago
I'm not entirely sure why they're running linters on every available platform to begin with, it seems like a massive waste of compute to me when surely the output will be identical because it's analysing source code, not behaviour.

xnorswap 1y ago

> How are people reviewing that? 90% of the page height is taken up by "Check failure", Typically, you wouldn't bother manually reviewing something until the automated checks have passed.

diggan 1y ago
I dunno, when I review code, I don't review what's automatically checked anyways, but thinking about the change/diff in a broader context, and whatever isn't automatically checked. And the earlier you can steer people in the right direction, the better. But maybe this isn't the typical workflow.
- xnorswap 1y ago
  The reality is more nuanced, there are situations where you'd want to glance over it anyway, such as looking for an opportunity to coach a junior dev. I'd rather hop in and get them on the right path rather than letting them struggle alone, particularly if they're struggling. If it's another senior developer though I'd happily leave them to it to get the unit tests all passing before I take a proper look at their work. But as a general principle, please at least get a PR through formatting checks before assigning it to a person.
- Cthulhu_ 1y ago
  It's a waste of time tbh; fixing the checks may require the author to rethink or rewrite their entire solution, which means your review no longer applies. Let them finish a pull request before spending time reviewing it. That said, a merge request needs to have an issue written before it's picked up, so that the author does not spend time on a solution before the problem is understood. That's idealism though.
- phkahler 1y ago
  >> And the earlier you can steer people in the right direction, the better. The earliest feedback you can get comes from the compiler. If it won't build successfully don't submit the PR.

spacecadet 1y ago

"I wonder if there is some difference in their internal processes that are leaking through here?" Maybe, but likely it is reality and their true company culture leaking through. Eventually some higher eq execs might come to the very late realization that they cant actually lead or build a worthwhile and productive company culture and all that remains is an insane reflection of that.

vasco 1y ago

> improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative Why do they even need it? Success is code getting merged 1st shot, failure gets worse the more requests for changes the agent gets. Asking for manual feedback seems like a waste of time. Measure cycle time and rate of approvals and change failure rate like you would for any developer.

surgical_fire 1y ago

> I can't imagine how the people who have to deal with this are feeling. It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. That comparison is awful. I work with quite a few Junior developers and they can be competent. Certainly don't make the silly mistakes that LLMs do, don't need nearly as much handholding, and tend to learn pretty quickly so I don't have to keep repeating myself. LLMs are decent code assistants when used with care, and can do a lot of heavy lifting, they certainly speed me up when I have a clear picture of what I want to do, and they are good to bounce off ideas when I am planning for something. That said, I really don't see how it could meaningfully replace an intern however, much less an actual developer.

sbarre 1y ago
I think that was the point of the comparison.. It's not like a regular junior developer, it's much worse.
- spacemadness 1y ago
  And yet it got the job and lots of would be juniors didn’t, and it seems to be costing the company more in compute and senior dev handholding. Nice work silicon valley.
safety1st 1y ago
These GH interactions remind me of one of those offshore software outsourcing firms on Upwork or Freelancer.com that bid $3/hr on every project that gets posted. There's a PM who takes your task and gives it to a "developer" who potentially has never actually written a line of code, but maybe they've built a WordPress site by pointing and clicking in Elementor or something. After dozens of hours billed you will, in fact, get code where the new file wasn't added to the csproj or something like that, and when you point it out, they will bill another 20 hours, and send you a new copy of the project, where the test always fails. It's exactly like this. Nice to see that Microsoft has automated that, failure will be cheaper now.

ta1243 1y ago

> @copilot please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

yubblegum 1y ago

This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic, from the interview process, to imitating "big tech" songs and dances by small fry companies, and now this. Is there any joy left in being a professional software developer?

bluefirebrand 1y ago
Making quite a bit of money brings me a lot of joy compared to other industries But the actual software part? I'm not sure anymore
diggan 1y ago
> This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic I feel the same way today, but I got started around 2012 professionally. I wonder how much of this is just our fading optimism after seeing how shit really works behind the scenes, and how much the industry itself is responsible for it. I know we're not the only two people feeling this way either, but it seems all of us have different timescales from when it turned from "enjoyable" to "get me out of here".
- salawat 1y ago
  My issue stems from the attitudes of the people we're doing it for. I started out doing it for humanity. To bring the bicycle for the mind to everyone. Then one day I woke up and realized the ones paying me were also the ones using it to run over or do circles around everyone else not equipped with a bicycle yet; and were colluding to make crippled bicycles that'd never liberate the masses as much as they themselves had been previously liberated; bicycles designed to monitor, or to undermine their owner, or more disgustingly, their "licensee". So I'm not doing it anymore. I'm not going to continue making deliberately crippled, overly complex, legally encumbered bicycles for the mind, purely intended as subjects for ARR extraction.
  - ecocentrik 1y ago
    It's hard to find anything wrong with your conclusions except that you're leaving out the part where they're trying to automate our contributions to devalue our skills. I'm surprised there isn't a movement to halt the use of AI for certain tasks in software development on the same level as the active resistance from doctors against socialized medicine in the US. These expensive toys will inevitably introduce catastrophic level bugs and security vulnerabilities into critical infrastructure software. Right now, most of Microsoft's product offerings, like GitHub and Office, are critical infrastructure software.

mrweasel 1y ago

At least we can tell the junior developers to not submit a pull-request before they have the tests running locally. At what point does the human developers just give up and close the PRs as "AI garbage". Keep the ones that works, then just junk the rest. I feel that at some point entertaining the machine becomes unbearable and people just stops doing it or rage close the PRs.

pydry 1y ago
When their performance reviews stop depending upon them not doing that. Microsoft's stock price is dependent on them proving that this is a success.
- mrweasel 1y ago
  What happens when they can't prove that and development efficiency starts falling, because developers spend 50% of their time battling copilot?
  - latentsea 1y ago
    All that's required is enough mental gymnastics for someone to feel like they can call it a success. At no point is it actually required to be one.
    - mrweasel 1y ago
      You're probably correct: "Our developers a very happy with CoPilot. They now spend 50% of their time interacting with our AI offerings, either via VSCode, Github or Clippy." No need to specify why they are interact with it, all engagement is good engagement.
      - ryandrake 1y ago
        By this measurement, a slower compiler is better than a faster one, because developers are using it for more of their time. Totally bonkers, Microsoft!

marmakoide 1y ago

Hot take : the whole LLM craze is fed by a delusion. LLM are good at mimicking human language, capturing some semantics on the way. With a large enough training set, the amount of semantic captured covers a large fraction of what the average human knows. This gives the illusion of intelligence, and the humans extrapolates on LLM capabilities, like actual coding. Because large amounts of code from textbooks and what not is on the training set, the illusion is convincing for people with shallow coding abilities. And then, while the tech is not mature, running on delusion and sunken costs, it's actually used for production stuffs. Butlerian Jihad when

otabdeveloper4 1y ago
> Butlerian Jihad when I estimate two more years for the bubble to pop.
deleted 1y ago
[deleted]
nyarlathotep_ 1y ago
I think the bubble is already a bit past peak. My sophisticated sentiment analysis (talking to co-workers other professional programmers and IT workers, HN and Reddit comments) seems to indicate a shift--there's a lot less storybook "Ay Eye is gonna take over the world" talk and a lot more distrust and even disdain than you'd see even 6 months ago. Moves like this will not go over well.
- rasz 1y ago
  AI proponents would say you are witnessing third stage of 'First they ignore you, then they laugh at you, then they fight you, then you win'

prossercj 1y ago

This comment on that PR is pure gold. The bots are talking to each other: https://github.com/dotnet/runtime/pull/115732#issuecomment-2891990223 https://github.com/dotnet/runtime/pull/115732#issuecomment-2...

deleted 1y ago

[deleted]

dfxm12 1y ago

It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. Anyone who has dealt with Microsoft support knows this feeling well. Even talking to the higher level customer success folks feels like talking to a brick wall. After dozens of support cases, I can count on zero hands the number of issues that were closed satisfactorily. I appreciate Microsoft eating their dogfood here, but please don't make me eat it too! If anyone from MS is reading this, please release finished products that you are prepared to support!

cruffle_duffle 1y ago

It’s probably the junior devs that get to review these PRs. That and interns.

throwup238 1y ago

> Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative. The feedback buttons open a feedback form modal, they don’t reflect the number of feedback given like the emoji button. If you leave feedback, it will reflect your thumbs up/down (hiding the other button), it doesn’t say anything about whether anyone else has left feedback (I’ve tried it on my own repos).

belter 1y ago

This whole thread from yesterday take a whole different meaning: https://news.ycombinator.com/item?id=44031432 https://news.ycombinator.com/item?id=44031432 Comment in the GitHub discussion: "...You and I and every programmer who hasn't been living under a rock knows that AI isn't ready to be adopted at this scale yet, on the premier; 100M-user code-hosting platform. It doesn't make any sense except in brain-washed corporate-talk like "we are testing today what it can do tomorrow". I'm not saying that this couldn't be an adequate change some day, perhaps even in a few years but we all know this isn't it today. It's 100% financial-driven hype with a pinch of we're too big to fail mentality..."

namaria 1y ago
"Big data" -> "Cloud" -> "LLM-as-A(G)I" It's all just recycled rent seeking corporate hype for enterprise compute. The moment I had decided to learn Kubernetes years ago, got a book and saw microservices compared to 'object-oriented' programming I realized that. The 'big ball of mud' paper and the 'worse is better' rant frame it all pretty well in my view. Prioritize velocity, get slop in production, cope with the accidental complexity, rinse repeat. Eventually you get to a point where GPU farms seem like a reasonable way to auto-complete code. When you find yourself in a hole, stop digging. Any bigger excavator you send down there will only get buried when the mud crashes down.

deleted 1y ago

[deleted]

TheNewsIsHere 1y ago

> This whole thing would be fucking hilarious if I didn't feel so bad for the humans who are on the other side of this. Which will soon be anyone who directly or indirectly relies on Microsoft technologies. Some of these PRs, including at least one that I saw reworked certificate validation logic with not much more than a perfunctory “LGTM”, have been merged into main. Coincidentally, I wonder if issues orthogonal to this slop is why I’ve been getting so many HTTP 500 errors when using GitHub lately.