9 ms·

Field experimental evidence of AI on knowledge worker productivity and quality

139 points by CharlesW 3y ago

datadrivenangel 3y ago
"Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores. For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI." So AI makes consultants faster but worse. As a consultant, this should make my expertise even more valuable.
- tempest_ 3y ago
  Only if being wrong actually matters.
  - datadrivenangel 3y ago
    The research was partially conducted by Boston Consulting Group, so in this case being wrong doesn't really matter as long as the Partners are smooth and the associates create billable hours.
- CharlesW 3y ago
  > So AI makes consultants faster but worse. That was just for tasks not suited to GPT-4 (and I assume, LLMs in general). > "For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI." For tasks suited to GPT-4, consultants produced 40% higher quality results by leveraging GPT-4. My take on that is that people who have at least some understanding of how LLMs work will be at a significant advantage, while people who think LLMs "think" will misuse LLMs and get predictably worse results.
  - ParetoOptimal 3y ago
    > while people who think LLMs "think" will misuse LLMs and get predictably worse results. I've found that now that I see LLM's less as thinking and more as stochiastic parrot I get a lot less use from them because I give 3-5 turns at most versus 10 turns.
    - NemoNobody 3y ago
      Well, maybe a balance between a stoic parrot and a "thinking machine" might be the best approach. You know, everything in moderation and what not.
      - ParetoOptimal 3y ago
        Yes, ideally. The problem for me is it used to be an addition to my energy and confidence, but when you "take the magic out of it" it doesn't help out there so much. In antirez's blog he mentioned how it made things "not worth it, worth it" and I agreed, but that happens much less often if you think an LLM helping you is a fluke.
- NemoNobody 3y ago
  Seriously, you cherry picked the paragraph you use to cite your opinion, which contrary to the finding of the article itself and completely ignore the the first part of the second sentence of your paragraph "for a task selected outside the frontier" - so, something beyond the capabilities of AI at this time ONLY decreased their correct solutions 19 percentage points. 43% better it made below average people, 17% better it made above average people. Same paragraph.
  - steve1977 3y ago
    BCG consultants. Not average people.
  - datadrivenangel 3y ago
    You know, I missed that part. Oops. Looking into the full paper, it looks like the task identified inside the frontier is much more broken down, with 18 numbered steps. The outside the frontier task is two bullet points. So for clearly scoped tasks, ChatGPT is hands down good. For less well scoped work it reduces quality. Interestingly, the authors also looked at the amount of text retained from ChatGPT, and found that people who received ChatGPT training ended up retaining a larger portion of the ChatGPT output in their final answers and performed better in the quality assessment.
- quickcheque 3y ago
  ' As a consultant, this should make my expertise even more valuable. ' Consultant logic at its finest.
  - j4yav 3y ago
    More garbage outputs being produced faster, paired with expertise which makes it credibly sound like more fast garbage will solve any remaining issues, probably _is_ approaching peak consultant value.
    - datadrivenangel 3y ago
      Only if that's what the clients want or need to justify actual improvements! Usually the goal is to produce less garbage outputs slower.
      - j4yav 3y ago
        The goal isn’t to generate billable hours?
        datadrivenangel 3y ago
        We like to find win-win situations.
ThrowawayTestr 3y ago
I just started a new job and used chatgpt to create some cool Excel automation with Python.
Ecoste 3y ago
I probably should know but I don't even have the vaguest of ideas. What do consultants actually do?
- paulsutter 3y ago
  Bill. Sometimes hourly, sometimes by project. They wake up every morning hell bent on maintaining existing customer accounts, growing existing accounts, and sometimes adding a new account. They maintain a professional demeanor, with just the right kind of smalltalk. Finally, they produce a report that justifies whatever was the decision that the CEO already made before he engaged the consultants Thats pretty much the whole job
  - sockaddr 3y ago
    Yeah I’ve struggled to understand what people mean by consultant and once heard that it was basically a person or group you can hire to take the blame for an unpopular decision and/or to weaponize it against others in the company since it was “someone else” that made the call.
- jLaForest 3y ago
  Tell management who to fire and who to outsource
- tokai 3y ago
  Provides you with a neat document that states what you wanted to do is the right thing to do.
- fhd2 3y ago
  I'm a consultant - for about a year now. Not at BCG or anywhere, I'm a one man company. What I do is pretty similar to what I did before in my CTO roles, just a bit more strategy and less execution. Clients come to me with questions and problems and I help them solve them based on my experience. If you've got multiple C levels, middle management and a board, you're in a similarly consulting position also as a full time CTO from what I've seen. I don't go anywhere _near_ LLMs for this, I'd find it somewhere between disrespectful and unethical. They're paying for _my_ expertise. I can imagine it could make a decent alternative to Google and Wikipedia for research purposes, but I'd have to double check it all anyway. I don't see how it'd make my work any easier.
mediaman 3y ago
This study had management consultants use GPT4 on a variety of different management consultant tasks. Some were more creative in nature, and some were more analytical. For example, ideas on marketing sneakers versus an analysis of store performance across a retailer's portfolio. In general they found that GPT4 was helpful for creative tasks, but didn't help much (and in fact reduced quality) for analytical questions. I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise. Whether it is useful or not comes down to whether engineers can harness it within a pipeline of content and tools. For example, when engineers create a system to summarize issues a customer has had from the CRM, that can help a customer service person be more informed. Structured semantic search on a knowledge base can help them also find the right solution to common customer problems. McKinsey made a retrieval augmented generation system that searched all their research reports and created summaries of content they could use, so a consultant could quickly find prior work that would be relevant on a client project. If they built that correctly, I imagine that is pretty useful. GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application. Many of the studies so far only look at vanilla GPT4 via ChatGPT, and it seems unlikely that, if LLMs do transform the workplace, that a standalone ChatGPT is what it will look like.
- ebiester 3y ago
  The first generation of company-focused llm will need to be bespoke. Teams involved with this will learn the necessary abstractions and will build new companies to lower the on-ramp or join the LLM providers to do the same.
- Fin_Code 3y ago
  This is entirely unsurprising. A language generator is not designed for analytical analysis. The fact it codes well is what I think is throwing people. Since code is a form of language it works well for the language model. The issue with code will come from larger context where analytical thought is needed and it will break down.
totoglazer 3y ago
Note this paper was published in September 2023.
clbrmbr 3y ago
Can someone explain Centaurs vs Cyborgs? (I didn’t make an account to read the full paper…)
araes 3y ago
The actual paper can be found at: https://www.iab.cl/wp-content/uploads/2023/11/SSRN-id4573321.pdf https://www.iab.cl/wp-content/uploads/2023/11/SSRN-id4573321... Note: After reading the paper, there are serious methodology issues. ChatGPT without any human involvement (simply reading the task as instructions and producing an answer) produced a better "quality" result than any human involvement in many tasks. My read: ChatGPT produces what we pre-view as "correct". They even kind of state this: "when human subjects use ChatGPT there is a reduction in the variation in the eventual ideas they produce. This result is perhaps surprising one would assume that ChatGPT, with its expansive knowledge base, would instead be able to produce many very distinct ideas, compared to human subjects alone." My read: The humans converge towards what is already viewed as "correct". Its like being in a business and your boss already knows exactly what your boss actually wants, and any variation you produce is automatically "bad" anyways. There is also a drastic difference in the treatment of the in/out/on "frontier" subject. Lots of talk about how great adding AI is to "inside frontier" tasks, no similar graphs/charts for "outside frontier" tasks. Finally, if these people are consultants and business profs, they produce some crazily bad charts. Figure 3 in the appendix is so difficult to read.
gandalfgeek 3y ago
If you don't want to read the full thing, Mollick's substack has more accessible summary of this paper: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the... There are two sides to this thing. The first author of this paper has also written another paper titles "Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment". Abstract: "As AI quality increases, humans have fewer incentives to exert effort and remain attentive, allowing the AI to substitute, rather than augment their performance... I found that subjects with higher quality AI were less accurate in their assessments of job applications than subjects with lower quality AI. On average, recruiters receiving lower quality AI exerted more effort and spent more time evaluating the resumes, and were less likely to automatically select the AI-recommended candidate. The recruiters collaborating with low-quality AI learned to interact better with their assigned AI and improved their performance. Crucially, these effects were driven by more experienced recruiters. Overall, the results show that maximizing human/AI performance may require lower quality AI, depending on the effort, learning, and skillset of the humans involved." https://static1.squarespace.com/static/604b23e38c22a96e9c78879e/t/62d5d9448d061f7327e8a7e7/1658181956291/Falling+Asleep+at+the+Wheel+-+Fabrizio+DellAcqua.pdf https://static1.squarespace.com/static/604b23e38c22a96e9c788...
- esafak 3y ago
  If you don't need a skill, you won't practice it, and it will atrophy. This frees you to learn different things. Like how most software engineers today don't know computer engineering (hardware) but they can write more complex applications.
  - SkyBelow 3y ago
    That is if the AI fully replaces the need for a skill. What happens if it mostly replaces it, enough so the skill atrophies, but not enough that the user never needs the skill?
    - esafak 3y ago
      Good point. That sounds like a temporary product problem, like a level 2-3 self-driving car, which still needs you to pay attention. I think you also need to consider whether you can afford not to have the skill; what would happen if the AI were taken away, or malfunctioned? Airplane pilots are an example of this. In that case you simply have to learn the skill as if the AI will not be there.
rich_sasha 3y ago
I am yet to to come across a programming task where ChatGPT saved me time. For a start, most of my work is about using internal APIs. But nevermind. Sometimes I come across a generic programming problem. ChatGPT get me to 80% of a solution. Getting it from there to a 100% solution takes as long as doing it from scratch. Just my $0.02
- snapcaster 3y ago
  Have you considered that it may be a personal skill issue? I'm not saying it is, but it seems like people with this opinion never seem to consider maybe they aren't using it properly or don't get it
  - elzbardico 3y ago
    The reverse could also be true. Some people could be not able to figure out that the code GPT produced is not good because they lack the skill level to review it efficiently.
  - rich_sasha 3y ago
    Well, maybe. But it seems to me this somewhat defeats the purpose. If I tell ChatGPT to write me, say, a sorting algorithm (which I'm sure it can do), but if I ask it _wrongly_ it gives me a deceptively looking lemon of a solution, that's a liability. Conversely, could you share an example of a nontrivial, practical programming problem ChatGPT can solve, but only if you use it right?
    - mewpmewp2 3y ago
      For coding it's mostly Copilot that saves time. Although ChatGPT in some cases as well, but not as reliably or frequently.
      - throwaway4aday 3y ago
        Agree, Copilot is a huge speed boost for stubbing things out, auto-generating the dead simple parts, and refactoring. ChatGPT is awesome for rubberducking, pair programming, brainstorming, and figuring out what that thing that you kind of remember is called. We aren't at the point where we can just say "robot, do my work" but it's for sure good enough to take care of the boring stuff and be a sounding board.
freedryk 3y ago
The biggest question I see with this study is it doesn't seem like subjects had access to tools outside of the AI. Were the subjects without AI able to do google searches? If not, then what is the performance gain of the AI users over people who can google stuff?
siliconc0w 3y ago
There is a lot of noise from VC-types about how AI copilots turn random outsourced devs into 10x engineers. My experience is that it's pretty good at writing boilerplate, that is - it saves me typing what I would expended little thought in typing. I suspect this is where most of the productivity statistics are coming from. Co-pilot especially seems prone to hallucinating APIs or suggesting entire functions that are wrong, repetitive, or even dangerous. This is the case even for side projects where I'm working with public python libraries which should be like the best case scenario. It's generally better than IDE autocomplete when I already am familiar with the API and so can catch the errors. I will say GPT-4 is mostly better than referencing stack-overflow or library documentation. Maybe once that gets fast/cheap enough to use as co-pilot we'll see some of these mythical productivity gains.
- smackeyacky 3y ago
  I don't think so. I think it will be more like a flare that burns out. The reason being that the source of material for GPT-4 is rotting underneath. As the internet gets ruined by AI garbage, the quality of output of AI will get worse, not better. edit: This is a modern day tower of Babel event. We had a magnificent resource and then polluted it with garbage and now it's quickly becoming useless.
  - throwaway4aday 3y ago
    This is wrong for at least two reasons: 1) The data scraped from the web is filtered, deduped, ranked, and cleaned in a variety of other ways before being used for training. While quantity is necessary for a model to learn the structure of language, quality is even more important once you have a model that can produce coherent output so a lot of work goes into grooming the data. 2) There have been a bunch of papers and training runs that show synthetic data created specifically for training a model is as good or better than scraped human produced data. The importance of scraped web content is quickly declining because you can now generate infinite higher quality examples using existing trained models. The only relevance it has now is for knowledge about new developments and that is a much easier stream to filter since most of the important stuff comes from official sources and you don't need as many variations since you can just generate your own using one or more examples.
RecycledEle 3y ago
From the article: "consultants using AI were significantly more productive (they completed 12.2% more tasks on average, and completed task 25.1% more quickly), and produced significantly higher quality results (more than 40% higher quality compared to a control group)."
lgleason 3y ago
I was underwhelmed when using it for code generation of things like tests.
dragoncrab 3y ago
I don't want to be rude, but given the average quality of material produced by big consulting companies I have encountered over my career, I'm not that surprised that GPT4 beats them.
cleandreams 3y ago
I've been an engineer at FAANG. I think the problems consultants face are simpler than the ones confronting engineers who need to become expert at million line code bases. There are coding standards, styles, security practices. I think AI would help when boiler plate is needed but otherwise it would be a dicier proposition. But there is another issue that I think is long term. The apprenticeship of a top engineer is difficult. If the result of "AI shortcuts" is a less skilled engineer profession there will be considerable downsides in the future. I have read that the ability of people to navigate is negatively impacted by a reliance on google maps.
- deleted 3y ago
  [deleted]

139 points by CharlesW 3y ago

"Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores. For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI." So AI makes consultants faster but worse. As a consultant, this should make my expertise even more valuable.

tempest_ 3y ago
Only if being wrong actually matters.
- datadrivenangel 3y ago
  The research was partially conducted by Boston Consulting Group, so in this case being wrong doesn't really matter as long as the Partners are smooth and the associates create billable hours.
CharlesW 3y ago
> So AI makes consultants faster but worse. That was just for tasks not suited to GPT-4 (and I assume, LLMs in general). > "For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI." For tasks suited to GPT-4, consultants produced 40% higher quality results by leveraging GPT-4. My take on that is that people who have at least some understanding of how LLMs work will be at a significant advantage, while people who think LLMs "think" will misuse LLMs and get predictably worse results.
- ParetoOptimal 3y ago
  > while people who think LLMs "think" will misuse LLMs and get predictably worse results. I've found that now that I see LLM's less as thinking and more as stochiastic parrot I get a lot less use from them because I give 3-5 turns at most versus 10 turns.
  - NemoNobody 3y ago
    Well, maybe a balance between a stoic parrot and a "thinking machine" might be the best approach. You know, everything in moderation and what not.
    - ParetoOptimal 3y ago
      Yes, ideally. The problem for me is it used to be an addition to my energy and confidence, but when you "take the magic out of it" it doesn't help out there so much. In antirez's blog he mentioned how it made things "not worth it, worth it" and I agreed, but that happens much less often if you think an LLM helping you is a fluke.
NemoNobody 3y ago
Seriously, you cherry picked the paragraph you use to cite your opinion, which contrary to the finding of the article itself and completely ignore the the first part of the second sentence of your paragraph "for a task selected outside the frontier" - so, something beyond the capabilities of AI at this time ONLY decreased their correct solutions 19 percentage points. 43% better it made below average people, 17% better it made above average people. Same paragraph.
- steve1977 3y ago
  BCG consultants. Not average people.
- datadrivenangel 3y ago
  You know, I missed that part. Oops. Looking into the full paper, it looks like the task identified inside the frontier is much more broken down, with 18 numbered steps. The outside the frontier task is two bullet points. So for clearly scoped tasks, ChatGPT is hands down good. For less well scoped work it reduces quality. Interestingly, the authors also looked at the amount of text retained from ChatGPT, and found that people who received ChatGPT training ended up retaining a larger portion of the ChatGPT output in their final answers and performed better in the quality assessment.
quickcheque 3y ago
' As a consultant, this should make my expertise even more valuable. ' Consultant logic at its finest.
- j4yav 3y ago
  More garbage outputs being produced faster, paired with expertise which makes it credibly sound like more fast garbage will solve any remaining issues, probably _is_ approaching peak consultant value.
  - datadrivenangel 3y ago
    Only if that's what the clients want or need to justify actual improvements! Usually the goal is to produce less garbage outputs slower.
    - j4yav 3y ago
      The goal isn’t to generate billable hours?
      - datadrivenangel 3y ago
        We like to find win-win situations.

ThrowawayTestr 3y ago

I just started a new job and used chatgpt to create some cool Excel automation with Python.

Ecoste 3y ago

I probably should know but I don't even have the vaguest of ideas. What do consultants actually do?

paulsutter 3y ago
Bill. Sometimes hourly, sometimes by project. They wake up every morning hell bent on maintaining existing customer accounts, growing existing accounts, and sometimes adding a new account. They maintain a professional demeanor, with just the right kind of smalltalk. Finally, they produce a report that justifies whatever was the decision that the CEO already made before he engaged the consultants Thats pretty much the whole job
- sockaddr 3y ago
  Yeah I’ve struggled to understand what people mean by consultant and once heard that it was basically a person or group you can hire to take the blame for an unpopular decision and/or to weaponize it against others in the company since it was “someone else” that made the call.
jLaForest 3y ago
Tell management who to fire and who to outsource
tokai 3y ago
Provides you with a neat document that states what you wanted to do is the right thing to do.
fhd2 3y ago
I'm a consultant - for about a year now. Not at BCG or anywhere, I'm a one man company. What I do is pretty similar to what I did before in my CTO roles, just a bit more strategy and less execution. Clients come to me with questions and problems and I help them solve them based on my experience. If you've got multiple C levels, middle management and a board, you're in a similarly consulting position also as a full time CTO from what I've seen. I don't go anywhere _near_ LLMs for this, I'd find it somewhere between disrespectful and unethical. They're paying for _my_ expertise. I can imagine it could make a decent alternative to Google and Wikipedia for research purposes, but I'd have to double check it all anyway. I don't see how it'd make my work any easier.

mediaman 3y ago

This study had management consultants use GPT4 on a variety of different management consultant tasks. Some were more creative in nature, and some were more analytical. For example, ideas on marketing sneakers versus an analysis of store performance across a retailer's portfolio. In general they found that GPT4 was helpful for creative tasks, but didn't help much (and in fact reduced quality) for analytical questions. I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise. Whether it is useful or not comes down to whether engineers can harness it within a pipeline of content and tools. For example, when engineers create a system to summarize issues a customer has had from the CRM, that can help a customer service person be more informed. Structured semantic search on a knowledge base can help them also find the right solution to common customer problems. McKinsey made a retrieval augmented generation system that searched all their research reports and created summaries of content they could use, so a consultant could quickly find prior work that would be relevant on a client project. If they built that correctly, I imagine that is pretty useful. GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application. Many of the studies so far only look at vanilla GPT4 via ChatGPT, and it seems unlikely that, if LLMs do transform the workplace, that a standalone ChatGPT is what it will look like.

ebiester 3y ago
The first generation of company-focused llm will need to be bespoke. Teams involved with this will learn the necessary abstractions and will build new companies to lower the on-ramp or join the LLM providers to do the same.
Fin_Code 3y ago
This is entirely unsurprising. A language generator is not designed for analytical analysis. The fact it codes well is what I think is throwing people. Since code is a form of language it works well for the language model. The issue with code will come from larger context where analytical thought is needed and it will break down.

totoglazer 3y ago

Note this paper was published in September 2023.

clbrmbr 3y ago

Can someone explain Centaurs vs Cyborgs? (I didn’t make an account to read the full paper…)

araes 3y ago

The actual paper can be found at: https://www.iab.cl/wp-content/uploads/2023/11/SSRN-id4573321.pdf https://www.iab.cl/wp-content/uploads/2023/11/SSRN-id4573321... Note: After reading the paper, there are serious methodology issues. ChatGPT without any human involvement (simply reading the task as instructions and producing an answer) produced a better "quality" result than any human involvement in many tasks. My read: ChatGPT produces what we pre-view as "correct". They even kind of state this: "when human subjects use ChatGPT there is a reduction in the variation in the eventual ideas they produce. This result is perhaps surprising one would assume that ChatGPT, with its expansive knowledge base, would instead be able to produce many very distinct ideas, compared to human subjects alone." My read: The humans converge towards what is already viewed as "correct". Its like being in a business and your boss already knows exactly what your boss actually wants, and any variation you produce is automatically "bad" anyways. There is also a drastic difference in the treatment of the in/out/on "frontier" subject. Lots of talk about how great adding AI is to "inside frontier" tasks, no similar graphs/charts for "outside frontier" tasks. Finally, if these people are consultants and business profs, they produce some crazily bad charts. Figure 3 in the appendix is so difficult to read.

gandalfgeek 3y ago

If you don't want to read the full thing, Mollick's substack has more accessible summary of this paper: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the... There are two sides to this thing. The first author of this paper has also written another paper titles "Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment". Abstract: "As AI quality increases, humans have fewer incentives to exert effort and remain attentive, allowing the AI to substitute, rather than augment their performance... I found that subjects with higher quality AI were less accurate in their assessments of job applications than subjects with lower quality AI. On average, recruiters receiving lower quality AI exerted more effort and spent more time evaluating the resumes, and were less likely to automatically select the AI-recommended candidate. The recruiters collaborating with low-quality AI learned to interact better with their assigned AI and improved their performance. Crucially, these effects were driven by more experienced recruiters. Overall, the results show that maximizing human/AI performance may require lower quality AI, depending on the effort, learning, and skillset of the humans involved." https://static1.squarespace.com/static/604b23e38c22a96e9c78879e/t/62d5d9448d061f7327e8a7e7/1658181956291/Falling+Asleep+at+the+Wheel+-+Fabrizio+DellAcqua.pdf https://static1.squarespace.com/static/604b23e38c22a96e9c788...

esafak 3y ago
If you don't need a skill, you won't practice it, and it will atrophy. This frees you to learn different things. Like how most software engineers today don't know computer engineering (hardware) but they can write more complex applications.
- SkyBelow 3y ago
  That is if the AI fully replaces the need for a skill. What happens if it mostly replaces it, enough so the skill atrophies, but not enough that the user never needs the skill?
  - esafak 3y ago
    Good point. That sounds like a temporary product problem, like a level 2-3 self-driving car, which still needs you to pay attention. I think you also need to consider whether you can afford not to have the skill; what would happen if the AI were taken away, or malfunctioned? Airplane pilots are an example of this. In that case you simply have to learn the skill as if the AI will not be there.

rich_sasha 3y ago

I am yet to to come across a programming task where ChatGPT saved me time. For a start, most of my work is about using internal APIs. But nevermind. Sometimes I come across a generic programming problem. ChatGPT get me to 80% of a solution. Getting it from there to a 100% solution takes as long as doing it from scratch. Just my $0.02

snapcaster 3y ago
Have you considered that it may be a personal skill issue? I'm not saying it is, but it seems like people with this opinion never seem to consider maybe they aren't using it properly or don't get it
- elzbardico 3y ago
  The reverse could also be true. Some people could be not able to figure out that the code GPT produced is not good because they lack the skill level to review it efficiently.
- rich_sasha 3y ago
  Well, maybe. But it seems to me this somewhat defeats the purpose. If I tell ChatGPT to write me, say, a sorting algorithm (which I'm sure it can do), but if I ask it _wrongly_ it gives me a deceptively looking lemon of a solution, that's a liability. Conversely, could you share an example of a nontrivial, practical programming problem ChatGPT can solve, but only if you use it right?
  - mewpmewp2 3y ago
    For coding it's mostly Copilot that saves time. Although ChatGPT in some cases as well, but not as reliably or frequently.
    - throwaway4aday 3y ago
      Agree, Copilot is a huge speed boost for stubbing things out, auto-generating the dead simple parts, and refactoring. ChatGPT is awesome for rubberducking, pair programming, brainstorming, and figuring out what that thing that you kind of remember is called. We aren't at the point where we can just say "robot, do my work" but it's for sure good enough to take care of the boring stuff and be a sounding board.

freedryk 3y ago

The biggest question I see with this study is it doesn't seem like subjects had access to tools outside of the AI. Were the subjects without AI able to do google searches? If not, then what is the performance gain of the AI users over people who can google stuff?

siliconc0w 3y ago

There is a lot of noise from VC-types about how AI copilots turn random outsourced devs into 10x engineers. My experience is that it's pretty good at writing boilerplate, that is - it saves me typing what I would expended little thought in typing. I suspect this is where most of the productivity statistics are coming from. Co-pilot especially seems prone to hallucinating APIs or suggesting entire functions that are wrong, repetitive, or even dangerous. This is the case even for side projects where I'm working with public python libraries which should be like the best case scenario. It's generally better than IDE autocomplete when I already am familiar with the API and so can catch the errors. I will say GPT-4 is mostly better than referencing stack-overflow or library documentation. Maybe once that gets fast/cheap enough to use as co-pilot we'll see some of these mythical productivity gains.

smackeyacky 3y ago
I don't think so. I think it will be more like a flare that burns out. The reason being that the source of material for GPT-4 is rotting underneath. As the internet gets ruined by AI garbage, the quality of output of AI will get worse, not better. edit: This is a modern day tower of Babel event. We had a magnificent resource and then polluted it with garbage and now it's quickly becoming useless.
- throwaway4aday 3y ago
  This is wrong for at least two reasons: 1) The data scraped from the web is filtered, deduped, ranked, and cleaned in a variety of other ways before being used for training. While quantity is necessary for a model to learn the structure of language, quality is even more important once you have a model that can produce coherent output so a lot of work goes into grooming the data. 2) There have been a bunch of papers and training runs that show synthetic data created specifically for training a model is as good or better than scraped human produced data. The importance of scraped web content is quickly declining because you can now generate infinite higher quality examples using existing trained models. The only relevance it has now is for knowledge about new developments and that is a much easier stream to filter since most of the important stuff comes from official sources and you don't need as many variations since you can just generate your own using one or more examples.

RecycledEle 3y ago

From the article: "consultants using AI were significantly more productive (they completed 12.2% more tasks on average, and completed task 25.1% more quickly), and produced significantly higher quality results (more than 40% higher quality compared to a control group)."

lgleason 3y ago

I was underwhelmed when using it for code generation of things like tests.

dragoncrab 3y ago

I don't want to be rude, but given the average quality of material produced by big consulting companies I have encountered over my career, I'm not that surprised that GPT4 beats them.

cleandreams 3y ago

I've been an engineer at FAANG. I think the problems consultants face are simpler than the ones confronting engineers who need to become expert at million line code bases. There are coding standards, styles, security practices. I think AI would help when boiler plate is needed but otherwise it would be a dicier proposition. But there is another issue that I think is long term. The apprenticeship of a top engineer is difficult. If the result of "AI shortcuts" is a less skilled engineer profession there will be considerable downsides in the future. I have read that the ability of people to navigate is negatively impacted by a reliance on google maps.

deleted 3y ago
[deleted]