9 ms·

Workers with less experience gain the most from generative AI

153 points by diskmuncher 3y ago

teaearlgraycold 3y ago
This makes sense to me. When using an LLM for programming it’s the most useful when teaching me the basics of a new language or framework. For example, it’s very helpful for giving me introductory information on x86-64. But much less useful at writing recursive higher-kinded types in TypeScript.
- gibspaulding 3y ago
  Yep, it's been great for me for knocking together scripts in Python where I know there will be a way to do something like open a csv file and read it into a list of dicts, but am not familiar enough with the ecosystem yet to know exactly what packages might be useful or how to invoke them. I could work the same thing out from stack overflow, but it would take a lot longer.
  - ZiiS 3y ago
    But did it do you any favours? The increadably well written official python csv doc manages to suggest using iterators rather then loading what may eventually be too much data into a list, Points out that the are dialects of CSV and some common ones you might use. https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/csv.html
    - ehsanu1 3y ago
      If it works, and it's a one-off script, why do I care?
- deleted 3y ago
  [deleted]
outofpaper 3y ago
To be clear, this is about contact center agents and other jobs of a related level.
timetraveller26 3y ago
I guess I should change jobs to something I am not good at
camhart 3y ago
My experience with Bard/ChatGPT has been that you'll easily shoot your foot off if you don't know what you're doing. Dangerous for workers with less experience. I see claims that it makes you 7-8x more effective. That hasn't been my experience. Maybe 5-10% improvement at best. Instead of Googling myself, LLMs can sometimes give me the answer more quickly than I could find it. Once you ask it anything not easily scraped on the internet it hallucinates like crazy and sounds so confident about it. Maybe I'm just horrible at prompting, but I can't help but feel we're still N breakthroughs away from having it really impact dev jobs.
- skepticATX 3y ago
  The funny thing about the 10x claims (I've even seen folks say 100x) is that results like these would be readily apparent. Even a broad 1.5x productivity increase would be world changing. And yet, we clearly are not seeing this currently. I'm not saying it will never happen, just that it will be very obvious if it does. 5-10% sounds about right to me.
  - larve 3y ago
    The problem with these claims is that you can't really quantify a "10x" change. I have found a lot of emergent benefits to using LLMs. For example, because my cognitive load of wrangling APIs and understanding and refactoring legacy code and all the other nonsense of my day to day can be so heavily delegated, I actually feel refreshed at the end of the day, and can bang out a decently chunked feature on my opensource software on the couch (admittedly also boilerplate heavy code). This means that I went from 0 opensource commits to 4000 since chatgpt came out. Not just that, but I've gotten not only more adventurous, but have the time to consider doing drastic refactors and spend much more time thinking about my software. I won't call it 10x or 100x, because that wouldn't mean anything, but surely it is a paradigm shift for me, completely world changing.
    - pm_me_your_quan 3y ago
      I'd be really curious if you're willing to expand more on how it has helped with those workflows. Do you copy/paste chunks in and ask it to explain them? Have it try to refactor them and then clean up?
jdm2212 3y ago
Claims like this are so dumb. Call center work has a very low ceiling on skill, so obviously people who are maxed out can't benefit. But in any knowledge economy type job, there is no ceiling on skill. Creative, curious, intelligent people who are already high skill are also going to be the most effective at using a new and weird and tricky tool in interesting ways to develop their skills. Generative AI will make some old skill obsolete, but people who are generically good at picking up new skills will be much more effective at wielding it. Those are the same people who were good at picking up the old skills, too.
daniel-cussen 3y ago
[dead]
b112 3y ago
Lose the most, because they're going to learn less, and do less, themselves.
jonplackett 3y ago
This is exactly the opposite of my experience with it. Getting it to code, it can save me, a fairly experienced coder, quite a lot of time because it can do some boring things - boilerplate stuff, or things I can’t be bothered to look up. And I can tell if it’s done it right or not. I’ve seen people with less coding experience use it and just take its suggestions at face value with painful results and definite detriment to their learning. The best test I’ve seen for what it’s good at is things that are hard to do, but easy to check.
- adamwk 3y ago
  I think my best experience with chatgpt is when I was digging around a typescript library without knowing the language. it was very convenient to enter type signatures and learn what they mean. I rarely am doing work like this where it’s worth opening chatgpt though.
- achates 3y ago
  The study in the article was on call center workers so I wouldn't necessarily expect it to scale to more complex tasks.
- dclowd9901 3y ago
  I like to use it as a jumping off point for a big project. We’re migrating from server round trip navigation to client side navigation and there’s a lot of steps to think through. It helps me get past the “blank paper” phase much faster.
- AbrahamParangi 3y ago
  My experience is similar. It’s a multiplier. The more capable you already are, the more you can get out of it. Perhaps not surprising in that all tools work this way.
mdp2021 3y ago
As others are saying/hinting, workers with less ability to vet outputs may overlook crucial errors. Take for example this automated help that I found only minutes ago - and seems to prove that generators can be racist (...I am joking): https://ibb.co/v3VRTQQ https://ibb.co/v3VRTQQ
ignoramous 3y ago
It could be that folks who perhaps aren't too married to their current workflow are adopting to this newer paradigm faster? For example, for places where English isn't the first language, LLMs are widely used for content-based tasks (posts, emails, support), because waiting 10s for Bard or GPT to output something higher quality is worth their time. Perhaps these folks end up being more attuned than usual, more in harmony and sync with the LLMs? Speaking from experience, it took a while for me to move from vim to vscode, from Javascript to Typescript. We probably prefer our silos and feel uber comfortable in existing setups.
Barrin92 3y ago
it should be pretty obvious, it's true for devs as well. ChatGPT helped me greatly with getting started with Qt/Qml because I'd never used it before, it's fantastic for exploration. I've worked about a decade in Python and there it does little for me. In fact often code can look very unidiomatic. It makes sense, these models are large and generic and so whether you're ahead or below an average level of knowledge usually determines how much you learn from the output. When people claim that technology disadvantages workers it usually just reflects their prejudices about working people. So called knowledge workers like to think only they can leverage tech while the call center workers are fixed automatons. But in reality people at a lower skill ceiling have a much easier time leveling up. ChatGPT when it works well is basically a Young Lady's Illustrated Primer (https://en.wikipedia.org/wiki/The_Diamond_Age https://en.wikipedia.org/wiki/The_Diamond_Age). Giving underskilled people free access to information will always benefit them the most.
smcleod 3y ago
Using ML tooling can be incredibly useful for new and experienced folks, - but in different ways. New folks are more likely to ask questions they’d otherwise feel uncomfortable asking but are far more at risk of being lead down the wrong path and skipping over gaining a fundamental understanding of how things work . It’s a balance of risk vs reward. I’ve been in tech 18~ years and get a lot out of copilot and chatGPT every day.
awinter-py 3y ago
ah their test environment was a call center feels like there is a viewpoint floating around that chatgpt is valuable in part because of the RLHF work but does this makes it generally capable? or just prove that you can hire essentially the people you would hire to staff a call center (language skills + good generalists) to create a language model that behaves essentially like a call center (refuses to take any actions that are not allowed)
pfisherman 3y ago
My experience building AI powered tools for scientists has been the opposite. Other commenters have pointed out the difficulties users with limited domaine expertise having in parsing the output and distinguishing what is useful, novel, interesting from what is trivial or incorrect. This slows down the iterative process of learning how to use the tool as well as refining queries because they don’t have a big enough database of ground truth knowledge to calibrate against. That being said these can work well as educational tools to rapidly get non-experts to a point where they can participate in conversations with experts. In that regard one might say that the bulk of the value is in the non-expert users because they are far more prevalent than experts. The biggest advantage I saw for the expert researchers was that they knew (1) which big questions to ask, and (2) how to break down those big questions into smaller, more precise questions.
patrick451 3y ago
One big difference between googling and asking an LLM is that it tends to be much easier to judge the legitimacy of information from a google result through informal heuristics. This is because google just links to a website, and there are all sorts of other factors that you use to subconsciously rank how reliable information on a particular website is, that mostly have nothing to do with the raw information. For example, we all have different weights for how much trust information we get from, e.g, Wikipedia, stackoverflow, high profile bloggers in your domain, conspiracy websites, listacles, blog-spam sites etc, marketing materal etc. If the info is on a social site, is highly upvoted? Is the info from an academic journal, arxiv.org, an academic blog, or a wordoc you downloaded from scribd [1]? Yes, this is judging a book by it's cover, but it's a heuristic that tends to work well. By contrast, LLMs present all information to you with the same confidence in the same homogeneous interface. There is no external context. So all your normal heuristics for judging reliability are broken. - [1] https://xkcd.com/1301/ https://xkcd.com/1301/
throwawayswiss 3y ago
It can't even get regular expressions correct in my experience
- svnt 3y ago
  imo this is the most human thing about it.
- ehsanu1 3y ago
  I've had it fail to get a basic regex right, but simultaneously, found it able to model my custom problem using the z3 constraint solcing framework, which I'd have struggled to do myself. Regular expressions are probably harder for it to get right easily, at least not without prompting it to first think through step by step ehat each piece of a regex should be or something.
croes 3y ago
The question is if companies still hire less experienced people or if the experienced ones are equally productive without them because of ChatGPT.
famouswaffles 3y ago
These are results from a model deployed from 2020 to 2021. Early GPT-3 days. We've come a long way since then.
wigster 3y ago
i tried it with an old, less popular language, Progress 4GL, and it just invented pure bullshit answers. when i asked are you sure about that it owned up and and said sorry. wtf?
javajosh 3y ago
Your standard full-stack dev is going to have "less experience" with approximately 80% of her codebase, by dint of having a finite life-span. ChatGPT is a faster and more convenient SO in that (very common) case. I, for example, have used it with great effect writing bash scripts that I would not dare to write otherwise. It's also not bad at bashing out the first use of a new library. I mean, I have ended up throwing that code out 100% of the time, but it's still useful.
somishere 3y ago
Reading the comments here made me think of a dishwashing analogy. I grew up without a dishwasher (when we asked our parents why we didn't have one my dad responded, "what do you mean? We have six!" referring to us kids). As an adult we have one in the house but it is rarely used. Washing dishes is a menial task but I enjoy it. It's hard to articulate exactly why. Needless to say I'm quite an experienced dish washer, there's an art to it. My process is to collect any dishes that aren't already stacked, fill the sink with just enough water, wash everything well (cutlery in first, out second last, washed individually), then wipe down all the benches and sink. I leave the dishes to drip dry. They get put away later. When we use the washing machine I still have to do most of the steps above. Except the washing which is 40-120mins of free time for me. But then I have an extra step of checking each dish or piece of cutlery while I'm putting it away. Most things get washed well, but about 10-15% of items have food baked onto them that I then need to soak or rewash (with a more abrasive scrubber than I'd usually use). Maybe this is down to my lack of experience stacking? The whole process takes longer, but it's (arguably) lower touch and (I hear) uses less water. Which is better? In a commercial kitchen a dishwasher for sure, efficiency at all costs. But do we need to remove all menial tasks from our workflows? I'm not so sure.
- barrkel 3y ago
  If you have 10-15% of items with food stuck on them after a wash, you need either a better dishwasher, better detergent (there is a noticeable range in quality) or both.
  - somishere 3y ago
    It's quite a high quality dishwasher .. multi-drawer job. But I will look into the detergent :)
- harrisi 3y ago
  I've gone back and forth with dishwashers my whole life, spending years with followed by years without, etc. and I finally am team dishwasher. There are a few tips I've learned, such as immediately rinsing a dish after using (I don't bother scrubbing), running the hot water for a minute before starting, using the pre-wash detergent and shine stuff appropriately. I also don't bother entirely filling it up to run it - which may seem wasteful, but I'd imagine filling it completely and having to rewash some percent of them ends up being more wasteful. Also, just like getting a manual dishwashing routine figured out, dishwashers are all different and need to be learned like any tool. Understanding where pieces are, which cycles run when, which compartments open and close and even how they do it, are all important. It really is best to think of it as a tool. Some useful videos: https://youtu.be/_rBO8neWw04 https://youtu.be/_rBO8neWw04 https://youtu.be/Ll6-eGDpimU https://youtu.be/Ll6-eGDpimU
simonw 3y ago
"The co-authors found that contact center agents with access to a conversational assistant saw a 14% boost in productivity, with the largest gains impacting new or low-skilled workers. In other words, the workers were upskilled, not replaced, thanks to the technology." I don't think the paper matches the assertion made in the title of this article - it's talking about a very specific use-case for generative AI, which won't necessarily generalize to "in all cases, workers with less experience gain the most from these tools".
dclowd9901 3y ago
As a frontend dev of 12 years, I’ve been getting a lot of utility out of it helping me write Jenkins pipeline code. And learning a lot of groovy, Java and Jenkins patterns along the way.
siliconc0w 3y ago
Currently, it's basically entry level for anything mildly complicated and it can't troubleshoot itself, so it needs to be coached into giving the right answer. So it's great for boilerplate and API lookups but if you were using a typed language before and had a decent IDE, this was already a solved problem. It seems impressive because it's very fast but that advantage goes away when you then have to be like, "hmm, well that's not quite right" and then you coach it into being correct (or close enough that you stop). If your team can't autocomplete APIs, and if you can't run everything locally so you can step through your code with a debugger, and if you don't have good tests so you need an AI to tell when you broke something (see also profiling, telemetry, release quickly and confidently, etc) - you're going to get a million percent more mileage just doing these sort of basics correctly (and fast) vs investing in AI because the open secret is that most of engineering time is spent reading and debugging code rather than writing it. This should be table stakes but so many engineering teams (even at bigtech companies) can't do the above. So they're all signing up for copilot but they still iterate with logging statements and catch regressions in production.
hmage 3y ago
Lifehack, add this to the end of your question: Start your response with "Let's work this out in a step by step way to be sure we have the right answer:" and work this out in a step by step way to be sure you have the right answer It will improve your answers significantly (for example, on word math problems, solve rate went from 18% to 79%). Source: https://github.com/openai/openai-cookbook/blob/main/techniques_to_improve_reliability.md https://github.com/openai/openai-cookbook/blob/main/techniqu...
black_13 3y ago
[dead]
AbrahamParangi 3y ago
The title is misleading. The experience of call center workers cannot be generalized to all workers.
PeterStuer 3y ago
In my personal experience chatgpt-4 has been a great addition to the toolbox, both for coding assistance and research. You do have to have sufficient grounding in the subjects to be able to evaluate the responses critically though. However, in all cases I have seen a very substantial decline of the capabilities of chatgpt-4 with the last few releases. E.g. it used to get code snippets most often right before. Whereas now it tends to be wrong most of the time. Usually it conflates capabilities of several distict libraries, just hallucinating (extrapolating) some non existing functions or attributes. I personally suspect they are 'cleansing' the training data, and/or driving severe 'schizophrenia' into the model through conflicting RLHF.
vbezhenar 3y ago
My usage of ChatGPT is as follows: 1. Google replacement for simple queries. Like "how do I HTTP POST with go?". Saves like 30 seconds of browsing. 2. Google replacement for vague questions. Like "Which ingress controllers are popular and what's their strengths". 3. Procrastination eliminators. Like "I need an identifier for ..., can you suggest me some proper names?". 4. Boring tasks which could be solved with enough bash scrpipts or regexes, but with ChatGPT it's often faster. 5. Most impressive to me: "I want to format the following YAML according to the following rules: <vague imprecise rules>. Example: ....". This sometimes saves me really lots of time, because it acts like a human and able to follow imprecise instructions. I could either do it myself spending lots of time or write some formatter spending, again, lots of time. I was never able to utilize it for "smart" tasks, like if I couldn't solve something and it solved it for me. It's not smart yet. But it's an useful tool in my toolbelt, definitely worth $20 and saves me time and sanity for some boring tasks.
gloosx 3y ago
It is fascinating how far the article is away from the paper. This all looks set up just to market something. First of all, the paper only provides suggestive(!) evidence that customer support agents(!) work faster and better if AI is writing the text instead of them (what a breakthrough). The researchers seem to be affiliated with "Fortune 500 enterprise software company that provides AI-based customer service support software" and half of the paper is describing what is AI and how it helps.

153 points by diskmuncher 3y ago

This makes sense to me. When using an LLM for programming it’s the most useful when teaching me the basics of a new language or framework. For example, it’s very helpful for giving me introductory information on x86-64. But much less useful at writing recursive higher-kinded types in TypeScript.

gibspaulding 3y ago
Yep, it's been great for me for knocking together scripts in Python where I know there will be a way to do something like open a csv file and read it into a list of dicts, but am not familiar enough with the ecosystem yet to know exactly what packages might be useful or how to invoke them. I could work the same thing out from stack overflow, but it would take a lot longer.
- ZiiS 3y ago
  But did it do you any favours? The increadably well written official python csv doc manages to suggest using iterators rather then loading what may eventually be too much data into a list, Points out that the are dialects of CSV and some common ones you might use. https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/csv.html
  - ehsanu1 3y ago
    If it works, and it's a one-off script, why do I care?
deleted 3y ago
[deleted]

outofpaper 3y ago

To be clear, this is about contact center agents and other jobs of a related level.

timetraveller26 3y ago

I guess I should change jobs to something I am not good at

camhart 3y ago

My experience with Bard/ChatGPT has been that you'll easily shoot your foot off if you don't know what you're doing. Dangerous for workers with less experience. I see claims that it makes you 7-8x more effective. That hasn't been my experience. Maybe 5-10% improvement at best. Instead of Googling myself, LLMs can sometimes give me the answer more quickly than I could find it. Once you ask it anything not easily scraped on the internet it hallucinates like crazy and sounds so confident about it. Maybe I'm just horrible at prompting, but I can't help but feel we're still N breakthroughs away from having it really impact dev jobs.

skepticATX 3y ago
The funny thing about the 10x claims (I've even seen folks say 100x) is that results like these would be readily apparent. Even a broad 1.5x productivity increase would be world changing. And yet, we clearly are not seeing this currently. I'm not saying it will never happen, just that it will be very obvious if it does. 5-10% sounds about right to me.
- larve 3y ago
  The problem with these claims is that you can't really quantify a "10x" change. I have found a lot of emergent benefits to using LLMs. For example, because my cognitive load of wrangling APIs and understanding and refactoring legacy code and all the other nonsense of my day to day can be so heavily delegated, I actually feel refreshed at the end of the day, and can bang out a decently chunked feature on my opensource software on the couch (admittedly also boilerplate heavy code). This means that I went from 0 opensource commits to 4000 since chatgpt came out. Not just that, but I've gotten not only more adventurous, but have the time to consider doing drastic refactors and spend much more time thinking about my software. I won't call it 10x or 100x, because that wouldn't mean anything, but surely it is a paradigm shift for me, completely world changing.
  - pm_me_your_quan 3y ago
    I'd be really curious if you're willing to expand more on how it has helped with those workflows. Do you copy/paste chunks in and ask it to explain them? Have it try to refactor them and then clean up?

jdm2212 3y ago

Claims like this are so dumb. Call center work has a very low ceiling on skill, so obviously people who are maxed out can't benefit. But in any knowledge economy type job, there is no ceiling on skill. Creative, curious, intelligent people who are already high skill are also going to be the most effective at using a new and weird and tricky tool in interesting ways to develop their skills. Generative AI will make some old skill obsolete, but people who are generically good at picking up new skills will be much more effective at wielding it. Those are the same people who were good at picking up the old skills, too.

daniel-cussen 3y ago

[dead]

b112 3y ago

Lose the most, because they're going to learn less, and do less, themselves.

jonplackett 3y ago

This is exactly the opposite of my experience with it. Getting it to code, it can save me, a fairly experienced coder, quite a lot of time because it can do some boring things - boilerplate stuff, or things I can’t be bothered to look up. And I can tell if it’s done it right or not. I’ve seen people with less coding experience use it and just take its suggestions at face value with painful results and definite detriment to their learning. The best test I’ve seen for what it’s good at is things that are hard to do, but easy to check.

adamwk 3y ago
I think my best experience with chatgpt is when I was digging around a typescript library without knowing the language. it was very convenient to enter type signatures and learn what they mean. I rarely am doing work like this where it’s worth opening chatgpt though.
achates 3y ago
The study in the article was on call center workers so I wouldn't necessarily expect it to scale to more complex tasks.
dclowd9901 3y ago
I like to use it as a jumping off point for a big project. We’re migrating from server round trip navigation to client side navigation and there’s a lot of steps to think through. It helps me get past the “blank paper” phase much faster.
AbrahamParangi 3y ago
My experience is similar. It’s a multiplier. The more capable you already are, the more you can get out of it. Perhaps not surprising in that all tools work this way.

mdp2021 3y ago

As others are saying/hinting, workers with less ability to vet outputs may overlook crucial errors. Take for example this automated help that I found only minutes ago - and seems to prove that generators can be racist (...I am joking): https://ibb.co/v3VRTQQ https://ibb.co/v3VRTQQ

ignoramous 3y ago

It could be that folks who perhaps aren't too married to their current workflow are adopting to this newer paradigm faster? For example, for places where English isn't the first language, LLMs are widely used for content-based tasks (posts, emails, support), because waiting 10s for Bard or GPT to output something higher quality is worth their time. Perhaps these folks end up being more attuned than usual, more in harmony and sync with the LLMs? Speaking from experience, it took a while for me to move from vim to vscode, from Javascript to Typescript. We probably prefer our silos and feel uber comfortable in existing setups.

Barrin92 3y ago

it should be pretty obvious, it's true for devs as well. ChatGPT helped me greatly with getting started with Qt/Qml because I'd never used it before, it's fantastic for exploration. I've worked about a decade in Python and there it does little for me. In fact often code can look very unidiomatic. It makes sense, these models are large and generic and so whether you're ahead or below an average level of knowledge usually determines how much you learn from the output. When people claim that technology disadvantages workers it usually just reflects their prejudices about working people. So called knowledge workers like to think only they can leverage tech while the call center workers are fixed automatons. But in reality people at a lower skill ceiling have a much easier time leveling up. ChatGPT when it works well is basically a Young Lady's Illustrated Primer (https://en.wikipedia.org/wiki/The_Diamond_Age https://en.wikipedia.org/wiki/The_Diamond_Age). Giving underskilled people free access to information will always benefit them the most.

smcleod 3y ago

Using ML tooling can be incredibly useful for new and experienced folks, - but in different ways. New folks are more likely to ask questions they’d otherwise feel uncomfortable asking but are far more at risk of being lead down the wrong path and skipping over gaining a fundamental understanding of how things work . It’s a balance of risk vs reward. I’ve been in tech 18~ years and get a lot out of copilot and chatGPT every day.

awinter-py 3y ago

ah their test environment was a call center feels like there is a viewpoint floating around that chatgpt is valuable in part because of the RLHF work but does this makes it generally capable? or just prove that you can hire essentially the people you would hire to staff a call center (language skills + good generalists) to create a language model that behaves essentially like a call center (refuses to take any actions that are not allowed)

pfisherman 3y ago

My experience building AI powered tools for scientists has been the opposite. Other commenters have pointed out the difficulties users with limited domaine expertise having in parsing the output and distinguishing what is useful, novel, interesting from what is trivial or incorrect. This slows down the iterative process of learning how to use the tool as well as refining queries because they don’t have a big enough database of ground truth knowledge to calibrate against. That being said these can work well as educational tools to rapidly get non-experts to a point where they can participate in conversations with experts. In that regard one might say that the bulk of the value is in the non-expert users because they are far more prevalent than experts. The biggest advantage I saw for the expert researchers was that they knew (1) which big questions to ask, and (2) how to break down those big questions into smaller, more precise questions.

patrick451 3y ago

One big difference between googling and asking an LLM is that it tends to be much easier to judge the legitimacy of information from a google result through informal heuristics. This is because google just links to a website, and there are all sorts of other factors that you use to subconsciously rank how reliable information on a particular website is, that mostly have nothing to do with the raw information. For example, we all have different weights for how much trust information we get from, e.g, Wikipedia, stackoverflow, high profile bloggers in your domain, conspiracy websites, listacles, blog-spam sites etc, marketing materal etc. If the info is on a social site, is highly upvoted? Is the info from an academic journal, arxiv.org, an academic blog, or a wordoc you downloaded from scribd [1]? Yes, this is judging a book by it's cover, but it's a heuristic that tends to work well. By contrast, LLMs present all information to you with the same confidence in the same homogeneous interface. There is no external context. So all your normal heuristics for judging reliability are broken. - [1] https://xkcd.com/1301/ https://xkcd.com/1301/

throwawayswiss 3y ago

It can't even get regular expressions correct in my experience

svnt 3y ago
imo this is the most human thing about it.
ehsanu1 3y ago
I've had it fail to get a basic regex right, but simultaneously, found it able to model my custom problem using the z3 constraint solcing framework, which I'd have struggled to do myself. Regular expressions are probably harder for it to get right easily, at least not without prompting it to first think through step by step ehat each piece of a regex should be or something.

croes 3y ago

The question is if companies still hire less experienced people or if the experienced ones are equally productive without them because of ChatGPT.

famouswaffles 3y ago

These are results from a model deployed from 2020 to 2021. Early GPT-3 days. We've come a long way since then.

wigster 3y ago

i tried it with an old, less popular language, Progress 4GL, and it just invented pure bullshit answers. when i asked are you sure about that it owned up and and said sorry. wtf?

javajosh 3y ago

Your standard full-stack dev is going to have "less experience" with approximately 80% of her codebase, by dint of having a finite life-span. ChatGPT is a faster and more convenient SO in that (very common) case. I, for example, have used it with great effect writing bash scripts that I would not dare to write otherwise. It's also not bad at bashing out the first use of a new library. I mean, I have ended up throwing that code out 100% of the time, but it's still useful.

somishere 3y ago

Reading the comments here made me think of a dishwashing analogy. I grew up without a dishwasher (when we asked our parents why we didn't have one my dad responded, "what do you mean? We have six!" referring to us kids). As an adult we have one in the house but it is rarely used. Washing dishes is a menial task but I enjoy it. It's hard to articulate exactly why. Needless to say I'm quite an experienced dish washer, there's an art to it. My process is to collect any dishes that aren't already stacked, fill the sink with just enough water, wash everything well (cutlery in first, out second last, washed individually), then wipe down all the benches and sink. I leave the dishes to drip dry. They get put away later. When we use the washing machine I still have to do most of the steps above. Except the washing which is 40-120mins of free time for me. But then I have an extra step of checking each dish or piece of cutlery while I'm putting it away. Most things get washed well, but about 10-15% of items have food baked onto them that I then need to soak or rewash (with a more abrasive scrubber than I'd usually use). Maybe this is down to my lack of experience stacking? The whole process takes longer, but it's (arguably) lower touch and (I hear) uses less water. Which is better? In a commercial kitchen a dishwasher for sure, efficiency at all costs. But do we need to remove all menial tasks from our workflows? I'm not so sure.

barrkel 3y ago
If you have 10-15% of items with food stuck on them after a wash, you need either a better dishwasher, better detergent (there is a noticeable range in quality) or both.
- somishere 3y ago
  It's quite a high quality dishwasher .. multi-drawer job. But I will look into the detergent :)
harrisi 3y ago
I've gone back and forth with dishwashers my whole life, spending years with followed by years without, etc. and I finally am team dishwasher. There are a few tips I've learned, such as immediately rinsing a dish after using (I don't bother scrubbing), running the hot water for a minute before starting, using the pre-wash detergent and shine stuff appropriately. I also don't bother entirely filling it up to run it - which may seem wasteful, but I'd imagine filling it completely and having to rewash some percent of them ends up being more wasteful. Also, just like getting a manual dishwashing routine figured out, dishwashers are all different and need to be learned like any tool. Understanding where pieces are, which cycles run when, which compartments open and close and even how they do it, are all important. It really is best to think of it as a tool. Some useful videos: https://youtu.be/_rBO8neWw04 https://youtu.be/_rBO8neWw04 https://youtu.be/Ll6-eGDpimU https://youtu.be/Ll6-eGDpimU

simonw 3y ago

"The co-authors found that contact center agents with access to a conversational assistant saw a 14% boost in productivity, with the largest gains impacting new or low-skilled workers. In other words, the workers were upskilled, not replaced, thanks to the technology." I don't think the paper matches the assertion made in the title of this article - it's talking about a very specific use-case for generative AI, which won't necessarily generalize to "in all cases, workers with less experience gain the most from these tools".

dclowd9901 3y ago

As a frontend dev of 12 years, I’ve been getting a lot of utility out of it helping me write Jenkins pipeline code. And learning a lot of groovy, Java and Jenkins patterns along the way.

siliconc0w 3y ago

Currently, it's basically entry level for anything mildly complicated and it can't troubleshoot itself, so it needs to be coached into giving the right answer. So it's great for boilerplate and API lookups but if you were using a typed language before and had a decent IDE, this was already a solved problem. It seems impressive because it's very fast but that advantage goes away when you then have to be like, "hmm, well that's not quite right" and then you coach it into being correct (or close enough that you stop). If your team can't autocomplete APIs, and if you can't run everything locally so you can step through your code with a debugger, and if you don't have good tests so you need an AI to tell when you broke something (see also profiling, telemetry, release quickly and confidently, etc) - you're going to get a million percent more mileage just doing these sort of basics correctly (and fast) vs investing in AI because the open secret is that most of engineering time is spent reading and debugging code rather than writing it. This should be table stakes but so many engineering teams (even at bigtech companies) can't do the above. So they're all signing up for copilot but they still iterate with logging statements and catch regressions in production.

hmage 3y ago

Lifehack, add this to the end of your question: Start your response with "Let's work this out in a step by step way to be sure we have the right answer:" and work this out in a step by step way to be sure you have the right answer It will improve your answers significantly (for example, on word math problems, solve rate went from 18% to 79%). Source: https://github.com/openai/openai-cookbook/blob/main/techniques_to_improve_reliability.md https://github.com/openai/openai-cookbook/blob/main/techniqu...

black_13 3y ago

[dead]

AbrahamParangi 3y ago

The title is misleading. The experience of call center workers cannot be generalized to all workers.

PeterStuer 3y ago

In my personal experience chatgpt-4 has been a great addition to the toolbox, both for coding assistance and research. You do have to have sufficient grounding in the subjects to be able to evaluate the responses critically though. However, in all cases I have seen a very substantial decline of the capabilities of chatgpt-4 with the last few releases. E.g. it used to get code snippets most often right before. Whereas now it tends to be wrong most of the time. Usually it conflates capabilities of several distict libraries, just hallucinating (extrapolating) some non existing functions or attributes. I personally suspect they are 'cleansing' the training data, and/or driving severe 'schizophrenia' into the model through conflicting RLHF.

vbezhenar 3y ago

My usage of ChatGPT is as follows: 1. Google replacement for simple queries. Like "how do I HTTP POST with go?". Saves like 30 seconds of browsing. 2. Google replacement for vague questions. Like "Which ingress controllers are popular and what's their strengths". 3. Procrastination eliminators. Like "I need an identifier for ..., can you suggest me some proper names?". 4. Boring tasks which could be solved with enough bash scrpipts or regexes, but with ChatGPT it's often faster. 5. Most impressive to me: "I want to format the following YAML according to the following rules: <vague imprecise rules>. Example: ....". This sometimes saves me really lots of time, because it acts like a human and able to follow imprecise instructions. I could either do it myself spending lots of time or write some formatter spending, again, lots of time. I was never able to utilize it for "smart" tasks, like if I couldn't solve something and it solved it for me. It's not smart yet. But it's an useful tool in my toolbelt, definitely worth $20 and saves me time and sanity for some boring tasks.

gloosx 3y ago

It is fascinating how far the article is away from the paper. This all looks set up just to market something. First of all, the paper only provides suggestive(!) evidence that customer support agents(!) work faster and better if AI is writing the text instead of them (what a breakthrough). The researchers seem to be affiliated with "Fortune 500 enterprise software company that provides AI-based customer service support software" and half of the paper is describing what is AI and how it helps.