7 ms·

If you don't want to read the full thing, Mollick's substack has more accessible summary of this paper: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-o

If you don't want to read the full thing, Mollick's substack has more accessible summary of this paper: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the... There are two sides to this thing. The first author of this paper has also written another paper titles "Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment". Abstract: "As AI quality increases, humans have fewer incentives to exert effort and remain attentive, allowing the AI to substitute, rather than augment their performance... I found that subjects with higher quality AI were less accurate in their assessments of job applications than subjects with lower quality AI. On average, recruiters receiving lower quality AI exerted more effort and spent more time evaluating the resumes, and were less likely to automatically select the AI-recommended candidate. The recruiters collaborating with low-quality AI learned to interact better with their assigned AI and improved their performance. Crucially, these effects were driven by more experienced recruiters. Overall, the results show that maximizing human/AI performance may require lower quality AI, depending on the effort, learning, and skillset of the humans involved." https://static1.squarespace.com/static/604b23e38c22a96e9c78879e/t/62d5d9448d061f7327e8a7e7/1658181956291/Falling+Asleep+at+the+Wheel+-+Fabrizio+DellAcqua.pdf https://static1.squarespace.com/static/604b23e38c22a96e9c788...

esafak 3y ago
If you don't need a skill, you won't practice it, and it will atrophy. This frees you to learn different things. Like how most software engineers today don't know computer engineering (hardware) but they can write more complex applications.
- SkyBelow 3y ago
  That is if the AI fully replaces the need for a skill. What happens if it mostly replaces it, enough so the skill atrophies, but not enough that the user never needs the skill?
  - esafak 3y ago
    Good point. That sounds like a temporary product problem, like a level 2-3 self-driving car, which still needs you to pay attention. I think you also need to consider whether you can afford not to have the skill; what would happen if the AI were taken away, or malfunctioned? Airplane pilots are an example of this. In that case you simply have to learn the skill as if the AI will not be there.
    - hn_throwaway_99 3y ago
      > That sounds like a temporary product problem The jury is still out on how "temporary" of a problem that will be. Improvements have been made but nobody really knows how to get an LLM, for example, to just say "I don't know".
      - Retric 3y ago
        And this is before people start poisoning the well. Google search worked better in a world without Google search. I suspect many AI systems will run into similar issues.
        hn_throwaway_99 3y ago
        > Google search worked better in a world without Google search. Oooh, I really like that. Very succinct and topical statement of Goodhart's Law.
  - deleted 3y ago
    [deleted]
  - taway_6PplYu5 3y ago
    Oh, you mean like self-driving cars?
- vlovich123 3y ago
  Depends on what you mean by computer engineering but they should definitely have a good mental model of how computers work. Otherwise your complex applications will have trouble scaling. That can matter less in many cases but early architecture mistakes can easily put you down an evolutionary dead end forcing you to experience exponentially rising maintenance costs to add new features / fix bugs or rewrite the thing by someone who understands computer architecture better (+ take advantage of a better understanding the problem domain)
  - WJW 3y ago
    I see what you mean but there's a huge gap between a "good enough" mental model of how computers work and the actual in-depth details that GP means and that have been successfully abstracted away from us. For example: the amount of cases when I have needed to know about the details of how the transistors in RAM chips are laid out have been extremely minimal. Nor have I ever had to worry about the encoding scheme used by the transceivers for the optical fibers that my TCP packets move across. Obviously there jobs out there that do have to worry about these things (mostly at companies like TSMC and Cisco), but in practice I doubt regular engineers at even the largest scale software companies have to worry about stuff like that.
- hn_throwaway_99 3y ago
  But that is the exact problem with AI - 99% of the time it's good, so your skills atrophy, but then there's that one percent of the time it decides the reflective side of that 18-wheeler is the sun and drives you into it. That's very different from, say, a compiler, which as a programmer I completely trust to turn my source code into machine code accurately.
  - tikhonj 3y ago
    > completely trust to turn my source code into machine code accurately Say hello to undefined behavior :) Appropriately enough, much of the reason undefined behavior is a consistent problem is exactly the same—it can do something reasonable 99% of the time and then totally screw you over out of nowhere.
    - Jensson 3y ago
      Undefined behavior is a defined thing, you know about it so you can avoid it if you want, or look up what the compiler you use does in that case to see if it is useful to you. No such thing for a black box AI model.
  - BeetleB 3y ago
    You're just mirroring the parent example. Prior to AI (or even computers), this was a problem. I used to be a more conventional engineering guy. We do a ton of calculus to get our degree. My first job - you just didn't need calculus (although all the associated courses in school used it to develop the theory). As a result, the one offs (less than once a year), when a problem did come up where calculus knowledge was helpful, they couldn't do the task and would go to that one guy (me) who still remembered calculus. BTW, I guarantee all the employees there got A's on their calculus courses.
    - hn_throwaway_99 3y ago
      > You're just mirroring the parent example. Prior to AI (or even computers), this was a problem. Not at all, and your argument isn't the same thing. The problem with current AI, which is very different from past forms of automation, is that, essentially, its failure mode is undefined. Past forms of automation, for example, did not "hallucinate" seemingly correct but false answers. Even taking your own example, it was obvious to your colleagues that they didn't know how to get the answer. So they did the correct thing - they went to someone who knew how to do it. What they did not do, and which is what most current AI solutions will do, is just "wing it" with a solution that is wrong but looks correct to those without more expertise.
- warner25 3y ago
  This a nice, tight statement of the big picture. This isn't new, like since LLMs were born, either. In the aviation community, and probably many other domains, there has been a running debate for decades about the pros and cons of introducing more automation into the cockpit and what still should or shouldn't be taught and practiced in training (to be able to effectively recognize and handle the edge cases where things fail). And in military aviation, this isn't just a question of safety; it's about productivity just like it is for knowledge workers. If an aircraft can fly itself most of the time, the crew can do more tactical decision-making and use more sensors and weapons. Instead of just flying their own aircraft, the crew can also control a number of unmanned aircraft with which they're teamed up.
digging 3y ago
While this is great information, I wish more could be said about Centaurs & Cyborgs, the title of the piece. There's a brief example of a type of task that each approach excels at, but I'd definitely like more. The crux of this piece is that our human judgment of when/how to use AI tools is the most important factor in work quality when we're on the edge of the frontier. But there's no analysis of whether centaurs or cyborgs did better at those outside-the-frontier tasks; it's not clear why those categories are even mentioned since they appear to have no relevance to the preceding research results. And as the article mentions, the frontier is "invisible" (or at least difficult to see); learning to detect tasks that are in, on, or outside of the frontier seems like an immensely important skill. (I also realize it may become completely obsolete in <5 years as the frontier expands exponentially.) I understand the goal of this research was not to find these "edges" and to determine how we can improve our judgment about when & how to use AI. But after reading these strong results, that's definitely the only thing I'm interested in. I use AI almost not at all in my work, web development. It has been most useful to me in getting me unstuck from a thorny under-documented problem. Over a year after ChatGPT released, I still don't know if modern LLMs can actually be a force multiplier for my work or if I'm correctly judging that they're not appropriate for the majority of my tasks. The latter seems increasingly unlikely as capabilities advance.
dang 3y ago
We've changed the URL from that to https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321. Thanks! (In cases like this, it's nice to have both the readable summary and the link to the paper.)
SlightlyLeftPad 3y ago
I think this a fascinating, yet unsurprising finding. We’ve known for a long time that automation leads to skillset atrophy on the part of the user. Perhaps most famously, in aviation where some automated systems make complicated procedures so easy that pilots effectively lose the skills to do the proper procedure themselves. The discussions I hope are happening is the prospect of intentional capability limits of AI in critical industries/areas where humans must absolutely and always intervene. It seems analogous to having a superpower without knowing how to control it, creating a potentially deadly situation at best. Then perhaps more long term at the generational scale, will humans cognitively devolve to a point where AI is making all of our decisions for us? Essentially we make ourselves obsolete and eventually we’re reclassified as “regular/inferior” species of animals. Profit and return on investments here cannot get in the way of handling these details with great oversight and responsibly.
- ethbr1 3y ago
  That opines interestingly on only allowing full self driving cars on pre-approved, high-assurance routes. It either needs to be 100% or (bad). Anywhere in the grey zone, and you retard human performance while still occasionally needing it.
  - SlightlyLeftPad 3y ago
    This is a particularly relevant use-case for sure. I think we will see a huge skillset disparity between those of us who grew up driving a regular car versus the incoming generations who will grow up with cars that drive themselves. Whether that ends up being a big deal or not, it’s hard to say.
- supriyo-biswas 3y ago
  For some more reading material on this subject, see Ironies of Automation[1]. [1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...

esafak 3y ago

If you don't need a skill, you won't practice it, and it will atrophy. This frees you to learn different things. Like how most software engineers today don't know computer engineering (hardware) but they can write more complex applications.

SkyBelow 3y ago
That is if the AI fully replaces the need for a skill. What happens if it mostly replaces it, enough so the skill atrophies, but not enough that the user never needs the skill?
- esafak 3y ago
  Good point. That sounds like a temporary product problem, like a level 2-3 self-driving car, which still needs you to pay attention. I think you also need to consider whether you can afford not to have the skill; what would happen if the AI were taken away, or malfunctioned? Airplane pilots are an example of this. In that case you simply have to learn the skill as if the AI will not be there.
  - hn_throwaway_99 3y ago
    > That sounds like a temporary product problem The jury is still out on how "temporary" of a problem that will be. Improvements have been made but nobody really knows how to get an LLM, for example, to just say "I don't know".
    - Retric 3y ago
      And this is before people start poisoning the well. Google search worked better in a world without Google search. I suspect many AI systems will run into similar issues.
      - hn_throwaway_99 3y ago
        > Google search worked better in a world without Google search. Oooh, I really like that. Very succinct and topical statement of Goodhart's Law.
- deleted 3y ago
  [deleted]
- taway_6PplYu5 3y ago
  Oh, you mean like self-driving cars?
vlovich123 3y ago
Depends on what you mean by computer engineering but they should definitely have a good mental model of how computers work. Otherwise your complex applications will have trouble scaling. That can matter less in many cases but early architecture mistakes can easily put you down an evolutionary dead end forcing you to experience exponentially rising maintenance costs to add new features / fix bugs or rewrite the thing by someone who understands computer architecture better (+ take advantage of a better understanding the problem domain)
- WJW 3y ago
  I see what you mean but there's a huge gap between a "good enough" mental model of how computers work and the actual in-depth details that GP means and that have been successfully abstracted away from us. For example: the amount of cases when I have needed to know about the details of how the transistors in RAM chips are laid out have been extremely minimal. Nor have I ever had to worry about the encoding scheme used by the transceivers for the optical fibers that my TCP packets move across. Obviously there jobs out there that do have to worry about these things (mostly at companies like TSMC and Cisco), but in practice I doubt regular engineers at even the largest scale software companies have to worry about stuff like that.
hn_throwaway_99 3y ago
But that is the exact problem with AI - 99% of the time it's good, so your skills atrophy, but then there's that one percent of the time it decides the reflective side of that 18-wheeler is the sun and drives you into it. That's very different from, say, a compiler, which as a programmer I completely trust to turn my source code into machine code accurately.
- tikhonj 3y ago
  > completely trust to turn my source code into machine code accurately Say hello to undefined behavior :) Appropriately enough, much of the reason undefined behavior is a consistent problem is exactly the same—it can do something reasonable 99% of the time and then totally screw you over out of nowhere.
  - Jensson 3y ago
    Undefined behavior is a defined thing, you know about it so you can avoid it if you want, or look up what the compiler you use does in that case to see if it is useful to you. No such thing for a black box AI model.
- BeetleB 3y ago
  You're just mirroring the parent example. Prior to AI (or even computers), this was a problem. I used to be a more conventional engineering guy. We do a ton of calculus to get our degree. My first job - you just didn't need calculus (although all the associated courses in school used it to develop the theory). As a result, the one offs (less than once a year), when a problem did come up where calculus knowledge was helpful, they couldn't do the task and would go to that one guy (me) who still remembered calculus. BTW, I guarantee all the employees there got A's on their calculus courses.
  - hn_throwaway_99 3y ago
    > You're just mirroring the parent example. Prior to AI (or even computers), this was a problem. Not at all, and your argument isn't the same thing. The problem with current AI, which is very different from past forms of automation, is that, essentially, its failure mode is undefined. Past forms of automation, for example, did not "hallucinate" seemingly correct but false answers. Even taking your own example, it was obvious to your colleagues that they didn't know how to get the answer. So they did the correct thing - they went to someone who knew how to do it. What they did not do, and which is what most current AI solutions will do, is just "wing it" with a solution that is wrong but looks correct to those without more expertise.
warner25 3y ago
This a nice, tight statement of the big picture. This isn't new, like since LLMs were born, either. In the aviation community, and probably many other domains, there has been a running debate for decades about the pros and cons of introducing more automation into the cockpit and what still should or shouldn't be taught and practiced in training (to be able to effectively recognize and handle the edge cases where things fail). And in military aviation, this isn't just a question of safety; it's about productivity just like it is for knowledge workers. If an aircraft can fly itself most of the time, the crew can do more tactical decision-making and use more sensors and weapons. Instead of just flying their own aircraft, the crew can also control a number of unmanned aircraft with which they're teamed up.

digging 3y ago

While this is great information, I wish more could be said about Centaurs & Cyborgs, the title of the piece. There's a brief example of a type of task that each approach excels at, but I'd definitely like more. The crux of this piece is that our human judgment of when/how to use AI tools is the most important factor in work quality when we're on the edge of the frontier. But there's no analysis of whether centaurs or cyborgs did better at those outside-the-frontier tasks; it's not clear why those categories are even mentioned since they appear to have no relevance to the preceding research results. And as the article mentions, the frontier is "invisible" (or at least difficult to see); learning to detect tasks that are in, on, or outside of the frontier seems like an immensely important skill. (I also realize it may become completely obsolete in <5 years as the frontier expands exponentially.) I understand the goal of this research was not to find these "edges" and to determine how we can improve our judgment about when & how to use AI. But after reading these strong results, that's definitely the only thing I'm interested in. I use AI almost not at all in my work, web development. It has been most useful to me in getting me unstuck from a thorny under-documented problem. Over a year after ChatGPT released, I still don't know if modern LLMs can actually be a force multiplier for my work or if I'm correctly judging that they're not appropriate for the majority of my tasks. The latter seems increasingly unlikely as capabilities advance.

dang 3y ago

We've changed the URL from that to https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321. Thanks! (In cases like this, it's nice to have both the readable summary and the link to the paper.)

SlightlyLeftPad 3y ago

I think this a fascinating, yet unsurprising finding. We’ve known for a long time that automation leads to skillset atrophy on the part of the user. Perhaps most famously, in aviation where some automated systems make complicated procedures so easy that pilots effectively lose the skills to do the proper procedure themselves. The discussions I hope are happening is the prospect of intentional capability limits of AI in critical industries/areas where humans must absolutely and always intervene. It seems analogous to having a superpower without knowing how to control it, creating a potentially deadly situation at best. Then perhaps more long term at the generational scale, will humans cognitively devolve to a point where AI is making all of our decisions for us? Essentially we make ourselves obsolete and eventually we’re reclassified as “regular/inferior” species of animals. Profit and return on investments here cannot get in the way of handling these details with great oversight and responsibly.

ethbr1 3y ago
That opines interestingly on only allowing full self driving cars on pre-approved, high-assurance routes. It either needs to be 100% or (bad). Anywhere in the grey zone, and you retard human performance while still occasionally needing it.
- SlightlyLeftPad 3y ago
  This is a particularly relevant use-case for sure. I think we will see a huge skillset disparity between those of us who grew up driving a regular car versus the incoming generations who will grow up with cars that drive themselves. Whether that ends up being a big deal or not, it’s hard to say.
supriyo-biswas 3y ago
For some more reading material on this subject, see Ironies of Automation[1]. [1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...