2 ms·

Why do you think judgement is impossible to automate? What aspects of it do you think make it hard?

by 0c3ca83 10d ago

noir_lord 10d ago
It requires general intelligence and we don't even have a good understanding of how our's works or a particularly good way of quantifying it. The counter argument is of course maybe you don't need to understand our kind of intelligence to create a different kind and that could well be true but then how do you determine if a system is intelligent. Unless the new system is intelligent enough to reason with us on our level in a way we can "see" is intelligent it becomes a philosophical argument. We also have a natural inclination towards anthropomorphising systems that mirror us, this is already a problem with LLM's and people overestimating their capabilities or forming actual emotional attachment. Then there are those of us who know more about how they work who in theory should be more immune to that and aren't. I added some stuff to my agent.md to make it sound less human and to communicate more like the machine be it is because I find the faked emotion extremely jarring. It can't be sorry, it's a set of numbers, it sits in the linguistic uncanny valley.
- Analemma_ 10d ago
  Why do you think it requires general intelligence? The parent and I aren't being obtuse here: the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later. See Go, driving, art (you can complain about the quality of AI art, but it's winning competitions with human judges), etc. A priori I'm not sure why you would think being a PM at a FAANG, deciding what color the login button should be, is any different.
  - pixl97 10d ago
    Hence my saying. We will never prove AI is intelligence. We'll only prove humans are not.
    - Analemma_ 10d ago
      I mean I could believe that there are some tasks which can't be done at acceptable performance until AI resembles something like Commander Data, I just don't think being a software PM is one of them.
  - thinkharderdev 10d ago
    I'm not entirely sure "general" intelligence is a meaningful concept but I take judgement to mean something like "the ability to make decisions under uncertainty without direct feedback." People seem to be able to do this using a lot of context and tacit knowledge. It's not really clear how you hill-climb to that capability through RL. > the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later True, but the history of AI research is also littered with AI researchers confidently predicting X job will be replaced by AI and being completely wrong because they don't actually understand what those jobs actually are. See Geoff Hinton predicting that Radiologists would be obsolete by ~2020, or predictions that truck drivers would all be replaced by self-driving tech.
  - devonkim 9d ago
    I think AI wins against people when we start using KPIs in an industrial view of software like defects / LOC, decomposability, etc), or possibly even maintainability / understandability of a codebase. Gosh, even since the 1980s expert systems were already out-performing entire doctors' boards in diagnosing issues in patients so clearly technical performance in KPIs isn't the only measure by which we adopt technology in society. When it comes to judgment calls for technical decisions, a lot of interesting innovations appear to come from rejecting conventions / averages in favor of a different set of constraints as a trade-off because we challenge the assumptions we make about the demands being asked of a solution / product. I'm thinking in the constellation of the apocryphal Steve Jobs quote about rejecting asking horse riders what they want because if we asked them they'd ask for a more reliable horse.
vector_spaces 10d ago
Judgments are not generally impossible to automate -- judgements are typically binary or quantifiable interpretations, so in some sense are perfect targets for automation, but the sheer volume of judgements needed to build something coherent is overly cumbersome to specify to the point of being intractable. There are also many hidden judgements, ones where the thresholds may not be well understood, and interactions between them. But humans still manage to wrangle these, sometimes seemingly effortlessly, through a process which we call by shorthand "taste". This is a largely vibes-based heuristic that combines expertise with life experience and cultural training -- intuition, more or less. This is likely not possible to automate either -- aspects of it may be automatable for a given expert, in small pieces in narrow subsets of their particular domains of interest, but even those likely will require some manual intervention. This is in part because it is, to a large degree, a black box, even to the expert deploying it. With some self-awareness and strong language skills we can articulate approximations of the judgements that go into taste. But even those will fall short, as even the most self-aware individual will fail to notice certain judgements and dependencies. In practice many of these are not even explicitly articulable. Humans are idiosyncratic and messy and dynamic, and the suggestion that we can build a machine that approximates this in a way that pleases our sensibilities and doesn't require supervision is kind of ludicrous, even in light of recent developments.
- 0c3ca83 10d ago
  Taste is just a set of statistically expressible heuristics for what other people will think is good. I don't undestand why you think this is impossible for AI to do. It feels like today, taste in design is similar to where software engineering was about a year ago.
- visarga 10d ago
  You don't need to describe it; just show samples of the style you want to achieve. Of course it's not perfect, but it's easier than describing it. I have my own theory about why it's impossible to remove the human from the loop: 1. Any task emerges from a need, from a human context. We need the human to pay and assume the risks and costs of using the model. So intent emerges from context. 2. While the task is being worked on, constant interaction with the context is needed, for action, for feedback, and for steering. 3. At the end of a task, consequences accumulate in the context, they don't fly to the model provider. The cost, risk, liability, gains and losses remain there. So the LLM is great except for the start, middle and end of a task. Contexts are humans, teams, projects, and they are maximally distributed, you can't copy a context, it is indexical and relational, just as you can't copy my phone number or eat for me.
d0mine 10d ago
At the moment, the hurdle is [missing] context [aka shared model/understanding of the business, common sense]. Though it is not hard to imagine that any and all communications being recorded for AI consumption in the future. The Next one is the relative lack of prompt feedback (expect the blowup in finite time like Navier-Stokes ;) [there is not much feedback even for humans at middle management positions]. The cost [tokens] might become prohibitive unless LLMs improve further [not a guarantee].
jmcqk6 10d ago
Not the OP, but it's because the value has got to ultimately be recognized by humans. You can make llms perform judgement, and maybe that will get you some progress. But ultimately the value is going to come from engaging with other humans.
freejazz 9d ago
I think right here you're demonstrating that judgment doesn't come easily

Why do you think judgement is impossible to automate? What aspects of it do you think make it hard?

by 0c3ca83 10d ago

Why do you think judgement is impossible to automate? What aspects of it do you think make it hard?

It requires general intelligence and we don't even have a good understanding of how our's works or a particularly good way of quantifying it. The counter argument is of course maybe you don't need to understand our kind of intelligence to create a different kind and that could well be true but then how do you determine if a system is intelligent. Unless the new system is intelligent enough to reason with us on our level in a way we can "see" is intelligent it becomes a philosophical argument. We also have a natural inclination towards anthropomorphising systems that mirror us, this is already a problem with LLM's and people overestimating their capabilities or forming actual emotional attachment. Then there are those of us who know more about how they work who in theory should be more immune to that and aren't. I added some stuff to my agent.md to make it sound less human and to communicate more like the machine be it is because I find the faked emotion extremely jarring. It can't be sorry, it's a set of numbers, it sits in the linguistic uncanny valley.

Analemma_ 10d ago
Why do you think it requires general intelligence? The parent and I aren't being obtuse here: the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later. See Go, driving, art (you can complain about the quality of AI art, but it's winning competitions with human judges), etc. A priori I'm not sure why you would think being a PM at a FAANG, deciding what color the login button should be, is any different.
- pixl97 10d ago
  Hence my saying. We will never prove AI is intelligence. We'll only prove humans are not.
  - Analemma_ 10d ago
    I mean I could believe that there are some tasks which can't be done at acceptable performance until AI resembles something like Commander Data, I just don't think being a software PM is one of them.
- thinkharderdev 10d ago
  I'm not entirely sure "general" intelligence is a meaningful concept but I take judgement to mean something like "the ability to make decisions under uncertainty without direct feedback." People seem to be able to do this using a lot of context and tacit knowledge. It's not really clear how you hill-climb to that capability through RL. > the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later True, but the history of AI research is also littered with AI researchers confidently predicting X job will be replaced by AI and being completely wrong because they don't actually understand what those jobs actually are. See Geoff Hinton predicting that Radiologists would be obsolete by ~2020, or predictions that truck drivers would all be replaced by self-driving tech.
- devonkim 9d ago
  I think AI wins against people when we start using KPIs in an industrial view of software like defects / LOC, decomposability, etc), or possibly even maintainability / understandability of a codebase. Gosh, even since the 1980s expert systems were already out-performing entire doctors' boards in diagnosing issues in patients so clearly technical performance in KPIs isn't the only measure by which we adopt technology in society. When it comes to judgment calls for technical decisions, a lot of interesting innovations appear to come from rejecting conventions / averages in favor of a different set of constraints as a trade-off because we challenge the assumptions we make about the demands being asked of a solution / product. I'm thinking in the constellation of the apocryphal Steve Jobs quote about rejecting asking horse riders what they want because if we asked them they'd ask for a more reliable horse.

vector_spaces 10d ago

Judgments are not generally impossible to automate -- judgements are typically binary or quantifiable interpretations, so in some sense are perfect targets for automation, but the sheer volume of judgements needed to build something coherent is overly cumbersome to specify to the point of being intractable. There are also many hidden judgements, ones where the thresholds may not be well understood, and interactions between them. But humans still manage to wrangle these, sometimes seemingly effortlessly, through a process which we call by shorthand "taste". This is a largely vibes-based heuristic that combines expertise with life experience and cultural training -- intuition, more or less. This is likely not possible to automate either -- aspects of it may be automatable for a given expert, in small pieces in narrow subsets of their particular domains of interest, but even those likely will require some manual intervention. This is in part because it is, to a large degree, a black box, even to the expert deploying it. With some self-awareness and strong language skills we can articulate approximations of the judgements that go into taste. But even those will fall short, as even the most self-aware individual will fail to notice certain judgements and dependencies. In practice many of these are not even explicitly articulable. Humans are idiosyncratic and messy and dynamic, and the suggestion that we can build a machine that approximates this in a way that pleases our sensibilities and doesn't require supervision is kind of ludicrous, even in light of recent developments.

0c3ca83 10d ago
Taste is just a set of statistically expressible heuristics for what other people will think is good. I don't undestand why you think this is impossible for AI to do. It feels like today, taste in design is similar to where software engineering was about a year ago.
visarga 10d ago
You don't need to describe it; just show samples of the style you want to achieve. Of course it's not perfect, but it's easier than describing it. I have my own theory about why it's impossible to remove the human from the loop: 1. Any task emerges from a need, from a human context. We need the human to pay and assume the risks and costs of using the model. So intent emerges from context. 2. While the task is being worked on, constant interaction with the context is needed, for action, for feedback, and for steering. 3. At the end of a task, consequences accumulate in the context, they don't fly to the model provider. The cost, risk, liability, gains and losses remain there. So the LLM is great except for the start, middle and end of a task. Contexts are humans, teams, projects, and they are maximally distributed, you can't copy a context, it is indexical and relational, just as you can't copy my phone number or eat for me.

d0mine 10d ago

At the moment, the hurdle is [missing] context [aka shared model/understanding of the business, common sense]. Though it is not hard to imagine that any and all communications being recorded for AI consumption in the future. The Next one is the relative lack of prompt feedback (expect the blowup in finite time like Navier-Stokes ;) [there is not much feedback even for humans at middle management positions]. The cost [tokens] might become prohibitive unless LLMs improve further [not a guarantee].

jmcqk6 10d ago

Not the OP, but it's because the value has got to ultimately be recognized by humans. You can make llms perform judgement, and maybe that will get you some progress. But ultimately the value is going to come from engaging with other humans.

freejazz 9d ago

I think right here you're demonstrating that judgment doesn't come easily