3 ms·

Finally some real pushback to the whole agentic mania - from an actor who is incentivized to push the narrative. Following the recent apple paper - some realism

Finally some real pushback to the whole agentic mania - from an actor who is incentivized to push the narrative. Following the recent apple paper - some realism is being injected into the hype. 58% success rate on a task is close to a coin flip. and 35% success rate on multiturn. >80% success rate on workflows could make that a reasonable usecase (eg, form filling) with some human supervision.

bigbuppo 1y ago
If it were an employee it would have been fired already, unless it were a nepo hire, and in someways, it is.
- sieabahlpark 1y ago
  [dead]
- onlyrealcuzzo 1y ago
  It might depend how much this employee costs. Your incentive to fire an employee who isn't great and costs $1 per day is much less than an incentive to fire one who isn't great and costs $1000 per day...
  - bigbuppo 1y ago
    There's a reason why I post the entire script to Bee Movie in every single AI-powered chat out there...
- AbstractH24 1y ago
  As someone who works in and around tons of nepo babies, this comment led to a lot of reflection on the similarities/differences. Wins the award for "deep comment of the week"
onlyrealcuzzo 1y ago
> 58% success rate on a task is close to a coin flip. Why does a single-step task imply a coinflip to you? There are more than two possible choices for an instruction like: "Lookup the status of order X".
- skywhopper 1y ago
  50% chance of being right is equivalent to a coin-flip.
  - onlyrealcuzzo 1y ago
    You don't have a 50% chance of being right rolling an N-sided weighted die.
    - lossolo 1y ago
      Regardless of what N is, if there's only one correct order status, you're left with just two choices: right or wrong.
      - onlyrealcuzzo 1y ago
        No, if there are 100 order statuses there are 99 wrong choices and 1 right choice. Additionally, the distribution of the choices is not guaranteed to be equal. If you assume equal distribution, you have a 1% chance of being right and a 99% chance of being wrong.
        lossolo 1y ago
        There is sample space (choices) so for example 100 different status labels and event space (how the system grades your choice), so right and wrong. My statement is true no matter how many choices are there, or how skewed the probabilities are. Your count of 99 incorrect labels is perfectly fine but it lives in sample space. Arguing that there are 99 incorrect answers doesn't refute that evaluation is binary. So counting 99 wrong labels tells us how many ways you can miss, but probability is assigned, not counted. Once a choice is made the system collapses everything to the two outcomes "correct" or "incorrect", and if the right label happens to have 50 % probability then the situation is mathematically identical to a coin flip, regardless of how many other labels sit on the die. Example with a weighted die and 99 incorrect answers: Die Faces: 100 Weights: Right status face = 0.50, the other 99 faces share the other 0.50 P(correct) = 0.50 -> exactly the coin-flip The 1/N rule only applies when all faces are equally likely, once you introduce weights, the number of faces no longer tells you the probability.
AbstractH24 1y ago
What is their incentive to share this data? I’m not really understanding They’ve leaned so hard into AI and agentforce that it doesn’t make sense to shoot themselves in the foot. Except that Hubspot, their main competitor on the SMB/MM/startup side recently announced a deep integration with ChatGPT. Still seems like a shot in the foot in an effort to undercut a growing competitor in a part of the market that theyd be better off exiting.

bigbuppo 1y ago

If it were an employee it would have been fired already, unless it were a nepo hire, and in someways, it is.

sieabahlpark 1y ago
[dead]
onlyrealcuzzo 1y ago
It might depend how much this employee costs. Your incentive to fire an employee who isn't great and costs $1 per day is much less than an incentive to fire one who isn't great and costs $1000 per day...
- bigbuppo 1y ago
  There's a reason why I post the entire script to Bee Movie in every single AI-powered chat out there...
AbstractH24 1y ago
As someone who works in and around tons of nepo babies, this comment led to a lot of reflection on the similarities/differences. Wins the award for "deep comment of the week"

onlyrealcuzzo 1y ago

> 58% success rate on a task is close to a coin flip. Why does a single-step task imply a coinflip to you? There are more than two possible choices for an instruction like: "Lookup the status of order X".

skywhopper 1y ago
50% chance of being right is equivalent to a coin-flip.
- onlyrealcuzzo 1y ago
  You don't have a 50% chance of being right rolling an N-sided weighted die.
  - lossolo 1y ago
    Regardless of what N is, if there's only one correct order status, you're left with just two choices: right or wrong.
    - onlyrealcuzzo 1y ago
      No, if there are 100 order statuses there are 99 wrong choices and 1 right choice. Additionally, the distribution of the choices is not guaranteed to be equal. If you assume equal distribution, you have a 1% chance of being right and a 99% chance of being wrong.
      - lossolo 1y ago
        There is sample space (choices) so for example 100 different status labels and event space (how the system grades your choice), so right and wrong. My statement is true no matter how many choices are there, or how skewed the probabilities are. Your count of 99 incorrect labels is perfectly fine but it lives in sample space. Arguing that there are 99 incorrect answers doesn't refute that evaluation is binary. So counting 99 wrong labels tells us how many ways you can miss, but probability is assigned, not counted. Once a choice is made the system collapses everything to the two outcomes "correct" or "incorrect", and if the right label happens to have 50 % probability then the situation is mathematically identical to a coin flip, regardless of how many other labels sit on the die. Example with a weighted die and 99 incorrect answers: Die Faces: 100 Weights: Right status face = 0.50, the other 99 faces share the other 0.50 P(correct) = 0.50 -> exactly the coin-flip The 1/N rule only applies when all faces are equally likely, once you introduce weights, the number of faces no longer tells you the probability.

AbstractH24 1y ago

What is their incentive to share this data? I’m not really understanding They’ve leaned so hard into AI and agentforce that it doesn’t make sense to shoot themselves in the foot. Except that Hubspot, their main competitor on the SMB/MM/startup side recently announced a deep integration with ChatGPT. Still seems like a shot in the foot in an effort to undercut a growing competitor in a part of the market that theyd be better off exiting.