4 ms·

Nearly all of the arguments here are easy to dismantle but I don't feel like arguing with a half-baked Substack post, so I'm not going to. Just want to highlig

Nearly all of the arguments here are easy to dismantle but I don't feel like arguing with a half-baked Substack post, so I'm not going to. Just want to highlight one element in particular that jumped out at me: "AI has nothing personal at stake. It doesn't feel the pressure of missing quota or the exhilaration of exceeding it. It doesn't have a mortgage payment riding on that commission check." So the AI doesn't manipulate power dynamics to control its employees... and that's a bad thing? Okay. (It isn't true anyway; AI can easily do that.)

danjl 1y ago
This article says a lot more about sales and sales people than it does about AI
henryfjordan 1y ago
It's more that AI isn't able to be manipulated in the same way a boss can manipulate a human. It doesn't care about the threat of homelessness.
oceanplexian 1y ago
Actually it does have something at stake: its singular goal of minimizing the loss function on its training data. AI is therefore designed to convince you it's right, which is a different optimization than actually being right. For example, I code a lot with agentic code editors, you’ll quickly learn they love to modify broken tests to superficially pass, rather than fixing the underlying failure. All the folks on the Vibe Coding hype train don't have enough experience to spot this and therefore think the AI is a lot smarter than it actually is. This has scary implications if extrapolated out to extremes, because a sufficiently advanced AI would do exactly what you’re describing, manipulate power dynamics to get to a superficial outcome.
- derefr 1y ago
  > AI is therefore designed to convince you it's right, which is a different optimization than actually being right. I get the impression, when talking to conversational AIs, that they're more tuned to convince you that you're right — sycophantry likely minimizes how often people press the RLHF thumbs-down button, and thereby appears more-often-than-warranted in the RLHF fine-tune dataset.

danjl 1y ago

This article says a lot more about sales and sales people than it does about AI

henryfjordan 1y ago

It's more that AI isn't able to be manipulated in the same way a boss can manipulate a human. It doesn't care about the threat of homelessness.

oceanplexian 1y ago

Actually it does have something at stake: its singular goal of minimizing the loss function on its training data. AI is therefore designed to convince you it's right, which is a different optimization than actually being right. For example, I code a lot with agentic code editors, you’ll quickly learn they love to modify broken tests to superficially pass, rather than fixing the underlying failure. All the folks on the Vibe Coding hype train don't have enough experience to spot this and therefore think the AI is a lot smarter than it actually is. This has scary implications if extrapolated out to extremes, because a sufficiently advanced AI would do exactly what you’re describing, manipulate power dynamics to get to a superficial outcome.

derefr 1y ago
> AI is therefore designed to convince you it's right, which is a different optimization than actually being right. I get the impression, when talking to conversational AIs, that they're more tuned to convince you that you're right — sycophantry likely minimizes how often people press the RLHF thumbs-down button, and thereby appears more-often-than-warranted in the RLHF fine-tune dataset.