4 ms·

> Why would a carpentry shop buy hundreds of thousands of dollars of power tools without consulting with their employees to see what they actually need to get t

> Why would a carpentry shop buy hundreds of thousands of dollars of power tools without consulting with their employees to see what they actually need to get their job done more effectively? The logic of buying the tools then forcing the employees to use them "or else" is completely backwards in any sane world. For one, software tools are cheap, especially with OSS in the mix. You're buying one "tool" and paying for operational expenses that scale with total usage across all company. But secondly, and more importantly, the "consulting" and discussing was done over the period of last 3 years, by ~1 year ago the high-level conclusions were pretty much locked in, the worthiness of the adoption was blindingly obvious at that point, so I can see why tokenmaxxing would be where this ended up, even though (here I disagree with the article a bit) the tools aren't at the "compounding correctness" stage just yet. It's really quite simple: the stick didn't work (telling people in increasingly direct ways to try using AI for stuff), so they tried the carrot. $deity knows a good chunk of engineers will inadvertently fall for any trick that involves a scoreboard. That holds even when they're perfectly aware they're being tricked. > If you want to see if the tools work, why don't you just ask your employees? Like any normal employer would? Again, they did that, they've been doing it continuously over past 3 years. Some people are excited, some people don't care, but some - a population that's definitely overrepresented in HN comments - just stubbornly refuse to try. Now that the answers are in, and they speak in favor of AI, the companies are doing what "any normal employer would": trying to get the stubborn employers to do their job they way their bosses want them to. (In fact, normal employers would be more eager to fire people who keep refusing top-down instructions - but it's also obvious this technology is experimental; the models and harnesses get more powerful faster than people can learn to use them - so carrots make more sense than sticks in this transition period. Stubborn people begrudgingly using those tools offer an entirely unique perspective and explore use cases and approaches that you won't get from excited adopters.)

oblio 4mo ago
> the worthiness of the adoption was blindingly obvious at that point Everything is so "blindingly obvious" yet nobody can point to ANY serious peer reviewed studies that prove it. I'm patient, I'll wait.
- TeMPOraL 4mo ago
  You don't need peer reviewed studies to tell you water is wet. Peer review is a technique to get evidence from data when SNR is low. It's not "science", it's just a technique. So is "throwing shit at a wall and seeing what sticks". Don't turn techniques into rituals, and science into religion.
  - walt_grata 4mo ago
    Vibes are not evidence, neither is a curated demo. You need actual measured evidence that has an adversarial review to actually prove something without falling to confirmation bias.
    - TeMPOraL 4mo ago
      Proof is not binary, it depends on the claim and the constraints you put around it, and the nature of the subject of your claim. Most of the general LLM discourse in our industry is still closer to "proof of the pudding is in the eating" than to "double-blind studies on large cohorts, p<0.01, effect size is still so small that result is useless in practice"[0]. And we're not talking about curated demos either - most of the contested value can be proven for your own specific cases with little to no expenditure of money and time, at a PoC level (it gets more expensive once you try to operationalize it and find kinks that are hard to iron out). And that is, the article claims (and I agree), the point of last 6-12 months of tokenmaxxing policies and top-down push - it's putting pressure on people to actually go and do those PoC-s for themselves, because just giving the opportunity and permission turned out insufficient for significant part of the workforce. -- [0] - Ironically, I remember it was the opposite around the time GPT-4 came out. Back then people talked more about specific claims and demanded measured evidence, because it was hard to get the models to reliably do something interesting. But now that the models can handle bad prompting and can understand you even when you're drunk, suddenly people are denying the general capability of LLMs and asking for randomized control trials. (For double irony, nowadays one can just ask an LLM for randomized trials; the current SOTA models will happily design you a bespoke eval pipeline if you ask them to.)

oblio 4mo ago

> the worthiness of the adoption was blindingly obvious at that point Everything is so "blindingly obvious" yet nobody can point to ANY serious peer reviewed studies that prove it. I'm patient, I'll wait.

TeMPOraL 4mo ago
You don't need peer reviewed studies to tell you water is wet. Peer review is a technique to get evidence from data when SNR is low. It's not "science", it's just a technique. So is "throwing shit at a wall and seeing what sticks". Don't turn techniques into rituals, and science into religion.
- walt_grata 4mo ago
  Vibes are not evidence, neither is a curated demo. You need actual measured evidence that has an adversarial review to actually prove something without falling to confirmation bias.
  - TeMPOraL 4mo ago
    Proof is not binary, it depends on the claim and the constraints you put around it, and the nature of the subject of your claim. Most of the general LLM discourse in our industry is still closer to "proof of the pudding is in the eating" than to "double-blind studies on large cohorts, p<0.01, effect size is still so small that result is useless in practice"[0]. And we're not talking about curated demos either - most of the contested value can be proven for your own specific cases with little to no expenditure of money and time, at a PoC level (it gets more expensive once you try to operationalize it and find kinks that are hard to iron out). And that is, the article claims (and I agree), the point of last 6-12 months of tokenmaxxing policies and top-down push - it's putting pressure on people to actually go and do those PoC-s for themselves, because just giving the opportunity and permission turned out insufficient for significant part of the workforce. -- [0] - Ironically, I remember it was the opposite around the time GPT-4 came out. Back then people talked more about specific claims and demanded measured evidence, because it was hard to get the models to reliably do something interesting. But now that the models can handle bad prompting and can understand you even when you're drunk, suddenly people are denying the general capability of LLMs and asking for randomized control trials. (For double irony, nowadays one can just ask an LLM for randomized trials; the current SOTA models will happily design you a bespoke eval pipeline if you ask them to.)