5 ms·
A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.
by reconnecting 7mo ago
A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.
- danielbln 7mo agoI don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.
- reconnecting 7mo agoThis is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result. Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.
- huflungdung 7mo ago[dead]
- orf 7mo agoIt’s exactly the same as A/B testing an interface. This is just testing 4 variants of a “page” (the plan), measuring how many people pressed “continue”.
- stavros 7mo agoYou've groupped LLMs into the wrong set. LLMs are closer to people than to machines. This argument is like saying "I want my tools to be reliable, like my light switch, and my personal assistant wasn't, so I fired him". Not to mention that of course everyone A/B tests their output the whole time. You've never seen (or implemented) an A/B test where the test was whether to improve the way e.g. the invoicing software generates PDFs?
- applfanboysbgon 7mo ago> LLMs are closer to people than to machines. jfc. I don't have anything to say to this other than that it deserves calling out. > You've never seen (or implemented) an A/B test where the test was whether to improve the way e.g. the invoicing software generates PDFs? I have never in my life seen or implemented an a/b test on a tool used by professionals. I see consumer-facing tests on websites all the time, but nothing silently changing the software on your computer. I mean, there are mandatory updates, which I do already consider to be malware, but those are, at least, not silent.
- johnisgood 7mo agoWhy are you calling it out? You are interpreting the statement too literally. The point is probably about behavior, not nature. LLMs do not always produce identical outputs for identical prompts, which already makes them less like deterministic machines and superficially closer to humans in interaction. That is it. The comparison can end here.
- applfanboysbgon 7mo agoThey actually can, though. The frontier model providers don't expose seeds, but for inferencing LLMs on your own hardware, you can set a specific seed for deterministic output and evaluate how small changes to the context change the output on that seed. This is like suggesting that Photoshop would be "more like a person than a machine" if they added a random factor every time you picked a color that changed the value you selected by +-20%, and didn't expose a way to lock it. "It uses a random number generator, therefore it's people" is a bit of a stretch.
- deleted 7mo ago[deleted]
- johnisgood 7mo agoYou are right, I was wrong. I think anthropomorphizing LLMs to begin with is kind of silly. The whole "LLMs are closer to people than to machines" comparison is misleading, especially when the argument comes down to output variability. Their outputs can vary in ways that superficially resemble human variability, but variability alone is a poor analogy for humanness. A more meaningful way to compare is to look at functional behaviors such as "pattern recognition", "contextual adaptation", "generalization to new prompts", and "multi-step reasoning". These behaviors resemble aspects of human capabilities. In particular, generalization allows LLMs to produce coherent outputs for tasks they were not explicitly trained on, rather than just repeating training data, making it a more meaningful measure than randomness alone. That said, none of this means LLMs are conscious, intentional, or actually understanding anything. I am glad you brought up the seed and determinism point. People should know that you can make outputs fully predictable, so the "human-like" label mostly only shows up under stochastic sampling. It is far more informative to look at real functional capabilities instead of just variability, and I think more people should be aware of this.
- doc_ick 7mo agoAs far as I can tell, llms never give the exact same output every time.
- johnisgood 7mo ago> same input, different result. What is your point? You get this from LLMs. It does not mean that it is not useful.
- freeone3000 7mo agoYes! And it was bad then too!! I want software that does a specific list of things, doesn’t change, and preferentially costs a known amount.
- NotGMan 7mo agoBy that definition humans are not professional since we hallucinate and make mistakes all the time.
- dkersten 7mo agoAnthropic have done a lot of things that would give me pause about trusting them in a professional context. They are anything but transparent, for example about the quota limits. Their vibe coded Claude code cli releases are a buggy mess too. Also the model quality inconsistency: before a new model release, there’s a week or two where their previous model is garbage. A/B testing is fine in itself, you need to learn about improvements somehow, but this seems to be A/B testing cost saving optimisations rather than to provide the user with a better experience. Less transparency is rarely good. This isn’t what I want from a professional tool. For business, we need consistency and reliability.
- r_lee 7mo ago> vibe coded Claude code cli releases are a buggy mess too this is what gets me. are they out of money? are so desperate to penny pinch that they can't just do it properly? what's going on in this industry?
- macNchz 7mo agoI’m a huge user of AI coding tools but I feel like there has been some kind of a zeitgeist shift in what is acceptable to release across the industry. Obviously it’s a time of incredibly rapid change and competition, but man there is some absolute garbage coming out of companies that I’d expect could do better without much effort. I find myself asking, like, did anyone even do 5 minutes of QA on this thing?? How has this major bug been around for so long? “It’s kind of broken, maybe they will fix it at some point,” has become a common theme across products from all different players, from both a software defect and service reliability point of view.
- r_lee 7mo agoI mean it's like, really they don't even need agentic AI or whatever, they could literally just employ devs and it wouldn't make a difference like, they'll drop $100 billion on compute, but when it comes to devs who make their products, all of a sudden they must desperately cut costs and hire as little as possible to me it makes no sense from a business perspective. Same with Google, e.g. YouTube is utterly broken, slow and laggy, but I guess because you're forced to use it, it doesn't matter. But still, if you have these huge money stockpiles, why not deploy it to improve things? It wouldn't matter anyways, it's only upside
- hrmtst93837 7mo ago[flagged]
- onion2k 7mo agoA professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof. The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use A/B tests on users to make Photoshop silently change the hue a user selects to be a little brighter, or Word could change the look of document titles, or a game could make enemies a bit stronger (fyi, this does actually happen - players get boosts on their first few rounds in online games to stop them being put off playing). The complaint is about A/B tests with no visible warnings, not AI.
- reconnecting 7mo agoThere's a distinction worth making here. A/B testing the interface button placement, hue of a UI element, title styling — is one thing. But you wouldn't accept Photoshop silently changing your #000000 to #333333 in the actual file. That's your output, not the UI around it. That's what LLMs do. The randomness isn't in the wrapper, it's in the result you take away.
- doc_ick 7mo agoIt’s an assistant, answering your question and running some errands for you. If you give it blind permission to do a task, then you’re not worrying about what it does.
- some_random 7mo agoThat's not what they're doing, they are trying to use plan mode to plan out a task. I don't know where you could have got the idea that they were blindly doing anything.
- doc_ick 7mo agoThe plan mode uses sub tasks to read, find, list, or grow out certain information. I would imagine those tasks would be covered by the dubiously discovered a/b testing
- WillAdams 7mo agoYeah, I've been using Copilot to process scans of invoices and checks (w/ a pen laid across the account information) converted to a PDF 20 at a time and it's pretty rare for it to get all 20, but it's sufficiently faster than opening them up in batches of 50 and re-saving using the Invoice ID and then using a .bat file to rename them (and remembering to quite Adobe Acrobat after each batch so that I don't run into the bug in it where it stops saving files after a couple of hundred have been so opened and re-saved).
- _heimdall 7mo agoLLMs are nondeterministic by design, but that has nothing to do with A/B testing.
- ordersofmag 7mo agoAny tool that auto-updates carries the implication that behavior will change over time. And one criteria for being a skilled professional is having expert understanding of ones tools. That includes understanding the strengths and weaknesses of the tools (including variability of output) and making appropriate choices as a result. If you don't feel you can produce professional code with LLM's then certainly you shouldn't use them. That doesn't mean others can't leverage LLM's as part of their process and produce professional results. Blindly accepting LLM output and vibe coding clearly doesn't consistently product professional results. But that's different than saying professionals can't use LLM in ways that are productive.
- johnisgood 7mo agoWell put. I would upvote this many times if I could.
- Mtinie 7mo agoWhat would you do differently if LLM outputs were deterministic? Perhaps I approach this from a different perspective than you do, so I’m interested to understand other viewpoints. I review everything that my models produce the same way I review work from my coworkers: Trust but verify.
- croes 7mo agoThat’s not a problem of LLMs but of using services provided by others. How often were features changed or deactivated by cloud services?