4 ms·
Who cares about benchmarks? These things still cost me time because of hallucinations.
by croes 2y ago
Who cares about benchmarks?
These things still cost me time because of hallucinations.
- cheema33 2y agoHumans hallucinate as well. Benchmarks count.
- croes 2y agoBut with less power consumption.
- golergka 2y agoI'm willing to bet $100 that a human consumes at least 10x times energy than a latest LLama (picking the open source model so that it can be easily verified) to produce two pages of text. All of this "AI is destroying the environment and consuming too much power" is about total consumption, both training and inference. Inference itself is cheap and green.
- croes 2y agoYeah, too bad it’s about quality not quantity.
- idiotsecant 2y agoThose goalposts just keep sliding
- brulard 2y agoWhat percentage of humans produce high-quality text?
- gkbrk 2y agoLess than half, and that's being generous.
- golergka 2y agoWe can control for quality too, of you want. A lot of real life uses for Chatgpt are really trivial. I regularly ask it for some basic recipes based on my groceries and likes, quality is basically 100% hits so far.
- Mekoloto 2y agoIt doesn't matter if it costs time. It matters if it is better than what you have. If it breaks a cup but is 10x cheaper than a human, go figer
- ban-evader 2y agoYou’re a very poor user of LLMs if they’re not a net time saver for you.
- Cheer2171 2y agoSo the No True Scotsman fallacy of LLM productivity?
- gdhkgdhkvff 2y agoMost people do see productivity gains from using LLMs correctly. Myself included. Just because some people don’t learn how to use them correctly doesn’t mean LLMs aren’t helpful. It’s like when internet search came out and a handful of laggards tried it once, failed to get the exact perfect result, and declared “internet search is useless”. Using a tool wrong is not evidence of the tool being useless, it’s evidence that you need to learn how to use the tool.
- smeeger 2y agohallucinations are literally the finger in the dam. if these models could sense when an output is well-founded and simply say “i dont know” otherwise… say goodbye to your job
- gdhkgdhkvff 2y agoGoogling a question and finding an incorrect answer every now and then doesn’t mean that googling is useless. It means that you need to learn how to use google. Trust but verify. Use it for scenarios where you aren’t looking for it to be the trusted fact checker. It excels at brainstorming, not at fact giving.
- aprilthird2021 2y agoI agree with your last sentence strongly, but a lot of the benchmarks are based on factual accuracy