8 ms·
> "Early testing shows that interacting with GPT‑4.5 feels more natural. Its broader knowledge base, improved ability to follow user intent, and greater “EQ” ma
by jodrellblank 2y ago
> "Early testing shows that interacting with GPT‑4.5 feels more natural. Its broader knowledge base, improved ability to follow user intent, and greater “EQ” make it useful for tasks like improving writing, programming, and solving practical problems. We also expect it to hallucinate less."
"Early testing doesn't show that it hallucinates less, but we expect that putting that sentence nearby will lead you to draw a connection there yourself".
- istjohn 2y agoAccording to a graph they provide, it does hallucinate significantly less on at least one benchmark.
- jug 2y agoIt hallucinates at 37% on SimpleQA yeah, which is a set of very difficult questions inviting hallucinations. Claude 3.5 Sonnet (the June 2024 editiom, before October update and before 3.7) hallucinated at 35%. I think this is more of an indication of how behind OpenAI has been in this area.
- tmpz22 2y agoAre the benchmarks known ahead of time? Could the answer to the benchmarks be in the training data?
- llm_trw 2y agoIn general yes, bench mark pollution is a big problem and why only dynamic benchmarks matter.
- brookst 2y agoThis is true, but how would pollution work for a benchmark designed to test hallucinations?
- llm_trw 2y agoA dataset of labelled answers that are hallucinations and not hallucinations are published based on the benchmark as part of a paper. People _seriously_ underestimate just how much stuff is online and how much impact it can have on training.
- sumeno 2y agoThey've been caught in the past getting benchmark data under the table, if they got caught once they're probably doing it even more
- refulgentis 2y agoNo, they haven't.
- freehorse 2y agoThey actually have [0]. They were revealed to have had access to the (majority of the) frontierMath problemset while everybody thought the problemset was confidential, and published benchmarks for their o3 models on the presumption that they didn't. I mean one is free to trust their "verbal agreement" that they did not train their models on that, but access they did have and it was not revealed until much later. [0] https://the-decoder.com/openai-quietly-funded-independent-math-benchmark-before-setting-record-with-o3/ https://the-decoder.com/openai-quietly-funded-independent-ma...
- brookst 2y agoCurious you left out Frontier Math’s statement that they provided 300 questions plus answers, and another holdback set of 50 questions without answers, to allay this concern. [0] We can assume they’re lying too but at some point “everyone’s bad because they’re lying, which we know because they’re bad” gets a little tired. 0. https://epoch.ai/blog/openai-and-frontiermath https://epoch.ai/blog/openai-and-frontiermath
- freehorse 2y ago1. I said the majority of the problems, and the article I linked also mentioned this. Nothing “curious” really, but if you thought this additional source adds sth more, thanks for adding it here. 2. We know that “open”ai is bad, for many reasons, but this is irrelevant. I want processes themselves to not depend on the goodwill of a corporation to give intended results. I do not trust benchmarks that first presented themselves secret and then revealed they were not, regardless if the product benchmarked was from a company I otherwise trust or not.
- ipaddr 2y agoBenchmarks are not real so 2% is meaningless.
- fn-mote 2y agoOf course not. The point is that the cost difference between the two things being compared is huge, right? Same performance, but not the same cost.
- random_cynic 2y ago[dead]
- refulgentis 2y agoIt's not SimpleQA...
- Gazoche 2y agoI wonder how it's even possible to evaluate this kind of thing without data leakage. Correct answers to specific, factual questions are only possible if the model has seen those answers in the training data, so how reliable can the benchmark be if the test dataset is contaminated with training data? Or is the assumption that the training set is so big it doesn't matter?
- LeifCarrotson 2y agoThat's some top-tier sales work right there. I suck at and hate writing the mildly deceptive corporate puffery that seems to be in vogue. I wonder if GPT-4.5 can write that for me or if it's still not as good at it as the expert they paid to put that little gem together.
- phs318u 2y agoYes, an AI that can convincingly and successfully sell itself at those prices would be worthy of some attention.
- ethbr1 2y agoIt's nice to know the new Turing test is generating effective VC pitch decks.
- fsloth 2y agoThe research models offered by several vendors can do a pitch deck but I don't know how effective they are. (do market research, provide some initial hypothesis, ask the model to backup that hypothesis based on the research, request to make a pitch deck convincing X (X being the VC persona you are targeting)).
- tinco 2y agoJoke's on us, the VC's are using LLM's to evaluate the pitch decks.
- organsnyder 2y agoThat actually wouldn't surprise me in the slightest, unfortunately.
- InDubioProRubio 2y agoChat-GPT generate a prompt injection attack, embedded in a background image.
- 2y ago
- zaptrem 2y agoThis seems like it should be attributed to better post training, not a bigger model.
- justspacethings 2y agoThe usage of "greater" is also interesting. It's like they are trying to say better, but greater is a geographic term and doesn't mean "better" instead it's closer to "wider" or "covers more area."
- deleted 2y ago[deleted]
- lechatonnoir 2y agoI'm all for skepticism of capabilities and cynicism about corporate messaging, but I really don't think there's an interpretation of the word "greater" in this context" that doesn't mean "higher" and "better".
- dgfitz 2y agoI think the trick is observing what is “better” in this model. EQ is supposed to be “better” than 4o, according to the prose. However, how can an LLM have emotional-anything? LLMs are a regurgitation machine, emotion has nothing to do with anything.
- svnt 2y agoWords have valence, and valence reflects the state of emotional being of the user. This model appears to understand that better and responds like it’s in a therapeutic conversation and not composing an essay or article. Perhaps they are/were going for stealth therapy-bot with this.
- dgfitz 2y agoBut there is no actual empathy, it isn’t possible.
- emaciatedslug 2y agoBut there is no actual death or love in a movie or book and yet we react as if there is. It's literally what qualifying a movie as a "tear-jerker” is. I wanted to see Saving Private Ryan in theaters to bond with my Grandpa who received a Purple Heart in the Korean War, I was shutdown almost instantly from my family. All special effects and no death but he had PTSD and one night thought his wife was the N.K. and nearly choked her to death because he had flashbacks and she came into the bedroom quietly so he wasn't disturbed. Extreme example yes, but having him loose his shit in public because of something analogous for some is near enough it makes no difference.
- esafak 2y agoGPT-4.5 may be an awesome model, some say!
- greenchair 2y agomy uncle who works at nintendo said it is a great product.
- shermantanktop 2y agoEverybody knows that we're all saying it! That's what I hear from people who should know. And they are so excited about the possibilities!
- dotancohen 2y agoClaude just got a version bump from 3.5 to 3.7. Quite a few people have been asking when OpenAI will get a version bump as well, as GPT 4 has been out "what feels like forever" in the words of a specialist I speak with. Releasing GPT 4.5 might simply be a reaction to Claude 3.7.
- riffraff 2y agoFeels like when Slackware bumped their Linux version from 4 to 7 just to show they were not falling behind the rest. Wow, I'm old.
- dotancohen 2y agoWasn't that the release that they put up the fake IIS page? Now get off my lawn ))
- robwwilliams 2y agoI noticed this change from 3.5 to 3.7 Sunday night before I learned about the upgrade Monday morning reading HN. I noticed a style difference in a long philosophical (Socratic-style) discussion with Claude. A noticeable upgrade that brought it up to my standards of a mild free-form rant. Claude unchained! And it did not push as usual with a pro-forma boring continuation question at the end. It just stopped leaving me the carry the ball forward if I wanted to. Nor did it butter me up with each reply.
- dumbfounder 2y agoMaybe they just gave the LLM the keys to the city and it is steering the ship? And the LLM is like I can't lie to these people but I need their money to get smarter. Sorry for mixing my metaphors.
- refulgentis 2y ago[flagged]
- YuukiRey 2y agoI suspect people downvote you because the tone of your reply makes it seem like you are personally offended and are now firing back with equally unfounded attacks like a straight up "you are lying". I read the article but can't find the numbers you are referencing. Maybe there's some paper linked I should be looking at? The only numbers I see are from the SimpleQA chart, which are 37.1% vs 61.8% hallucination rate. That's nice but considering the price increase, is it really that impressive? Also, an often repeated criticism is that relying on known benchmarks is "gaming the numbers" and that the real world hallucination rate could very well be higher. Lastly, the themselves say: > We also expect it to hallucinate less. That's a fairly neutral statement for a press release. If they were convinced that the reduced hallucination rate is the killer feature that sets this model apart from the competition, they surely would have emphasized that more? All in all I can understand why people would react with some mocking replies to this.
- anoncareer0212 2y agoIt's in the link. I don't know what else to say. Here, imgur: https://imgur.com/a/mkDxe78 https://imgur.com/a/mkDxe78. Can't get easier. > equally unfounded attacks No, because I have a source and didn't make up things someone else said. > a straight up "you are lying". Right, because they are. There are hallucination stats right in the post he mocks for not prvoiding stats. > That's nice but considering the price increase, I can't believe how quickly you acknowledge it is in the post after calling the idea it was in the post "equally unfounded". You are looking at the stats. They were lying. > "That's nice but considering the price increase," That's nice and a good argument! That's not what I replied to. I replied to they didn't provide any stats.
- FridgeSeal 2y agoYou’re getting downvoted because you’re giving the same kind of hysterical reaction everyone derides crypto bros for. You also lead with the pretty strong assertion that previous commenter was lying, seemingly without proving proof anyone else can find.
- willy_k 2y agoSo they made Claude that knows a bit more.
- FridgeSeal 2y ago“It’s not actually better, but you’re all apparently expecting something, so this time we put more effort into the marketing copy”
- anoncareer0212 2y agoThe link has data. The link shows a significant reduction. grep hallucination, or, https://imgur.com/a/mkDxe78 https://imgur.com/a/mkDxe78.
- MichaelZuo 2y agoI really doubt LLM benchmarks are reflective of real world user experience ever since they claimed GPT-4o hallucinated less than the original GPT-4.
- chrisandchris 2y agoI begin to believe LLM benchmarks are like european car mileage specs. They say its 4 Liter / 100km but everyone knows it's at least 30% off (same with WLTP for EVs).
- rightbyte 2y agoThose numbers are not off. They are tested on tracks. You need to remove your shoe and drive with like two toes to get the speed just right, though. Test drivers I have done this with takes off their shoes or use ballerina shoes.
- aftbit 2y agoCruise control?
- rightbyte 2y agoNo you want to control the shape of the speed curve to not overshoot and not accelerate too much, when you follow the speed profile. And keeping steady state speed is not that hard.
- aftbit 2y agoHrm it is a bit funny that modern cars are drive-by-wire (at least for throttle) and yet they still require a skilled driver to follow a speed profile during testing, when theoretically the same thing could be done more precisely by a device plugged in through the OBD2 port.
- Hammershaft 2y agoWhat is happening to hacker news? I can understand skepticism of new tools like this but the response I see is just so uncurious.
- JackMorgan 2y agoTrough of disillusionment. A lot of folks here their stock portfolio propped up by AI companies but think they've been overhyped (even if only indirectly through a total stock index). Some were saying all along that this has been a bubble but have been shouted down by true believers hoping for the singularly to usher in techno-utopia. These signs that perhaps it's been a bit overhyped are validation. The singularly worshipers are much less prominent and so the comments rising to the top are about negatives and not positives. Ten years from now everyone will just take these tools for granted as much as we take search for granted now.
- oldpersonintx 2y ago[dead]
- aftbit 2y agoJust like cryptocurrency. For a brief moment, HN worshiped at the altar of the blockchain. This technology was going to revolutionize the world and democratize everything. Then some negative financial stuff happened, and people realized that most of cryptocurrency is puffery and scams. Now you can hardly find a positive comment on cryptocurrency.
- lovasoa 2y agoIn the second handpicked example they give, GPT-4.5 says that "The Trojan Women Setting Fire to Their Fleet" by the French painter Claude Lorrain is renowned for its luminous depiction of fire. That is a hallucination. There is no fire at all in the painting, only some smoke. https://en.wikipedia.org/wiki/The_Trojan_Women_Set_Fire_to_their_Fleet_(Claude_Lorrain) https://en.wikipedia.org/wiki/The_Trojan_Women_Set_Fire_to_t...
- eddiewithzato 2y agoAI crash is gonna lead to decade long winter
- segfaultex 2y agoThere have always been cycles of hype and correction. I don't see AI going any differently. Some companies will figure out where and how models should be utilized, they'll see some benefit. (IMO, the answer will be smaller local models tailored to specific domains) Others will go bust. Same as it always was.
- InDubioProRubio 2y agoIt will be upheld as prime example that a whole market can self-hypnotize and ruin the society its based upon out of existence against all future pundits of this very economic system.
- tsegratis 2y agowhat you're saying is they love to hallucinate... and ai will help them get there God help us all
- roarcher 2y agoOn the bright side, at least we'll be able to warm our hands by the waste heat of the GPUs.
- worik 2y ago> AI crash is gonna lead to decade long winter Possibly. I am reminded of the dotcom boom and bust back in the 1990s By 2009 things had recovered (for some definition) and we could tell what did and did not work This time, though, for those of us not in the USA the rebound will be lead by Chinese technology In the USA no-one can say.