3 ms·

[flagged]

by panarky 11mo ago

[flagged]

Xss3 11mo ago
This comment was written by an AI specifically instructed to be more concise than usual.
thejarren 11mo ago
[flagged]
- sindriava 11mo ago
  You seem very comfortable making unfounded claims. I don't think this is very constructive or adds much to the discussion. While we can debate the stylistic changes of the previous commenter, you seem to be discounting the rate at which the writing style of various LLMs has backpropagated into many peoples' brains.
  - roncesvalles 11mo ago
    Also discounting the fact that people actually do talk like that. In fact, these days I have to modify my prose to be intentionally less LLM-like lest the reader thinks it's LLM output.
  - ewoodrich 11mo ago
    I can sympathize with being mistakingly accused of using LLM output, but as a reader the above format of "Its not x - it's y" repeated multiple times for artificial dramatic emphasis to make a pretty mundane point that could use 1/3 the length grates on me like reading LinkedIn or marketing voice whether it's AI or not (and it's almost always AI anyway). I've seen fairly niche subreddits go from enjoyable and interesting to ruined by being clogged with LLM spam that sounds exactly like this so my tolerance for reading it is incredibly low, especially on HN, and I'll just dismiss it. I probably lose the occasionally legitimate original observation now and then but in a world where our attention is being hijacked by zero effort spam everywhere you look I just don't have the time or energy to avoid that heuristic.
- panarky 11mo ago
  Hmmm, I wrote those words myself, maybe I've spent too much time with LLMs and now I'm talking like them?? I'd be interested in any evidence-based arguments you might have beyond attacking my writing style and insinuating bad intent. I found this commenter had sage advice about how to use HN well, I try to follow it: https://news.ycombinator.com/item?id=38944467 https://news.ycombinator.com/item?id=38944467
  - momojo 11mo ago
    You mentioned "step change" twice. Maybe a once over next time? My favorite Mark Twain quote is (very paraphrased) "My apologies, had I more time, I would have written a shorter letter".
    - versteegen 11mo ago
      I thought the repetition was intentional.
  - thejarren 11mo ago
    I’ll take you at your word, sorry for the incorrect callout. Your comment format appeared malicious, so my response wasn’t an attempt at being “snarky”, just acting defensively. I like the HN Rules/Guidelines.
  - Aeolun 11mo ago
    “If you gaze long into an abyss, the abyss also gazes into you.”
    - gnabgib 11mo ago
      Is that you Nietzsche? Or are you Magog https://andromeda.fandom.com/wiki/Spirit_of_the_Abyss https://andromeda.fandom.com/wiki/Spirit_of_the_Abyss
      - deleted 11mo ago
        [deleted]
- ineedasername 11mo ago
  1) Models learn these patterns from common human usage. They are in the wild, and as such there will be people who use them naturally. 2) Now, given its for-some-reason-ubiquitous choice by models, it is also a phrasing that many more people are exposed to, every day. Language is contagious. This phrasing is approaching herd levels, meaning models trained from up-to-the-moment web content will start to see it as less distinctly salient. Eventually, there will be some other high-signal novel phrase with high salience, and the attention heads will latch on to it from the surrounding context, and then that will be the new AI shibboleth. It's just how language works. We see it in the mixes between generations when our kids pick up new lingo, and then it stops being in-group for them when it spreads too far.. Skibidi, 6 7, etc. It's just how language works, and a generation ago the internet put it on steroids. Now? Even faster.
legel 11mo ago
Thanks for reporting these metrics and drawing the conclusion of an underlying breakthrough in search. In his Nobel Prize winning speech, Demis Hassabis ends by discussing how he sees all of intelligence as a big tree-like search process. https://youtube.com/watch?v=YtPaZsasmNA&t=1218 https://youtube.com/watch?v=YtPaZsasmNA&t=1218
- derangedHorse 11mo ago
  The one thing I got out of the MIT OpenCourseWare AI course by Patrick Winston was that all of AI could be framed as a problem of search. Interesting to see Demis echo that here.
vitorgrs 11mo ago
I usually ask a simple question that ALL the models get wrong: List of mayor of my city [Londrina]. ALL the models (offine) get wrong. And I mean, all the models. The best that I could, it's o3 I believe, saying it couldn't give a good answer for that, and told to access the city website. Gemini 3 somehow is able to give a list of mayors, including details on who got impeached, etc. This should be a simple answer, because all the data is on wikipedia, that certainly the models are trained on, but somehow most models don't manage to give that answer right, because... it's just a irrelevant city in a huge dataset. But somehow, Gemini 3 did it. Edit: Just asked "Cool places to visit in Londrina" (In portuguese), and it was also 99% right, unlike other models, who just create stuff. The only thing wrong here, it mentioned sakuras in a lake... Maybe it confused with Brazilian ipês, which are similar, and indeed the city it's full of them. It seems to have a visual understanding, imo.
- arach 11mo ago
  thanks for sharing, very interesting example
- guiambros 11mo ago
  Ha, I just did the same with my hometown (Guaiba, RS), a city that is 1/6th of Londrina, and its wikipedia page in English hasn't been updated in years, and still has the wrong mayor (!). Gemini 3 nailed on the first try, included political affiliation, and added some context on who they competed with and won over in each of the last 3 elections. And I just did a fun application with AI Studio, and it worked on first shot. Pretty impressive. (disclaimer: Googler, but no affiliation with Gemini team)
- usef- 11mo ago
  Pure fact-based, niche questions like that aren't really the focus of most providers any more from what I've heard, since they can be solved more reliably by integrating search tools (and all providers now have search). I wouldn't be surprised if the smallest models can answer fewer such (fact-only) questions over time offline as they distill/focus them more thoroughly on logic etc.
- belter 11mo ago
rafaelmn 11mo ago
It tells me that the benchmark is probably leaking into training data, and going to the benchmark site : > Model was published after the competition date, making contamination possible. Aside from eval on most of these benchmarks being stupid most of the time, these guys have every incentive to cheat - these aren't some academic AI labs, they have to justify hundreds of billions being spent/allocated in the market. Actually trying the model on a few of my daily tasks and reading the reasoning traces all I'm seeing is same old, same old - Claude is still better at "getting" the problem.
Der_Einzige 11mo ago
You clearly AI generated this comment.
thomasahle 11mo ago
From my understanding, Google put online the largest RL cluster in the world not so long ago. It's not surprising they do really well on things that are "easy" to RL, like math or SimpleQA
red75prime 11mo ago
> To succeed this well in math, you can't just do better probabilistic generation, you need verifiable search. You say "probabilistic generation" like it's some kind of a limitation. What is exactly the limiting factor here? [(0.9999, "4"), (0.00001, "four"), ...] is a valid probability distribution. The sampler can be set to always choose "4" in such cases.
coldtea 11mo ago
>This is not an incremental advance. It is a step change. This indicates a new discovery, not just more data or more compute. To succeed this well in math, you can't just do better probabilistic generation, you need verifiable search. You need to verify what you're doing, detect when you make a mistake, and backtrack to try a different approach. Loos like AI slop
- TaupeRanger 11mo ago
  It obviously is.
calmoo 11mo ago
Your comment is AI generated
- tim333 11mo ago
  I'll give you the style is like an LLM but the thoughts seem a bit unlike one. I mean the MathArena Apex results indicating a new discovery rather than more data is definitely a hypothesis. Also panarky denies it.
dbbk 11mo ago
Aren't you just describing tool calls?

[flagged]

by panarky 11mo ago

[flagged]

Xss3 11mo ago

This comment was written by an AI specifically instructed to be more concise than usual.

thejarren 11mo ago

[flagged]

sindriava 11mo ago
You seem very comfortable making unfounded claims. I don't think this is very constructive or adds much to the discussion. While we can debate the stylistic changes of the previous commenter, you seem to be discounting the rate at which the writing style of various LLMs has backpropagated into many peoples' brains.
- roncesvalles 11mo ago
  Also discounting the fact that people actually do talk like that. In fact, these days I have to modify my prose to be intentionally less LLM-like lest the reader thinks it's LLM output.
- ewoodrich 11mo ago
  I can sympathize with being mistakingly accused of using LLM output, but as a reader the above format of "Its not x - it's y" repeated multiple times for artificial dramatic emphasis to make a pretty mundane point that could use 1/3 the length grates on me like reading LinkedIn or marketing voice whether it's AI or not (and it's almost always AI anyway). I've seen fairly niche subreddits go from enjoyable and interesting to ruined by being clogged with LLM spam that sounds exactly like this so my tolerance for reading it is incredibly low, especially on HN, and I'll just dismiss it. I probably lose the occasionally legitimate original observation now and then but in a world where our attention is being hijacked by zero effort spam everywhere you look I just don't have the time or energy to avoid that heuristic.
panarky 11mo ago
Hmmm, I wrote those words myself, maybe I've spent too much time with LLMs and now I'm talking like them?? I'd be interested in any evidence-based arguments you might have beyond attacking my writing style and insinuating bad intent. I found this commenter had sage advice about how to use HN well, I try to follow it: https://news.ycombinator.com/item?id=38944467 https://news.ycombinator.com/item?id=38944467
- momojo 11mo ago
  You mentioned "step change" twice. Maybe a once over next time? My favorite Mark Twain quote is (very paraphrased) "My apologies, had I more time, I would have written a shorter letter".
  - versteegen 11mo ago
    I thought the repetition was intentional.
- thejarren 11mo ago
  I’ll take you at your word, sorry for the incorrect callout. Your comment format appeared malicious, so my response wasn’t an attempt at being “snarky”, just acting defensively. I like the HN Rules/Guidelines.
- Aeolun 11mo ago
  “If you gaze long into an abyss, the abyss also gazes into you.”
  - gnabgib 11mo ago
    Is that you Nietzsche? Or are you Magog https://andromeda.fandom.com/wiki/Spirit_of_the_Abyss https://andromeda.fandom.com/wiki/Spirit_of_the_Abyss
    - deleted 11mo ago
      [deleted]
ineedasername 11mo ago
1) Models learn these patterns from common human usage. They are in the wild, and as such there will be people who use them naturally. 2) Now, given its for-some-reason-ubiquitous choice by models, it is also a phrasing that many more people are exposed to, every day. Language is contagious. This phrasing is approaching herd levels, meaning models trained from up-to-the-moment web content will start to see it as less distinctly salient. Eventually, there will be some other high-signal novel phrase with high salience, and the attention heads will latch on to it from the surrounding context, and then that will be the new AI shibboleth. It's just how language works. We see it in the mixes between generations when our kids pick up new lingo, and then it stops being in-group for them when it spreads too far.. Skibidi, 6 7, etc. It's just how language works, and a generation ago the internet put it on steroids. Now? Even faster.

legel 11mo ago

Thanks for reporting these metrics and drawing the conclusion of an underlying breakthrough in search. In his Nobel Prize winning speech, Demis Hassabis ends by discussing how he sees all of intelligence as a big tree-like search process. https://youtube.com/watch?v=YtPaZsasmNA&t=1218 https://youtube.com/watch?v=YtPaZsasmNA&t=1218

derangedHorse 11mo ago
The one thing I got out of the MIT OpenCourseWare AI course by Patrick Winston was that all of AI could be framed as a problem of search. Interesting to see Demis echo that here.

vitorgrs 11mo ago

I usually ask a simple question that ALL the models get wrong: List of mayor of my city [Londrina]. ALL the models (offine) get wrong. And I mean, all the models. The best that I could, it's o3 I believe, saying it couldn't give a good answer for that, and told to access the city website. Gemini 3 somehow is able to give a list of mayors, including details on who got impeached, etc. This should be a simple answer, because all the data is on wikipedia, that certainly the models are trained on, but somehow most models don't manage to give that answer right, because... it's just a irrelevant city in a huge dataset. But somehow, Gemini 3 did it. Edit: Just asked "Cool places to visit in Londrina" (In portuguese), and it was also 99% right, unlike other models, who just create stuff. The only thing wrong here, it mentioned sakuras in a lake... Maybe it confused with Brazilian ipês, which are similar, and indeed the city it's full of them. It seems to have a visual understanding, imo.

arach 11mo ago
thanks for sharing, very interesting example
guiambros 11mo ago
Ha, I just did the same with my hometown (Guaiba, RS), a city that is 1/6th of Londrina, and its wikipedia page in English hasn't been updated in years, and still has the wrong mayor (!). Gemini 3 nailed on the first try, included political affiliation, and added some context on who they competed with and won over in each of the last 3 elections. And I just did a fun application with AI Studio, and it worked on first shot. Pretty impressive. (disclaimer: Googler, but no affiliation with Gemini team)
usef- 11mo ago
Pure fact-based, niche questions like that aren't really the focus of most providers any more from what I've heard, since they can be solved more reliably by integrating search tools (and all providers now have search). I wouldn't be surprised if the smallest models can answer fewer such (fact-only) questions over time offline as they distill/focus them more thoroughly on logic etc.
belter 11mo ago

rafaelmn 11mo ago

It tells me that the benchmark is probably leaking into training data, and going to the benchmark site : > Model was published after the competition date, making contamination possible. Aside from eval on most of these benchmarks being stupid most of the time, these guys have every incentive to cheat - these aren't some academic AI labs, they have to justify hundreds of billions being spent/allocated in the market. Actually trying the model on a few of my daily tasks and reading the reasoning traces all I'm seeing is same old, same old - Claude is still better at "getting" the problem.

Der_Einzige 11mo ago

You clearly AI generated this comment.

thomasahle 11mo ago

From my understanding, Google put online the largest RL cluster in the world not so long ago. It's not surprising they do really well on things that are "easy" to RL, like math or SimpleQA

red75prime 11mo ago

> To succeed this well in math, you can't just do better probabilistic generation, you need verifiable search. You say "probabilistic generation" like it's some kind of a limitation. What is exactly the limiting factor here? [(0.9999, "4"), (0.00001, "four"), ...] is a valid probability distribution. The sampler can be set to always choose "4" in such cases.

coldtea 11mo ago

>This is not an incremental advance. It is a step change. This indicates a new discovery, not just more data or more compute. To succeed this well in math, you can't just do better probabilistic generation, you need verifiable search. You need to verify what you're doing, detect when you make a mistake, and backtrack to try a different approach. Loos like AI slop

TaupeRanger 11mo ago
It obviously is.

calmoo 11mo ago

Your comment is AI generated

tim333 11mo ago
I'll give you the style is like an LLM but the thoughts seem a bit unlike one. I mean the MathArena Apex results indicating a new discovery rather than more data is definitely a hypothesis. Also panarky denies it.

dbbk 11mo ago

Aren't you just describing tool calls?