5 ms·

I do miss the earlier "heavy" models that had encyclopedic knowledge vs the new "lighter" models that rely on web search. Relying on web search surfaces a shal

by psadri 1y ago

I do miss the earlier "heavy" models that had encyclopedic knowledge vs the new "lighter" models that rely on web search. Relying on web search surfaces a shallow layer of knowledge (thanks to SEO and all the other challenges of ranking web results) vs having ingested / memorized basically the entirety of human written knowledge beyond what's typically reachable within the first 10 results of a web search (eg: digitized offline libraries).

hamdingers 1y ago
I feel the opposite. Before I can use information from a model's "internal" knowledge I have to engage in independent research to verify that it's not a hallucination. Having an LLM generate search strings and then summarize the results does that research up front and automatically, I need only click the sources to verify. Kagi Assistant does this really well.
- beefnugs 1y ago
  So does anyone have any good examples of it effectively avoiding the blogspam and SEO? Or being fooled by it? How often either way?
  - simonw 1y ago
    Here's a good article about Google AI mode usually managing to spot and avoid social media misinformation but occasionally falling for it: https://open.substack.com/pub/mikecaulfield/p/is-the-llm-response-wrong-or-have https://open.substack.com/pub/mikecaulfield/p/is-the-llm-res...
  - 15123123aa 1y ago
    I find one thing it doesn't do very well is avoiding marketing articles pushed by a brand itself. e.g. if I search is X better than Y, very likely landing on articles by makers of brand X and Y and not a 3rd party reviewer. When I manually search on Google I can spot marketing articles just by the URL.
    - simonw 1y ago
      Have you tried that with GPT-5 Thinking or is this based on your experience with older versions of ChatGPT + search?
  - coffeefirst 1y ago
    Bulk search is the only thing where I’ve been consistently impressed with LLMs. But, like the parent, I’m using the Kagi assistant. So the answer here might be “search for 5 things and pull the relevant results” works incredibly well, but first you have to build an extremely good search engine that lets the user filter out spam sites. That said, this isn’t magic, it’s just automated an hour of googling. If the content doesn’t exist you won’t find it.
gerdesj 1y ago
"encyclopedic knowledge" Have you just hallucinated that?
ants_everywhere 1y ago
Most real knowledge is stored outside the head, so intelligent agents can't rely solely on what they've remembered. That's why libraries are so fundamental to universities.
mastercheif 1y ago
I kept search off for a long time due to it tanking the quality of the responses from ChatGPT. I recently added the following to my custom instructions to get the best of both worlds: # Modes When the user enters the following strings you should follow the following mode instructions: 1. "xz": Use the web tool as needed when developing your answer. 2. "xx": Exclusively use your own knowledge instead of searching the internet. By default use mode "xz". The user can switch between modes during a chat session. Stay with the current mode until the user explicitly switches modes.
stephen_cagle 1y ago
I think this is partially something I have felt myself as well. It would be interesting if these lighter web search models would highlight the distinction between information that has been seen elsehwere vs information that is novel for each page? Like, a view that lets me look at the things that have been asserted and see how many of the different pages show those facts asserted (vs unmentioned vs contradicted).
killerstorm 1y ago
These models are still available: GPT-4.5, Gemini 2.5 Pro (at least the initial version - not sure if they optimized it away). From what I can tell, they are pretty damn big. Grok 4 is quite large too.
simianwords 1y ago
There is a tradeoff here: the non search models are internally heavy but the search models are light but also depend on real data. I keep switching between both but I think I'm starting to prefer the lighter one that is based on the sources instead.

by psadri 1y ago

hamdingers 1y ago

I feel the opposite. Before I can use information from a model's "internal" knowledge I have to engage in independent research to verify that it's not a hallucination. Having an LLM generate search strings and then summarize the results does that research up front and automatically, I need only click the sources to verify. Kagi Assistant does this really well.

beefnugs 1y ago
So does anyone have any good examples of it effectively avoiding the blogspam and SEO? Or being fooled by it? How often either way?
- simonw 1y ago
  Here's a good article about Google AI mode usually managing to spot and avoid social media misinformation but occasionally falling for it: https://open.substack.com/pub/mikecaulfield/p/is-the-llm-response-wrong-or-have https://open.substack.com/pub/mikecaulfield/p/is-the-llm-res...
- 15123123aa 1y ago
  I find one thing it doesn't do very well is avoiding marketing articles pushed by a brand itself. e.g. if I search is X better than Y, very likely landing on articles by makers of brand X and Y and not a 3rd party reviewer. When I manually search on Google I can spot marketing articles just by the URL.
  - simonw 1y ago
    Have you tried that with GPT-5 Thinking or is this based on your experience with older versions of ChatGPT + search?
- coffeefirst 1y ago
  Bulk search is the only thing where I’ve been consistently impressed with LLMs. But, like the parent, I’m using the Kagi assistant. So the answer here might be “search for 5 things and pull the relevant results” works incredibly well, but first you have to build an extremely good search engine that lets the user filter out spam sites. That said, this isn’t magic, it’s just automated an hour of googling. If the content doesn’t exist you won’t find it.

gerdesj 1y ago

"encyclopedic knowledge" Have you just hallucinated that?

ants_everywhere 1y ago

Most real knowledge is stored outside the head, so intelligent agents can't rely solely on what they've remembered. That's why libraries are so fundamental to universities.

mastercheif 1y ago

I kept search off for a long time due to it tanking the quality of the responses from ChatGPT. I recently added the following to my custom instructions to get the best of both worlds: # Modes When the user enters the following strings you should follow the following mode instructions: 1. "xz": Use the web tool as needed when developing your answer. 2. "xx": Exclusively use your own knowledge instead of searching the internet. By default use mode "xz". The user can switch between modes during a chat session. Stay with the current mode until the user explicitly switches modes.

stephen_cagle 1y ago

I think this is partially something I have felt myself as well. It would be interesting if these lighter web search models would highlight the distinction between information that has been seen elsehwere vs information that is novel for each page? Like, a view that lets me look at the things that have been asserted and see how many of the different pages show those facts asserted (vs unmentioned vs contradicted).

killerstorm 1y ago

These models are still available: GPT-4.5, Gemini 2.5 Pro (at least the initial version - not sure if they optimized it away). From what I can tell, they are pretty damn big. Grok 4 is quite large too.

simianwords 1y ago

There is a tradeoff here: the non search models are internally heavy but the search models are light but also depend on real data. I keep switching between both but I think I'm starting to prefer the lighter one that is based on the sources instead.