5 ms·

with web search, is knowledge cutoff really relevant anymore? Or is this more of a comment on how long it took them to do post-training?

deleted 1y ago
[deleted]
mastercheif 1y ago
In my experience, web search often tanks the quality of the output. I don't know if it's because of context clogging or that the model can't tell what's a high quality source from garbage. I've defaulted to web search off and turn it on via the tools menu as needed.
- bangaladore 1y ago
  I feel the same. LLMs using web search ironically seem to have less thoughtful output. Part of the reason for using LLMs is to explore somewhat novel ideas. I think with web search it aligns too strongly to the results rather than the overall request making it a slow search-engine.
  - troyvit 1y ago
    That makes sense. They're doing their interpretation on the fly for one thing. For another just because they now have data that is 10 months more recent than their cutoff they don't have any of the intervening information. That's gotta make it tough.
- ActionHank 1y ago
  I also find that it gets way more snarky. The internet brings that bad taint.
- gorkish 1y ago
  Web search often tanks the quality of MY output these days too. Context clogging seems a reasonable description of what I experience when I try to use the normal web.
  - clbrmbr 1y ago
    THIS. I do my best work after a long vigorous walk and contemplation, while listening to Bach sipping espresso. (Not exaggerating much.) If I go on HN or slack or ClickUp or work email, context is slammed and I cannot do /clear so fast. Even looking up something quick on the web or an LLM causes a dirtying.
- jjice 1y ago
  I haven't tried ChatGPT web search, but my experience with Claude web search is very good. It's actually what sold me and made me start using LLMs as part of my day to day. The citations they leave (I assume ChatGPT does the same) are killer for making sure I'm not being BSd on certain points.
  - illiac786 1y ago
    That’s interesting. I use the API and there are zero citations with Claude, charGPT and Gemini. Only Kagi assistant gives me some, which is why I prefer it when researching facts. What software to you use? The native Claude app? What subscription do you have?
    - jjice 1y ago
      Claude directly (web and mobile) with the Pro ($20) subscriptions. I found it very similar to Kagi Assistant (which I also use).
  - nicce 1y ago
    How often you actually check the citations? They seems to confidentally cite things but then they also say different things what source has.
    - jjice 1y ago
      It depends on the question. I was having a casual chat with my dad and we wondered how Apple's revenue was split amongst products, and it was just to chat about so I didn't check. On the other hand, I got an overview of Postgres RLS and I checked the majority of those citations since those answers were going to be critical.
- manmal 1y ago
  Web search is super important for frameworks that are not (sufficiently?) in the training data. o3 often pulls info from Swift forums to find and fix obscure Swift concurrency issues for me.
  - fmos 1y ago
    In my experience none of the frontier models I tried (o3, Opus 4, Gemini 2.5 Pro) was able to solve Swift concurrency issues, with or without web search. At least not sufficiently for Swift 6 language mode. They don’t seem to have a mental model of the whole concept and how things (actors, isolation, Tasks) need to play together.
    - elpakal 1y ago
      > They don’t seem to have a mental model of the whole concept and how things (actors, isolation, Tasks) need to play together. to be fair, does anyone ¯\_(ツ)_/¯
      - manmal 1y ago
        This. It’s a bunch of rules you need to juggle in your head.
- throw310822 1y ago
  Completely opposite experience here (with Claude). Most of my googling is now done through Claude- it can find and digest a d compile information much quicker and better than I'd do myself. Without web search you're basically asking an LLM to pull facts out of its ass- good luck with trusting the results.
- pbronez 1y ago
  Kagi really helps with this. They built a good search engine first, then wired it up to AI stuff.
diegocg 1y ago
I wonder if it would even be helpful because they avoid the increasing AI content
- rapind 1y ago
  This is what I was thinking. Eventually most new material could be AI produced (including a lot of slop).
joshuacc 1y ago
Still relevant, as it means that a coding agent is more likely to get things right without searching. That saves time, money, and improves accuracy of results.
MisterSandman 1y ago
It still is, not all queries trigger web search, and it takes more tokens and time to do research. ChatGPT will confidently give me outdated information, and unless I know it’s wrong and ask it to research, it wouldn’t know it is wrong. Having a more recent knowledge base can be very useful (for example, knowing who the president is without looking it up, making references to newer node versions instead of old ones)
havefunbesafe 1y ago
Question: do web search results that GPT kick back get "read" and backpropagated into the model?
- CamperBob2 1y ago
  Right now nothing affects the underlying model weights. They are computed once during pretraining at enormous expense, adjusted incrementally during training, and then left untouched until the next frontier model is built. Being able to adjust the weights will be the next big leap IMO, maybe the last one. It won't happen in real time but periodically, during intervals which I imagine we'll refer to as "sleep." At that point the model will do everything we do, at least potentially.
clickety_clack 1y ago
The biggest issue I can think of is code recommendations with out of date versions of packages. Maybe the quality of code has deteriorated in the past year and scraping github is not as useful to them anymore?
deleted 1y ago
[deleted]
seanw265 1y ago
Knowledge cutoff isn’t a big deal for current events. Anything truly recent will have to be fed into the context anyway. Where it does matter is for code generation. It’s error-prone and inefficient to try teaching a model how to use a new framework version via context alone, especially if the model was trained on an older API surface.
bearjaws 1y ago
Falling back to web search is a crutch, its slower and often bloats context resulting in worse output.
CharlieDigital 1y ago
Yes, because it may not know that it needs to do a web search for the most relevant information.
WorldPeas 1y ago
The problem, perhaps illusory that it's easy to fix, is that the model will choose solutions that are a year old, e.g. thinking database/logger versions from December '24 are new and usable in a greenfield project despite newer quarterly LTS releases superseding them. I try to avoid humanizing these models, but could it be that in training/posttraining one could make it so the timestamp is fed in via the system prompt and actually respected? I've begged models to choose "new" dependencies after $DATE but they all still snap back to 2024
stevage 1y ago
I've been having a lot of issues with chatgpt's knowledge of DuckDb being out of date. It doesn't think DuckDb enforces foreign keys, for instance.
roflyear 1y ago
Yes, totally. The model will not know about new versions of libraries, features recently deprecated, etc..
alfalfasprout 1y ago
It absolutely is, for example, even in coding where new design patterns or language features aren't easy to leverage. Web search enables targeted info to be "updated" at query time. But it doesn't get used for every query and you're practically limited in how much you can query.
m3kw9 1y ago
Web pages become prompt, so you still need the model to analyze
richardw 1y ago
Isn’t this an issue with eg Cloudflare removing a portion of the web? I’m all for it from the perspective of people not having their content repackaged by an LLM, but it means that web search can’t check all sources.

deleted 1y ago

[deleted]

mastercheif 1y ago

In my experience, web search often tanks the quality of the output. I don't know if it's because of context clogging or that the model can't tell what's a high quality source from garbage. I've defaulted to web search off and turn it on via the tools menu as needed.

bangaladore 1y ago
I feel the same. LLMs using web search ironically seem to have less thoughtful output. Part of the reason for using LLMs is to explore somewhat novel ideas. I think with web search it aligns too strongly to the results rather than the overall request making it a slow search-engine.
- troyvit 1y ago
  That makes sense. They're doing their interpretation on the fly for one thing. For another just because they now have data that is 10 months more recent than their cutoff they don't have any of the intervening information. That's gotta make it tough.
ActionHank 1y ago
I also find that it gets way more snarky. The internet brings that bad taint.
gorkish 1y ago
Web search often tanks the quality of MY output these days too. Context clogging seems a reasonable description of what I experience when I try to use the normal web.
- clbrmbr 1y ago
  THIS. I do my best work after a long vigorous walk and contemplation, while listening to Bach sipping espresso. (Not exaggerating much.) If I go on HN or slack or ClickUp or work email, context is slammed and I cannot do /clear so fast. Even looking up something quick on the web or an LLM causes a dirtying.
jjice 1y ago
I haven't tried ChatGPT web search, but my experience with Claude web search is very good. It's actually what sold me and made me start using LLMs as part of my day to day. The citations they leave (I assume ChatGPT does the same) are killer for making sure I'm not being BSd on certain points.
- illiac786 1y ago
  That’s interesting. I use the API and there are zero citations with Claude, charGPT and Gemini. Only Kagi assistant gives me some, which is why I prefer it when researching facts. What software to you use? The native Claude app? What subscription do you have?
  - jjice 1y ago
    Claude directly (web and mobile) with the Pro ($20) subscriptions. I found it very similar to Kagi Assistant (which I also use).
- nicce 1y ago
  How often you actually check the citations? They seems to confidentally cite things but then they also say different things what source has.
  - jjice 1y ago
    It depends on the question. I was having a casual chat with my dad and we wondered how Apple's revenue was split amongst products, and it was just to chat about so I didn't check. On the other hand, I got an overview of Postgres RLS and I checked the majority of those citations since those answers were going to be critical.
manmal 1y ago
Web search is super important for frameworks that are not (sufficiently?) in the training data. o3 often pulls info from Swift forums to find and fix obscure Swift concurrency issues for me.
- fmos 1y ago
  In my experience none of the frontier models I tried (o3, Opus 4, Gemini 2.5 Pro) was able to solve Swift concurrency issues, with or without web search. At least not sufficiently for Swift 6 language mode. They don’t seem to have a mental model of the whole concept and how things (actors, isolation, Tasks) need to play together.
  - elpakal 1y ago
    > They don’t seem to have a mental model of the whole concept and how things (actors, isolation, Tasks) need to play together. to be fair, does anyone ¯\_(ツ)_/¯
    - manmal 1y ago
      This. It’s a bunch of rules you need to juggle in your head.
throw310822 1y ago
Completely opposite experience here (with Claude). Most of my googling is now done through Claude- it can find and digest a d compile information much quicker and better than I'd do myself. Without web search you're basically asking an LLM to pull facts out of its ass- good luck with trusting the results.
pbronez 1y ago
Kagi really helps with this. They built a good search engine first, then wired it up to AI stuff.

diegocg 1y ago

I wonder if it would even be helpful because they avoid the increasing AI content

rapind 1y ago
This is what I was thinking. Eventually most new material could be AI produced (including a lot of slop).

joshuacc 1y ago

Still relevant, as it means that a coding agent is more likely to get things right without searching. That saves time, money, and improves accuracy of results.

MisterSandman 1y ago

It still is, not all queries trigger web search, and it takes more tokens and time to do research. ChatGPT will confidently give me outdated information, and unless I know it’s wrong and ask it to research, it wouldn’t know it is wrong. Having a more recent knowledge base can be very useful (for example, knowing who the president is without looking it up, making references to newer node versions instead of old ones)

havefunbesafe 1y ago

Question: do web search results that GPT kick back get "read" and backpropagated into the model?

CamperBob2 1y ago
Right now nothing affects the underlying model weights. They are computed once during pretraining at enormous expense, adjusted incrementally during training, and then left untouched until the next frontier model is built. Being able to adjust the weights will be the next big leap IMO, maybe the last one. It won't happen in real time but periodically, during intervals which I imagine we'll refer to as "sleep." At that point the model will do everything we do, at least potentially.

clickety_clack 1y ago

The biggest issue I can think of is code recommendations with out of date versions of packages. Maybe the quality of code has deteriorated in the past year and scraping github is not as useful to them anymore?

deleted 1y ago

[deleted]

seanw265 1y ago

Knowledge cutoff isn’t a big deal for current events. Anything truly recent will have to be fed into the context anyway. Where it does matter is for code generation. It’s error-prone and inefficient to try teaching a model how to use a new framework version via context alone, especially if the model was trained on an older API surface.

bearjaws 1y ago

Falling back to web search is a crutch, its slower and often bloats context resulting in worse output.

CharlieDigital 1y ago

Yes, because it may not know that it needs to do a web search for the most relevant information.

WorldPeas 1y ago

The problem, perhaps illusory that it's easy to fix, is that the model will choose solutions that are a year old, e.g. thinking database/logger versions from December '24 are new and usable in a greenfield project despite newer quarterly LTS releases superseding them. I try to avoid humanizing these models, but could it be that in training/posttraining one could make it so the timestamp is fed in via the system prompt and actually respected? I've begged models to choose "new" dependencies after $DATE but they all still snap back to 2024

stevage 1y ago

I've been having a lot of issues with chatgpt's knowledge of DuckDb being out of date. It doesn't think DuckDb enforces foreign keys, for instance.

roflyear 1y ago

Yes, totally. The model will not know about new versions of libraries, features recently deprecated, etc..

alfalfasprout 1y ago

It absolutely is, for example, even in coding where new design patterns or language features aren't easy to leverage. Web search enables targeted info to be "updated" at query time. But it doesn't get used for every query and you're practically limited in how much you can query.

m3kw9 1y ago

Web pages become prompt, so you still need the model to analyze

richardw 1y ago

Isn’t this an issue with eg Cloudflare removing a portion of the web? I’m all for it from the perspective of people not having their content repackaged by an LLM, but it means that web search can’t check all sources.