3 ms·

Great article, even if it will be interesting to see whether things continue to develop in such a direction or not. > There's a version of this future where th

by msdz 2mo ago

Great article, even if it will be interesting to see whether things continue to develop in such a direction or not. > There's a version of this future where the model card stops listing a knowledge cutoff at all, because what's left in the weights goes stale on a scale of years instead of weeks. Future? Even just recently I’ve read of two approaches to this problem: Cactus have come up with Needle [0][1], which is their tool-calling focused 14 MB model (still an LLM!) – no world knowledge engrained. And instead of say, tool call structure, VibeThinker [2][3] focuses on reasoning over world knowledge. Combine these two approaches with a reliable search tool/a safe way of accessing the internet for the model, and you’ve got a probably slightly slower model for factual questions, which on the upside however doesn’t hallucinate. [0] https://cactuscompute.com/needle https://cactuscompute.com/needle [1] https://news.ycombinator.com/item?id=49246804 https://news.ycombinator.com/item?id=49246804 [2] https://arxiv.org/abs/2606.16140 https://arxiv.org/abs/2606.16140 [3] https://news.ycombinator.com/item?id=48639240 https://news.ycombinator.com/item?id=48639240

kennywinker 2mo ago
That kind of setup is super dependent on a search engine, and search keeps getting worse.
- msdz 2mo ago
  Generally I agree, with two caveats/thoughts: Providing not just any a baseline, but a correct and useful one, is ever more important the less the model is grounded in world knowledge – misunderstandings probably compound faster if there is no general grasp of (broadly) “life on earth”, or computers, or whatever. And secondly, I think (consumer-oriented) search becoming worse and worse is a challenge that’s mostly solvable (but far from solved!) for the big labs: (Mostly) trusted or even editorialized/reviewed sources like published work, Wikipedia, etc. is something they could index internally, it doesn’t need to come from a random blog site on the public internet. Furthermore, there’s a whole slew of companies specializing in crawling-for-LLM (i.e., bypassing bot protections) now as well.
- cyanydeez 2mo ago
  the technicals of search did not get worse. the economics of the search provider did. If government wasn't stuck in the 50s mindset, search provider would have been nationalized just like a utility.
  - kennywinker 2mo ago
    I think the techinicals of search did get worse - the ai sloppification of the internet made the needle harder to find in the haystack.
    - cyanydeez 2mo ago
      that's not the techicals, thats the content. That could be overcome via robust "Library of Congress" like filtering and evaluating. We don't really have that much more good information now than 30 years ago. We had an explosion of garbage that google saw as an oppotunity to sell us on, rather than filter out. Google saw it as their job to keep us on search longer, so the SEO garbage filling the search feed benefited them.
- jcfrei 2mo ago
  search isnt that super hard to solve. the problem is that incentives are misaligned - which means companies who sell something have an incentive to share their index with the corporation that pays to be the default search provider. for a simple search engine that just delivers facts there's very little incentive to keep its index up-to-date: because that company wont get the apis to retrieve the content efficiently, so it needs more compute, bandwidth and storage to run.
- skissane 2mo ago
  > search keeps getting worse For a lot of use cases, you don't need a general purpose search engine – a search over a curated knowledge base works even better. There are plenty of freely available data sets you can use, depending on the application; plus in many cases you will want to use internal-only knowledge bases containing non-public information (e.g. documentation for a corporation's internal systems and procedures). There are also many paid subscription domain-specific knowledge services available.

by msdz 2mo ago

kennywinker 2mo ago

That kind of setup is super dependent on a search engine, and search keeps getting worse.

msdz 2mo ago
Generally I agree, with two caveats/thoughts: Providing not just any a baseline, but a correct and useful one, is ever more important the less the model is grounded in world knowledge – misunderstandings probably compound faster if there is no general grasp of (broadly) “life on earth”, or computers, or whatever. And secondly, I think (consumer-oriented) search becoming worse and worse is a challenge that’s mostly solvable (but far from solved!) for the big labs: (Mostly) trusted or even editorialized/reviewed sources like published work, Wikipedia, etc. is something they could index internally, it doesn’t need to come from a random blog site on the public internet. Furthermore, there’s a whole slew of companies specializing in crawling-for-LLM (i.e., bypassing bot protections) now as well.
cyanydeez 2mo ago
the technicals of search did not get worse. the economics of the search provider did. If government wasn't stuck in the 50s mindset, search provider would have been nationalized just like a utility.
- kennywinker 2mo ago
  I think the techinicals of search did get worse - the ai sloppification of the internet made the needle harder to find in the haystack.
  - cyanydeez 2mo ago
    that's not the techicals, thats the content. That could be overcome via robust "Library of Congress" like filtering and evaluating. We don't really have that much more good information now than 30 years ago. We had an explosion of garbage that google saw as an oppotunity to sell us on, rather than filter out. Google saw it as their job to keep us on search longer, so the SEO garbage filling the search feed benefited them.
jcfrei 2mo ago
search isnt that super hard to solve. the problem is that incentives are misaligned - which means companies who sell something have an incentive to share their index with the corporation that pays to be the default search provider. for a simple search engine that just delivers facts there's very little incentive to keep its index up-to-date: because that company wont get the apis to retrieve the content efficiently, so it needs more compute, bandwidth and storage to run.
skissane 2mo ago
> search keeps getting worse For a lot of use cases, you don't need a general purpose search engine – a search over a curated knowledge base works even better. There are plenty of freely available data sets you can use, depending on the application; plus in many cases you will want to use internal-only knowledge bases containing non-public information (e.g. documentation for a corporation's internal systems and procedures). There are also many paid subscription domain-specific knowledge services available.