3 ms·
> LLMs are, at their core, search tools. This is the wrong take. Search tools are deterministic unless you purposely inject random weights into the ranking. W
by sdesol 2y ago
> LLMs are, at their core, search tools.
This is the wrong take. Search tools are deterministic unless you purposely inject random weights into the ranking. With search tools, the same search query will always yield the same search result, provided they are designed too and/or the underlying data has not changed.
With LLMs, I can ask the exact same question and get a different response, even if the data has not changed.
- Scene_Cast2 2y agoThe randomness comes from sampling. With local LLMs, you can fix the random seed, or even disable sampling all together - both will get you determinism. I agree that LLMs are not search tools, but for very different reasons.
- klabb3 2y agoSemantics. It may be able to get deterministic but it’s unstable wrt unrelated changes in the training data, no? If I add a page about sausages to a search index, the results for ”ski jacket” will be unaffected. In a practical sense, LLMs are non-deterministic. I mean, ChatGPT even has a ”regenerate” button to expose this ”turbulence” as a feature.
- User23 2y agoHence n-grams rather than documents. Also what's with using "semantics" as a dismissal when the technology we're talking about is the most semantically relevant search ever made.
- sdesol 2y agoThanks for the info on local LLMs. Based on my chats with multiple LLMs, the biggest issue appears to be hardware. Non-deterministic hardware: All LLMs mentioned that modern computing hardware, such as GPUs or TPUs, can introduce non-determinism due to factors like parallel processing, caching, or numerical instability. This can make it challenging to achieve determinism, even with fixed random seeds or deterministic algorithms. You can find the summary of my chats https://beta.gitsense.com/?chat=1c3e69f9-7b8b-48a3-8b99-bb1bdf191e60 https://beta.gitsense.com/?chat=1c3e69f9-7b8b-48a3-8b99-bb1b.... If you scroll to the top and click on the "Conversation" link in the first message, you can read the individual responses.