4 ms·

The randomness comes from sampling. With local LLMs, you can fix the random seed, or even disable sampling all together - both will get you determinism. I agre

• klabb3 2y ago
  Semantics. It may be able to get deterministic but it’s unstable wrt unrelated changes in the training data, no? If I add a page about sausages to a search index, the results for ”ski jacket” will be unaffected. In a practical sense, LLMs are non-deterministic. I mean, ChatGPT even has a ”regenerate” button to expose this ”turbulence” as a feature.
  • User23 2y ago
    Hence n-grams rather than documents. Also what's with using "semantics" as a dismissal when the technology we're talking about is the most semantically relevant search ever made.
• sdesol 2y ago
  Thanks for the info on local LLMs. Based on my chats with multiple LLMs, the biggest issue appears to be hardware. Non-deterministic hardware: All LLMs mentioned that modern computing hardware, such as GPUs or TPUs, can introduce non-determinism due to factors like parallel processing, caching, or numerical instability. This can make it challenging to achieve determinism, even with fixed random seeds or deterministic algorithms. You can find the summary of my chats https://beta.gitsense.com/?chat=1c3e69f9-7b8b-48a3-8b99-bb1bdf191e60 https://beta.gitsense.com/?chat=1c3e69f9-7b8b-48a3-8b99-bb1b.... If you scroll to the top and click on the "Conversation" link in the first message, you can read the individual responses.