4 ms·
> usually resulting in one word repeating ad infinitum I've had that using gemini (via windsurf). Doesn't seem to happen with other models. No idea if there's
by _joel 1y ago
> usually resulting in one word repeating ad infinitum
I've had that using gemini (via windsurf). Doesn't seem to happen with other models. No idea if there's any correlation but it's an interesting failure mode.
- mattnewton 1y agoThis is usually a symptom of greedy sampling (always picking the most probable token) on smaller models. It's possible that configuration had different sampling defaults, ie. was not using top p or temperature. I'm not familiar with distributed-llama but from searching the git repo it looks like it at least takes a --temperature flag and probably has one for top p. I'd recommend rerunning the benchmarks with the sampling methods explicitly configured the same in each tool. It's tempting to benchmark with all the nondeterminism turned off, but I think it's less useful since in practice for any model you're self hosting for real work you're going to probably want top-p sampling or something like it and you want to benchmark the implementation of that too. I've never seen gemini do this though, that'd be kinda wild if they shipped something that samples that way. I wonder if windsurf was sending a different config over the api or if this was a different bug.
- mrbungie 1y agoYep, sometimes Gemini for some reason ends up in what I call "ergodic self-flagellation". Here are some examples: https://www.reddit.com/r/GeminiAI/comments/1lxqbxa/i_am_actually_terrified/ https://www.reddit.com/r/GeminiAI/comments/1lxqbxa/i_am_actu...
- justinclift 1y agoI've seen that occasionally with one of the deepseek models when using the default Ollama context size of 4096, rather than whatever the model's preferred context size was. After having that happen, I switched my stuff to check the model's preferred context size, then set the context size to match, before using any given model.