5 ms·
Gemini last models: temperature, top_p, and top_k are deprecated and ignored
- greatgib 3mo ago1. Sampling parameter deprecation (temperature, top_p, top_k) temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests.
- tolugenius 3mo ago> To improve determinism, define a system instruction with explicit rules for your specific use case. Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
- janalsncm 3mo agoIt is guaranteed to work worse than top_k=1, that’s for sure.
- furyofantares 3mo agoDepends on what you mean by determinism.
- sara_mo 3mo ago[flagged]
- tough 3mo agofwiw sonnet-5 also drops temperature (sonne-4 had it)
- aesthesia 3mo agoMy guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.
- pixelmelt 3mo agoI'm inclined to agree given how unstable Gemma 4 is when not using the "official" sampler settings
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- impulser_ 3mo agoGood. These have been basically useless for the past few generations of models, and most of the time made the model perform worst.
- kouteiheika 3mo agoObligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb08b94bce20d5397 https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
- NooneAtAll3 3mo agowhere can one learn what top_k and top_p mean?
- cratermoon 3mo agohttps://www.ibm.com/docs/en/watsonx/saas?topic=prompts-model-parameters-prompting#more-options-for-sampling-decoding https://www.ibm.com/docs/en/watsonx/saas?topic=prompts-model...
- krapht 3mo agoironically, any frontier LLM will easily generate a tutorial at any detail you like explaining what these are. if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
- barkingcat 3mo agoIs an llm able to explain to itself what these parameters do, and change its own parameter settings?
- chronogram 3mo agoAn LLM is stateless so every time you send a message a command is run with the parameters, the prompts, and a copy of the chat history when using one of those chat interfaces, and then it returns text. If the output includes explaining it to an LLM that would then be included in the copy of the chat history placed in the new command. It would be a bit like opening a text editor and typing in that you want to increase the font size. Someone external to the text would have to come along and click the font options.
- gdiamos 3mo agothank god, these parameters are so confusing
- salamo 3mo agoPossible reasons: - They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate. - They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection". [1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of-n-sampling https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
- regularfry 3mo agoCouple of other more businessy reasons: - SynthID hides the watermark in the sampling RNG. No randomness -> no watermark. - If you want to distil on the model outputs, you want temp=0 outputs. No temp=0 -> worse distillation.
- orbital-decay 3mo agoYou probably don't want temp 0, especially with Gemini which often fails in the greedy sampling mode in practice, often in ridiculous ways (e.g. multiple thousand token loops). As another comment says it's pretty brittle and this is even reflected in their docs somewhere IIRC. You want the "normal" temperature (whatever it is) and oversampling if necessary. Certain open models have/had the temperature locked on the official APIs, I assume they just have sampling incompatible with static temperature or do some fancy speculative decoding. It's clearly not to hide anything, as the weights are open and there are always alternative providers.
- charcircuit 3mo agoAlong with everything else. These parameters can make speculative decoding less accurate increasing the inference cost.
- nichohel 3mo ago"Last" or "latest"? Those are rather different.
- froh 3mo agohehe... fwiw, the un-editorialized title is "Using the *latest" Gemini models" I speculate OP wanted to put focus on their chosen detail in the title...
- greatgib 3mo agoIndeed, the focus was on this section but it is direct link to the relevant paragraph. For the "last/latest", it is the result of trying to trim the title enough to enter in the size limited title box and then I didn't notice the typo. At first I wanted to put the 3.5/3.6 model number because "latest" is not clear and time dependant but it was too long then.
- david-gpu 3mo agoThe OP does not appear to be a native English speaker. And, for example, languages like Spanish don't clearly differentiate between "last" and "latest".
- henriquecm8 3mo agoI am not a Spanish speaker, but Portuguese is usually similar. In it we say "último" for last, and "mais recente" for latest, so while there isn't a single word translating, we still have its equivalent. I imagine it's the same for Spanish.
- david-gpu 3mo agoIn Spanish, "último" is ambiguous, and can mean either last or most recent. Funnily, in Galician there is a distinction between "último" (most recent) and "derradeiro" (final). I would have guessed that it would be the same in Portuguese, given how closely related the two languages are.
- herpderperator 3mo agoI'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
- StrauXX 3mo agoI'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software. I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.
- franky47 3mo ago> To improve determinism, define a system instruction with explicit rules for your specific use case. "Please be deterministic".
- rf15 3mo agoplease oh god please, I don't know how to
- Zambyte 3mo agoIn all seriousness, this seems like it could be fine if done well. You can just have a model do a pass over the system prompt and set reasonable parameters based on that. That's probably not what they're doing, but it could be.
- dev_dan_2 3mo ago> improve determinism Oh well. I might be to picky here, but how I see things, determinism cannot be improved or worsened, but achieved or not achieved. Or Partially archieved, when analyzing a system that has both components that are deterministic or non deterministic. There are times to think in absolutes, and when talking about deterministic behavior of technical systems, this is one of them. Join the sith side, we have cookies, and when we say we have cookies, we do.
- gregdaniels421 3mo agoProbably what is meant is consistency instead of determinism
- fjdjshsh 3mo agoWould "decrease randomness" be acceptable to you?
- dev_dan_2 3mo agoYes! The "-ness" implies some degree of spectrum already, and from what I know and understand, one can talk about different degrees/classes of randomness. "decreasing entropy" would work too, I guess (Intuitively: If one would open a bet on a system outcome, how high would the quotas be? If there is some degree of randomness in the system, it would make sense to put money on multiple different outcomes to minimize ones losses. Only if there is no randomness at all, it would be rational to bet all money on the one option we expect). Another further distinction would be between "true randomness" and "We do not know all input factors yet, hence, the outcome appears random to us". The former is something that genuinely exists in the universe as far as I understand physics (i.e. even if all input parameters are known, it is not possible to predict the outcome reliably, which seems to be the case for quantum physics, which I have the surfacest or surface levels of understandings about), the later boils down to reducing "unknown unkowns". In practise, this means stuff like using seeds for random number generators, fixing things that look random like day of the week of a DateTime.now(), and so on. Depending on what you do, it might be never possible to get rid of all randomness, but I am 100% convinced that you can always contain it into a well-defined cage, and control everything else down to the last bit. That also made me think of the opposite: "Increasing predictability" would be acceptable for me too, and a deterministic process/component/system/whatever is one that is at maximum possible predictability for me then.
- double2025 3mo ago[flagged]
- bob1029 3mo ago> To improve determinism, define a system instruction with explicit rules for your specific use case. What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one. The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
- orbital-decay 3mo agoCurrent post-trained models are stochastically and semantically collapsed to a large extent, you'll never get meaningful semantic diversity if your prompt stays the same. This makes the temperature brittle, the model becomes dumb much earlier than meaningfully diverse. There are fancy samplers that claim to overcome this (comments ITT mentioned them), but a) you need a local inference stack to use them and b) I've yet to see any definitive evidence they add any deep semantic diversity without dumbing the model down, most of what I've seen is pretty superficial, including distribution-aware samplers. >I'm currently adding random variation to prompts This is better than naive temperature tweaks, if done right.
- arthiarumugam18 3mo ago[flagged]
- marcinjachymiak 3mo agoIt's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).
- kittikitti 3mo agoThe API endpoints are less useful if you can't even fine-tune these parameters. These endpoints used to be developer oriented and this sort of change would be a huge cause of concern. If you primarily vibe-code, this is not a problem but understand that they're making their product worse on purpose.
- greatgib 3mo agoSame thing as the new APIs like assistant API that rip us of the control of the history of messages for the context that is now hidden server side.