8 ms·
Embarrassingly simple self-distillation improves code generation
- roger_ 7mo agoSkimmed this but don't have an intuitive understanding of why this works and how temperature and truncation factor in.
- jofzar 7mo ago> simple self-distillation (SSD): Sorry apple, SSD is already taken, you can't use that acronym.
- ape4 7mo agoATT=All TLAs are Taken
- love2read 7mo agoYou're right, I offer these alternatives: Consistency Preservation Update (CPU) Guided Probability Update (GPU) History-aware Distillation Driving (HDD) Probability Smoothing Update (PSU)
- drittich 7mo agoI used to invent TLAs on the spot for fun, and when someone asked what it was, would respond, "It's a PUA", eventually revealing that meant "previously unknown acronym". It was even more annoying that it sounds.
- deleted 7mo ago[deleted]
- dist-epoch 7mo ago[flagged]
- 0x3f 7mo agoThat's... almost every AI paper.
- deleted 7mo ago[deleted]
- amelius 7mo agoSo "Made in China, designed by Apple in California" should be: "Made in China, designed by Chinese people in California"?
- ptidhomme 7mo agoI used to have the opposite rule in my signal processing field : the more Chinese names, the less innovation was there. They seemed like they had to be churning out papers and any little adaptation to existing research triggered a new publication. But it may have changed now.
- avaer 7mo agoI definitely pay more attention to papers affiliated with Chinese companies; the economics seem to be more conducive to doing good academic work and publishing it. I would say the same for companies like Apple (where TFA came from). But to filter based on author's names sounds pretty darn racist.
- 0x3f 7mo agoHaven't read the paper yet, but it is interesting how seemingly simple many breakthroughs in ML are. Even transformers are like that. Maybe it's hindsight bias. I suppose we just don't have a deeper underlying theory to lean on and help us 'design' anything.
- christophilus 7mo agoA lot of discoveries are like that. In fact, simplicity is often the hallmark of correctness, and complexity is often a sign that our understanding is incomplete and we’re still stumbling towards the right model. Not always, but often. It’s been a good rule of thumb in my programming career.
- heeton 7mo ago100%. I have a guiding approach when solving problems: keep reframing and exploring until the solution becomes obvious. I often find, if I've got a complicated solution, it’s because I haven’t fully examined the problem.
- Teever 7mo agoA designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away. -- Antoine de Saint-Exupery
- GandalfHN 7mo ago[flagged]
- GandalfHN 7mo ago[flagged]
- GandalfHN 7mo ago[flagged]
- khalic 7mo agoIncredible, will translate to better coding models in the near future. We really need to develop better tools to understand what's happening inside these NNs. Working with high-D spaces is not something we're good at, and we're basically throwing stuff at it and seeing if it sticks.
- politelemon 7mo agoIt's cringe worthy to see that the original paper itself is editorialised. Title should be: Simple Self-Distillation Improves Code Generation
- StevenWaterman 7mo ago"Embarrassingly" has a history as a technically meaningful word roughly equivalent to "maximally", see "Embarrassingly parallel" https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel
- Aurornis 7mo agoThe phrase embarrassingly parallel has a history in computer science. Many computer science paper titles allude to past titles in other CS papers. Calling it “cringe worthy” is unnecessarily mean. There is context and history you don’t understand.
- gottheUIblues 7mo ago"Embarrassingly" considered harmful?
- cbm-vic-20 7mo ago"Embarrassingly" considered harmful is all you need.
- TeMPOraL 7mo agoProgramming Introduction to "Embarrasingly" considered harmful is all you need in 21 hours.
- rzzzt 7mo agoCringeworthily parallel, not even serial
- deleted 7mo ago[deleted]
- ape4 7mo agoShouldn't a scientific paper be using metric units (like 30T) rather than 30B. There are two distinct billions. https://en.wikipedia.org/wiki/Billion https://en.wikipedia.org/wiki/Billion
- mikkupikku 7mo agoObjective one should be to communicate effectively, not confuse everybody.
- unknownx113 7mo agothat disqualifies like 80% of papers lmao
- mikkupikku 7mo agoLol, you're probably not wrong. But have you ever noticed that the most important papers tend to be on the clear and readable side of things? It's as if researchers understand that being understood is important, but deemphasize that when the paper itself isn't important in the first place. (Maybe if they're only publishing to not perish, not being understood is actually a goof thing from their perspective?)
- bensyverson 7mo agoReally fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compromise; we call this tension the precision-exploration conflict. In other words, just like us, the model needs to shift from "exploration" in "fork" mode (divergent thinking to produce a creative solution) to "precision" in "lock" mode (producing syntactically correct code). What this paper shows is that their simple technique (SSD) can improve the ranking of optimal tokens in both lock and fork positions, meaning the model is more likely to explore when it should be exploring, and more likely to be precise when it needs to be. I love that we're still learning the emergent properties of LLMs!
- stingraycharles 7mo agoSeems like this is true for not just code but for all content being generated? Albeit for code it’s more well-defined, but the fork / lock mechanism works for a lot more problem domains.
- bensyverson 7mo agoThat would seem intuitively true; it certainly applies to written language, where a clause could go off in another direction, but at other positions the correct grammar/syntax is unambiguous.
- bryanrasmussen 7mo agothinking - well if we think of lock as happening in a narrative, then I think we can see there can be points where "everything you know is wrong" which essentially allows you to go back into a sort of fork mode and work towards another lock. Completely artistic creation, creating something that does not exist and that cannot produce things out of itself, means that locking can be more diffuse, not as settled.
- 7mo ago
- wg0 7mo agoAfter TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more generous usage limits in the future and power users would be mostly running their own models anyway. Anyone using these models as "non-deterministic transpilers" from natural language to code (experienced engineers who can write code themselves) would probably not be paying to any AI providers. [0] https://www.youtube.com/watch?v=-_hC-C_Drcw https://www.youtube.com/watch?v=-_hC-C_Drcw
- spiderfarmer 7mo agoI always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.
- BarryMilo 7mo agoI seem to remember that's one of the first things they tried, but the general models tended to win out. Turns out there's more to learn from all code/discussions than from just JS.
- justinlivi 7mo agoFrom my own empirical research, the generalized models acting as specialists outperform both the tiny models acting as specialists and the generalist models acting as generalists. It seems that if peak performance is what you're after, then having a broad model act as several specialized models is the most impactful.
- nareyko 7mo ago[dead]
- smallerize 7mo agoI don't suppose they published the improved models?
- l5870uoo9y 7mo ago> Our method, simple self-distillation (SSD), is embarrassingly simple: sample solutions from the base model with specified temperature and truncation, then fine-tune on those raw, unverified samples via standard cross-entropy loss. So you prompt the base model for answer and then rerun the prompt with the answer from the first run?
- ACCount37 7mo agoNo. There's no "answer" really. They use self-distillation to shift the output distribution of the model towards that of the same model, but running with different temperature/truncation settings in sampling. This effectively "folds" the logit tail truncation behavior into the model itself. Not entirely unlike a few "model controlled sampling settings" things I've seen in what it does, but different in execution.
- 3rd3 7mo agoIsn't that "scheduled sampling"? In that case they also shift the input distribution toward that of the model, which possibly is even more crucial than shifting the output distribution?
- zug_zug 7mo agoYeah basically. You use the outputs from the first run (right or wrong) as answers for the second training run, and repeat. Magically it works. That's what's so surprising. I guess a theory is because there are so many diverse ways to be wrong that they don't accumulate error... still seems surprising and would be interesting to see if it works in other domains.
- yesexactly 7mo agoYeah basically. It's annoying as hell how much euphemistic language is used. They say "embarassingly simple" but they really mean "something everyone already knows" They have made 0 discoveries
- drooby 7mo agoFascinating... This feels eerily similar to sleep consolidation or synaptic pruning
- ACCount37 7mo agoI don't see much similarity? Unless you're looking at self-distillation in general and not just this use of it.
- oliver236 7mo agoHow not? I think the analogy is actually pretty specific to this paper, not just self-distillation in general. During sleep your brain replays experiences but noisy and distorted. The replays are often incoherent as narratives (dreams are weird). But the consolidation still works because the value isn't in the narrative coherence, it's in the activation patterns at each moment. Important pathways get strengthened, weak ones get pruned. Section 4.4 of this paper is what makes the connection click. They cranked training temperature to 2.0 with no truncation. 62% of the sampled outputs had no extractable code. Coherent Python that devolves into multilingual gibberish halfway through. The model still improved (+5.7pp pass@1). This makes no sense if you think the model is learning from good code examples. But it makes a lot of sense if you think of it as the model replaying its own knowledge back to itself in a noisy/distorted form, and the replay process strengthening what matters (sharp distributions at "lock" positions where one token is correct, broad distributions at "fork" positions where multiple approaches work) while pruning what doesn't (distractor tails). The model doesn't learn anything new. It just wakes up performing better because what it already knew got cleaned up. How is this comment not at number 1??
- ACCount37 7mo agoThis is a property of self-distillation. Self-distillation shifts the behavior of the model towards that of the model + steering. As such, you don't strictly "need" the tokens to be in-domain for it to work. The logits are a vessel for transferring the steering into the model's internals. The tokens can be gibberish. What transfers isn't whether they're gibberish or not, but how the flavor of model predictions, if given gibberish, differs from that of an unsteered version of itself. In this specific case, the behavioral difference comes from the "temperature-shifted, truncated samples" in the "teacher" sampling strategy, and it is that difference that is internalized by the "student" model.
- vishnugupta 7mo agoCan someone please eli5 this to a friend web developer? I read the abstract but couldn’t understand much.
- unknownx113 7mo agoyou're probably overcomplicating it; as the paper says, it's embarrassingly simple: given a problem set, generate a response for each problem with a fixed temperature and truncation - then fine tune the model on the generations. Their hypothesis as to why this works requires a bit more knowledge about model architecture, but basically when a model generates code some positions have only one right answer and some have many valid options - but the model has to use one global confidence setting for both. Sampling with a specific temperature + a garbage-token filter, then training on those outputs, teaches the model to internalize 'be precise where there's one answer, stay open-minded where there are several' — without anyone labeling which is which. Note that there's a lot more nuance to this and I simplified a lot.
- useful 7mo agoif the probability mass is on a single token, its a precise answer like `1 + 1 = ` if next token predicted shares probability with other token, then there are multiple answers like `position: ` you can generate and train answers by exploring on varying the length of the code generated
- zug_zug 7mo agoELI 5 You teach the machine by asking it to solve some problems, and then whatever answer it gives you say "That's exactly right. Now we train on those answers YOU just gave me" (even if they are wrong) and repeat. Somehow THAT works over time.
- pithtkn 7mo ago[dead]
- usermac 7mo ago[dead]
- xbmcuser 7mo agoSo the chances of Singularity went up.
- hu3 7mo agoOr down if this research leads to a local minima.
- an0malous 7mo agoI’d like to understand AI research better and I recall some posts a while back where someone collected all the key papers that one should read, but I don’t remember enough to be able to find it. Does anyone know what I’m talking about and could link me to that post?
- zug_zug 7mo agoThis might sound paradoxical -- but any decent LLM will be happy to explain all the papers to you at great depth, and read new ones, and translate the math into simpler concepts and such. It'll also happily recommend relevant math to study, or give training problems, or whatever you want.
- dzjkb 7mo agoperhaps it was Ilya Sutskever's list? https://github.com/pageman/sutskever-30-implementations https://github.com/pageman/sutskever-30-implementations
- an0malous 7mo agoYes I think so, thank you!
- 4b11b4 7mo agoSelf-consistency meets fine-tuning?
- ultramann 7mo agoMaybe not the thing I should be focusing on, but I was surprised this paper came from apple. I was under the impression that apples ai/LLM research was far behind the curve. I get that research is a rising tides lifts all boats situation, I just thought that I had seen lots of negative news about apples progress in the front, and heuristically haven’t seen many (any?) apple research papers make it the front page of hacker news. Wondering if anyone more familiar with apple/ai research could comment on this?
- bensyverson 7mo agoApple routinely makes hn's front page for their AI research [0][1], particularly related to their work with small on-device models. [0] https://news.ycombinator.com/item?id=46117802 https://news.ycombinator.com/item?id=46117802 [1] https://news.ycombinator.com/item?id=47107974 https://news.ycombinator.com/item?id=47107974
- antirez 7mo agoAnother potentially usable trick is the following: based on the observation that longer token budget improves model performances, one could generate solutions using a lot of thinking budget, then ask the LLM to turn the trace into a more compact one, and later SFT on that. That said, I have the feeling the result of the paper will likely be hard to apply in practice without affecting other capabilities, and/or not superior to other techniques that provide similar improvement in sampling.
- VoqalAI 7mo ago[dead]
- fooker 7mo agoI'm excited for the long tail of techniques like this that are going to be discovered over the next several decades that's going to make this technology eventually run on a toaster!
- augment_me 7mo agoIsn't this was DeepSeek + Kimi did to Claude?
- itmitica 7mo agoIt’s an interesting claim, and the reported benchmark gains are large, but it is still an April 1, 2026 arXiv preprint, so I’d treat it as promising rather than settled.
- robwwilliams 7mo agoVery cool. An evolutionary biologist would say: Welcome to the party! Mutation rate modulation is the AI engineers’ heat. And selection does the trimming of the outliers. Some more serious biomorphic thinking and we may get to the next big insight courtesy of 3+ billion years of evolution—- evolution that enabled a great ape species to write a paper like this and build LMM’s like Gemma4 that totally rock on a 3.5 pound MacBookPro M5 Max with 128 GB of RAM.
- p1esk 7mo agoIt’s so ironic that Apple still publishes AI research and OpenAI does not.
- dhruv3006 7mo agoI find it ironic too - there was no need for OpenAI to not publish really.
- michaelcampbell 7mo agoThey have no marketplace to religiously defend for it...yet.
- dwa3592 7mo agoCan anyone help clarify these doubts - I didn't see any information about how different the test/benchmark set is from the training set. It feels like an important gap to not fill in a ML paper. What if there is an overlap between the problems in the test set and the training set?? What is the decontamination strategy of going from LCBv5 to LCBv6 ?
- techpulselab 7mo ago[dead]
- porridgeraisin 7mo agoThere's an obvious baseline which seems missing If you sample from the base model with T=1.6, top_k=20, top_p=0.8, i.e, the decode settings used for the distillation's ground truth, does it match the SSD'd model + some decoding? Performance wise. Their sweep is missing this. And only covers "standard" decoding settings.
- crustycoder 7mo ago"SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6" I know virtually nothing about this area but my naive take is that something that means it still only passes tests around half the time doesn't seem like a particularly big jump forwards. What am I missing?
- SEMW 7mo agoThere's no shortage of benchmarks (coding or otherwise) that any competent coding model will now pass with ~100%. But no-one quotes those any more because if everyone passes them, they don't serve any useful purpose in discriminating between different models or identifying advancements So people switch to new benchmarks which either have more difficult tasks or some other artificial constraints that make them in some way harder to pass, until the scores are low enough that they're actually discriminating between models. and a 50% score is in some sense ideal for that - there's lots of room for variance around 50%. (whether the thing they're measuring is something that well correlates to real coding performance is another question) So you can't infer anything in isolation from a given benchmark score being only 50% other than that benchmarks are calibrated to make such scores the likely outcome
- crustycoder 7mo agoSo it's the relative and not the absolute diff that matters - thanks.
- martinrolph 7mo agoThink of it less like a test suite and more like an exam. If you're trying to differentiate between the performance of different people/systems/models, you need to calibrate the difficulty accordingly. When designing a benchmark, a pass rate of roughly 50% is useful because it gives you the most information about the relative performance of different models. If the pass rate is 90%+ too often, that means the test is too easy: you're wasting questions asking the model to do things we already know it can do, and getting no extra information. And if it's too low then you're wasting questions at the other end, trying to make it do impossible tasks.
- uduni 7mo agoIt's crazy how much better you can make LLM output just by asking "is this the most elegant solution?" In a loop (Not fine tuning, but interesting none the less. If a model can so easily find a more elegant solution, why didn't it pick that in the first place?)
- suzzer99 7mo agoIME human developers also span a spectrum on this. On one end, you have devs who might meditate half a day on different solutions before writing a line of code. On the other end are devs who run full speed ahead with the first working solution that comes to mind. LLMs in their current form are mostly the latter.
- jditu 7mo ago[dead]
- noman-land 7mo agoThe elegant solution rarely happens on the first try. Many times you need to first arrive at a solution, and then keep iterating on it until it's elegant. Akin to "sorry I didn't have time to write a shorter letter".
- aplomb1026 7mo ago[dead]
- yubainu 7mo ago[dead]
- aiiaro 7mo ago[flagged]
- mickdarling 7mo agoI'm working on a tool to determine which portions of an LLM process can be optimized, and how to measure that optimization and check whether it's optimizable at all. The shaping pattern that they talk about here is directly relevant and makes a whole lot more processes potentially optimizable by looking at the pattern rather than if the metrics just go up or down.
- gavinray 7mo agoWhy have we been fed the narrative that training models on their own output progressively degrades quality? It's the first thing anyone would think of (like a self-hosted compiler) but everything I've read said "it doesn't work." EDIT: For context: > Shumailov et al. (2024) — "AI models collapse when trained on recursively generated data" (Nature, 2024)
- zyklu5 7mo agoTheir explanation for why their idea (SSD) might work - precision-exploration conflict hypothesis - is something adaptive decoding also tries to solve. https://ai.meta.com/research/publications/adaptive-decoding-via-latent-preference-optimization/ https://ai.meta.com/research/publications/adaptive-decoding-...
- andy_xor_andrew 7mo agoI've been wondering about adaptive decoding! It seems obvious to me that at some points during decoding (reasoning, "creative thinking") you would want a higher temperature, while at other points (emitting syntactically correct code, following a plan that was already established) you would want lower temperature.
- krackers 7mo agoSeems similar to entropix? https://github.com/xjdr-alt/entropix https://github.com/xjdr-alt/entropix
- OxfordOutlander 7mo agoSo... it's like a golfer who hits thousands of balls into an open field without ever once aiming for a hole. The relentless repetition flawlessly locks in their foundational muscle memory and basic swing mechanics, so when they finally step up to a real course, they don't have to waste a single thought on how to hold the club. Their basic swing is completely automatic - they can confidently take the creative, high-risk shot required to actually sink a hole-in-one.
- thunky 7mo ago> The relentless repetition flawlessly locks in their foundational muscle memory and basic swing mechanics If only this were true there wouldn't be an army of duffers who after a lifetime of "training" still dig a trench in front of the ball every time they play.
- drdrek 7mo agoThis is the "Factors" Bonanza in finance all over again. You get a generally useful model, then you over-fit it to some criteria and announce advancement in the field, then it performs worse in real life. New infinite academic article glitch just dropped boys!
- maxbeech 7mo ago[dead]
- hnretards 7mo agoI've been doing something even better than this for years using only Mistral 7b. My local running Mistral 7b is a 100x better at modern JavaScript than any model on the market, mainly just from RAG on my own code samples. That's basically what they are describing with "post-training", the TLDR is that code especially of a certain style is vastly simpler than written language. You really don't need a huge model or data centers etc. you just need a small but good model like Mistral 7b and literally a few good samples. But you guys keep doing you lol. A bunch of non-devs trying to solve code is pretty funny to watch.
- hooloovoo_zoo 7mo agoOne sentence summary: We fine-tuned a general-purpose model to produce valid benchmark code results and it got better at producing benchmark code results; we didn't bother to evaluate it on anything the model used to be good at.
- andy_xor_andrew 7mo agoNot really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the same temp/topk sweep of the original. So no, they are not fine-tuning a general purpose model to produce "valid benchmark code results."
- hooloovoo_zoo 7mo agoThey are training the model to 1. Produce code (as opposed to answer a question, write a poem, etc.) 2. Produce long enough output to be a valid solution. So they are doing exactly what I said. Cheers.
- mememememememo 7mo agoIn layman, they are putting wet tyres on when it is raining and saying the car performs better over the next lap?
- fpgaminer 7mo agoNot only that, they additionally ran an experiment with the training temperature turned way up (2.0) and truncation turned off such that the majority of SFT examples were incoherent (63% IIRC). Yet the model finetuned on these broken examples still improved over baseline.
- krackers 7mo ago
- Lerc 7mo agoThis is the natural conclusion of what was really claimed about model collapse, and indeed natural evolution. Making an imperfect copy while invoking a selection mechanism is evolution. Some of the claims about models training on their own data, in their enthusiasm to frame it as a failure, went further to suggest that it magnified biases. I had my doubts about their conclusions. If it were true, it would be a much greater breakthrough because the ability to magnify a property represents a way to measure a weak version that property. The ability to do that would mean they would have found a way to provide a training signal to avoid bias. It would be great if that's what they did but I suspect there would have been more news about it. Perhaps this paper will put to rest the notion that AI output is useless as training data. It has only ever been the case that it was useless as an indiscriminate source of data.
- aplomb1026 7mo ago[dead]
- hackermeows 7mo agowhat is the big deal with obsidian ? I see a lot of people use it but I'm more than happy with giving an LLM a local sqlite table , embedding api and asking the agent to maintain its own memory
- teleforce 7mo agoIt seems that self-distillation is the way to go for LLM. Self-distillation has been shown recently as very efficient and effective back in January this year by MIT and ETH team in their Self-Distillation Fine-Tuning (SDFT) LLM system [1],[2]. This paper is also their closest competitor named On-Policy Self-Distillation in the comparison table. I hope they keep the original work real name that is Self-Distillation Fine-Tuning or SDFT. Imagine later paper citing this very paper as cross-entropy self-distillation instead of their very own given name Simple Self-Distillation or SSD. Although I'd have admitted it's a lousy name that breaks the namespace with common SSD nomenclature for solid-dtate drive, as others have rightly pointed. I think they should given the proper credit to this earlier seminal earlier on SDFT but apparently they just put it as one as of the systems in their benchmark but not explaining much of the connection and lineage which is a big thing in research publication. [1] Self-Distillation Enables Continual Learning: https://arxiv.org/abs/2601.19897 https://arxiv.org/abs/2601.19897 [2] Self-Distillation Enables Continual Learning: https://self-distillation.github.io/SDFT.html https://self-distillation.github.io/SDFT.html
- krackers 7mo agoGood explainer for on-policy self distillation from the authors https://x.com/siyan_zhao/status/2014372747862999382#m https://x.com/siyan_zhao/status/2014372747862999382#m
- neuzhou 7mo ago[flagged]
- try-working 7mo agomost codebases dont have traces to train on. if you use rlm-workflow you will build up rich traceability in the form of requirements, plans, implementation artifacts, along with worktree diffs. with these, you can then use self-distillation on models or use autoagent to improve your harness. https://github.com/doubleuuser/rlm-workflow https://github.com/doubleuuser/rlm-workflow
- Sim-In-Silico 7mo ago[dead]
- naasking 7mo agoIt's interesting that LLMs improve skills, especially on harder problems, just by practicing them. That's effectively what's going on.
- namuol 7mo ago> sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning It’s all moonspeak to me. I tried reading other comments that explain this and they all sounded different or contradictory. I’ve studied ML as a hobby years ago but this was before the LLM explosion. Guess I need to start over again?
- joshuaisaact 7mo agoThis was a really interesting paper but there's a massive gap in what they didn't try, which is inference-time temperature changes based on the fork/lock distinction. Maybe I'll try that myself, because it feels like it could be a great source of improvements. It would be really useful to see adaptive per-token sampling as an additional decode-only baseline.
- grumbelbart 7mo agoIs this some kind of calibration then? I'd expect that the probabilities automatically adjust during training, such that in "lock" mode, for example, syntax-breaking tokens have a very low probability and would not be picked even wich higher temperature.
- tatrions 7mo ago[flagged]
- BitsAndObjects 7mo agoHow is this not equivalent to training the model on the test data set? Yes it performs better at generating code for the target problems, but seemingly by becoming more tuned to the specific context of those problems (“context aware”), which suggests to me it would not generalise to real-world usage?
- LeonTing1010 7mo ago[flagged]
- revv00 7mo agoIndeed very insightful findings. Just wonder how it will affect visual reason tasks and other reasoning tasks.