3 ms·
It's not fundamental at all, that's what randomized sampling is for. Try tinkering with a base model and you'll be surprised how diverse it is. The semantic col
by orbital-decay 1mo ago
It's not fundamental at all, that's what randomized sampling is for. Try tinkering with a base model and you'll be surprised how diverse it is. The semantic collapse happens in post-training that is using the current methods.
- BoredomIsFun 1mo ago> Try tinkering with a base model and you'll be surprised how diverse it is. Have you tried? I have. Not much different from RLHFed; full of tics and slop, similar but slightly different from intsruction posttrains.
- orbital-decay 1mo agoYes, although admittedly I haven't tried recent ones which have a bunch of synthetic data in them (mainly to aid the reasoning), and are usually only available after mid-training. One look at the logits/output distribution and it's clear the base is pretty different.