The "weights" bit is interesting. HuggingFace's SFTTrainer allows you to train on completions only if you want, but while that makes sense for us humans, LLMs a

The "weights" bit is interesting. HuggingFace's SFTTrainer allows you to train on completions only if you want, but while that makes sense for us humans, LLMs are generally better of learning to predict the entire input. This allows you to get the best of both worlds.