3 ms·

I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it lead

I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?

m00x 2y ago
Not instruct tuning, you use it in general training. If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.