3 ms·

> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: <start_of_turn> > E

> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: <start_of_turn> > End of conversation turn: <end_of_turn> > Beginning of sequence: <bos> > End of sequence: <eos> You know I keep wondering why <bos> and <eos> tokens are even a thing in general. No model is tuned to keep generating multiple turns after its <end_of_turn> equivalent is sent, and what's the point of <bos> when you're parsing the entire context anyway. If it's an attempt to ignore text before it... then why is that text there? Just remove it from context, you're throwing away compute.

danielmarkbruce 2y ago
think about training.
- moffkalast 2y ago
  I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
  - m00x 2y ago
    Not instruct tuning, you use it in general training. If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.
alekandreev 2y ago
Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute. To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not related.
- thomasahle 2y ago
  You can just do that my shaping the attention mask, no? That also gives you an actual guarantee that no information is leaked between conversations.
  - suryabhupa 2y ago
    In practice, and at scale, that's exactly what having <bos> and <eos> tokens allow you to easily and programmatically do.
  - danielmarkbruce 2y ago
    You can't pack multiple examples into a single row of a matrix without knowing where one begins and one ends.

danielmarkbruce 2y ago

think about training.

moffkalast 2y ago
I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
- m00x 2y ago
  Not instruct tuning, you use it in general training. If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.

alekandreev 2y ago

Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute. To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not related.

thomasahle 2y ago
You can just do that my shaping the attention mask, no? That also gives you an actual guarantee that no information is leaked between conversations.
- suryabhupa 2y ago
  In practice, and at scale, that's exactly what having <bos> and <eos> tokens allow you to easily and programmatically do.
- danielmarkbruce 2y ago
  You can't pack multiple examples into a single row of a matrix without knowing where one begins and one ends.