2 ms·

I kind of assumed the model would process the text 'directly', from what I understand, wouldn't this be biasing the input based on how you tokenise as it's loss

• convexstrictly 3mo ago
  Tokenization is done on the CPU. Models never see the raw characters. That's why you get trick questions like the number of r's in strawberry. There are many research papers on models using characters directly. One challenge is that effective context length is smaller.