4 ms·

Can't you tokenize in preloading on demand?

• marcelroed 3mo ago
  You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult. In practice every training project I've worked on does tokenization in a separate data processing phase.