3 ms·

You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.

• washadjeffmad 3y ago
  > ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step. If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step. People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority. That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".