4 ms·
I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to
by spywaregorilla 3y ago
I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.
- ftxbro 3y ago> "I don't think that is intuitive at all." That's exactly the point. Every intuition is always on the side of feature engineering.
- spywaregorilla 3y agoNot really.
- whimsicalism 3y agoYou would be wrong at the scales we are talking about. The whole point is that it is unintuitive.
- yorwba 3y agoIf you replace a tokenizer with 5 bytes per token on average by a byte-level representation, you now need 5 times as much memory and (depending on the specifics of the attention mechanism) 11 to 25 times as much compute. At the scales we're talking about, that's quite a hefty price to pay, and it doesn't even take into account that you might need more layers to replace the processing that was implicitly done by the tokenizer.