8 ms·
But it could be. It's just less efficient.
by davrosthedalek 5mo ago
But it could be. It's just less efficient.
- jrahmy 5mo agoI don't see how. You could ask a neural network to do the tokenization I suppose, but in doing so you'd have to convert the prompt into tokens via the same deterministic process the network was trained on, essentially just moving the exact same process up one layer.
- deleted 5mo ago[deleted]