3 ms·
Does anyone have any understanding why these models don't simply output unicode, chunked into 12 or 16 bit words or whatever? The token lists are 100.000 tokens
by isoprophlex 3y ago
Does anyone have any understanding why these models don't simply output unicode, chunked into 12 or 16 bit words or whatever? The token lists are 100.000 tokens long, 65336 positions derived from bit sequences shouldn't be a problem right?