13 ms·
What is 16k referring to here
by dmix 4y ago
What is 16k referring to here
- refulgentis 4y agoI’ve begun to grok it as “the amount of ram I have to play in before I have to start sharding work” more literally and correctly, it’s the maximum number of tokens in the input and output, combined, where a token is 4/3 of a word So we’re shifting from 5K words maximum to 40K (per sibling comment, who pointed out 32K context leaked as well)
- MattRix 4y agoA minor correction: a token is 3/4 of a word. ie. it’s slightly smaller than a word, not larger.