3 ms·
The problem is that tokens don't all equate to the same size. A megabyte of some random json is a LOT more tokens than a megabyte of "Moby Dick".
by jermaustin1 1y ago
The problem is that tokens don't all equate to the same size. A megabyte of some random json is a LOT more tokens than a megabyte of "Moby Dick".