3 ms·
For 5 more tokens (i.e. two space, four space, eight space, etc.), they could have converted up to 32 spaces into log2(N) tokens. e.g. 31 spaces would be 16 +
by denverllc 3y ago
For 5 more tokens (i.e. two space, four space, eight space, etc.), they could have converted up to 32 spaces into log2(N) tokens.
e.g. 31 spaces would be 16 + 8 + 4 + 2 + 1 = 5 tokens; 15 spaces would be 8 + 4 + 2 + 1, etc; but for a lot of code the spaces are an even multiple (such as when a tab = 4 spaces or tab = 2 spaces) so even fewer would be required.
If you took all of the Python code and queried how many spaces are used, I would wager most lines have four spaces, a few have eight, and even fewer have more than that.
If the Python code were written with four spaces = tab, then it wouldn't have cost any more tokens than squashing the repeated spaces (since a "four-space" token consumes the same as a "one-space" token).