4 ms·
what constitutes a token is super unintuitive. my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph
by jasonjmcghee 3y ago
what constitutes a token is super unintuitive.
my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters
In either case, I do consistently see spaces between characters included as part of the token following the space.
" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
- jasonjmcghee 3y agoNew winner " communication" at 14