BPE (byte pair encoding) as used in state of the art neural translation models , maximize the information per token by greedy iterative merging of the most comm

BPE (byte pair encoding) as used in state of the art neural translation models , maximize the information per token by greedy iterative merging of the most common (lowest information) tokens, until a predefined max number of tokens is reached https://en.m.wikipedia.org/wiki/Byte_pair_encoding https://en.m.wikipedia.org/wiki/Byte_pair_encoding