It is impossible to create a perfect hash function, unless you have the complete dictionary (otherwise there is collision risk). If you have the full dictionary

It is impossible to create a perfect hash function, unless you have the complete dictionary (otherwise there is collision risk). If you have the full dictionary then there are perfect hash algorithms. So, in essence, you want to add metadata to switch dictionaries, it is feasible: {Dictionary}{id}. You would have to consider out of dictionary words too. By the way, note that UTF-8 is "compressed" (unlike UTF-32). Some languages don't use spaces to separate words, so you would have metadata problems in common languages like Chinese. What I have done, and works pretty well, is using dictionary compression: sort a dictionary by frequency, replace the word with the rank in the dictionary (in base64 or something high), use spaces to separate words and use a dedicated symbol to escape out of dictionary words. Oh, and I was compressing lower case, mostly Spanish words. It gave me around 50% compression AND it can be used to run NLP algorithms without decompressing the data (like word2vec or TF-IDF).