3 ms·
Size, perhaps? Remember, most of the world does not speak English. I bet you baidu uses UTF-16 for its indexes.
by maaku 15y ago
Size, perhaps? Remember, most of the world does not speak English. I bet you baidu uses UTF-16 for its indexes.
- burgerbrain 15y agoIs size actually a concern in the modern world? With transmission and storage size is practically a nonissue and we have proper rather cheap compression anyway. And in memory? We have gigabytes of ram, who cares? Of the potential situations where it may become an issue that I can think of, pretty much none of them are on end user hardware.
- maaku 15y agoI reiterate my example of baidu. Just because size isn't a problem for your application, doesn't mean there aren't others out there dealing with petabytes of text.
- burgerbrain 15y ago"Of the potential situations where it may become an issue that I can think of, pretty much none of them are on end user hardware." Do what you need to on your companies machines, but keep it out of ecosystems it has no reason to be in.
- hrktb 15y agothe other way round, having a wildly variable size depending on the character can be a weakness. In japanese EUC was sometime used despite its shortcomings to force 16bit by character.
- pornel 15y agoBaidu uses GBK, which is more efficient than UTF-16, doesn't suffer from Han unification, and has ASCII compatibility similar to UTF-8.