4 ms·

ASCII is 7-bit (encoded in 8 bits - the high bit is ignored) and UTF-8 takes 8 bits for most characters, but can take 16+ bits for some characters. Node is bui

by mhansen 16y ago

ASCII is 7-bit (encoded in 8 bits - the high bit is ignored) and UTF-8 takes 8 bits for most characters, but can take 16+ bits for some characters. Node is built for massive scalability on applications that (mostly) pass text from one source to another. Thus, having to convert the encoding of every string that passes through node can be a bottlenck. Felixge has a good writeup of this: http://debuggable.com/posts/streaming-utf-8-with-node-js:4bf28e8b-a290-432f-a222-11c1cbdd56cb http://debuggable.com/posts/streaming-utf-8-with-node-js:4bf...

pmjordan 16y ago
UTF-8 takes 8 bits for most characters It should be noted that "most" here presumably means "most characters in an average English or western/central European language text" as out of the ~2^21 (~2 million) Unicode code points, only 128 are represented using 8 bits in UTF-8.
- pornel 16y ago
  It doesn't matter. Whenever ASCII is an option, UTF-8 is optimal too. ASCII is not an option for languages other than average English with poor typography and inability to deal with foreign names and addresses (e.g. LinkedIn made horrible mistake of using Latin1 initially. I still have contacts with &xxxx; visible in their names). I think node.js should use UTF-8 by default, and require users to consciously switch bottleneck parts of their apps to ASCII.
  - pmjordan 16y ago
    I wasn't stating my opinion in my last post, just facts/clarifications. But yes, I agree that UTF-8 would be a better default than ASCII unless someone provides hard evidence that encoding/decoding is a severe performance bottleneck in most real applications. (even then, I'd default to the "correct", not the fastest)
Sephr 16y ago
Felix doesn't seem to realize that JavaScript already has native functions for this. All of his code can be simplified to decodeURIComponent(escape(utf8ByteString)).

by mhansen 16y ago

pmjordan 16y ago

UTF-8 takes 8 bits for most characters It should be noted that "most" here presumably means "most characters in an average English or western/central European language text" as out of the ~2^21 (~2 million) Unicode code points, only 128 are represented using 8 bits in UTF-8.

pornel 16y ago
It doesn't matter. Whenever ASCII is an option, UTF-8 is optimal too. ASCII is not an option for languages other than average English with poor typography and inability to deal with foreign names and addresses (e.g. LinkedIn made horrible mistake of using Latin1 initially. I still have contacts with &xxxx; visible in their names). I think node.js should use UTF-8 by default, and require users to consciously switch bottleneck parts of their apps to ASCII.
- pmjordan 16y ago
  I wasn't stating my opinion in my last post, just facts/clarifications. But yes, I agree that UTF-8 would be a better default than ASCII unless someone provides hard evidence that encoding/decoding is a severe performance bottleneck in most real applications. (even then, I'd default to the "correct", not the fastest)

Sephr 16y ago

Felix doesn't seem to realize that JavaScript already has native functions for this. All of his code can be simplified to decodeURIComponent(escape(utf8ByteString)).