4 ms·

> I agree that UTF-8 is the better default Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the bett

> I agree that UTF-8 is the better default Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the better default can be different. > side from it being the default on Windows Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.

outworlder 8y ago
Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.
- Const-me 8y ago
  > Better default for everything. Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server. When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value. Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-jni-utf-8-strings/ http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.
Kwpolska 8y ago
> python 3 (unless compiled with UTF32 support but the default is UTF16) Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).
int_19h 8y ago
> python 3 (unless compiled with UTF32 support but the default is UTF16) The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.
masklinn 8y ago
> python 3 (unless compiled with UTF32 support but the default is UTF16) Not the case since PEP 393 / Python 3.3. Also pretty much all Linux distros used wide builds. > Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings. Most of these don't actually use UTF16 but UCS2 with surrogates (sometimes nicknamed WTF16).
- Const-me 8y ago
  > Most of these don't actually use UTF16 but UCS2 with surrogates Most of these actually use UTF16. .NET has System.Globalization.StringInfo class, Java has string methods like String.codePointCount. Operator [] returns 2-byte code points in these languages because backward compatibility. Newer languages don’t need to be backward compatible see e.g. https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html#ID290 https://docs.swift.org/swift-book/LanguageGuide/StringsAndCh... but their internal format is still UTF-16. If you need to interop with any of these languages, your life will be much easier if your C++ code uses UTF-16 as well.
  - masklinn 8y ago
    > Most of these actually use UTF16. All of .Net, Java and Javascript[0] allow unpaired surrogates, meaning none of them uses UTF-16. I expect the same happen for Symbian, OpenOffice, Qt, … So did Python on pre-FSR narrow builds. Swift is migrating to UTF-8 as part of its ABI stabilisation: https://forums.swift.org/t/string-s-abi-and-utf-8/17676 https://forums.swift.org/t/string-s-abi-and-utf-8/17676 [0] unclear for ObjC/NSString but given the string interface leaks it being an array of utf-16 code units I'd expect the same edit: https://stackoverflow.com/a/33558934/8182118 https://stackoverflow.com/a/33558934/8182118 indicates that both NSString and Swift strings can contain unpaired surrogates, so not UTF-16 either.
    - Const-me 8y ago
      > meaning none of them uses UTF-16 Depends on your definition of “uses”, and I disagree with yours. It’s always technically possible to make invalid data. For example, you can write a web app that will send Content-type: application/json and invalid utf-8 in the response. Yes, technically these programming languages allow unpaired surrogates in strings. This is not necessarily a bad thing, there’re valid uses for such strings. For example, you can concatenate strings without decoding UTF16 into code points, i.e. the code will be slightly faster, but while you’re concatenating, in some moment of time the destination will contain an unpaired surrogate. I hope you agree golang uses utf-8 strings. Despite that, the unicode/utf8 standard package has ValidString function, which means you can still create invalid strings in go: https://golang.org/src/unicode/utf8/utf8.go https://golang.org/src/unicode/utf8/utf8.go Does it mean golang uses wtf-8 instead of utf-8?

outworlder 8y ago

Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.

Const-me 8y ago
> Better default for everything. Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server. When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value. Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-jni-utf-8-strings/ http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.

Kwpolska 8y ago

> python 3 (unless compiled with UTF32 support but the default is UTF16) Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).

int_19h 8y ago

> python 3 (unless compiled with UTF32 support but the default is UTF16) The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.

masklinn 8y ago

> python 3 (unless compiled with UTF32 support but the default is UTF16) Not the case since PEP 393 / Python 3.3. Also pretty much all Linux distros used wide builds. > Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings. Most of these don't actually use UTF16 but UCS2 with surrogates (sometimes nicknamed WTF16).

Const-me 8y ago
> Most of these don't actually use UTF16 but UCS2 with surrogates Most of these actually use UTF16. .NET has System.Globalization.StringInfo class, Java has string methods like String.codePointCount. Operator [] returns 2-byte code points in these languages because backward compatibility. Newer languages don’t need to be backward compatible see e.g. https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html#ID290 https://docs.swift.org/swift-book/LanguageGuide/StringsAndCh... but their internal format is still UTF-16. If you need to interop with any of these languages, your life will be much easier if your C++ code uses UTF-16 as well.
- masklinn 8y ago
  > Most of these actually use UTF16. All of .Net, Java and Javascript[0] allow unpaired surrogates, meaning none of them uses UTF-16. I expect the same happen for Symbian, OpenOffice, Qt, … So did Python on pre-FSR narrow builds. Swift is migrating to UTF-8 as part of its ABI stabilisation: https://forums.swift.org/t/string-s-abi-and-utf-8/17676 https://forums.swift.org/t/string-s-abi-and-utf-8/17676 [0] unclear for ObjC/NSString but given the string interface leaks it being an array of utf-16 code units I'd expect the same edit: https://stackoverflow.com/a/33558934/8182118 https://stackoverflow.com/a/33558934/8182118 indicates that both NSString and Swift strings can contain unpaired surrogates, so not UTF-16 either.
  - Const-me 8y ago
    > meaning none of them uses UTF-16 Depends on your definition of “uses”, and I disagree with yours. It’s always technically possible to make invalid data. For example, you can write a web app that will send Content-type: application/json and invalid utf-8 in the response. Yes, technically these programming languages allow unpaired surrogates in strings. This is not necessarily a bad thing, there’re valid uses for such strings. For example, you can concatenate strings without decoding UTF16 into code points, i.e. the code will be slightly faster, but while you’re concatenating, in some moment of time the destination will contain an unpaired surrogate. I hope you agree golang uses utf-8 strings. Despite that, the unicode/utf8 standard package has ValidString function, which means you can still create invalid strings in go: https://golang.org/src/unicode/utf8/utf8.go https://golang.org/src/unicode/utf8/utf8.go Does it mean golang uses wtf-8 instead of utf-8?