11 ms·
PEP 686 – Make UTF-8 mode default
- Macha 2y ago> And many other popular programming languages, including Node.js, Go, Rust, and Java uses UTF-8 by default. Oh, I missed Java moving from UTF-16 to UTF-8.
- PurpleRamen 2y agoSeems it happened two years ago, with Java 18.
- rootext 2y agoIt seems you are mixing two things: inner string representation and read/write encoding. Java has never used UTF-16 as default for the second.
- cryptonector 2y agoNot even on Windows?
- layer8 2y agoNo, file I/O on Windows in general doesn’t use UTF-16, but the regional code page, or nowadays UTF-8 if the application decides so.
- int_19h 2y agoDepends on what you define as "file I/O", though. NTFS filenames are UTF-16 (or rather UCS2). As far as file contents, there isn't really a standard, but FWIW for a long time most Windows apps - Notepad being the canonical example when asked to save anything as "Unicode" would save it as UTF-16.
- layer8 2y agoI'm talking about the default behavior of Microsoft's C runtime (MSVCRT.DLL) that everyone is/was using. UTF-16 text files are rather rare, as is using Notepad's UTF-16 options. The only semi-common use I know of is *.reg files saved from regedit. One issue with UTF-16 is that it has two different serializations (BE and LE), and hence generally requires a BOM to disambiguate.
- TheCycoONE 2y agoPowershell use to output utf-16 by default on Windows. It might still but it's been awhile since I needed to try.
- int_19h 2y agoThen you're talking about the C stdlib, which, yeah, is meant to use the locale-specific encoding on any platform, so it's not really a Windows thing specifically. But even then someone could use the CRT but call wfopen() rather than fopen() etc - this was actually not uncommon for Windows software precisely because it let you handle Unicode without having to work with Win32 API directly. Microsoft's implementation of fopen() also supports "ccs=..." to open Unicode text files in Unicode, and interestingly "ccs=UNICODE" will get you UTF-16LE, not UTF-8 (but you can do "ccs=UTF-8"). .NET also has this weird naming quirk where Encoding.Unicode is UTF-16, although there at least UTF-8 is the default for all text I/O classes like StreamReader if you don't specify the encoding. Still, many people didn't know better, and so some early .NET software would use UTF-16 for text I/O for no reason other than its developers believing that Encoding.Unicode is obviously what they are supposed to be using to "support Unicode", and so explicitly passing it everywhere.
- Dwedit 2y agoOr possibly confusing it with JavaScript, which treats strings as sequences of UTF-16 characters?
- hashmash 2y agoWith Java, the default encoding when converting bytes to strings was originally platform independent, but now it's UTF-8. UTF-16 and latin-1 encodings are (still*) used internally by the String class, and the JVM uses a modified UTF-8 encoding like it always has. * The String class originally only used UTF-16 encoding, but since Java 9 it also uses a single-byte-per-character latin-1 encoding when possible.
- Myrmornis 2y agoHm TIL, I thought that the string encoding argument to .decode() and .encode() was required, but now I see it defaults to "utf-8". Did that change at some point?
- _ache_ 2y agoYou can verify on the documentation by switching the version. So ... since 3.2: https://docs.python.org/3.2/library/stdtypes.html#bytes.decode https://docs.python.org/3.2/library/stdtypes.html#bytes.deco... In 3.1 it was the default encoding of string (the type str I guess). https://docs.python.org/3.1/library/stdtypes.html#bytes.decode https://docs.python.org/3.1/library/stdtypes.html#bytes.deco...
- aktiur 2y ago> In 3.1 it was the default encoding of string (the type str I guess). No, what was used was what sys.getdefaultencoding(), which was already UTF-8 in 3.1 (I checked the source code). At that time, the format used for representing `str` objects in memory depended on if you used a "narrow" (UTF-16) or "wide" (UTF-32) build of Python. Fortunately, wide and narrow builds were abandonned in Python 3.2, with a new way of representing strings : current Python will use ASCII if there's no non-ASCII char, UCS-2 –UTF-16 without surrogate pairs — if there is no codepoint higher than U+FFFF, and UTF-32 else. But that did not exist in 3.1, where you could either use the "narrow" build of python (that used UTF-16) or the "wide" build (that used UTF-32). See this article for a good overview of the history of strings in Python : https://tenthousandmeters.com/blog/python-behind-the-scenes-9-how-python-strings-work/ https://tenthousandmeters.com/blog/python-behind-the-scenes-...
- _ache_ 2y agoThank you ! The documentation was misleading about "default encoding of string".
- int_19h 2y agoThe simple thing to remember is that for all versions of Python going back 12 years, there's no such thing as "default encoding of string". A Python string is defined as a sequence of 32-bit Unicode codepoints, and that is how Python code perceives it in all respects. How it is stored internally is an implementation detail that does not affect you.
- Affric 2y agoMake UTF-8 default on Windows
- johannes1234321 2y agoSince Windows Version 1903 (May 2019 Update) they push for Utf-8. But Windows is a big pile of compatible legacy.
- tedivm 2y agoThat's exactly what this proposal (which has been accepted) is going to do.
- lolinder 2y agoI think they mean that the Windows operating system should default to UTF-8.
- pjc50 2y agoIn addition to ApiFunctionA and ApiFunctionW, introduce ApiFunction8? (times whole API surface) Introduce a #define UNICODE_NO_REALLY_ALL_UNICODE_WE_MEAN_IT_THIS_TIME ?
- cryptonector 2y agoApiFunctionA is UTF-8 capable. Needs a run-time switch too, not just compile-time.
- garaetjjte 2y agoIt's now possible, but for years the excuse was that MBCS encodings only supported characters up to 2 bytes.
- ComputerGuru 2y agoOnly under windows 11, I believe. And that switch is off by default.
- lexicality 2y ago> Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify encoding="utf-8" when they read text files encoded in UTF-8 "forget" or possibly simply aren't made well enough aware? I genuinely thought that python would only use UTF-8 for everything unless you explicitly ask it to do otherwise.
- deleted 2y ago[deleted]
- aktiur 2y agoIt actually depends! `bytes.decode` (and `str.encode`) have used UTF-8 as a default since at least Python 3. However, the default encoding used for decoding the name of files use ` sys.getfilesystemencoding()`, which is also UTF-8 on Windows and macos, but will vary with the locale on linux (specifically with CODESET). Finally, `open` will directly use `locale.getencoding()`.
- jillesvangurp 2y agoNot relying on flaky system defaults is a good thing. These things have a way of turning around and being different than what you assume them to be. A few years ago I was dealing with Ubuntu and some init.d scripts. One issue I ran into was that some script we used to launch Java (this was before docker) was running as root (bad, I know) and with a shell that did not set UTF-8 as the default like would be completely normal for regular users. And of course that revealed some bad APIs that we were using in Java that use the os default. Most of these things have variants that allow you to set the encoding at this point and a lot of static code checkers will warn you if you use the wrong one. But of course it only takes one place for this to start messing up content. These days it's less of an issue but I would simply not rely on the os to get this right ever for this. Most uses of encodings other than UTF-8 are extremely likely to be unintentional at this point. And if it is intentional, you should be very explicit about it and not rely on weird indirect configuration through the OS that may or may not line up. So, good change. Anything that breaks over this is probably better off with the simple fix added. And it's not worth leaving everything else as broken as it is with content corruption bugs just waiting to happen.
- ok_computer 2y agoI was using .gitignore generated by an aliased touch function in powershell. Despite my best efforts, I could not get git to respect its gitignore. Figured out the touched text file was utf-16 and basically not respected at all. Lesson learned I uuchanged a system default to utf-8 but just rely on my text editor now.
- account42 2y agoGlobal locales were a mistake in general, not just the encoding part. printf("%f", 4.2) should not magically output different strings depending on the environment, that just causes more problems than it solves. Instead you should have to explicitly pass the local information (or relevant parts of it) to functions that you want to make locale-dependent.
- Euphorbium 2y agoI thought it was default since python 3.
- lucb1e 2y agoYou may be thinking of strings where the u"" prefix was made obsolete in python3. Then again, trying on Python 2.7 just now, typing "éķů" results in it printing the UTF-8 bytes for those characters so I don't actually know what that u prefix ever did, but one of the big py2-to-3 changes was strings having an encoding and byte strings being for byte sequences without encodings This change seems to be about things like open('filename', mode='r') mainly on Windows where the default encoding is not UTF-8 and so you'd have to specify open('filename', mode='r', encoding='UTF-8')
- aktiur 2y ago> strings having an encoding and byte strings being for byte sequences without encodings You got it kind of backwards. `str` are sequence of unicode codepoints (not UTF-8, which is a specific encoding for unicode codepoints), without reference to any encoding. `bytes` are arbitrary sequence of octets. If you have some `bytes` object that somehow stands for text, you need to know that it is text and what its encoding is to be able to interpret it correctly (by decoding it to `str`). And, if you got a `str` and want to serialize it (for writing or transmitting), you need to choose an encoding, because different encodings will generate different `bytes`. As an example : >>> "évènement".encode("utf-8") b'\xc3\xa9v\xc3\xa8nement' >>> "évènement".encode("latin-1") b'\xe9v\xe8nement'
- lucb1e 2y ago> `str` are sequence of unicode codepoints [...] without reference to any encoding I guess I see it from the programmer's perspective: to handle bytes coming from the disk/network as a string, I need to specify an encoding, so they are (to me) byte sequences with an encoding assigned. Didn't realize strings don't have an encoding in Python's internal string handling but are, instead, something like an array of integers pointing to unicode code points. Not sure if this viewpoint means I am getting it backwards but I can see how that was phrased poorly on my part!
- nerdponx 2y agoDefault text file encoding being platform-dependent always drove me nuts. This is a welcome change. I also appreciate that they did not attempt to tackle filesystem encoding here, which is a separate issue that drives me nuts, but separately.
- layer8 2y agoHistorically it made sense, when most software was local-only, and text files were expected to be in the local encoding. Not just platform-dependent, but user’s preferred locale-dependent. This is also how the C standard library operates. For example, on Unix/Linux, using iso-8859-1 was common when using Western-European languages, and in Europe it became common to switch to iso-8859-15 after the Euro was introduced, because it contained the € symbol. UTF-8 only began to work flawlessly in the later aughts. Debian switched to it as the default with the Etch release in 2010.
- anthk 2y agoEmacs was amazing for that; builtin text encoders/decoders/transcoders for everything.
- hollerith 2y agoMy experience was that brittleness around text encoding in Emacs (versions 22 and 23 or so) was a constant source of annoyance for years. IIRC, the main way this brittleness bit me was that every time a buffer containing a non-ASCII character was saved, Emacs would engage me in a conversation (which I found tedious and distracting) about what coding system I would like to use to save the file, and I never found a sane way to configure it to avoid such conversations even after spending hours learning about how Emacs does coding systems: I simply had to wait (a year or 3) for a new version of Emacs in which the code for saving buffers worked better. I think some people like engaging in these conversations with their computers even though the conversations are very boring and repetitive and that such conversation-likers are numerous among Emacs users or at least Emacs maintainers.
- a-french-anon 2y agoWhy not utf-8-sig, though? It handles optional BOMs. Had to fix a script last week that choked on it.
- otteromkram 2y agoOnly one script? Out of how many?
- hermitdev 2y agoNot the OP, but I see this pop up quite frequently in ETL, usually handling csv files.
- orf 2y agoBecause changing Python to silently prefixing all IO with an invisible BOM isn’t a good idea.
- int_19h 2y agoThe expectation isn't for it to generate BOM in the output, but to handle BOM gracefully when it occurs in the input.
- deleted 2y ago[deleted]
- shpx 2y ago> On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file https://docs.python.org/3/library/codecs.html https://docs.python.org/3/library/codecs.html The codec you're imagining would also make reading a file and writing it back change the file if it contains a BOM.
- int_19h 2y agoIndeed it would, but since codecs are only used for files that are semantically text, and in such files BOM is basically a legacy no-op marker, it's not actually a problem. Naive code using text I/O APIs would also have this issue with line endings, for example, so there's precedent for not providing the perfect roundtrip experience (that's what bytes I/O is for).
- anthk 2y agoOn UTF-8, the Linux framebuffer should had had a good utf8 support (a proper one, not 256/512 glyphs) long ago. Even GNU Hurd since 2007 or so it had a better 'terminal console' with UTF8 support. It's 2024.
- anonym29 2y ago[flagged]
- terr-dav 2y agoIt sounds like you're building an empire over there, albeit a tiny one with very large gates.
- deleted 2y ago[deleted]
- anordal 2y agoThe following heuristic has become increasingly true over the last couple of decades: If you have some kind of "charset" configuration anywhere, and it's not UTF-8, it's wrong. Python 2 was charset agnostic, so it always worked, but the improvement with Python 3 was not only an improvement – how to tell a Python 3 script from a Python 2 script? * If it contains the string "utf-8", it's Python3. * If it only works if your locale is C.UTF-8, it's Python3. Needless to say, I welcome this change. The way I understand it, it would "repair" Python 3.
- Aerbil313 2y agoNice. Now the only thing we need is JS to switch to UTF-8. But of course JS can't improve, because unlike any other programming language, we need to be compatible with code written in 1995.
- shpx 2y agoThis is about when you ask Python to open a file "as text", what encoding it will use by default. The internal representation of strings is a different matter and, like JavaScript, Python doesn't "just use UTF-8" for that.
- Animats 2y agoIs the internal encoding in CPython UTF-8 yet? You can index through Python strings with a subscript, but random access is rare enough that it's probably worthwhile to lazily index a string when needed. If you just need to advance or back up by 1, you don't need an index. So an internal representation of UTF-8 is quite possible.
- rogerbinns 2y agoThe PyUnicode object is what represents a str. If the UTF-8 bytes are ever requested, then a bytes object is created on demand and cached as part of the PyUnicode, being freed with the PyUnicode itself is freed. Separately from that the codepoints making up the string are stored in a straight forward array allowing random access. The size of each codepoint can be 1, 2, or 4 bytes. When you create a PyUnicode you have to specify the maximum codepoint value which is rounded up to 127, 255, 65535, or 1,114,111. That determines if 1, 2, or 4 bytes is used. If the maxiumum codepoint value is 127 then that array representation can be used for the UTF-8 directly. So the answer to your question is that many strings are stored as UTF-8 because all the codepoints are <= 127. Separately from that, advancing through strings should not be done by codepoints anyway. A user perceived character (aka grapheme cluster) is made up of one or more codepoints. For example an e with an accent could be the e codepoint followed by a combining accent codepoint. The phoenix emoji is really the bird emoji, a zero width joiner, and then fire emoji. Some writing systems used by hundreds of millions of people are similar to having consonants, with combining marks to represent vowels. This - - is 5 codepoints. There is a good blog post diving into it and how various languages report its "length". https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/ Source: I've just finished implementing Unicode TR29 which covers this for a Python C extension.