5 ms·

This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just

This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know. It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it. Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.

jameshart 11y ago
The problem with elegant backwards compatibility with ASCII is that few systems ever stuck to ASCII - everybody used different, incompatible extensions to ASCII, like the ISO 8859 family and codepage 1252. So by being elegantly backwards compatible with ASCII, UTF-8 also manages to be subtly incompatible with the majority of almost-aSCII data that exists, in ways that sometimes don't matter. Until they do.
- kps 11y ago
  ISO 8859 (or big brother ISO 10367) with ISO 4873 is fairly sane, backwards compatible with ASCII, distinguishable from UTF-8, and historically supported by X terminals… but not much else. ‘Plain ASCII’ is not often fully supported either; how many people even know that the standard included composing accented characters by overstriking (e.g. a BS " → ä)?
  - jameshart 11y ago
    The ASCII standard has nothing to say about backspace overstriking - that would be something defined by a terminal, file format, or wire specification. In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. Some terminal standards will do so on an LF, others just move the cursor down on LF; file and wire specs need to take a position on how they want to represent a line break; and now we get to the point where CRLF is a magic sequence needed to trigger a line break in some formats and it is completely unrelated to how a particular terminal behaves.
    - kps 11y ago
      > The ASCII standard has nothing to say about backspace overstriking It seems impossible to get ANSI copies of old standards (even for money) but the ECMA (1973) printing says: 3.2 Diacritical Signs (Positions: 2/2, 2/7, 2/12, 5/14, 6/0, 7/14) In the 7-bit character set, some printing symbols may be designed to permit their use for the composition of acce‐ nted letters when necessary for general interchange of information. A sequence of three characters, comprising a letter, BACKSPACE and one of these symbols, is needed for this composition; the symbol is then regarded as a diacrit‐ ical sign. It should be noted that these symbols take on their diacritical significance only when they precede or follow the character BACKSPACE; for example, the symbol corresponding to the code combination 2/7 normally has the significance of APOSTROPHE, but becomes the diacritical sign ACUTE ACCENT when preceded or followed by the character BACKSPACE. This is precisely the reason ASCII 1967 replaced ← with _ and ↑ with ^ (explicitly still “circumflex accent” in Unicode) and added ` (“grave accent”, likewise). ANSI made this optional in the 1986 revision, in §2.1.2 — “The use of BS for forming composite characters is not required.” — with a note that it would likely be removed from a future revision (but there never was another one). > In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. It does; it just says something slightly unfortunate about code 0x0A. CR Carriage Return A format effector which moves the active position to the first character position *on the same line*. LF Line Feed A format effector which advances the active position to the *same character position* of the next line. [Italics added] But then it says: The Format Effectors are intended for equipment in which horizontal and vertical movements are effected separately. If equipment requires the action of CARRIAGE RETURN to be comhined with a vertical movement, the Format Effector for that vertical movement may be used to effect the combined movement. For example, if NEW LINE (symbol NL, equivalent to CR + LF) is required, FE2 shall be used to represent it. This substitution requires agreement between the sender and the recipient of the data. The use of these combined functions may be restricted for international transmission on general switched telecommunication networks (telegraph and telephone networks). So CR LF will unambiguously get you the first position on the next line. The code for LF is allowed to be replaced by NL by “agreement”, but CR can't move to the next line.
ntauthority 11y ago
NT was initially designed (1989-1990) before UTF-8 came along. UTF-16 instead of UCS-2 was shoehorned in later with Win2k.
Someone 11y ago
Realistically, there is no way NT could have used UTF-8 in-memory. UTF-8 was scribbled on a placemat on September 2, 1992 and officially presented in January 1993 (https://en.m.wikipedia.org/wiki/UTF-8#History https://en.m.wikipedia.org/wiki/UTF-8#History) Windows NT was released in July 1993, but development started in 1989 (https://en.m.wikipedia.org/wiki/Windows_NT#Development https://en.m.wikipedia.org/wiki/Windows_NT#Development) Also, even after UTF-8 was accepted to be a good idea, it wasn't considered a good idea for in-memory usage; indexing UCS-2 encoded strings is way easier, and UTF-16 didn't exist yet. It arrived with Unicde 2 in July 1996 (https://en.m.wikipedia.org/wiki/UTF-16#History https://en.m.wikipedia.org/wiki/UTF-16#History)

jameshart 11y ago

The problem with elegant backwards compatibility with ASCII is that few systems ever stuck to ASCII - everybody used different, incompatible extensions to ASCII, like the ISO 8859 family and codepage 1252. So by being elegantly backwards compatible with ASCII, UTF-8 also manages to be subtly incompatible with the majority of almost-aSCII data that exists, in ways that sometimes don't matter. Until they do.

kps 11y ago
ISO 8859 (or big brother ISO 10367) with ISO 4873 is fairly sane, backwards compatible with ASCII, distinguishable from UTF-8, and historically supported by X terminals… but not much else. ‘Plain ASCII’ is not often fully supported either; how many people even know that the standard included composing accented characters by overstriking (e.g. a BS " → ä)?
- jameshart 11y ago
  The ASCII standard has nothing to say about backspace overstriking - that would be something defined by a terminal, file format, or wire specification. In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. Some terminal standards will do so on an LF, others just move the cursor down on LF; file and wire specs need to take a position on how they want to represent a line break; and now we get to the point where CRLF is a magic sequence needed to trigger a line break in some formats and it is completely unrelated to how a particular terminal behaves.
  - kps 11y ago
    > The ASCII standard has nothing to say about backspace overstriking It seems impossible to get ANSI copies of old standards (even for money) but the ECMA (1973) printing says: 3.2 Diacritical Signs (Positions: 2/2, 2/7, 2/12, 5/14, 6/0, 7/14) In the 7-bit character set, some printing symbols may be designed to permit their use for the composition of acce‐ nted letters when necessary for general interchange of information. A sequence of three characters, comprising a letter, BACKSPACE and one of these symbols, is needed for this composition; the symbol is then regarded as a diacrit‐ ical sign. It should be noted that these symbols take on their diacritical significance only when they precede or follow the character BACKSPACE; for example, the symbol corresponding to the code combination 2/7 normally has the significance of APOSTROPHE, but becomes the diacritical sign ACUTE ACCENT when preceded or followed by the character BACKSPACE. This is precisely the reason ASCII 1967 replaced ← with _ and ↑ with ^ (explicitly still “circumflex accent” in Unicode) and added ` (“grave accent”, likewise). ANSI made this optional in the 1986 revision, in §2.1.2 — “The use of BS for forming composite characters is not required.” — with a note that it would likely be removed from a future revision (but there never was another one). > In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. It does; it just says something slightly unfortunate about code 0x0A. CR Carriage Return A format effector which moves the active position to the first character position *on the same line*. LF Line Feed A format effector which advances the active position to the *same character position* of the next line. [Italics added] But then it says: The Format Effectors are intended for equipment in which horizontal and vertical movements are effected separately. If equipment requires the action of CARRIAGE RETURN to be comhined with a vertical movement, the Format Effector for that vertical movement may be used to effect the combined movement. For example, if NEW LINE (symbol NL, equivalent to CR + LF) is required, FE2 shall be used to represent it. This substitution requires agreement between the sender and the recipient of the data. The use of these combined functions may be restricted for international transmission on general switched telecommunication networks (telegraph and telephone networks). So CR LF will unambiguously get you the first position on the next line. The code for LF is allowed to be replaced by NL by “agreement”, but CR can't move to the next line.

ntauthority 11y ago

NT was initially designed (1989-1990) before UTF-8 came along. UTF-16 instead of UCS-2 was shoehorned in later with Win2k.

Someone 11y ago

Realistically, there is no way NT could have used UTF-8 in-memory. UTF-8 was scribbled on a placemat on September 2, 1992 and officially presented in January 1993 (https://en.m.wikipedia.org/wiki/UTF-8#History https://en.m.wikipedia.org/wiki/UTF-8#History) Windows NT was released in July 1993, but development started in 1989 (https://en.m.wikipedia.org/wiki/Windows_NT#Development https://en.m.wikipedia.org/wiki/Windows_NT#Development) Also, even after UTF-8 was accepted to be a good idea, it wasn't considered a good idea for in-memory usage; indexing UCS-2 encoded strings is way easier, and UTF-16 didn't exist yet. It arrived with Unicde 2 in July 1996 (https://en.m.wikipedia.org/wiki/UTF-16#History https://en.m.wikipedia.org/wiki/UTF-16#History)