4 ms·

It won't catch on for other reasons too, such as that existing software won't understand Unicode tags and will either not preserve them or break their forms so

It won't catch on for other reasons too, such as that existing software won't understand Unicode tags and will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here. Unicode tag codepoints simply cannot be reliable. Nor can Unicode tag codepoints provide sufficient functionality to replace XML (or HTML, or ...). Even the subset they can replace/displace would require revamping XML/XSLT processors in order to make XSLT able to see them -- that's a terrible imposition on the rest of the world. Some of us hate UTF-16, but there it is and we can't get rid of it. Others hate UTF-8, and.. the same thing applies. XML exists, it's widely used, it's not going away no matter how much anyone dislikes it. Moving parts of HTML or XML into Unicode will not work for the reasons given above. Please dedicate your energy and ideas to innovating in other ways than this.

derefr 8y ago
> will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here You're assuming a representation that encodes such markup as ["start of bold" codepoint], ["end of bold" codepoint]. I agree that this would be bad, for the same reason that the Unicode "language" tags were bad. There are other representation options, though. You can instead do exactly what Unicode's Ruby-character support does, and just make a markup annotation last for exactly one grapheme-cluster. A run of bold characters, then, is a run of grapheme-clusters where each grapheme-cluster has been individually bold-annotated. There's no real way to "break" that. It's inefficient, of course, but the point of it isn't to be a universally-optimal format, but a markup-format-of-last-resort when there's no higher-level markup layer in the document format to hold styling information. To be clear: there's demand for a "markup-format-of-last-resort." In fact, people are already abusing other Unicode features to create one, and are quite happy with the resulting "styled" Unicode documents, which are—horrifyingly—proliferating across the internet and breaking semantic machine-readability for any future archival uses of that text. If you can't guess, I'm talking about people replacing the Unicode codepoints of their text with entirely-different Unicode codepoints, that just happen to already look like the "styled text" version of the text they want. For example, you've got 𝓽𝓱𝓲𝓼 𝓽𝓮𝔁𝓽, ʇɥᴉs ʇǝxʇ, t̶h̶i̶s̶ ̶t̶e̶x̶t̶, etc. These are the outputs of "text generators" (online IMEs, essentially) that take your Unicode plaintext input text, and output Unicode plaintext output text that is "styled." Now picture a Unicode feature to obviate such "text generators"—where, instead of two codepoints ["𝓽"]["𝓱"] that are both 1. impossible to type on a keyboard and 2. not actually the same characters semantically as ["t"]["h"], you've instead got four codepoints: [next-with-italics]["t"][next-with-italics]["h"]. You typed ["t"]["h"], and then selected them both and hit the "I" button in your editor, and then copied the result, and that's what's now in the copy buffer. Nothing would break—text editors already understand how to keep Unicode combining diacritics together with their base codepoint just fine, and this is just another use of the same logic. Similarly, the NFD representation of [next-with-italics]["t"][next-with-italics]["h"] would just be... ["t"]["h"]. Stick your four-codepoint sequence in a DB fulltext field, and it'll index it no problem. > Nor can Unicode tag codepoints provide sufficient functionality to replace XML Oh, I would never suggest that. Structural markup has no place in Unicode. I just think that there is a thing that is a subset of "rich text" but a superset of "Unicode plaintext" and maps to exactly the sort of text-styles that a publishing editor laying out InDesign flows, would let their contributing writers use in the RTF documents data-bound to those flows; or which a blogging engine like Twitter would let their users use to style their tweets. Like I said, this doesn't belong in the Unicode standard. Such a format can be considered an additional format layer between Unicode plaintext and a document format. It just so happens, though, that you can take advantage of the Unicode-rendering capabilities of most rich-text layout engines, to support such markup. Same as how the VT100 spec took advantage of the logic that already existed to respond to newlines and such, and introduced inline escape sequences—nominally "a higher layer", but in practice a thing that is conflated with the layer below it, such that any terminal program that claims to output "plaintext" might actually be outputting plaintext-with-ANSI-escapes without mentioning it.
- dragonwriter 8y ago
 > I'm talking about something that you'd use in exactly the situations where people currently use "text generators" to output those abominations, The situations, AFAICT, are often to defeat text filters.
 - derefr 8y ago
 That use-case does certainly exist (it's what you see used on e.g. escort ads on websites where there shouldn't be escort ads), and obviously, such people wouldn't want to switch to using more machine-comprehensible styling. While this might represent a large percentage of the "pre-baked styling" Unicode text in the wild by volume of text, there aren't that many people who have created all this text (most of the corpus is spam repostings of the same messages with randomly-varied styling.) Rather, I think the majority of the authors of "pre-baked styling" Unicode text are simply people posting to websites like Twitter or Facebook, or a Disqus comments section, where styling is either not allowed, or doesn't have all the options the author wants. Or, to put that another way: the authors are mostly teenagers who care more about the aesthetics of their text than its comprehensibility. And that's surely the market that the Unicode consortium cares the most about—why else would we have so many emoji? ;)
- cryptonector 8y ago
 Attaching style to each glyph is problematic too, since now they can get spread around unintentionally, and we'll need new regexp code, and much else. No, the semantics of tagging inside Unicode are uniformly bad. I understand that people abuse Unicode already to get some such effects, but it's mostly a novelty, and I would not seriously use any software making use of these tricks. A standard way to mark up style in Unicode would certainly be better than Unicode abuse, but it would still come with a bunch of difficult associated issues -- are they really better than whatever issues you have with XML? and are they better for all of us?? BTW, games with BS (backspace) and overstrike for bold and underscore for underline still work in Unix terminals... That might be the best argument for having some stylistic composing codepoints in Unicode... after all, ASCII was designed in part to be usable with BS for expressing diacritical marks (accents, umlauts, cedilles, ...), making it a variable length internationalized encoding! But still, then less we complicate Unicode, the better.

derefr 8y ago

> will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here You're assuming a representation that encodes such markup as ["start of bold" codepoint], ["end of bold" codepoint]. I agree that this would be bad, for the same reason that the Unicode "language" tags were bad. There are other representation options, though. You can instead do exactly what Unicode's Ruby-character support does, and just make a markup annotation last for exactly one grapheme-cluster. A run of bold characters, then, is a run of grapheme-clusters where each grapheme-cluster has been individually bold-annotated. There's no real way to "break" that. It's inefficient, of course, but the point of it isn't to be a universally-optimal format, but a markup-format-of-last-resort when there's no higher-level markup layer in the document format to hold styling information. To be clear: there's demand for a "markup-format-of-last-resort." In fact, people are already abusing other Unicode features to create one, and are quite happy with the resulting "styled" Unicode documents, which are—horrifyingly—proliferating across the internet and breaking semantic machine-readability for any future archival uses of that text. If you can't guess, I'm talking about people replacing the Unicode codepoints of their text with entirely-different Unicode codepoints, that just happen to already look like the "styled text" version of the text they want. For example, you've got 𝓽𝓱𝓲𝓼 𝓽𝓮𝔁𝓽, ʇɥᴉs ʇǝxʇ, t̶h̶i̶s̶ ̶t̶e̶x̶t̶, etc. These are the outputs of "text generators" (online IMEs, essentially) that take your Unicode plaintext input text, and output Unicode plaintext output text that is "styled." Now picture a Unicode feature to obviate such "text generators"—where, instead of two codepoints ["𝓽"]["𝓱"] that are both 1. impossible to type on a keyboard and 2. not actually the same characters semantically as ["t"]["h"], you've instead got four codepoints: [next-with-italics]["t"][next-with-italics]["h"]. You typed ["t"]["h"], and then selected them both and hit the "I" button in your editor, and then copied the result, and that's what's now in the copy buffer. Nothing would break—text editors already understand how to keep Unicode combining diacritics together with their base codepoint just fine, and this is just another use of the same logic. Similarly, the NFD representation of [next-with-italics]["t"][next-with-italics]["h"] would just be... ["t"]["h"]. Stick your four-codepoint sequence in a DB fulltext field, and it'll index it no problem. > Nor can Unicode tag codepoints provide sufficient functionality to replace XML Oh, I would never suggest that. Structural markup has no place in Unicode. I just think that there is a thing that is a subset of "rich text" but a superset of "Unicode plaintext" and maps to exactly the sort of text-styles that a publishing editor laying out InDesign flows, would let their contributing writers use in the RTF documents data-bound to those flows; or which a blogging engine like Twitter would let their users use to style their tweets. Like I said, this doesn't belong in the Unicode standard. Such a format can be considered an additional format layer between Unicode plaintext and a document format. It just so happens, though, that you can take advantage of the Unicode-rendering capabilities of most rich-text layout engines, to support such markup. Same as how the VT100 spec took advantage of the logic that already existed to respond to newlines and such, and introduced inline escape sequences—nominally "a higher layer", but in practice a thing that is conflated with the layer below it, such that any terminal program that claims to output "plaintext" might actually be outputting plaintext-with-ANSI-escapes without mentioning it.

dragonwriter 8y ago
> I'm talking about something that you'd use in exactly the situations where people currently use "text generators" to output those abominations, The situations, AFAICT, are often to defeat text filters.
- derefr 8y ago
  That use-case does certainly exist (it's what you see used on e.g. escort ads on websites where there shouldn't be escort ads), and obviously, such people wouldn't want to switch to using more machine-comprehensible styling. While this might represent a large percentage of the "pre-baked styling" Unicode text in the wild by volume of text, there aren't that many people who have created all this text (most of the corpus is spam repostings of the same messages with randomly-varied styling.) Rather, I think the majority of the authors of "pre-baked styling" Unicode text are simply people posting to websites like Twitter or Facebook, or a Disqus comments section, where styling is either not allowed, or doesn't have all the options the author wants. Or, to put that another way: the authors are mostly teenagers who care more about the aesthetics of their text than its comprehensibility. And that's surely the market that the Unicode consortium cares the most about—why else would we have so many emoji? ;)
cryptonector 8y ago
Attaching style to each glyph is problematic too, since now they can get spread around unintentionally, and we'll need new regexp code, and much else. No, the semantics of tagging inside Unicode are uniformly bad. I understand that people abuse Unicode already to get some such effects, but it's mostly a novelty, and I would not seriously use any software making use of these tricks. A standard way to mark up style in Unicode would certainly be better than Unicode abuse, but it would still come with a bunch of difficult associated issues -- are they really better than whatever issues you have with XML? and are they better for all of us?? BTW, games with BS (backspace) and overstrike for bold and underscore for underline still work in Unix terminals... That might be the best argument for having some stylistic composing codepoints in Unicode... after all, ASCII was designed in part to be usable with BS for expressing diacritical marks (accents, umlauts, cedilles, ...), making it a variable length internationalized encoding! But still, then less we complicate Unicode, the better.