4 ms·

This wont catch on for the same reason noone uses U+001F or U+001E for their tabular data, because its not on the keyboard. Any extra encoding characters would

This wont catch on for the same reason noone uses U+001F or U+001E for their tabular data, because its not on the keyboard. Any extra encoding characters would need new keyboards, or maybe touch screens that are good enough to code with. It would be pretty neat though.

derefr 8y ago
You don't type the BIDI instructions on your keyboard; they're inserted automatically in response to script changes. You don't type Unicode non-breaking spaces with your keyboard—you ask your editor to insert one with an editor-specific command, or you use textual markup that compiles to a Unicode non-breaking space. Etc. Most of the Unicode markup features that already exist—other than newlines—form a backing layer: a standard interchange format for text, when it's not in the internal DOM+ropes representation that most text editors use. It's not a working representation, it's just a representation that text is rendered out to. (In fact, for a lot of these existing markup codepoints, when there's one sitting around "raw" in a text field—rather than having been parsed into a text-editor DOM node—you can't even position the cursor and then delete it. Such codepoints just sort of "stick" to the grapheme-clusters/edit-field boundaries on one or both sides of them, such that deleting the grapheme-cluster will maybe "garbage collect" the associated markup, or maybe leave it there forever. The editor doesn't expect that character to be in its text, breaking the layout. It expects to have parsed it out, and then to put it back in place when it saves the document. The markup codepoints exist to be inline instructions for rendering a run of "baked" text; they don't exist to be a convenient data structure for text-editors to do editing operations against.) So, the idea I'm proposing here, is not that you'd just "be able to type a codepoint"; but rather that: 1. plaintext editors like Notepad and TextEdit—and HTML's plain <textarea> controls—would now have buttons for simple text styling (because now they can!) 2. even OS plaintext edit controls would allow styling of their contents, through an OS-universal styling palette (like macOS's "Fonts panel") that would apply to the cursor/selection of the active window (like it does for macOS rich-text edit controls.) It would essentially be part of the IME, like macOS's minimized-form "Character Viewer" palette is, or like dead-key entry is on X11. 2. WYSIWYG editors for rich-text formats like RTF and Markdown would move to evolved, distinguished storage formats (e.g. ".rtfu" = "text/rtf+umark") where the choice-of-format has the meaning of canonicalizing all styling that is possible using Unicode markup codepoints, as Unicode markup codepoints. 3. The original formats these new formats evolved from ("text/rtf", "text/markdown", etc.) would now be defined specifically as formats that have no knowledge of the meaning of Unicode markup codepoints, such that if a file containing Unicode markup codepoints were to be "Open As"ed the old format, the codepoints would render as �s. (But if you just open the document without an "Open As", the editor would heuristically determine from the fact that the file contains markup codepoints, that the file is actually "text/rtf+umark", and would insist that you "Save As" the file with the new extension.) 4. Each OS's neutral "rich text" drag-object/paste-object format would switch from "text/rtf" to "text/rtf+umark". All text-editing controls on OS apps would be required to both produce this as on drag/copy (even if it's not their storage format), and accept+parse this on drop/paste (even if it's not their storage format.) 5. Each OS's neutral "plaintext" drag-object/paste-object format would become "text/plain+umark"—i.e. plaintext that preserves styling. Thus, copying from a rich-text edit control, to a plaintext edit control, would get you resulting text with editor-specific styling stripped, but retaining the Unicode markup codepoints. 6. If the OS supports arbitrary-Unicode filenames, it should probably accept Unicode filenames with markup codepoints in them, and should probably render that markup when displaying the filename. (Consider: at least in Linux, filenames can contain newlines, and the OS renders those.)
cryptonector 8y ago
It won't catch on for other reasons too, such as that existing software won't understand Unicode tags and will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here. Unicode tag codepoints simply cannot be reliable. Nor can Unicode tag codepoints provide sufficient functionality to replace XML (or HTML, or ...). Even the subset they can replace/displace would require revamping XML/XSLT processors in order to make XSLT able to see them -- that's a terrible imposition on the rest of the world. Some of us hate UTF-16, but there it is and we can't get rid of it. Others hate UTF-8, and.. the same thing applies. XML exists, it's widely used, it's not going away no matter how much anyone dislikes it. Moving parts of HTML or XML into Unicode will not work for the reasons given above. Please dedicate your energy and ideas to innovating in other ways than this.
- derefr 8y ago
 > will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here You're assuming a representation that encodes such markup as ["start of bold" codepoint], ["end of bold" codepoint]. I agree that this would be bad, for the same reason that the Unicode "language" tags were bad. There are other representation options, though. You can instead do exactly what Unicode's Ruby-character support does, and just make a markup annotation last for exactly one grapheme-cluster. A run of bold characters, then, is a run of grapheme-clusters where each grapheme-cluster has been individually bold-annotated. There's no real way to "break" that. It's inefficient, of course, but the point of it isn't to be a universally-optimal format, but a markup-format-of-last-resort when there's no higher-level markup layer in the document format to hold styling information. To be clear: there's demand for a "markup-format-of-last-resort." In fact, people are already abusing other Unicode features to create one, and are quite happy with the resulting "styled" Unicode documents, which are—horrifyingly—proliferating across the internet and breaking semantic machine-readability for any future archival uses of that text. If you can't guess, I'm talking about people replacing the Unicode codepoints of their text with entirely-different Unicode codepoints, that just happen to already look like the "styled text" version of the text they want. For example, you've got 𝓽𝓱𝓲𝓼 𝓽𝓮𝔁𝓽, ʇɥᴉs ʇǝxʇ, t̶h̶i̶s̶ ̶t̶e̶x̶t̶, etc. These are the outputs of "text generators" (online IMEs, essentially) that take your Unicode plaintext input text, and output Unicode plaintext output text that is "styled." Now picture a Unicode feature to obviate such "text generators"—where, instead of two codepoints ["𝓽"]["𝓱"] that are both 1. impossible to type on a keyboard and 2. not actually the same characters semantically as ["t"]["h"], you've instead got four codepoints: [next-with-italics]["t"][next-with-italics]["h"]. You typed ["t"]["h"], and then selected them both and hit the "I" button in your editor, and then copied the result, and that's what's now in the copy buffer. Nothing would break—text editors already understand how to keep Unicode combining diacritics together with their base codepoint just fine, and this is just another use of the same logic. Similarly, the NFD representation of [next-with-italics]["t"][next-with-italics]["h"] would just be... ["t"]["h"]. Stick your four-codepoint sequence in a DB fulltext field, and it'll index it no problem. > Nor can Unicode tag codepoints provide sufficient functionality to replace XML Oh, I would never suggest that. Structural markup has no place in Unicode. I just think that there is a thing that is a subset of "rich text" but a superset of "Unicode plaintext" and maps to exactly the sort of text-styles that a publishing editor laying out InDesign flows, would let their contributing writers use in the RTF documents data-bound to those flows; or which a blogging engine like Twitter would let their users use to style their tweets. Like I said, this doesn't belong in the Unicode standard. Such a format can be considered an additional format layer between Unicode plaintext and a document format. It just so happens, though, that you can take advantage of the Unicode-rendering capabilities of most rich-text layout engines, to support such markup. Same as how the VT100 spec took advantage of the logic that already existed to respond to newlines and such, and introduced inline escape sequences—nominally "a higher layer", but in practice a thing that is conflated with the layer below it, such that any terminal program that claims to output "plaintext" might actually be outputting plaintext-with-ANSI-escapes without mentioning it.

derefr 8y ago

You don't type the BIDI instructions on your keyboard; they're inserted automatically in response to script changes. You don't type Unicode non-breaking spaces with your keyboard—you ask your editor to insert one with an editor-specific command, or you use textual markup that compiles to a Unicode non-breaking space. Etc. Most of the Unicode markup features that already exist—other than newlines—form a backing layer: a standard interchange format for text, when it's not in the internal DOM+ropes representation that most text editors use. It's not a working representation, it's just a representation that text is rendered out to. (In fact, for a lot of these existing markup codepoints, when there's one sitting around "raw" in a text field—rather than having been parsed into a text-editor DOM node—you can't even position the cursor and then delete it. Such codepoints just sort of "stick" to the grapheme-clusters/edit-field boundaries on one or both sides of them, such that deleting the grapheme-cluster will maybe "garbage collect" the associated markup, or maybe leave it there forever. The editor doesn't expect that character to be in its text, breaking the layout. It expects to have parsed it out, and then to put it back in place when it saves the document. The markup codepoints exist to be inline instructions for rendering a run of "baked" text; they don't exist to be a convenient data structure for text-editors to do editing operations against.) So, the idea I'm proposing here, is not that you'd just "be able to type a codepoint"; but rather that: 1. plaintext editors like Notepad and TextEdit—and HTML's plain <textarea> controls—would now have buttons for simple text styling (because now they can!) 2. even OS plaintext edit controls would allow styling of their contents, through an OS-universal styling palette (like macOS's "Fonts panel") that would apply to the cursor/selection of the active window (like it does for macOS rich-text edit controls.) It would essentially be part of the IME, like macOS's minimized-form "Character Viewer" palette is, or like dead-key entry is on X11. 2. WYSIWYG editors for rich-text formats like RTF and Markdown would move to evolved, distinguished storage formats (e.g. ".rtfu" = "text/rtf+umark") where the choice-of-format has the meaning of canonicalizing all styling that is possible using Unicode markup codepoints, as Unicode markup codepoints. 3. The original formats these new formats evolved from ("text/rtf", "text/markdown", etc.) would now be defined specifically as formats that have no knowledge of the meaning of Unicode markup codepoints, such that if a file containing Unicode markup codepoints were to be "Open As"ed the old format, the codepoints would render as �s. (But if you just open the document without an "Open As", the editor would heuristically determine from the fact that the file contains markup codepoints, that the file is actually "text/rtf+umark", and would insist that you "Save As" the file with the new extension.) 4. Each OS's neutral "rich text" drag-object/paste-object format would switch from "text/rtf" to "text/rtf+umark". All text-editing controls on OS apps would be required to both produce this as on drag/copy (even if it's not their storage format), and accept+parse this on drop/paste (even if it's not their storage format.) 5. Each OS's neutral "plaintext" drag-object/paste-object format would become "text/plain+umark"—i.e. plaintext that preserves styling. Thus, copying from a rich-text edit control, to a plaintext edit control, would get you resulting text with editor-specific styling stripped, but retaining the Unicode markup codepoints. 6. If the OS supports arbitrary-Unicode filenames, it should probably accept Unicode filenames with markup codepoints in them, and should probably render that markup when displaying the filename. (Consider: at least in Linux, filenames can contain newlines, and the OS renders those.)

cryptonector 8y ago

It won't catch on for other reasons too, such as that existing software won't understand Unicode tags and will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here. Unicode tag codepoints simply cannot be reliable. Nor can Unicode tag codepoints provide sufficient functionality to replace XML (or HTML, or ...). Even the subset they can replace/displace would require revamping XML/XSLT processors in order to make XSLT able to see them -- that's a terrible imposition on the rest of the world. Some of us hate UTF-16, but there it is and we can't get rid of it. Others hate UTF-8, and.. the same thing applies. XML exists, it's widely used, it's not going away no matter how much anyone dislikes it. Moving parts of HTML or XML into Unicode will not work for the reasons given above. Please dedicate your energy and ideas to innovating in other ways than this.

derefr 8y ago
> will either not preserve them or break their forms so that you'll have the equivalent of stuff hereother stuff here You're assuming a representation that encodes such markup as ["start of bold" codepoint], ["end of bold" codepoint]. I agree that this would be bad, for the same reason that the Unicode "language" tags were bad. There are other representation options, though. You can instead do exactly what Unicode's Ruby-character support does, and just make a markup annotation last for exactly one grapheme-cluster. A run of bold characters, then, is a run of grapheme-clusters where each grapheme-cluster has been individually bold-annotated. There's no real way to "break" that. It's inefficient, of course, but the point of it isn't to be a universally-optimal format, but a markup-format-of-last-resort when there's no higher-level markup layer in the document format to hold styling information. To be clear: there's demand for a "markup-format-of-last-resort." In fact, people are already abusing other Unicode features to create one, and are quite happy with the resulting "styled" Unicode documents, which are—horrifyingly—proliferating across the internet and breaking semantic machine-readability for any future archival uses of that text. If you can't guess, I'm talking about people replacing the Unicode codepoints of their text with entirely-different Unicode codepoints, that just happen to already look like the "styled text" version of the text they want. For example, you've got 𝓽𝓱𝓲𝓼 𝓽𝓮𝔁𝓽, ʇɥᴉs ʇǝxʇ, t̶h̶i̶s̶ ̶t̶e̶x̶t̶, etc. These are the outputs of "text generators" (online IMEs, essentially) that take your Unicode plaintext input text, and output Unicode plaintext output text that is "styled." Now picture a Unicode feature to obviate such "text generators"—where, instead of two codepoints ["𝓽"]["𝓱"] that are both 1. impossible to type on a keyboard and 2. not actually the same characters semantically as ["t"]["h"], you've instead got four codepoints: [next-with-italics]["t"][next-with-italics]["h"]. You typed ["t"]["h"], and then selected them both and hit the "I" button in your editor, and then copied the result, and that's what's now in the copy buffer. Nothing would break—text editors already understand how to keep Unicode combining diacritics together with their base codepoint just fine, and this is just another use of the same logic. Similarly, the NFD representation of [next-with-italics]["t"][next-with-italics]["h"] would just be... ["t"]["h"]. Stick your four-codepoint sequence in a DB fulltext field, and it'll index it no problem. > Nor can Unicode tag codepoints provide sufficient functionality to replace XML Oh, I would never suggest that. Structural markup has no place in Unicode. I just think that there is a thing that is a subset of "rich text" but a superset of "Unicode plaintext" and maps to exactly the sort of text-styles that a publishing editor laying out InDesign flows, would let their contributing writers use in the RTF documents data-bound to those flows; or which a blogging engine like Twitter would let their users use to style their tweets. Like I said, this doesn't belong in the Unicode standard. Such a format can be considered an additional format layer between Unicode plaintext and a document format. It just so happens, though, that you can take advantage of the Unicode-rendering capabilities of most rich-text layout engines, to support such markup. Same as how the VT100 spec took advantage of the logic that already existed to respond to newlines and such, and introduced inline escape sequences—nominally "a higher layer", but in practice a thing that is conflated with the layer below it, such that any terminal program that claims to output "plaintext" might actually be outputting plaintext-with-ANSI-escapes without mentioning it.