6 ms·
Unicode should be for visible characters. Invisible characters are an abomination. So are ways to hide text by using Unicode so-called "characters" to cause the
by WalterBright 8mo ago
Unicode should be for visible characters. Invisible characters are an abomination. So are ways to hide text by using Unicode so-called "characters" to cause the cursor to go backwards.
Things that vanish on a printout should not be in Unicode.
Remove them from Unicode.
- abujazar 8mo agoInvisible characters are there for visible characters to be printed correctly...
- WalterBright 8mo agoI'll grant that a space and a newline are necessary. The rest, nope.
- abujazar 8mo agoYou're talking about a subset of ASCII then. Unicode is supposed to support different languages and advanced typography, for which those characters are necessary. You can't write e.g. Arabic or Hebrew without those "unnecessary" invisible characters.
- WalterBright 8mo agoPlease explain why an invisible zero width "character" is necessary.
- slim 8mo agoif you write كلب which is an arabic word written right to left in the middle of an english sentence, you want to preserve the order of the characters in the stream for computer processing purposes. meaning the chararacter ك must come before the ل and after the e and the space with respect to the memory layout. whereas when displayed, it must be inverted to be legible. the solution is to have an invisible character that indicates a switch in text direction. if you were wondering, the situation where you want to write text in a foreign language within your text is very common outside english speaking countries.
- WalterBright 8mo agoLook I'm writing sdrawkcab (amazingly, I did it without using Unicode!). Layout is the job of your text formatting program. It's easy to fix a text editor to support right-to-left text entry. The switch in text direction has resulted in malicious code injection attacks, as the reversed text becomes invisible. I had to change my compiler to reject those Unicode characters for that reason. It can be used in other cases to have hidden, malicious text. Have you checked your SQL code for invisible backwards text that injects malware?
- estebank 8mo ago> Look I'm writing sdrawkcab How would that work with Text-To-Speech output?
- WalterBright 8mo agoGood question! Two possibilities: 1. Tell the TTS program that the text is RTOL. 2. If the TTS program can speak Arabic, it can detect RTOL Arabic text. The only purpose for RTOL English I can think of is to insert hidden text for malicious purposes.
- slim 8mo agohow do you search for strings in the text ? how do you search for half the word ? as you do in autocomplete or in that search box in your browser
- slim 8mo agoI don't know what "sdrawkcab" means. I'm not a native english speaker, and nothing indicates that it's not a real word or that it is spelled backwards
- ted_dunning 8mo agoTo prevent ligatures from forming when you need that.
- WalterBright 8mo agoThat's the job of a typesetting language.
- krior 8mo agoTo mark linewrapping-breakpoints in strings.
- WalterBright 8mo agoLeave typesetting to a proper typesetting language, like Latex.
- krior 8mo agoAnd how do you call into the typesetting language? Slugging around byte-arrays?
- moritzruth 8mo agogreatidea,whoneedsspacesanyway
- WalterBright 8mo agoSpaces appear on a printout.
- ted_dunning 8mo agoAs do tabs, ems, ens and quads.
- WalterBright 8mo agoUnicode shouldn't be a typesetting language. The proper tool for that is Latex.
- WalterBright 8mo agoAnother dum dum Unicode idea is having multiple code points with identical glyphs. Rule of thumb: two Unicode sequences that look identical when printed should consist of the same code points.
- jeltz 8mo agoI don't think that would help much. There are also characters which are similar but not the same and I don't think humans can spot the differences unless they are actively looking for them which most of the time people are not. If only one of two glyphs which are similar appear in the text nobody would likely notice, expectation bias will fuck you over.
- WalterBright 8mo agoI wonder how anybody got by with printed books.
- nswango 8mo agoSo you think that the letters in the Greek and Cyrillic alphabets which are printed identically to the Latin A should not exist? And, for example, Greek words containing this letter should be encoded with a mix of Latin and Greek characters?
- WalterBright 8mo ago> So you think that the letters in the Greek and Cyrillic alphabets which are printed identically to the Latin A should not exist? Yes. Unicode should not be about semantic meaning, it should be about the visual. Like text in a book. > And, for example, Greek words containing this letter should be encoded with a mix of Latin and Greek characters? Yup. Consider a printed book. How can you tell if a letter is a Greek letter or a Latin letter? Those Unicode homonyms are a solution looking for a problem.
- Muromec 8mo ago>Yup. Consider a printed book. How can you tell if a letter is a Greek letter or a Latin letter? I can absolutely tell Cyrillic k from the lating к and latin u from the Cyrillic и. >should not be about semantic meaning, It's always better to be able to preserve more information in a text and not less.
- pvillano 8mo agoUnicode is "designed to support the use of text in all of the world's writing systems that can be digitized" Unicode needs tab, space, form feed, and carriage return. Unicode needs U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK to switch between left-to-right and right-to-left languages. Unicode needs U+115F HANGUL CHOSEONG FILLER and U+1160 HANGUL JUNGSEONG FILLER to typeset Korean. Unicode needs U+200C ZERO WIDTH NON-JOINER to encode that two characters should not be connected by a ligature. Unicode needs U+200B ZERO WIDTH SPACE to indicate a word break opportunity without actually inserting a visible space. Unicode needs MONGOLIAN FREE VARIATION SELECTORs to encode the traditional Mongolian alphabet.
- WalterBright 8mo ago[flagged]
- WalterBright 8mo agoLook Ma xt! N ! e tee S T larip (No Unicode needed.)
- bulbar 8mo agoThat's a very narrow view of the world. One example: In the past I have handled bilingual english-arabic files with switches within the same line and Arabic is written from left to right. There are also languages that are written from to to bottom. Unicode is not exclusively for coding, to the contrary, pretty sure it's only a small fraction of how Unicode is used. > Somehow people didn't need invisible characters when printing books. They didn't need computers either so "was seemingly not needed in the past" is not a good argument.
- pibaker 8mo ago> That's a very narrow view of the world. But not one that would surprise anyone familiar with WalterBright's antics on this website…
- 8mo ago
- uhoh-itsmaciek 8mo ago>Remove them from Unicode. Do you honestly think this is a workable solution?
- WalterBright 8mo agoYes, absolutely. See my other replies.
- luke-stanley 8mo agoSo we need a new standard problem due to the complexity of the last standard? Isn't unicode supposed to be a superset of ASCII, which already has control characters like new space, CR, and new lines? xD
- WalterBright 8mo agoThe only ones people use any more are newline and space. A tab key is fine in your editor, but it's been more or less abandoned as a character. I haven't used a form feed character since the 1970s.
- deleted 8mo ago[deleted]
- eviks 8mo agoSo you'd remove space and tab from Unicode?
- tetha 8mo agoThat ship has sailed, but I consider Unicode a good thing, yet I consider it problematic to support Unicode in every domain. I should be able to use Ü as a cursed smiley in text, and many more writing systems supported by Unicode support even more funny things. That's a good thing. On the other hand, if technical and display file names (to GUI users) were separate, my need for crazy characters in file names, code bases and such are very limited. Lower ASCII for actual file names consumed by technical people is sufficient to me.
- WalterBright 8mo ago> That ship has sailed Sure, but more crazy stuff gets added all the time.
- bawolff 8mo agoGood luck with that given there are invisible characters in ascii. Also this attack doesnt seem to use invisible characters just characters that dont have an assigned meaning.
- WalterBright 8mo agoThe only problematic one is CR which can be used to hide text on a glass terminal with a tty interface. I'd get rid of it if I could.
- ted_dunning 8mo agoNo need to remove them. Just make them visible for applications that don't need to render every language. Make that behavior optional as well in case you really want to name characters with Hangul or Tibetan. Some middle ground so that you can use greek letters in Julia might be nice as well. But I don't see any purpose in using the Personal Use Areas (PUA) in programming.