4 ms·

I think it's a good idea to write a plugin for any IDE to highlight those confusing characters.

samatman 2y ago
VSCode does this out of the box actually. Ended up putting a few on a whitelist while writing Julia, where it can get kind of ugly (puts a yellow box around them).
userbinator 2y ago
Using an ASCII-only font automatically shows all characters that IMHO should not be present in source code.
- metadat 2y ago
  Some platforms, such at python3 have full UTF-8 support already, so what is the problem?
  - userbinator 2y ago
    The one shown very clearly by this article.
    - metadat 2y ago
      Thanks usrbinator.. guilty grimace smile Maybe highlighting isn't such bad idea :)
    - keybored 2y ago
      The wrong values are from PDF files. Maybe you mean using a system-wide ASCII-only font but you finished your point with “should not be present in source code”. Source code wasn’t the problem here.
      - foobarchu 2y ago
        It very much is a problem in source code too though. It's unfortunately common in college courses (particularly non-CS courses with programming like bioinformatics) for instructors to distribute sample code as word docs. Cue students who can't run the code and don't know why because Word helpfully converted all double quotes to a "prettier" Unicode equivalent.
        keybored 2y ago
        Bizarrely I have experienced the same thing from Latex with its purpose-made code/literal blocks. But the most shocking thing are printed learning resources on things like Haskell where the code examples on purpose are some kind of typographic printout rather than just the symbols themselves!
- lifthrasiir 2y ago
  String literals frequently have non-ASCII characters to say the least.
  - deleted 2y ago
    [deleted]
- powersnail 2y ago
  That would make it impossible to edit non-ascii strings, like texts in foreign languages. As far as I know, most editors/IDE don't support switching fonts for string literals. It is more feasible for a syntax highlighter to highlight non-ascii characters outside of literals.
  - Someone 2y ago
    > As far as I know, most editors/IDE don't support switching fonts for string literals When asked to render an Unicode character that isn’t present in the font modern OSes will automatically pick a font that has it. https://en.wikipedia.org/wiki/Fallback_font https://en.wikipedia.org/wiki/Fallback_font: “A fallback font is a reserve typeface containing symbols for as many Unicode characters as possible. When a display system encounters a character that is not part of the repertoire of any of the other available fonts, a symbol from a fallback font is used instead. Typically, a fallback font will contain symbols representative of the various types of Unicode characters.” That can be avoided, for example by storing text as “one character per byte”, but I don’t think many editors do that nowadays.
    - powersnail 2y ago
      But that would not distinguish between chars inside a string literal and chars outside of a string literal.
- keybored 2y ago
  For every such Unicode problem (which is a data input^W source problem, not a programming source code error) there are fifty problems caused by the anemic ASCII character set like Unix toothpicks and three layers of escaping due to using too uniform delimiters. (Granted this is heavily biased since so much source code is ASCII-only so you don’t get many Unicode problems in the first place...)
- makeitdouble 2y ago
  A note on non-ascii in code: I thought of it as an abomination, until hitting test pattern descriptors. On a project targeted at a non English speaking devs with a strong domain knowledge requirement, writing the test patterns (endless arrays of input -> expected output sequences, interspersed with adjustment code) in the native language saves an incredible amount of time and effort, in particular as we don't need to translate obscure notions into even more obscure English. And that had very little downsides as it's not production running code, lining will still raise anything problematic, and and the whole thing is easier to get reviewed by non domain experts. We could have made a translation layer to have the content in a spreadsheet and convert it to test code, but that's not any more stable than having unicode names straight into the code.
  - nine_k 2y ago
    String constants / symbols is one domain, keywords and reserved characters, another. They should be checked for different things. E.g. spell-checking string constants as plain text if they look as plain text is helpful. Checking for non-ASCII quotes / dashes / other punctuation outside quoted strings, where they can only occur by mistake, is also helpful.
    - makeitdouble 2y ago
      My comment got mistakenly autocorrected (meant "linting" instead of "lining"), which is so on point given the subject. I agree, and think a decent linter can deal with these issues, and syntax highlighting as well. In particular these kind of rules tend to get complicated with many exceptions (down to specific folders needing dedicated rules), so doing it as lint and not at the language level gives a lot of freedom on where and how to apply the rules and raise warnings.
- oneeyedpigeon 2y ago
  It depends on whether you count html as "source code", but if so, then non-ASCII characters absolutely should be present!
- deleted 2y ago
  [deleted]
- PaulHoule 2y ago
  It's a very unpopular opinion but I use as much Unicode as I can in source code. In comments for instance I can write x² as well as italic and bold characters (would have demoed but HN filters out Unicode bold & italics) and I can write a test named processes中文Characters() and also write Java that looks like APL, add sigil characters in code generated stubs that will never conflict with other people's code because they're too afraid to use these characters, etc. https://github.com/paulhoule/ferocity/blob/main/ferocity-stdlib/src/test/java/com/ontology2/ferocity/stdlib/TestIllegalStateException.java https://github.com/paulhoule/ferocity/blob/main/ferocity-std... People will ask "how do you enter those characters?" and I say "I don't know but I can cut and paste them, they get offered by the autocomplete, etc."
  - Arnt 2y ago
    Hardly unpopular where I live. Lots of source code contains € and much else. Grepping for it in the code I worked on last week, I find non-ASCII character in dozens of tests, in some scripts that seem to be part of CI, in a comment about a locale-specific bug, and I stopped looking there. How to enter them? Well, the keys are on the keyboard.
    - PaulHoule 2y ago
      If you're in Euro land. I have a lot of personal interest in Chinese language content these days, I have no idea how to set up and use an "input method" but I either see the text I want in front of me or ask an LLM "How do I write X in Chinese?" and either way cut and paste.
      - sigseg1v 2y ago
        Chinese enter words into a keyboard using the same type of keyboard you would use in North America. The characters are entered as "pinyin" which is a romanized phonetic method of describing Chinese words. You should be able to enter it into your keyboard on Windows for example by enabling Simplified Chinese / pinyin in the language input settings.
      - 2y ago
MrJohz 2y ago
I know vscode had this feature built in, and it's come in handy a couple of times for me.

VSCode does this out of the box actually. Ended up putting a few on a whitelist while writing Julia, where it can get kind of ugly (puts a yellow box around them).

userbinator 2y ago

Using an ASCII-only font automatically shows all characters that IMHO should not be present in source code.

metadat 2y ago
Some platforms, such at python3 have full UTF-8 support already, so what is the problem?
- userbinator 2y ago
  The one shown very clearly by this article.
  - metadat 2y ago
    Thanks usrbinator.. guilty grimace smile Maybe highlighting isn't such bad idea :)
  - keybored 2y ago
    The wrong values are from PDF files. Maybe you mean using a system-wide ASCII-only font but you finished your point with “should not be present in source code”. Source code wasn’t the problem here.
    - foobarchu 2y ago
      It very much is a problem in source code too though. It's unfortunately common in college courses (particularly non-CS courses with programming like bioinformatics) for instructors to distribute sample code as word docs. Cue students who can't run the code and don't know why because Word helpfully converted all double quotes to a "prettier" Unicode equivalent.
      - keybored 2y ago
        Bizarrely I have experienced the same thing from Latex with its purpose-made code/literal blocks. But the most shocking thing are printed learning resources on things like Haskell where the code examples on purpose are some kind of typographic printout rather than just the symbols themselves!
lifthrasiir 2y ago
String literals frequently have non-ASCII characters to say the least.
- deleted 2y ago
  [deleted]
powersnail 2y ago
That would make it impossible to edit non-ascii strings, like texts in foreign languages. As far as I know, most editors/IDE don't support switching fonts for string literals. It is more feasible for a syntax highlighter to highlight non-ascii characters outside of literals.
- Someone 2y ago
  > As far as I know, most editors/IDE don't support switching fonts for string literals When asked to render an Unicode character that isn’t present in the font modern OSes will automatically pick a font that has it. https://en.wikipedia.org/wiki/Fallback_font https://en.wikipedia.org/wiki/Fallback_font: “A fallback font is a reserve typeface containing symbols for as many Unicode characters as possible. When a display system encounters a character that is not part of the repertoire of any of the other available fonts, a symbol from a fallback font is used instead. Typically, a fallback font will contain symbols representative of the various types of Unicode characters.” That can be avoided, for example by storing text as “one character per byte”, but I don’t think many editors do that nowadays.
  - powersnail 2y ago
    But that would not distinguish between chars inside a string literal and chars outside of a string literal.
keybored 2y ago
For every such Unicode problem (which is a data input^W source problem, not a programming source code error) there are fifty problems caused by the anemic ASCII character set like Unix toothpicks and three layers of escaping due to using too uniform delimiters. (Granted this is heavily biased since so much source code is ASCII-only so you don’t get many Unicode problems in the first place...)
makeitdouble 2y ago
A note on non-ascii in code: I thought of it as an abomination, until hitting test pattern descriptors. On a project targeted at a non English speaking devs with a strong domain knowledge requirement, writing the test patterns (endless arrays of input -> expected output sequences, interspersed with adjustment code) in the native language saves an incredible amount of time and effort, in particular as we don't need to translate obscure notions into even more obscure English. And that had very little downsides as it's not production running code, lining will still raise anything problematic, and and the whole thing is easier to get reviewed by non domain experts. We could have made a translation layer to have the content in a spreadsheet and convert it to test code, but that's not any more stable than having unicode names straight into the code.
- nine_k 2y ago
  String constants / symbols is one domain, keywords and reserved characters, another. They should be checked for different things. E.g. spell-checking string constants as plain text if they look as plain text is helpful. Checking for non-ASCII quotes / dashes / other punctuation outside quoted strings, where they can only occur by mistake, is also helpful.
  - makeitdouble 2y ago
    My comment got mistakenly autocorrected (meant "linting" instead of "lining"), which is so on point given the subject. I agree, and think a decent linter can deal with these issues, and syntax highlighting as well. In particular these kind of rules tend to get complicated with many exceptions (down to specific folders needing dedicated rules), so doing it as lint and not at the language level gives a lot of freedom on where and how to apply the rules and raise warnings.
oneeyedpigeon 2y ago
It depends on whether you count html as "source code", but if so, then non-ASCII characters absolutely should be present!
deleted 2y ago
[deleted]
PaulHoule 2y ago
It's a very unpopular opinion but I use as much Unicode as I can in source code. In comments for instance I can write x² as well as italic and bold characters (would have demoed but HN filters out Unicode bold & italics) and I can write a test named processes中文Characters() and also write Java that looks like APL, add sigil characters in code generated stubs that will never conflict with other people's code because they're too afraid to use these characters, etc. https://github.com/paulhoule/ferocity/blob/main/ferocity-stdlib/src/test/java/com/ontology2/ferocity/stdlib/TestIllegalStateException.java https://github.com/paulhoule/ferocity/blob/main/ferocity-std... People will ask "how do you enter those characters?" and I say "I don't know but I can cut and paste them, they get offered by the autocomplete, etc."
- Arnt 2y ago
  Hardly unpopular where I live. Lots of source code contains € and much else. Grepping for it in the code I worked on last week, I find non-ASCII character in dozens of tests, in some scripts that seem to be part of CI, in a comment about a locale-specific bug, and I stopped looking there. How to enter them? Well, the keys are on the keyboard.
  - PaulHoule 2y ago
    If you're in Euro land. I have a lot of personal interest in Chinese language content these days, I have no idea how to set up and use an "input method" but I either see the text I want in front of me or ask an LLM "How do I write X in Chinese?" and either way cut and paste.
    - sigseg1v 2y ago
      Chinese enter words into a keyboard using the same type of keyboard you would use in North America. The characters are entered as "pinyin" which is a romanized phonetic method of describing Chinese words. You should be able to enter it into your keyboard on Windows for example by enabling Simplified Chinese / pinyin in the language input settings.
    - 2y ago

MrJohz 2y ago

I know vscode had this feature built in, and it's come in handy a couple of times for me.