8 ms·

Warning, controversial: I'm actually against this. While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than be

Warning, controversial: I'm actually against this. While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0] Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter. Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY. And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to. [0] https://krebsonsecurity.com/2021/11/trojan-source-bug-threatens-the-security-of-all-code/ https://krebsonsecurity.com/2021/11/trojan-source-bug-threat...

Raymonf 4y ago
é is a valid character in latin-1, the current encoding in "many cases".
vips7L 4y ago
This isn’t about your codebase. It’s about the JDK’s codebase.
francoisdevlin 4y ago
I once wrote a math library with using the letters delta and epsilon. It was cute, but I had to copy/paste every time. Never, ever again.
- EntrePrescott 4y ago
  depending on your OS / windowing system, you actually have easy ways to quickly get greek letters out of your (non-greek) keyboard without having to copy-paste them: my favorite way is to assign a key or key combination of your choice to act as so-called dead_greek modifier, and then just press that modifier before a latin letter key to get the corresponding greek letter. For example, under linux/xorg, if you wanted AltGr+g to be your dead_greek, you can use xmodmap to set it so: xmodmap -e "$(xmodmap -pke | grep "keycode\s\+42" | sed -E 's/=((\s\S){4})\s\S+/=\1 dead_greek/')" you'll find more on that here: https://askubuntu.com/questions/787113/compose-dead-greek-with-compose-key https://askubuntu.com/questions/787113/compose-dead-greek-wi... alternative 1: assign a key as a direct modifier (not a dead key) and add key combination defs for that to generate the characters you want … alternative 2: switchable keyboard layouts
  - LtWorf 4y ago
    Does this work in wayland?
    - EntrePrescott 4y ago
      can't test it currently as the machine I have here doesn't run wayland but xorg, but iirc: the general idea of using a dead_greek modifier should also work on wayland, but assigning a key for it unfortunately cannot be done with the simple non-xkb legacy X11 keyboard layout tools like xmodmap that still work on xorg. That being said, wayland did take over part of xkb (the less old keyboard system) from xorg, and so from what I can gather from a quick search, the easiest way to assign a key as dead_greek in wayland would probably be with a file in $XDG_CONFIG_HOME/xkb/symbols/ like they do here for other key symbols: https://unix.stackexchange.com/questions/292868/how-to-customise-keyboard-mappings-with-wayland/698044#698044 https://unix.stackexchange.com/questions/292868/how-to-custo... I don't know any way to tell wayland to (re)load an xkb config on the fly (without logging out and back in) though. In particular, I doubt that setxkbmap would work for that like on xorg.
- LtWorf 4y ago
  I set my compose configuration. I just write <composekey>delta = δ, or <composekey>DELTA = Δ (and so on for the rest of the alphabet).
MaulingMonkey 4y ago
I can see wanting to stick to plain ASCII for identifiers, but what of embedded strings? Localization might prefer to be data-driven, but what of unit tests surrounding text management - including locale-specific sorting tests, glyph rendering, etc.? I don't think you're against making the source code UTF-8 per se, just "UTF-8" identifiers. Which seems quite valid. In rust you might write: #![forbid(non_ascii_idents)] pub fn fiancée() {} And get an appropriate error: error: identifier contains non-ASCII characters --> src/lib.rs:3:8 | 3 | pub fn fiancée() {} | ^^^^^^^ | note: the lint level is defined here --> src/lib.rs:1:11 | 1 | #![forbid(non_ascii_idents)] | ^^^^^^^^^^^^^^^^ Despite the source code indeed being UTF-8.
- geokon 4y ago
  Why would you not just put test input in a separate test-input file that you pipe in? That way you can test multiple different encodings and all sorts of edge cases While I'm also team ASCII, it's possible I'm not appreciating some edge case here.
  - MaulingMonkey 4y ago
    > Why would you not just put test input in a separate test-input file that you pipe in? You just added a dependency on File I/O (can't easily run the test on filesystemless embedded/wasm targets), deserialization, the current working directory if using relative paths, filesystem layout if using absolute paths - we lose type checking as part of our compile step, so deserialization might fail, we lose intellisense... > That way you can test multiple different encodings and all sorts of edge cases That can be done in code too. Let's take something concrete like https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/Locale/bcp47u/CurrencyTests.java https://github.com/openjdk/jdk/blob/master/test/jdk/java/uti... : {JPY, Locale.forLanguageTag("ja-JP-u-rg-uszzzz"), "\uffe5"}, {JPY, Locale.forLanguageTag("en-US-u-rg-jpzzzz"), "\u00a5"}, {JPY, Locale.forLanguageTag("ko-KR-u-rg-jpzzzz"), "JP\u00a5"}, Well, a mixture of half and fullwidth yen symbols (¥￥). A bit awkward to eyeball as they've been escaped via \u#### codes - dodging the whole "what encoding are our source files" problem - but given the graphical variants of that glyph, I could see keeping the escaped versions. Now, the array this is helping construct could certainly be deserialized from, say, a JSON or XML test-input file. But what would we gain, exactly, besides more boilerplate and context switching to wade through? > While I'm also team ASCII, it's possible I'm not appreciating some edge case here. To be clear I'm not saying you can't use files, and sometimes files are more appropriate and convenient than loading everything into code despite the caveats I mentioned... but I'm not seeing much of a boon for exiling test data to files here.
maxerickson 4y ago
What's the difference between a variable name that's obscure because it has a special character and a variable name that's obscure for some other reason? I wish a compose key was more widely available. It's actually moderately intuitive, on x windows, é is compose-e-'.
quadhome 4y ago
The JDK source already has non-latin characters. What’s being proposed is to normalise them to UTF-8. Are you “against” the link title or linked proposal?
- smarks 4y ago
  Yes, this. The files in the source repo are 99.9% ASCII. Anything that's not ASCII should be encoded in UTF-8 instead of some other encoding. The mailing list discussion refers to some issues we've had where characters in Shift-JIS (CP932) crept into the code base. I don't think anyone is advocating using non-ASCII characters, such as accented characters or emoji, in identifiers.
  - tialaramex 4y ago
    Exactly, and this isn't a proposal for everybody else's Java software, it's a proposal for the JDK itself. So this isn't even close to Rust's rule (all Rust source is UTF-8), let alone a declaration that Java is going to introduce data types with Korean names or new Russian method names on existing types, it's just hey, if any part of the JDK's own source code needs non-ASCII then it should be UTF-8, and if it doesn't need to be non-ASCII that's fine because ASCII is a strict subset of UTF-8.
qsort 4y ago
I actually agree with this. Any byte higher than 127 in any source file should be a compile-time error, and it is so in any project where I can make the call. > And yes, I know that there are codebases which are non-English Non-English speaker here. Use English for identifiers and comments. Put magic strings in files other than source files, which is what you should be doing anyway if you want to seriously support i18n.
- kevin_thibedeau 4y ago
  This makes it a PITA to use non-ASCII in string literals. They have applications beyond non-English language support. Unicode character escapes are opaque and impair readability. If a wizened language like C can do it gracefully, newer languages should be able to as well. For languages limited to Latin-1 specifically, I always put a © in the header block as a canary for possible misinterpretation as Unicode or something else.
- bobbylarrybobby 4y ago
  Asking all developers to read English to understand comments is asking a lot, no?
  - kaba0 4y ago
    I don’t know, if you want to find any form of official documentation for your dependencies/frameworks, you will have to know English either way. It is just the de facto language of IT, and I think it is fair to have every (even internal-only by a foreign language speaking group) codebase entirely in English. And I say it as someone whose native tongue is not English.
lmm 4y ago
> While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0] > Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter. If your search function can't handle different characters with the same meaning then it's already broken (e.g. usually you want search to be case insensitive). Likewise if similar-looking characters are an exploitable problem in your codebase then that problem already exists with e.g. l/I. > Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY. In any decent system you can type it with compose-'-e like it looks like. I'm all for making programmers dogfood the use of non-ascii characters (actually I'd like to see more use of non-unicode encodings, unless and until unicode stops screwing over Japanese). Maybe it'll give them more empathy for those of us who need to type things like this daily, and stop them doing things like flashy new packaging systems that break IME.
- josephg 4y ago
  > unless and until unicode stops screwing over Japanese Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?
  - rtpg 4y ago
    They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table. I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think
    - lmm 4y ago
gwbas1c 4y ago
> I have seen codebases in English with the aforementioned é in a function name Can that be blocked with something like .editorconfig?
- ywain 4y ago
  You'd need something that can actually parse the language to apply rules on language constructs like identifiers. I don't think EditorConfig supports this, but a linting tool would be able to.
  - gwbas1c 4y ago
    The dotnet ecosystem's formatting rules are all specified in .editorconfig.
layer8 4y ago
While I agree that sticking with ASCII sidesteps a lot of potential issues, I don’t agree that “I don’t know how to type that character” is a valid excuse nowadays. There are various ways, from Compose keys to builtin OS functionality, that entering accented characters, at the very least, shouldn’t be a major hurdle. This is to encourage people to look into this, it’s really not that difficult.
bombolo 4y ago
> Warning, controversial: I'm actually against this. Yes, please mr I-Only-Speak-English, enlighten us. /s > I feel that having UTF-8 in source code brings more risks than benefits. [0] While this is a really fun theoretical attack, has this ever been encountered? A such patch would have to go trough a diff anyway before being accepted. And certainly a "comment" that looks like code has a high chance of getting rejected? Does it make sense to restrict this feature (that is useful to everyone in the world, except those who only speak English) on account of a very theoretical risk? > Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` Just in Italy, it can be written as: fidanzata, zita, morosa, ragazza, picciotta. Certainly the accent isn't the main problem here? Do you want to eliminate all synonyms? All languages other than English? > I couldn't type it on my QWERTY. Have you encountered this situation often? If so, just remap your keyboard. I, for example, have a "w" key that is mostly used in shooter games, since the letter doesn't appear in my language. > And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to. Perhaps allowing unicode in source files could be a solution.
LtWorf 4y ago
I wrote a CLI public transport software [0] that prints emojis. How do you suggest I would achieve this in your non-unicode world? [0] https://github.com/ltworf/vasttrafik-cli https://github.com/ltworf/vasttrafik-cli edit: I had put the relevant emojis within the comment, as example, but it seems HN filtered them (angry emoji here)
GoblinSlayer 4y ago
English already allows spelling variations metre/meter, gray/grey, ctl/ctrl/control. How will you search for them?
- nayuki 4y ago
  Trivia: POV-Ray is one of the few languages that supports both color and colour as keywords. https://www.povray.org/documentation/3.7.0/r3_3.html#r3_3_1_7 https://www.povray.org/documentation/3.7.0/r3_3.html#r3_3_1_...
bmn__ 4y ago
> it also limits discoverability if `fiancee` now is written as `fiancée` […] Searching for the former does not discover the latter. Let's book this under "falsehoods programmers believe about Unicode". You can try either the code below or simply using your browser's search function for this very page and type in the string `fiancee` and see all variants, accent-marked or not, highlighted as results. #!/usr/bin/env perl use utf8; use Unicode::Collate; my $uc = Unicode::Collate->new( normalization => undef, level => 1 ); printf "pos %d len %d\n", $uc->index('Lettre à ma fiancée.docx', $_) for 'fiancée', 'fiancee', 'lettre a ma-Fiancee _DOCX_'; __END__ pos 12 len 7 pos 12 len 7 pos 0 len 24 Pardon me if I butchered the French grammar, I do not speak the language and can't easily verify whether I made a mistake in adapting the code from a year ago. <https://news.ycombinator.com/item?id=30405840 https://news.ycombinator.com/item?id=30405840>
- jraph 4y ago
  French is my main language, "Lettre à ma fiancée" seems fine.
- forgotpwd16 4y ago
  It isn't a falsehood but a valid remark regarding most tools. Tried searching "fiancee" in your example on VS Code and failed to find "fiancée". Of course you can consider "e=é" similar to how you can do case-insensitive search considering "e=E". Problem is most tools currently don't do this.
ulrikrasmussen 4y ago
I think you have misunderstood what this issue is about. The Java Language Specification already specifies that Java programs are written using the Unicode character set and that letters in identifiers can be drawn from the entire Unicode character set. The issue is that there are many ways to encode Unicode characters and that a single encoding is not used across the code base. The suggestion is merely to normalize this encoding so that all files use the same, so as to not confuse the tooling and complicate the build process. Edit: Source: https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.8 https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html...
jraph 4y ago
It's not about using more or less non-ASCII characters. It's about saving the source code files all in the same encoding (UTF-8) and standardizing around this encoding. They want get rid of an encoding mess. I don't know how one can be against this. Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters. It's probably just that the JDK is a very old code base that was written from different OSes, without much thought on the encoding. Now, they want to standardize while making sure not to break anything. What's more, the presence of non-ASCII characters in such a huge code base is almost inevitable so yeah, it matters if you want to avoid issues.
- hddqsb 4y ago
  Good points. > Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters. Nit: For a document that only contains ASCII characters, UTF-8 is identical to ASCII. So calling it UTF-8 is technically correct, but misleading -- you could equally well say the document is encoded using any of the other supersets of ASCII.
  - jraph 4y ago
    Indeed. I had that in mind when writing my comment, but it's good that you made this part explicit. You'd better have your editor / tools set to use UTF-8 though in case some non-ascii characters end up in your file or else you might be in for some pain.
dcpit 4y ago
Not so controversial. The vast majority of code is written in plain ASCII (and english), and I hope it will go on. Using non ASCII character in your code is a double failure. Take an example: "trouvé" instead of "found" as a variable name. 1- You're right. Most people won't be able to type "é", "à"... with their keyboard. Working on that code won't be easy. 2: What is the meaning of "trouvé" ? How people could figure out that variable name ? Average coder will search for english named functions or variable. Same things about domain names btw. PS : I am french.
tpoacher 4y ago
What you're against seems to be bad practices that could be more easily abused with UTF, rather than UTF itself. I think the benefits could outweigh the harms, but some of it would indeed depend on good conventions being followed. As for things like é, a function called fianc__e_accent_aigu__e would be far worse, and still a matter of convention. At least search engines could easily be configured to toggle accented letters as unaccented etc. And even nano has pretty good autocomplete facilities these days. Where I find a big benefit with utf names is in reproducing standard mathematical formulae in code. But I agree that this implies a good way to input the required symbols that doesnt require you to search utf charmaps.

Raymonf 4y ago

é is a valid character in latin-1, the current encoding in "many cases".

vips7L 4y ago

This isn’t about your codebase. It’s about the JDK’s codebase.

francoisdevlin 4y ago

I once wrote a math library with using the letters delta and epsilon. It was cute, but I had to copy/paste every time. Never, ever again.

EntrePrescott 4y ago
depending on your OS / windowing system, you actually have easy ways to quickly get greek letters out of your (non-greek) keyboard without having to copy-paste them: my favorite way is to assign a key or key combination of your choice to act as so-called dead_greek modifier, and then just press that modifier before a latin letter key to get the corresponding greek letter. For example, under linux/xorg, if you wanted AltGr+g to be your dead_greek, you can use xmodmap to set it so: xmodmap -e "$(xmodmap -pke | grep "keycode\s\+42" | sed -E 's/=((\s\S){4})\s\S+/=\1 dead_greek/')" you'll find more on that here: https://askubuntu.com/questions/787113/compose-dead-greek-with-compose-key https://askubuntu.com/questions/787113/compose-dead-greek-wi... alternative 1: assign a key as a direct modifier (not a dead key) and add key combination defs for that to generate the characters you want … alternative 2: switchable keyboard layouts
- LtWorf 4y ago
  Does this work in wayland?
  - EntrePrescott 4y ago
    can't test it currently as the machine I have here doesn't run wayland but xorg, but iirc: the general idea of using a dead_greek modifier should also work on wayland, but assigning a key for it unfortunately cannot be done with the simple non-xkb legacy X11 keyboard layout tools like xmodmap that still work on xorg. That being said, wayland did take over part of xkb (the less old keyboard system) from xorg, and so from what I can gather from a quick search, the easiest way to assign a key as dead_greek in wayland would probably be with a file in $XDG_CONFIG_HOME/xkb/symbols/ like they do here for other key symbols: https://unix.stackexchange.com/questions/292868/how-to-customise-keyboard-mappings-with-wayland/698044#698044 https://unix.stackexchange.com/questions/292868/how-to-custo... I don't know any way to tell wayland to (re)load an xkb config on the fly (without logging out and back in) though. In particular, I doubt that setxkbmap would work for that like on xorg.
LtWorf 4y ago
I set my compose configuration. I just write <composekey>delta = δ, or <composekey>DELTA = Δ (and so on for the rest of the alphabet).

MaulingMonkey 4y ago

I can see wanting to stick to plain ASCII for identifiers, but what of embedded strings? Localization might prefer to be data-driven, but what of unit tests surrounding text management - including locale-specific sorting tests, glyph rendering, etc.? I don't think you're against making the source code UTF-8 per se, just "UTF-8" identifiers. Which seems quite valid. In rust you might write: #![forbid(non_ascii_idents)] pub fn fiancée() {} And get an appropriate error: error: identifier contains non-ASCII characters --> src/lib.rs:3:8 | 3 | pub fn fiancée() {} | ^^^^^^^ | note: the lint level is defined here --> src/lib.rs:1:11 | 1 | #![forbid(non_ascii_idents)] | ^^^^^^^^^^^^^^^^ Despite the source code indeed being UTF-8.

geokon 4y ago
Why would you not just put test input in a separate test-input file that you pipe in? That way you can test multiple different encodings and all sorts of edge cases While I'm also team ASCII, it's possible I'm not appreciating some edge case here.
- MaulingMonkey 4y ago
  > Why would you not just put test input in a separate test-input file that you pipe in? You just added a dependency on File I/O (can't easily run the test on filesystemless embedded/wasm targets), deserialization, the current working directory if using relative paths, filesystem layout if using absolute paths - we lose type checking as part of our compile step, so deserialization might fail, we lose intellisense... > That way you can test multiple different encodings and all sorts of edge cases That can be done in code too. Let's take something concrete like https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/Locale/bcp47u/CurrencyTests.java https://github.com/openjdk/jdk/blob/master/test/jdk/java/uti... : {JPY, Locale.forLanguageTag("ja-JP-u-rg-uszzzz"), "\uffe5"}, {JPY, Locale.forLanguageTag("en-US-u-rg-jpzzzz"), "\u00a5"}, {JPY, Locale.forLanguageTag("ko-KR-u-rg-jpzzzz"), "JP\u00a5"}, Well, a mixture of half and fullwidth yen symbols (¥￥). A bit awkward to eyeball as they've been escaped via \u#### codes - dodging the whole "what encoding are our source files" problem - but given the graphical variants of that glyph, I could see keeping the escaped versions. Now, the array this is helping construct could certainly be deserialized from, say, a JSON or XML test-input file. But what would we gain, exactly, besides more boilerplate and context switching to wade through? > While I'm also team ASCII, it's possible I'm not appreciating some edge case here. To be clear I'm not saying you can't use files, and sometimes files are more appropriate and convenient than loading everything into code despite the caveats I mentioned... but I'm not seeing much of a boon for exiling test data to files here.

maxerickson 4y ago

What's the difference between a variable name that's obscure because it has a special character and a variable name that's obscure for some other reason? I wish a compose key was more widely available. It's actually moderately intuitive, on x windows, é is compose-e-'.

quadhome 4y ago

The JDK source already has non-latin characters. What’s being proposed is to normalise them to UTF-8. Are you “against” the link title or linked proposal?

smarks 4y ago
Yes, this. The files in the source repo are 99.9% ASCII. Anything that's not ASCII should be encoded in UTF-8 instead of some other encoding. The mailing list discussion refers to some issues we've had where characters in Shift-JIS (CP932) crept into the code base. I don't think anyone is advocating using non-ASCII characters, such as accented characters or emoji, in identifiers.
- tialaramex 4y ago
  Exactly, and this isn't a proposal for everybody else's Java software, it's a proposal for the JDK itself. So this isn't even close to Rust's rule (all Rust source is UTF-8), let alone a declaration that Java is going to introduce data types with Korean names or new Russian method names on existing types, it's just hey, if any part of the JDK's own source code needs non-ASCII then it should be UTF-8, and if it doesn't need to be non-ASCII that's fine because ASCII is a strict subset of UTF-8.

qsort 4y ago

I actually agree with this. Any byte higher than 127 in any source file should be a compile-time error, and it is so in any project where I can make the call. > And yes, I know that there are codebases which are non-English Non-English speaker here. Use English for identifiers and comments. Put magic strings in files other than source files, which is what you should be doing anyway if you want to seriously support i18n.

kevin_thibedeau 4y ago
This makes it a PITA to use non-ASCII in string literals. They have applications beyond non-English language support. Unicode character escapes are opaque and impair readability. If a wizened language like C can do it gracefully, newer languages should be able to as well. For languages limited to Latin-1 specifically, I always put a © in the header block as a canary for possible misinterpretation as Unicode or something else.
bobbylarrybobby 4y ago
Asking all developers to read English to understand comments is asking a lot, no?
- kaba0 4y ago
  I don’t know, if you want to find any form of official documentation for your dependencies/frameworks, you will have to know English either way. It is just the de facto language of IT, and I think it is fair to have every (even internal-only by a foreign language speaking group) codebase entirely in English. And I say it as someone whose native tongue is not English.

lmm 4y ago

> While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0] > Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter. If your search function can't handle different characters with the same meaning then it's already broken (e.g. usually you want search to be case insensitive). Likewise if similar-looking characters are an exploitable problem in your codebase then that problem already exists with e.g. l/I. > Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY. In any decent system you can type it with compose-'-e like it looks like. I'm all for making programmers dogfood the use of non-ascii characters (actually I'd like to see more use of non-unicode encodings, unless and until unicode stops screwing over Japanese). Maybe it'll give them more empathy for those of us who need to type things like this daily, and stop them doing things like flashy new packaging systems that break IME.

josephg 4y ago
> unless and until unicode stops screwing over Japanese Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?
- rtpg 4y ago
  They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table. I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think
  - lmm 4y ago

gwbas1c 4y ago

> I have seen codebases in English with the aforementioned é in a function name Can that be blocked with something like .editorconfig?

ywain 4y ago
You'd need something that can actually parse the language to apply rules on language constructs like identifiers. I don't think EditorConfig supports this, but a linting tool would be able to.
- gwbas1c 4y ago
  The dotnet ecosystem's formatting rules are all specified in .editorconfig.

layer8 4y ago

While I agree that sticking with ASCII sidesteps a lot of potential issues, I don’t agree that “I don’t know how to type that character” is a valid excuse nowadays. There are various ways, from Compose keys to builtin OS functionality, that entering accented characters, at the very least, shouldn’t be a major hurdle. This is to encourage people to look into this, it’s really not that difficult.

bombolo 4y ago

> Warning, controversial: I'm actually against this. Yes, please mr I-Only-Speak-English, enlighten us. /s > I feel that having UTF-8 in source code brings more risks than benefits. [0] While this is a really fun theoretical attack, has this ever been encountered? A such patch would have to go trough a diff anyway before being accepted. And certainly a "comment" that looks like code has a high chance of getting rejected? Does it make sense to restrict this feature (that is useful to everyone in the world, except those who only speak English) on account of a very theoretical risk? > Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` Just in Italy, it can be written as: fidanzata, zita, morosa, ragazza, picciotta. Certainly the accent isn't the main problem here? Do you want to eliminate all synonyms? All languages other than English? > I couldn't type it on my QWERTY. Have you encountered this situation often? If so, just remap your keyboard. I, for example, have a "w" key that is mostly used in shooter games, since the letter doesn't appear in my language. > And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to. Perhaps allowing unicode in source files could be a solution.

LtWorf 4y ago

I wrote a CLI public transport software [0] that prints emojis. How do you suggest I would achieve this in your non-unicode world? [0] https://github.com/ltworf/vasttrafik-cli https://github.com/ltworf/vasttrafik-cli edit: I had put the relevant emojis within the comment, as example, but it seems HN filtered them (angry emoji here)

GoblinSlayer 4y ago

English already allows spelling variations metre/meter, gray/grey, ctl/ctrl/control. How will you search for them?

nayuki 4y ago
Trivia: POV-Ray is one of the few languages that supports both color and colour as keywords. https://www.povray.org/documentation/3.7.0/r3_3.html#r3_3_1_7 https://www.povray.org/documentation/3.7.0/r3_3.html#r3_3_1_...

bmn__ 4y ago

> it also limits discoverability if `fiancee` now is written as `fiancée` […] Searching for the former does not discover the latter. Let's book this under "falsehoods programmers believe about Unicode". You can try either the code below or simply using your browser's search function for this very page and type in the string `fiancee` and see all variants, accent-marked or not, highlighted as results. #!/usr/bin/env perl use utf8; use Unicode::Collate; my $uc = Unicode::Collate->new( normalization => undef, level => 1 ); printf "pos %d len %d\n", $uc->index('Lettre à ma fiancée.docx', $_) for 'fiancée', 'fiancee', 'lettre a ma-Fiancee _DOCX_'; __END__ pos 12 len 7 pos 12 len 7 pos 0 len 24 Pardon me if I butchered the French grammar, I do not speak the language and can't easily verify whether I made a mistake in adapting the code from a year ago. <https://news.ycombinator.com/item?id=30405840 https://news.ycombinator.com/item?id=30405840>

jraph 4y ago
French is my main language, "Lettre à ma fiancée" seems fine.
forgotpwd16 4y ago
It isn't a falsehood but a valid remark regarding most tools. Tried searching "fiancee" in your example on VS Code and failed to find "fiancée". Of course you can consider "e=é" similar to how you can do case-insensitive search considering "e=E". Problem is most tools currently don't do this.

ulrikrasmussen 4y ago

I think you have misunderstood what this issue is about. The Java Language Specification already specifies that Java programs are written using the Unicode character set and that letters in identifiers can be drawn from the entire Unicode character set. The issue is that there are many ways to encode Unicode characters and that a single encoding is not used across the code base. The suggestion is merely to normalize this encoding so that all files use the same, so as to not confuse the tooling and complicate the build process. Edit: Source: https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.8 https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html...

jraph 4y ago

It's not about using more or less non-ASCII characters. It's about saving the source code files all in the same encoding (UTF-8) and standardizing around this encoding. They want get rid of an encoding mess. I don't know how one can be against this. Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters. It's probably just that the JDK is a very old code base that was written from different OSes, without much thought on the encoding. Now, they want to standardize while making sure not to break anything. What's more, the presence of non-ASCII characters in such a huge code base is almost inevitable so yeah, it matters if you want to avoid issues.

hddqsb 4y ago
Good points. > Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters. Nit: For a document that only contains ASCII characters, UTF-8 is identical to ASCII. So calling it UTF-8 is technically correct, but misleading -- you could equally well say the document is encoded using any of the other supersets of ASCII.
- jraph 4y ago
  Indeed. I had that in mind when writing my comment, but it's good that you made this part explicit. You'd better have your editor / tools set to use UTF-8 though in case some non-ascii characters end up in your file or else you might be in for some pain.

dcpit 4y ago

Not so controversial. The vast majority of code is written in plain ASCII (and english), and I hope it will go on. Using non ASCII character in your code is a double failure. Take an example: "trouvé" instead of "found" as a variable name. 1- You're right. Most people won't be able to type "é", "à"... with their keyboard. Working on that code won't be easy. 2: What is the meaning of "trouvé" ? How people could figure out that variable name ? Average coder will search for english named functions or variable. Same things about domain names btw. PS : I am french.

tpoacher 4y ago

What you're against seems to be bad practices that could be more easily abused with UTF, rather than UTF itself. I think the benefits could outweigh the harms, but some of it would indeed depend on good conventions being followed. As for things like é, a function called fianc__e_accent_aigu__e would be far worse, and still a matter of convention. At least search engines could easily be configured to toggle accented letters as unaccented etc. And even nano has pretty good autocomplete facilities these days. Where I find a big benefit with utf names is in reproducing standard mathematical formulae in code. But I agree that this implies a good way to input the required symbols that doesnt require you to search utf charmaps.