5 ms·

> let hello = String::from("שָׁלוֹם"); This brings back PTSD for anyone who implemented ad-hoc RTL and UTF-16 surrogate pairs support in an environment that la

> let hello = String::from("שָׁלוֹם"); This brings back PTSD for anyone who implemented ad-hoc RTL and UTF-16 surrogate pairs support in an environment that lacked them. Thank g-d I insisted that we at least omit diacritics as nobody uses them and they're a huge pain to implement and QA properly. Also, can we please, please, please keep source code files as pure ascii? Asking as a non-native english speaker. There're a lot of tools that we developers need to use on those, and their encoding issues tend to follow the Murphy's law and appear at the most inconvenient moment. Edit regarding localization: all projects that I've worked on had localization in separate json or XML files, with automatic import/export from spreadsheets that would be filled by third party localization company. This is the line of code I find my cursor on literally the moment I switch to HN: questIcon.Text.text = ("Quest:" + World.current[index].template.trueId + "_title").T(); Where T() is the extension method from localization system. Why on earth would you include user-facing strings in the code itself?

Dylan16807 9y ago
What tools care whether a file is ASCII or UTF-8?
wodenokoto 9y ago
> Also, can we please, please, please keep source code files as pure ascii How do you propose that we write programs that interact with non-ASCII compatible language users? All string literals are to be saved in external, somehow non-source code files? How would you go about writing a Chinese "Hello World"? In Python 3, I would write print("你好，世界！") I have no idea how you propose this should be done in pure ascii.
- masklinn 9y ago
  print("\u4f60\u597d\uff0c\u4e16\u754c\uff01") I'd guess?
  - matthewrudy 9y ago
    in which case you'll need to add a comment to remember what it says: print("\u4f60\u597d\uff0c\u4e16\u754c\uff01") # "你好，世界！"
    - masklinn 9y ago
      That makes the source file not-ascii again.
      - matthewrudy 9y ago
        Sure. That's my joke. You'd obfuscate the content, but to be practical you'd still have to know what it said. Hence my comment.
- smcl 9y ago
  I think the main point is to remove the Chinese characters in the string literal and refer to some object which contains the contents of some already-parsed localisation file - and that's the file where the Chinese chars characters would live. So you'd probably write something like: print(localisedStuff["zh"]["hello-world"]) This looks clunky in a little standalone example but if you can imagine having to build an app that supports many languages then this starts to make sense as you'd more likely have: print(localisedStuff[selectedLanguage]["hello-world"]) However I disagree with the original comment that non-ASCII chars shouldn't be supported in code - I think that Chinese, Arabic, Korean, Israeli (etc) speaking devs should be able to deal with string literals in whatever language they feel comfortable in.
- icebraining 9y ago
  I'm not opposed to unicode in source code, but we do put all string literals in external files, by using gettext. Your example would be: print(_("Hello World!")) And then a zh.po file with: #: file.py:1 msgid "Hello World!" msgstr "你好，世界！"
  - saurik 9y ago
    This assumes then programmer speaks English and Chinese is a "translation".
    - masklinn 9y ago
      Or even just the second part. One could imagine software with chinese as the reference and english as the translation. I guess that's one good thing about Java's resource bundles, though they don't help source readability.
    - icebraining 9y ago
      It's just a placeholder, you can use anything. The company I work for, and its clients, are not in English-speaking countries either.
pleasecalllater 9y ago
> Also, can we please, please, please keep source code files as pure ascii? Asking as a non-native english speaker. I'm a non native English speaker. That's why I hate tools, and websites which don't understand that people have different letters in names. Just use good tools.
- kiriakasis 9y ago
  I agree that tools should be better, but i believe that "just use good tools" might not be a viable solution for the parent.
  - pleasecalllater 9y ago
    No, there is no justification for bad tools. If someone makes a tool only for ASCII, fine, but don't expect that people will say it's a good tool. Even huge sites have problems when I want to use my real name, or the name of the city I live in. They are usually made by US companies, which is strange, as you have all kinds of people there, with different names. Unfortunately the reality is that those US companies don't care about supporting normal characters people use. Even websites like Upwork don't allow me to enter my real name. Isn't that strange? I have told them about that months ago, nobody cares. So getting to the real problem: supporting unicode is not as hard as it seems, just use from the beginning.
    - setr 9y ago
      >supporting unicode is not as hard as it seems, just use from the beginning. Retrofitting unicode support onto existing programs however is a hard problem for all unicode-lacking but otherwise well-developed tools, with no sufficiently useful and also unicode supporting alternive, that you'd like to use, whats your proposal? Go back in time and tell them to use unicode from the start? Reimpliment them yourself? Give up and say no useful tool for me? There are a hundred reasons certain tooling isn't viable for your usecase; unicode support is rarely even close to being an immediate dealbreaker. Particularly because it can be sufficiently worked around.
- 9y ago
enriquto 9y ago
> Also, can we please, please, please keep source code files as pure ascii? But why? In scientific computing (e.g., in C and in Python), I like to write my formulas using variable names like α, β, γ. Having to write alpha, beta, gamma uglyfies the formulas unacceptably. Support for non-ascii variable names is actually my main reason to use clang instead of gcc.
lucideer 9y ago
> can we please, please, please keep source code files as pure ascii? Asking as a non-native english speaker. There're a lot of tools If there are tools that break when this happens, those tools need to be fixed. Bugs in those tools need to be found (through usage) and reported. If we avoid the problem by strictly sticking to ascii in source files, that amounts to sweeping those bugs under the carpet. Furthermore, another commenter said "use good tools". The inverse of this—don't use bad tools—is important: developing tools that correctly handle encoding issues needs to be seen by developers as a priority. There needs to be a ramification (in the form of bug reports, complaints, etc.) if encoding issues are not considered by tool makers.
- golergka 9y ago
  This is all good and well, but there are also projects that need to be delivered in this imperfect world and imperfect tools that we have. And I'd rather assume the worst of the world in the beginning and take appropriate precautions than debug some tooling problem right on the day of the release.
  - lucideer 9y ago
    You're right, economic incentives do usually favour ignoring/avoiding difficult problems and increasing technical debt. but I'd generally like to approach that as a pragmatic balance between what should be done and what needs to be in the short-term.
    - golergka 9y ago
      > ignoring/avoiding difficult problems and increasing technical debt Technical debt, like any debt, shouldn't be blindly avoided, but instead managed in a smart way. But anyway - it is not my debt or problems to solve. Most of the tools I'm talking about are closed source, and I wouldn't be able to do anything with them except hacking around.

Dylan16807 9y ago

What tools care whether a file is ASCII or UTF-8?

wodenokoto 9y ago

> Also, can we please, please, please keep source code files as pure ascii How do you propose that we write programs that interact with non-ASCII compatible language users? All string literals are to be saved in external, somehow non-source code files? How would you go about writing a Chinese "Hello World"? In Python 3, I would write print("你好，世界！") I have no idea how you propose this should be done in pure ascii.

masklinn 9y ago
print("\u4f60\u597d\uff0c\u4e16\u754c\uff01") I'd guess?
- matthewrudy 9y ago
  in which case you'll need to add a comment to remember what it says: print("\u4f60\u597d\uff0c\u4e16\u754c\uff01") # "你好，世界！"
  - masklinn 9y ago
    That makes the source file not-ascii again.
    - matthewrudy 9y ago
      Sure. That's my joke. You'd obfuscate the content, but to be practical you'd still have to know what it said. Hence my comment.
smcl 9y ago
I think the main point is to remove the Chinese characters in the string literal and refer to some object which contains the contents of some already-parsed localisation file - and that's the file where the Chinese chars characters would live. So you'd probably write something like: print(localisedStuff["zh"]["hello-world"]) This looks clunky in a little standalone example but if you can imagine having to build an app that supports many languages then this starts to make sense as you'd more likely have: print(localisedStuff[selectedLanguage]["hello-world"]) However I disagree with the original comment that non-ASCII chars shouldn't be supported in code - I think that Chinese, Arabic, Korean, Israeli (etc) speaking devs should be able to deal with string literals in whatever language they feel comfortable in.
icebraining 9y ago
I'm not opposed to unicode in source code, but we do put all string literals in external files, by using gettext. Your example would be: print(_("Hello World!")) And then a zh.po file with: #: file.py:1 msgid "Hello World!" msgstr "你好，世界！"
- saurik 9y ago
  This assumes then programmer speaks English and Chinese is a "translation".
  - masklinn 9y ago
    Or even just the second part. One could imagine software with chinese as the reference and english as the translation. I guess that's one good thing about Java's resource bundles, though they don't help source readability.
  - icebraining 9y ago
    It's just a placeholder, you can use anything. The company I work for, and its clients, are not in English-speaking countries either.

pleasecalllater 9y ago

> Also, can we please, please, please keep source code files as pure ascii? Asking as a non-native english speaker. I'm a non native English speaker. That's why I hate tools, and websites which don't understand that people have different letters in names. Just use good tools.

kiriakasis 9y ago
I agree that tools should be better, but i believe that "just use good tools" might not be a viable solution for the parent.
- pleasecalllater 9y ago
  No, there is no justification for bad tools. If someone makes a tool only for ASCII, fine, but don't expect that people will say it's a good tool. Even huge sites have problems when I want to use my real name, or the name of the city I live in. They are usually made by US companies, which is strange, as you have all kinds of people there, with different names. Unfortunately the reality is that those US companies don't care about supporting normal characters people use. Even websites like Upwork don't allow me to enter my real name. Isn't that strange? I have told them about that months ago, nobody cares. So getting to the real problem: supporting unicode is not as hard as it seems, just use from the beginning.
  - setr 9y ago
    >supporting unicode is not as hard as it seems, just use from the beginning. Retrofitting unicode support onto existing programs however is a hard problem for all unicode-lacking but otherwise well-developed tools, with no sufficiently useful and also unicode supporting alternive, that you'd like to use, whats your proposal? Go back in time and tell them to use unicode from the start? Reimpliment them yourself? Give up and say no useful tool for me? There are a hundred reasons certain tooling isn't viable for your usecase; unicode support is rarely even close to being an immediate dealbreaker. Particularly because it can be sufficiently worked around.
9y ago

enriquto 9y ago

> Also, can we please, please, please keep source code files as pure ascii? But why? In scientific computing (e.g., in C and in Python), I like to write my formulas using variable names like α, β, γ. Having to write alpha, beta, gamma uglyfies the formulas unacceptably. Support for non-ascii variable names is actually my main reason to use clang instead of gcc.

lucideer 9y ago

> can we please, please, please keep source code files as pure ascii? Asking as a non-native english speaker. There're a lot of tools If there are tools that break when this happens, those tools need to be fixed. Bugs in those tools need to be found (through usage) and reported. If we avoid the problem by strictly sticking to ascii in source files, that amounts to sweeping those bugs under the carpet. Furthermore, another commenter said "use good tools". The inverse of this—don't use bad tools—is important: developing tools that correctly handle encoding issues needs to be seen by developers as a priority. There needs to be a ramification (in the form of bug reports, complaints, etc.) if encoding issues are not considered by tool makers.

golergka 9y ago
This is all good and well, but there are also projects that need to be delivered in this imperfect world and imperfect tools that we have. And I'd rather assume the worst of the world in the beginning and take appropriate precautions than debug some tooling problem right on the day of the release.
- lucideer 9y ago
  You're right, economic incentives do usually favour ignoring/avoiding difficult problems and increasing technical debt. but I'd generally like to approach that as a pragmatic balance between what should be done and what needs to be in the short-term.
  - golergka 9y ago
    > ignoring/avoiding difficult problems and increasing technical debt Technical debt, like any debt, shouldn't be blindly avoided, but instead managed in a smart way. But anyway - it is not my debt or problems to solve. Most of the tools I'm talking about are closed source, and I wouldn't be able to do anything with them except hacking around.