7 ms·

Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...

by umvi 5y ago

PeterisP 5y ago
Using non-ascii characters in file paths, toolchain config files and other non-display contexts is something every development team should explicitly, intentionally do in order to catch such bugs. "Asking for trouble" is a key part of testing. My suggestion would be for a QA person to have their username (and root folder of the testable project) to start with a space, and be followed by an accented letter, tab-symbol, apostrophe, an emoji, followed by an unicode RTL control character and some Arabic text.
fluxem 5y ago
Also spaces. I spent half an hour debugging why cmake cuda build was failing.
- munk-a 5y ago
  A lack of support for spaces at this point is unacceptable. I, personally, despise spaces in paths but on windows a whole bunch of default system paths already have spaces embedded in them in major ways... and let's not forget parens as well - thanks "Program Files (x86)"
b112 5y ago
This wouldn't have happened if using rust!
- nightfly 5y ago
  Can you knock it off??? This is even more annoying that out-of-place rust evangelism
  - GoblinSlayer 5y ago
    You know he's right. Look at all the rust converts preaching their dogmas here.
- burnished 5y ago
  Some of the other attempts are a little subtle, this one is a pretty blatant attempt to rile up the folks that are already angry about rust for whatever reason. Please stop.
jasonpeacock 5y ago
This is the modern, post-ASCII computing world, we should no longer be willing to settle for the lowest-common-denominator of ASCII-only strings. There's no excuse for actively supported, paid products to have these problems today.
- amenod 5y ago
  True. But these actively supported, paid products build upon layers and layers of no-longer-supported, free/opensource products. Good luck fixing them. Not saying that this is OK, just explaining why using non-ascii characters, in this day and age, is still asking for trouble.
  - SAI_Peregrinus 5y ago
    This is on the Windows version. Windows 2000 is when the OS changed to UTF-16 by default. Before that Windows NT was UCS-2, IIRC only the DOS-based Windows versions were Windows-1252 internally, starting from Windows 1.0. So while ł wasn't supported in Windows 1, characters like ñ were. Windows has literally NEVER been an ASCII-based OS.
    - horsawlarway 5y ago
      Sure, but having used a lot of the windows system apis (admittedly - a lot of years ago) it was a complete hodgepodge of which api would take a char vs a wchar, and then they tried to hide the whole thing behind tchar, which just made it even harder to keep track of. Basically - I agree: This shouldn't be a problem, and 7 months is a long time to wait for a basic fix. But there are a lot of footguns hanging around in windows code with respect to character encodings. Just looking at the first result on google for "c++ get windows home directory" shows this: https://docs.microsoft.com/en-us/windows/win32/api/userenv/nf-userenv-getuserprofiledirectorya https://docs.microsoft.com/en-us/windows/win32/api/userenv/n... Which takes a long pointer to tchar string (LPTSTR) - so this behavior is dependent on the unicode settings of the project at compile time, even today.
      - 5y ago
BiteCode_dev 5y ago
Unfortunately, it's true, most toolchains are stuck in the past, and don't deal with non-ascii characters or even spaces very well. In fact, I just learned that spaces in .deskop files values could cause trouble after a long debugging. But it's a shame. In Europe, we do have a lot of non-ascii characters everywhere. Ubuntu puts a "Vidéo" and a "Téléchargements" directory in my $HOME because I'm french. If I were to use my name as my username I would have even more troubles. I'm careful with not using special chars in names for work, but it feels like I'm a girl trying to not dress sexy in the wrong part of town: necessary, but I shouldn't have to do this, and it's definitely the others to blame. All in all, I thank the Gods of encoding for Python 3 unicode handling. Having a scripting language that does the right thing out of the box is wonderful on this side of the pond.
- mjevans 5y ago
  "The right thing" for filesystem entries is transparently copy, do not evaluate. A file path is a mem-copied, length value sized block of identifier you don't ever mangle. If you must mangle it, touch only the necessary areas as directed. (E.G. join with os.pathsep and do not normalize anything). Want to offer Unicode validation? Sure having that as an OPTION is fine. Forcing it means I can't rely on that tool to handle real world data which happens to not be valid but is still a valid file-system address.
  - BiteCode_dev 5y ago
    That the beauty of it with python: - if you want to treat paths like unicode strings, you can. Which is great for simple scripts where you don't want to deal with complexity. And 99% of the time, it's enough with modern OSes. - if you want to threat path as bags of raw bytes, you can. Which is necessary to transparently copy and do not evaluate, as you said, for covering edge cases. - if you need to actually deal with those as strings but don't want to loose data for edge cases, so a mix of the 2 above, you can use surrogate escapes
    - mjevans 5y ago
      Where does that filepath come from? A config file; are you going to do your text processing, and interaction, with other modules in byte[] arrays in Python 3+? Python 2's unicode model was _closer_ to correct, the trivial coercion between byte[] and Unicode. Conversion also shouldn't imply, force, or check Validation nor Normalization. Labeling a bytestream with an Encoding and validating / normalizing that encoding should be options. Operations on bytestreams with encoding related attributes should set them to either 'unknown' result or to a proper output type if they're aware the manipulations will still yield a valid encoding. Normalization is more complex, since Unicode strings can be normalized in different ways, then combined, and still be a valid string but no longer uniformly normalized.
gumby 5y ago
This is blaming the victim
- makeitdouble 5y ago
  This is reality though. As much as I wish we lived in a better world where name characters were better handled, using anything outside of [a-ZA-Z]{12} as a username is a world of hurt. Some people just realize it later than others. So yes, you shouldn't think of your handlename as your name, it's just another identifier, and choosing simple handle names is a life skill at this point.
- midev 5y ago
  No it's not. It's not even remotely victim blaming. At no point did anyone even remotely hint it was their fault. People have taken this to a ridiculous level. Giving practical advice about how to avoid problems is not blaming the victim. Telling my child to look both ways before crossing the street is not blaming him if he gets hit. It's not wanting to see him get hurt when the person actually at fault fucks up. Telling someone to avoid non-ASCII characters because a program can't handle it is not blaming them.... Your comment was unhelpful. Great! Not their fault! What now? It's also part of a larger trend that will lead to people being hurt.
  - gumby 5y ago
    The computer is supposed to make our lives better, we should not be required to make the computer’s (or programmers’) life better. Your attitude reminds me of the people in the 60s who included punch cards in the utility bills, marking them “DO NOT FOLD, SPLINDLE, OR MUTILATE” (it became a meme). Or the people in Spain who changed their alphabet to make it easier for computers to sort. My name appears differently in my passport, on plane tickets (not always using the same modification), and my green card. And for the latter two I left out the part of my name that can’t properly be represented at all in ASCII. And you are saying that somehow I am at fault?
    - midev 5y ago
      > The computer is supposed to make our lives better, we should not be required to make the computer’s (or programmers’) life better Yes... The computer is at fault for not supporting proper names... That's literally what everyone has said. Nobody blamed the user.... > Your attitude reminds me of the people in the 60s who included punch cards in the utility bills, marking them “DO NOT FOLD, SPLINDLE, OR MUTILATE” (it became a meme). The problem should be fixed, but it's not victim blaming to tell someone how to still submit their bill. > And you are saying that somehow I am at fault? No... I'm literally saying the opposite.... I think you need to reread your comment, then my comment. Telling you to omit those characters so you can still travel internationally is not blaming you in any way shape or form... Your criticism of practical advice being victim blaming is harmful and unhelpful.

Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...

by umvi 5y ago

Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...

PeterisP 5y ago

Using non-ascii characters in file paths, toolchain config files and other non-display contexts is something every development team should explicitly, intentionally do in order to catch such bugs. "Asking for trouble" is a key part of testing. My suggestion would be for a QA person to have their username (and root folder of the testable project) to start with a space, and be followed by an accented letter, tab-symbol, apostrophe, an emoji, followed by an unicode RTL control character and some Arabic text.

fluxem 5y ago

Also spaces. I spent half an hour debugging why cmake cuda build was failing.

munk-a 5y ago
A lack of support for spaces at this point is unacceptable. I, personally, despise spaces in paths but on windows a whole bunch of default system paths already have spaces embedded in them in major ways... and let's not forget parens as well - thanks "Program Files (x86)"

b112 5y ago

This wouldn't have happened if using rust!

nightfly 5y ago
Can you knock it off??? This is even more annoying that out-of-place rust evangelism
- GoblinSlayer 5y ago
  You know he's right. Look at all the rust converts preaching their dogmas here.
burnished 5y ago
Some of the other attempts are a little subtle, this one is a pretty blatant attempt to rile up the folks that are already angry about rust for whatever reason. Please stop.

jasonpeacock 5y ago

This is the modern, post-ASCII computing world, we should no longer be willing to settle for the lowest-common-denominator of ASCII-only strings. There's no excuse for actively supported, paid products to have these problems today.

amenod 5y ago
True. But these actively supported, paid products build upon layers and layers of no-longer-supported, free/opensource products. Good luck fixing them. Not saying that this is OK, just explaining why using non-ascii characters, in this day and age, is still asking for trouble.
- SAI_Peregrinus 5y ago
  This is on the Windows version. Windows 2000 is when the OS changed to UTF-16 by default. Before that Windows NT was UCS-2, IIRC only the DOS-based Windows versions were Windows-1252 internally, starting from Windows 1.0. So while ł wasn't supported in Windows 1, characters like ñ were. Windows has literally NEVER been an ASCII-based OS.
  - horsawlarway 5y ago
    Sure, but having used a lot of the windows system apis (admittedly - a lot of years ago) it was a complete hodgepodge of which api would take a char vs a wchar, and then they tried to hide the whole thing behind tchar, which just made it even harder to keep track of. Basically - I agree: This shouldn't be a problem, and 7 months is a long time to wait for a basic fix. But there are a lot of footguns hanging around in windows code with respect to character encodings. Just looking at the first result on google for "c++ get windows home directory" shows this: https://docs.microsoft.com/en-us/windows/win32/api/userenv/nf-userenv-getuserprofiledirectorya https://docs.microsoft.com/en-us/windows/win32/api/userenv/n... Which takes a long pointer to tchar string (LPTSTR) - so this behavior is dependent on the unicode settings of the project at compile time, even today.
    - 5y ago

BiteCode_dev 5y ago

Unfortunately, it's true, most toolchains are stuck in the past, and don't deal with non-ascii characters or even spaces very well. In fact, I just learned that spaces in .deskop files values could cause trouble after a long debugging. But it's a shame. In Europe, we do have a lot of non-ascii characters everywhere. Ubuntu puts a "Vidéo" and a "Téléchargements" directory in my $HOME because I'm french. If I were to use my name as my username I would have even more troubles. I'm careful with not using special chars in names for work, but it feels like I'm a girl trying to not dress sexy in the wrong part of town: necessary, but I shouldn't have to do this, and it's definitely the others to blame. All in all, I thank the Gods of encoding for Python 3 unicode handling. Having a scripting language that does the right thing out of the box is wonderful on this side of the pond.

mjevans 5y ago
"The right thing" for filesystem entries is transparently copy, do not evaluate. A file path is a mem-copied, length value sized block of identifier you don't ever mangle. If you must mangle it, touch only the necessary areas as directed. (E.G. join with os.pathsep and do not normalize anything). Want to offer Unicode validation? Sure having that as an OPTION is fine. Forcing it means I can't rely on that tool to handle real world data which happens to not be valid but is still a valid file-system address.
- BiteCode_dev 5y ago
  That the beauty of it with python: - if you want to treat paths like unicode strings, you can. Which is great for simple scripts where you don't want to deal with complexity. And 99% of the time, it's enough with modern OSes. - if you want to threat path as bags of raw bytes, you can. Which is necessary to transparently copy and do not evaluate, as you said, for covering edge cases. - if you need to actually deal with those as strings but don't want to loose data for edge cases, so a mix of the 2 above, you can use surrogate escapes
  - mjevans 5y ago
    Where does that filepath come from? A config file; are you going to do your text processing, and interaction, with other modules in byte[] arrays in Python 3+? Python 2's unicode model was _closer_ to correct, the trivial coercion between byte[] and Unicode. Conversion also shouldn't imply, force, or check Validation nor Normalization. Labeling a bytestream with an Encoding and validating / normalizing that encoding should be options. Operations on bytestreams with encoding related attributes should set them to either 'unknown' result or to a proper output type if they're aware the manipulations will still yield a valid encoding. Normalization is more complex, since Unicode strings can be normalized in different ways, then combined, and still be a valid string but no longer uniformly normalized.

gumby 5y ago

This is blaming the victim

makeitdouble 5y ago
This is reality though. As much as I wish we lived in a better world where name characters were better handled, using anything outside of [a-ZA-Z]{12} as a username is a world of hurt. Some people just realize it later than others. So yes, you shouldn't think of your handlename as your name, it's just another identifier, and choosing simple handle names is a life skill at this point.
midev 5y ago
No it's not. It's not even remotely victim blaming. At no point did anyone even remotely hint it was their fault. People have taken this to a ridiculous level. Giving practical advice about how to avoid problems is not blaming the victim. Telling my child to look both ways before crossing the street is not blaming him if he gets hit. It's not wanting to see him get hurt when the person actually at fault fucks up. Telling someone to avoid non-ASCII characters because a program can't handle it is not blaming them.... Your comment was unhelpful. Great! Not their fault! What now? It's also part of a larger trend that will lead to people being hurt.
- gumby 5y ago
  The computer is supposed to make our lives better, we should not be required to make the computer’s (or programmers’) life better. Your attitude reminds me of the people in the 60s who included punch cards in the utility bills, marking them “DO NOT FOLD, SPLINDLE, OR MUTILATE” (it became a meme). Or the people in Spain who changed their alphabet to make it easier for computers to sort. My name appears differently in my passport, on plane tickets (not always using the same modification), and my green card. And for the latter two I left out the part of my name that can’t properly be represented at all in ASCII. And you are saying that somehow I am at fault?
  - midev 5y ago
    > The computer is supposed to make our lives better, we should not be required to make the computer’s (or programmers’) life better Yes... The computer is at fault for not supporting proper names... That's literally what everyone has said. Nobody blamed the user.... > Your attitude reminds me of the people in the 60s who included punch cards in the utility bills, marking them “DO NOT FOLD, SPLINDLE, OR MUTILATE” (it became a meme). The problem should be fixed, but it's not victim blaming to tell someone how to still submit their bill. > And you are saying that somehow I am at fault? No... I'm literally saying the opposite.... I think you need to reread your comment, then my comment. Telling you to omit those characters so you can still travel internationally is not blaming you in any way shape or form... Your criticism of practical advice being victim blaming is harmful and unhelpful.