15 ms·

$ touch $'a.\xFF' $ find -name $'*.\xFF' ./a.? $ ./fd -e $'\xFF' error: invalid UTF-8 was detected in one or more arguments Usage: fd [OPTI

$ touch $'a.\xFF' $ find -name $'*.\xFF' ./a.? $ ./fd -e $'\xFF' error: invalid UTF-8 was detected in one or more arguments Usage: fd [OPTIONS] [pattern] [path]... For more information, try '--help'. $ $ touch III $ LC_ALL=tr_TR.UTF-8 find -iname 'iii' $ LC_ALL=tr_TR.UTF-8 ./fd 'iii' III $ Every fucking time

coldpie 2y ago
Can you elaborate on what's going on here? Something like "fd" is assuming your filenames are UTF-8, but you're actually using some other encoding?
- oguz-ismail 2y ago
  fd is assuming the argument to -e is valid UTF-8 while filenames can contain any byte but NUL and /
  - porridgeraisin 2y ago
    "I cook up impractical situations and then blame my tools for it" Nobody cares that valid filenames are anything except the null byte and /. Tell me one valid usecase for a non-UTF8 filename.
    - coldpie 2y ago
      UTF-8 is common now, but it hasn't always been. Wanting support for other encoding schemes is a valid ask (though, I think the OP was needlessly rude about it).
      - porridgeraisin 2y ago
        It's backwards compatible with ascii right? But yeah I suppose you would need support for all the other foreign-language encodings that came in between -- UCS-2 for example. But basically nobody does that. Glib (which drives all GTK apps' and various other apps file reading) doesn't support anything other than UTF8 filenames. At that point I'd consider the "migration" done and dusted.
        coldpie 2y ago
        The world is a lot more complicated & varied than you think :) I was digging around in some hard drives from 2004 just last weekend. At that time, lots of different encodings were common, especially internationally. Software archaelogy is a common hobby, it could be nice to be able to use a tool like this to search through old filesystems. "Not worth the effort" is definitely a valid response to the feature request, but that also doesn't mean there is absolutely no use for the feature.
        jcranmer 2y ago
        I can definitely see a use case for supporting non-UTF-8 pathnames on disk (primarily for archaeological purposes). In a UTF-8-path-only world, what I would do is have a mount option that says that the pathnames on disk are Latin-1 (so that \xff is mapped to U+00FF in UTF-8, which I'm too lazy to work its exact binary representation right now), and let the people doing archaeology on that write their own tools to remap the resulting mojibake pathnames into more readable ones. Not the cleanest solution, but there are ways to support non-UTF-8 disks even with UTF-8-only pathnames.
        porridgeraisin 2y ago
        Oh yeah I can imagine the pain for drives from that era. I remember reading that sometimes you need the right "codebook" - what was the word - installed and stuff like that.
        creeble 2y ago
        You do not have (or write programs for) filesystems that contain loads of ancient mp3 and wma files. It is the bane of my existence, but many programs support all the Latin-1 and other file name encodings that are incompatible with UTF-8, so users expect _your_ programs to work too. Now if you want me to actually _display_ them all correctly, tough turds...
        porridgeraisin 2y ago
        True. Btw curious, is there a defined encoding for text in mp3 metadata? Or is that a pain too.
    - shakna 2y ago
      Running a shell script went badly, generating a bunch of invalid files containing random data in their names, rather than one file containing random data. You wish to find and delete them all, now that they've turned your home directory into a monstrosity.
      - dleink 2y ago
        nah, eff all that. Roll back the snapshot.
- kccqzy 2y ago
  Right because file names are not guaranteed to be UTF-8. That's the reason Rust has str and then OsStr. You may assume Rust str is valid UTF-8 (unless you have unsafe code tricking it) but you may not assume OsStr is valid UTF-8. Here an invalid UTF-8 is passed via command line arguments. If it is desired to support this, the correct way is to use args_os https://doc.rust-lang.org/beta/std/env/fn.args_os.html https://doc.rust-lang.org/beta/std/env/fn.args_os.html which gives an iterator that yields OsString.
  - SAI_Peregrinus 2y ago
    No, OsStr is for OSes that don't encode strings as UTF-8, e.g. Windows by default. Use Path or PathBuf for paths, they're not strings. Pretty much every OS has either some valid strings that aren't valid paths (e.g. the filename `CON` on Windows is a valid string but not valid in a path), some valid paths that aren't valid strings (e.g. UNIX allows any byte other than 0x00 in paths, and any byte other than 0x00 or `/` in filenames, with no restrictions to any text encoding), or both.
- burntsushi 2y ago
  What's happening here is that fd's `-e/--extension` flag requires that its parameter value be valid UTF-8. The only case where this doesn't work is if you want to filter by a file path whose extension is not valid UTF-8. Needless to say, I can't ever think of a case where you'd really want to do this. But if you do, fd still supports it. You just can't use the `-e/--extension` convenience: $ touch $'a.\xFF' $ fd '.*\.(?-u:\xFF)' a.� That is, `fd` requires that the regex patterns you give it be valid UTF-8, but that doesn't mean the patterns themselves are limited to only matching valid UTF-8. You can disable Unicode mode and match raw bytes via escape sequences (that's what `(?-u:\xFF)` is doing). So as a matter of fact, the sibling comments get the analysis wrong here. `fd` doesn't assume all of its file paths are valid UTF-8. As demonstrated above, it handles non-UTF-8 paths just fine. But some conveniences, like specifying a file extension, do require valid UTF-8.
ivanjermakov 2y ago
I think it's common for tools to assume that file names are valid unicode, not surprized.
- SAI_Peregrinus 2y ago
  Common, but rather stupid. Filenames aren't even text. `fd` is written in Rust, and it uses std::path for paths, the regex pattern defaults to matching text. That said, it is possible by turning off the Unicode flag. `(?-u:\x??)` where `??` is a raw byte in hex. E.g. `(?-u:\xFF)` for OP. See "Opt out of Unicode support[1] in the regex docs. [1] https://docs.rs/regex/latest/regex/#opt-out-of-unicode-support https://docs.rs/regex/latest/regex/#opt-out-of-unicode-suppo...
  - burntsushi 2y ago
    What is text? Are the contents of files text? How does one determine if something is text? (I'm the author of ripgrep, and this is my way of gently suggesting that "filenames aren't even text" isn't an especially useful model.)
    - SAI_Peregrinus 2y ago
      Oh, I agree that "text" isn't well-defined. The best I can come up with is that "text" is a valid sequence of bytes when interpreted in some text encoding. I think that something designed to search filenames should clearly document how to search for all valid filenames in its help or manual, not require looking up the docs of a dependency. Filenames are paths, which are weird on every platform. 99% of the time you can search paths using some sort of text encoding, but IMO it should be pointed out in the man page that non-unicode filenames can actually be searched for. `fd`'s man page just links to the regex crate docs, it doesn't generate a new man page for those & name that. As for "filenames aren't even text" not being a useful model, to me text is a `&str` or `String` or `OsString`, filenames are a `Path` or `PathBuf`. We have different types for paths & strings because they represent different things, and have different valid contents. All I mean by that is the types are different, and the types you use for text shouldn't be the same as the types you use for paths.
packetlost 2y ago
Ok, but like, in practice this is a pretty weird edge case. It's impractical and usually worthless to have filenames that can't be described using the characters on a keyboard.
- Galanwe 2y ago
  Which keyboard ?
  - packetlost 2y ago
    Any of them. File names are in the vast majority of cases human readable in some character encoding, even UTF-8. You would be hard pressed to find a keyboard/character code that has characters that aren't represented in Unicode, but it doesn't matter, just honor the system locale.
- bravetraveler 2y ago
  Disagree, filesystems provide for both services and people... this is an imposition. I, a mere human, may need my tools to wrangle output generated by other software that has never once used a keyboard. Or a sensible character/set, bytes are bytes File extensions - or their names - mean absolutely nothing with ELF. Maybe $APPLICATION decides to use the filename to store non-ASCII/Unicode parity data... because it was developed in a closet with infinite funding. Who knows. Who's to say. Contrived, yes, but practical. Imposing isn't. The filesystem may contain more than this can handle.
  - packetlost 2y ago
    My point is that it's such a weird edge case in the first place that the chances of you needing to use a tool like fd/find in this way is vanishingly small. I agree with the general issue of treating file paths as encoded strings when they are not. Go is the worst offender here because it does it at the language level which is just egregious. Regardless, the point is moot because `fd` handles the filenames gracefully you just need to use a different flag [0]. [0]: https://news.ycombinator.com/item?id=43412190 https://news.ycombinator.com/item?id=43412190
149765 2y ago
Had to lookup regex crate docs, but it's possible: $ fd '(?-u:\xFF)' a.�
dspillett 2y ago
It isn't wrong, 0xff is invalid UTF8. Of course if your locale is not set to UTF8 then that is a potential problem.
- vsl 2y ago
  *nix filenames are series of bytes, not UTF-8 (or anything else) strings. If a find replacement doesn't accept valid (parts of) filenames as input, it's a bit unfortunate.
  - coldpie 2y ago
    If all you want to do is match against a sequence of bytes, sure. But when you want to start providing features like case-insensitivity, matching against file extensions, globbing, etc, then you have to declare what a given byte sequence actually represents, and that requires an encoding.
    - oguz-ismail 2y ago
      > when you want to start providing features like case-insensitivity fd does that for English only. See the III/iii case in my comment; iii capitalizes to İİİ in Turkish, there's no way to have fd respect that.
      - burntsushi 2y ago
        > fd does that for English only. That's false. Counter-example: $ touch 'Δ' $ fd δ Δ Your Turkish example doesn't work with `fd` because `fd` doesn't support specific locales or locale specific tailoring for case insensitive matching. It only supports what Unicode calls "simple case folding." It works for things far beyond English, as demonstrated above, but definitely misses some cases specific to particular locales.
        oguz-ismail 2y ago
        [flagged]

coldpie 2y ago

Can you elaborate on what's going on here? Something like "fd" is assuming your filenames are UTF-8, but you're actually using some other encoding?

oguz-ismail 2y ago
fd is assuming the argument to -e is valid UTF-8 while filenames can contain any byte but NUL and /
- porridgeraisin 2y ago
  "I cook up impractical situations and then blame my tools for it" Nobody cares that valid filenames are anything except the null byte and /. Tell me one valid usecase for a non-UTF8 filename.
  - coldpie 2y ago
    UTF-8 is common now, but it hasn't always been. Wanting support for other encoding schemes is a valid ask (though, I think the OP was needlessly rude about it).
    - porridgeraisin 2y ago
      It's backwards compatible with ascii right? But yeah I suppose you would need support for all the other foreign-language encodings that came in between -- UCS-2 for example. But basically nobody does that. Glib (which drives all GTK apps' and various other apps file reading) doesn't support anything other than UTF8 filenames. At that point I'd consider the "migration" done and dusted.
      - coldpie 2y ago
        The world is a lot more complicated & varied than you think :) I was digging around in some hard drives from 2004 just last weekend. At that time, lots of different encodings were common, especially internationally. Software archaelogy is a common hobby, it could be nice to be able to use a tool like this to search through old filesystems. "Not worth the effort" is definitely a valid response to the feature request, but that also doesn't mean there is absolutely no use for the feature.
        jcranmer 2y ago
        I can definitely see a use case for supporting non-UTF-8 pathnames on disk (primarily for archaeological purposes). In a UTF-8-path-only world, what I would do is have a mount option that says that the pathnames on disk are Latin-1 (so that \xff is mapped to U+00FF in UTF-8, which I'm too lazy to work its exact binary representation right now), and let the people doing archaeology on that write their own tools to remap the resulting mojibake pathnames into more readable ones. Not the cleanest solution, but there are ways to support non-UTF-8 disks even with UTF-8-only pathnames.
        porridgeraisin 2y ago
        Oh yeah I can imagine the pain for drives from that era. I remember reading that sometimes you need the right "codebook" - what was the word - installed and stuff like that.
      - creeble 2y ago
        You do not have (or write programs for) filesystems that contain loads of ancient mp3 and wma files. It is the bane of my existence, but many programs support all the Latin-1 and other file name encodings that are incompatible with UTF-8, so users expect _your_ programs to work too. Now if you want me to actually _display_ them all correctly, tough turds...
        porridgeraisin 2y ago
        True. Btw curious, is there a defined encoding for text in mp3 metadata? Or is that a pain too.
  - shakna 2y ago
    Running a shell script went badly, generating a bunch of invalid files containing random data in their names, rather than one file containing random data. You wish to find and delete them all, now that they've turned your home directory into a monstrosity.
    - dleink 2y ago
      nah, eff all that. Roll back the snapshot.
kccqzy 2y ago
Right because file names are not guaranteed to be UTF-8. That's the reason Rust has str and then OsStr. You may assume Rust str is valid UTF-8 (unless you have unsafe code tricking it) but you may not assume OsStr is valid UTF-8. Here an invalid UTF-8 is passed via command line arguments. If it is desired to support this, the correct way is to use args_os https://doc.rust-lang.org/beta/std/env/fn.args_os.html https://doc.rust-lang.org/beta/std/env/fn.args_os.html which gives an iterator that yields OsString.
- SAI_Peregrinus 2y ago
  No, OsStr is for OSes that don't encode strings as UTF-8, e.g. Windows by default. Use Path or PathBuf for paths, they're not strings. Pretty much every OS has either some valid strings that aren't valid paths (e.g. the filename `CON` on Windows is a valid string but not valid in a path), some valid paths that aren't valid strings (e.g. UNIX allows any byte other than 0x00 in paths, and any byte other than 0x00 or `/` in filenames, with no restrictions to any text encoding), or both.
burntsushi 2y ago
What's happening here is that fd's `-e/--extension` flag requires that its parameter value be valid UTF-8. The only case where this doesn't work is if you want to filter by a file path whose extension is not valid UTF-8. Needless to say, I can't ever think of a case where you'd really want to do this. But if you do, fd still supports it. You just can't use the `-e/--extension` convenience: $ touch $'a.\xFF' $ fd '.*\.(?-u:\xFF)' a.� That is, `fd` requires that the regex patterns you give it be valid UTF-8, but that doesn't mean the patterns themselves are limited to only matching valid UTF-8. You can disable Unicode mode and match raw bytes via escape sequences (that's what `(?-u:\xFF)` is doing). So as a matter of fact, the sibling comments get the analysis wrong here. `fd` doesn't assume all of its file paths are valid UTF-8. As demonstrated above, it handles non-UTF-8 paths just fine. But some conveniences, like specifying a file extension, do require valid UTF-8.

ivanjermakov 2y ago

I think it's common for tools to assume that file names are valid unicode, not surprized.

SAI_Peregrinus 2y ago
Common, but rather stupid. Filenames aren't even text. `fd` is written in Rust, and it uses std::path for paths, the regex pattern defaults to matching text. That said, it is possible by turning off the Unicode flag. `(?-u:\x??)` where `??` is a raw byte in hex. E.g. `(?-u:\xFF)` for OP. See "Opt out of Unicode support[1] in the regex docs. [1] https://docs.rs/regex/latest/regex/#opt-out-of-unicode-support https://docs.rs/regex/latest/regex/#opt-out-of-unicode-suppo...
- burntsushi 2y ago
  What is text? Are the contents of files text? How does one determine if something is text? (I'm the author of ripgrep, and this is my way of gently suggesting that "filenames aren't even text" isn't an especially useful model.)
  - SAI_Peregrinus 2y ago
    Oh, I agree that "text" isn't well-defined. The best I can come up with is that "text" is a valid sequence of bytes when interpreted in some text encoding. I think that something designed to search filenames should clearly document how to search for all valid filenames in its help or manual, not require looking up the docs of a dependency. Filenames are paths, which are weird on every platform. 99% of the time you can search paths using some sort of text encoding, but IMO it should be pointed out in the man page that non-unicode filenames can actually be searched for. `fd`'s man page just links to the regex crate docs, it doesn't generate a new man page for those & name that. As for "filenames aren't even text" not being a useful model, to me text is a `&str` or `String` or `OsString`, filenames are a `Path` or `PathBuf`. We have different types for paths & strings because they represent different things, and have different valid contents. All I mean by that is the types are different, and the types you use for text shouldn't be the same as the types you use for paths.

packetlost 2y ago

Ok, but like, in practice this is a pretty weird edge case. It's impractical and usually worthless to have filenames that can't be described using the characters on a keyboard.

Galanwe 2y ago
Which keyboard ?
- packetlost 2y ago
  Any of them. File names are in the vast majority of cases human readable in some character encoding, even UTF-8. You would be hard pressed to find a keyboard/character code that has characters that aren't represented in Unicode, but it doesn't matter, just honor the system locale.
bravetraveler 2y ago
Disagree, filesystems provide for both services and people... this is an imposition. I, a mere human, may need my tools to wrangle output generated by other software that has never once used a keyboard. Or a sensible character/set, bytes are bytes File extensions - or their names - mean absolutely nothing with ELF. Maybe $APPLICATION decides to use the filename to store non-ASCII/Unicode parity data... because it was developed in a closet with infinite funding. Who knows. Who's to say. Contrived, yes, but practical. Imposing isn't. The filesystem may contain more than this can handle.
- packetlost 2y ago
  My point is that it's such a weird edge case in the first place that the chances of you needing to use a tool like fd/find in this way is vanishingly small. I agree with the general issue of treating file paths as encoded strings when they are not. Go is the worst offender here because it does it at the language level which is just egregious. Regardless, the point is moot because `fd` handles the filenames gracefully you just need to use a different flag [0]. [0]: https://news.ycombinator.com/item?id=43412190 https://news.ycombinator.com/item?id=43412190

149765 2y ago

Had to lookup regex crate docs, but it's possible: $ fd '(?-u:\xFF)' a.�

dspillett 2y ago

It isn't wrong, 0xff is invalid UTF8. Of course if your locale is not set to UTF8 then that is a potential problem.

vsl 2y ago
*nix filenames are series of bytes, not UTF-8 (or anything else) strings. If a find replacement doesn't accept valid (parts of) filenames as input, it's a bit unfortunate.
- coldpie 2y ago
  If all you want to do is match against a sequence of bytes, sure. But when you want to start providing features like case-insensitivity, matching against file extensions, globbing, etc, then you have to declare what a given byte sequence actually represents, and that requires an encoding.
  - oguz-ismail 2y ago
    > when you want to start providing features like case-insensitivity fd does that for English only. See the III/iii case in my comment; iii capitalizes to İİİ in Turkish, there's no way to have fd respect that.
    - burntsushi 2y ago
      > fd does that for English only. That's false. Counter-example: $ touch 'Δ' $ fd δ Δ Your Turkish example doesn't work with `fd` because `fd` doesn't support specific locales or locale specific tailoring for case insensitive matching. It only supports what Unicode calls "simple case folding." It works for things far beyond English, as demonstrated above, but definitely misses some cases specific to particular locales.
      - oguz-ismail 2y ago
        [flagged]