4 ms·

Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so thi

Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so this went in my pile of things to like about Rust. Apparently Rust only chose relatively narrowly to have &str (the string slice type, which is guaranteed to be UTF-8 text) at all, rather than just &[u8] (a slice of bytes) everywhere and to my mind that's a pretty serious benefit. The choice to have this in a language with the safe/ unsafe distinction works very nicely because in so many languages you'd have this promised UTF-8 type and then in practice everybody and their dog uses the unsafe assumed conversion because it's easier, but in Rust you're pulled up short because that conversion needs an unsafe block, your local style may require (and good practice certainly does) that you address this with a safety comment explaining why it's OK and... it just isn't, in most cases. So you write the safe conversion instead unless you really need not to. This is a really nice nudge, you can do the Wrong Thing™, but it's just easier not to.

masfuerte 3y ago
Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString. MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.
- tialaramex 3y ago
  OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither. In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck? But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.
  - pornel 3y ago
    IMHO Rust got OSString wrong – it indirectly promised that UTF-8 can always be cast to it without any copying or allocations, so on Windows it has to use WTF-8 rather than UCS2/UTF-16. Instead of being OS’s preferred string type, it’s merely a trick to preserve unpaired surrogates.
    - tialaramex 3y ago
      Huh. Where is the "indirect promise" ? Is there a (conventionally hinted as "free", so it's not good if there's one which isn't very cheap) as function? Like as_os_string or something?
      - LegionMammal978 3y ago
        It's because str implements AsRef<OsStr> [0]. The function signature promises that whenever you have a borrowed &'a str, the standard library can give you a borrowed &'a OsStr with the same data. Since references can't have destructors (they don't own the data like an OsString does), it means that the standard library can't give you a newly-allocated string without leaking it. Since obviously it isn't going to do that, the &OsStr must instead just act as a view into the underlying &str. And the conversion can't enforce any extra restrictions on the input string without breaking backward compatibility. The overall effect is that whatever format OsStr uses, it has to be a superset of UTF-8. [0] https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#impl-AsRef%3COsStr%3E-for-str https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#i...
        tialaramex 3y ago
        Ah, an AsRef<OsStr>, yep that would do it. Thanks.
    - jcranmer 3y ago
      I wouldn't say it's necessarily wrong; it may be (accidental) foresight. Windows has added a UTF-8 code page, which means you can get as near enough as makes no difference full support with the A functions instead of the W functions. That said, even now in 2024, it's not clear how much of a bet Windows is making on UTF-8 versus UTF-16.
- pjmlp 3y ago
  The first time I had to deal with this was on Turbo Pascal for Windows, where we got C strings alongside Pascal ones. Then when 32 bit came to be we had the other variations on top. Got to love leaky abstractions.

masfuerte 3y ago

Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString. MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.

tialaramex 3y ago
OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither. In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck? But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.
- pornel 3y ago
  IMHO Rust got OSString wrong – it indirectly promised that UTF-8 can always be cast to it without any copying or allocations, so on Windows it has to use WTF-8 rather than UCS2/UTF-16. Instead of being OS’s preferred string type, it’s merely a trick to preserve unpaired surrogates.
  - tialaramex 3y ago
    Huh. Where is the "indirect promise" ? Is there a (conventionally hinted as "free", so it's not good if there's one which isn't very cheap) as function? Like as_os_string or something?
    - LegionMammal978 3y ago
      It's because str implements AsRef<OsStr> [0]. The function signature promises that whenever you have a borrowed &'a str, the standard library can give you a borrowed &'a OsStr with the same data. Since references can't have destructors (they don't own the data like an OsString does), it means that the standard library can't give you a newly-allocated string without leaking it. Since obviously it isn't going to do that, the &OsStr must instead just act as a view into the underlying &str. And the conversion can't enforce any extra restrictions on the input string without breaking backward compatibility. The overall effect is that whatever format OsStr uses, it has to be a superset of UTF-8. [0] https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#impl-AsRef%3COsStr%3E-for-str https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#i...
      - tialaramex 3y ago
        Ah, an AsRef<OsStr>, yep that would do it. Thanks.
  - jcranmer 3y ago
    I wouldn't say it's necessarily wrong; it may be (accidental) foresight. Windows has added a UTF-8 code page, which means you can get as near enough as makes no difference full support with the A functions instead of the W functions. That said, even now in 2024, it's not clear how much of a bet Windows is making on UTF-8 versus UTF-16.
pjmlp 3y ago
The first time I had to deal with this was on Turbo Pascal for Windows, where we got C strings alongside Pascal ones. Then when 32 bit came to be we had the other variations on top. Got to love leaky abstractions.