4 ms·

> Wait, why is that a problem? Using UTF-8 only is the closest to programmer's ideal. I have a lot of archived data with various unknown encodings. APIs that b

by josefx 2y ago

> Wait, why is that a problem? Using UTF-8 only is the closest to programmer's ideal. I have a lot of archived data with various unknown encodings. APIs that blindly try to force an encoding when all I have is "bytes" tend to be a major pain and I usually end up bruteforcing my way around them with things like base64.

Ygg2 2y ago
If that's the case the winning move is get the bytes, and convert them to UTF8 or just process it as bytes. &str is a wrapper around &[u8] with strict UTF8 guarantees. Modern systems should be able to convert various encodings at GiB/s into UTF8.
- josefx 2y ago
  > If that's the case the winning move is get the bytes, and convert them to UTF8 That would require knowing the original encoding. > or just process it as bytes As long as the APIs you have to use take bytes and not u8 strings. > Modern systems should be able to convert various encodings at GiB/s into UTF8. They might even guess the correct source encoding a quarter of the time and it will break any legacy application that still has to interact with the data. I guess that would be a win-win situation for Rust.
  - dralley 2y ago
    >That would require knowing the original encoding. Or just use a library that can detect the encoding, and spit out utf-8. There's several of those.
    - josefx 2y ago
      Yes, you can try to automatically guess the wrong encoding based on statistics that only work some of the time when given large amounts of free flowing text.
  - Ygg2 2y ago
    > That would require knowing the original encoding. If you don't know that's one more reason to get bytes and try to figure out encoding. Usually using lib like encodings.rs or WTF8.rs > As long as the APIs you have to use take bytes and not utf8 strings. You can convert one into the other, albeit converting to str requires a check.
aldanor 2y ago
That's why byte strings exist. You don't know the encoding? Well, it's a binary chunk and nothing else.

by josefx 2y ago

Ygg2 2y ago

If that's the case the winning move is get the bytes, and convert them to UTF8 or just process it as bytes. &str is a wrapper around &[u8] with strict UTF8 guarantees. Modern systems should be able to convert various encodings at GiB/s into UTF8.

josefx 2y ago
> If that's the case the winning move is get the bytes, and convert them to UTF8 That would require knowing the original encoding. > or just process it as bytes As long as the APIs you have to use take bytes and not u8 strings. > Modern systems should be able to convert various encodings at GiB/s into UTF8. They might even guess the correct source encoding a quarter of the time and it will break any legacy application that still has to interact with the data. I guess that would be a win-win situation for Rust.
- dralley 2y ago
  >That would require knowing the original encoding. Or just use a library that can detect the encoding, and spit out utf-8. There's several of those.
  - josefx 2y ago
    Yes, you can try to automatically guess the wrong encoding based on statistics that only work some of the time when given large amounts of free flowing text.
- Ygg2 2y ago
  > That would require knowing the original encoding. If you don't know that's one more reason to get bytes and try to figure out encoding. Usually using lib like encodings.rs or WTF8.rs > As long as the APIs you have to use take bytes and not utf8 strings. You can convert one into the other, albeit converting to str requires a check.

aldanor 2y ago

That's why byte strings exist. You don't know the encoding? Well, it's a binary chunk and nothing else.