3 ms·

It just doesn't match.

by burntsushi 2y ago

camgunz 2y ago
Maybe! Or maybe another tool is like, "oof that's some junk in there, let's pop that out"
- burntsushi 2y ago
  I'm the author of Rust's regex engine. Treat me as an oracle. What is your specific concern? I would really strongly recommend you read my bstr blog post first though. It should answer a lot of your questions and I hope presents a nuanced data driven analysis.
  - camgunz 2y ago
    > I'm the author of Rust's regex engine. Treat me as an oracle. What is your specific concern? I know and I'm terrified haha. That's a very generous offer; I don't disagree w/ your post or OP's basic premise that validation isn't absolutely necessary. I also think it's perfectly reasonable for regexes to not match on invalid bytes (what, we should just ignore stuff now? Total madness). Like, if you don't like it, consider living less dangerously and validating! My overarching worry is that a lot of engineers will read this and think, "validation is indeed for wimps" and suddenly a lot of invalid UTF-8 will end up in databases, tape backups, and S3 tripping us up for years to come. I probably could be mollified with a small title tweak like "In some cases (and you'll know if if you have one) you can intentionally not validate UTF-8 strings, but throw some salt over your shoulder first." I also think there's plenty of weird interactions like with normalization or basic string manipulation people should think about before walking down this road. > I hope presents a nuanced data driven analysis I was actually sold early on when you pointed out that filesystems are creepy attics full of exotic data; perfect use case for not assuming/forcing any kind of string encoding and a really good blueprint for applying solid engineering principles to give the people what they want.
    - burntsushi 2y ago
      I think I would point to Go here as a good data point for mollification. Its `string` data is conventionally UTF-8. Go has been a huge player in web services over the last decade and a half, and I'm not sure I've seen a rise in the problem of junk data. The people behind Go were also the people behind UTF-8 (Ken Thompson and Rob Pike). Go's `string` data type is thoughtfully designed. If you iterate over the characters in a string that contains invalid UTF-8, you get replacement codepoints, just like in bstr. Go has transcoding and normalization libraries that work just fine too. I don't think there are much if any weird interactions with normalization or other such things. When you're doing Unicode operations on a conventionally UTF-8 string, you just treat invalid UTF-8 as U+FFFD. That's really all there is to it. Everything else works. Unicode algorithms are defined over codepoints. So all you need is a semantic for how to convert invalid sequences to codepoints. And Unicode (Chapter 3, Section 9) provides exactly that for you. Or in some cases, you can just leave the invalid UTF-8 as-is. The last example in bstr's docs for its "to_lowercase" routine[1], for example, demonstrates how you can perform a Unicode-aware algorithm on a string that contains invalid UTF-8. We've all been burned by mojibake before. I get it. I'm not totally discounting your viewpoint. I do tend to think that "required UTF-8" is probably the better thing for a primitive string data type. But you really do absolutely need to also then support conventional UTF-8. Otherwise you're cutting off a ton of very valid use cases, and potentially forcing others to either take a huge perf hit or to do something dangerous like `unsafe { std::str::from_utf8_unchecked(..) }`. I've seen people do the unchecked variant before just because they had a `&[u8]` but needed it to be a `&str` because they wanted to treat it as a string without doing a second pass over the data. And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-) [1]: https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#examples-58 https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#exampl...
      - camgunz 2y ago
        The Go thing (+ invalid sequence conversion) is a really good point. If OP's point was "don't validate just convert invalid sequences when you need to" or "think twice before forcing validation into your primitive string type" I'd buy in. I also think your choice there to just write the invalid bytes out unchanged also makes a ton of sense. You're right that there's more options than "yolo no validate" or "validate or face death". > And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-) Woof man yeah, that's what I was getting at with you're forcing people to basically implement things twice; it's not the best! This was a really good conversation; before this I was 100% in agreement w/ you on "'required UTF-8' is probably the better thing for a primitive string type" but now I see there's a fair amount of pragmatic issues with that. Definitely a lot to mull over here; thanks :)
  - paulddraper 2y ago
    Does . match one bytes or one characters?
    - burntsushi 2y ago
      `.`, which is `(?u:.)` by default, matches the UTF-8 encoding of any Unicode scalar value except for the newline terminator. `(?-u:.)` matches any individual byte except for the newline terminator. You can't use `(?-u:.)` when searching `&str` because it could match invalid UTF-8. `Regex::new` will fail if you try. You can only use it when searching `&[u8]` via `regex::bytes::Regex`.

It just doesn't match.

camgunz 2y ago

Maybe! Or maybe another tool is like, "oof that's some junk in there, let's pop that out"

burntsushi 2y ago
I'm the author of Rust's regex engine. Treat me as an oracle. What is your specific concern? I would really strongly recommend you read my bstr blog post first though. It should answer a lot of your questions and I hope presents a nuanced data driven analysis.
- camgunz 2y ago
  > I'm the author of Rust's regex engine. Treat me as an oracle. What is your specific concern? I know and I'm terrified haha. That's a very generous offer; I don't disagree w/ your post or OP's basic premise that validation isn't absolutely necessary. I also think it's perfectly reasonable for regexes to not match on invalid bytes (what, we should just ignore stuff now? Total madness). Like, if you don't like it, consider living less dangerously and validating! My overarching worry is that a lot of engineers will read this and think, "validation is indeed for wimps" and suddenly a lot of invalid UTF-8 will end up in databases, tape backups, and S3 tripping us up for years to come. I probably could be mollified with a small title tweak like "In some cases (and you'll know if if you have one) you can intentionally not validate UTF-8 strings, but throw some salt over your shoulder first." I also think there's plenty of weird interactions like with normalization or basic string manipulation people should think about before walking down this road. > I hope presents a nuanced data driven analysis I was actually sold early on when you pointed out that filesystems are creepy attics full of exotic data; perfect use case for not assuming/forcing any kind of string encoding and a really good blueprint for applying solid engineering principles to give the people what they want.
  - burntsushi 2y ago
    I think I would point to Go here as a good data point for mollification. Its `string` data is conventionally UTF-8. Go has been a huge player in web services over the last decade and a half, and I'm not sure I've seen a rise in the problem of junk data. The people behind Go were also the people behind UTF-8 (Ken Thompson and Rob Pike). Go's `string` data type is thoughtfully designed. If you iterate over the characters in a string that contains invalid UTF-8, you get replacement codepoints, just like in bstr. Go has transcoding and normalization libraries that work just fine too. I don't think there are much if any weird interactions with normalization or other such things. When you're doing Unicode operations on a conventionally UTF-8 string, you just treat invalid UTF-8 as U+FFFD. That's really all there is to it. Everything else works. Unicode algorithms are defined over codepoints. So all you need is a semantic for how to convert invalid sequences to codepoints. And Unicode (Chapter 3, Section 9) provides exactly that for you. Or in some cases, you can just leave the invalid UTF-8 as-is. The last example in bstr's docs for its "to_lowercase" routine[1], for example, demonstrates how you can perform a Unicode-aware algorithm on a string that contains invalid UTF-8. We've all been burned by mojibake before. I get it. I'm not totally discounting your viewpoint. I do tend to think that "required UTF-8" is probably the better thing for a primitive string data type. But you really do absolutely need to also then support conventional UTF-8. Otherwise you're cutting off a ton of very valid use cases, and potentially forcing others to either take a huge perf hit or to do something dangerous like `unsafe { std::str::from_utf8_unchecked(..) }`. I've seen people do the unchecked variant before just because they had a `&[u8]` but needed it to be a `&str` because they wanted to treat it as a string without doing a second pass over the data. And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-) [1]: https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#examples-58 https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#exampl...
    - camgunz 2y ago
      The Go thing (+ invalid sequence conversion) is a really good point. If OP's point was "don't validate just convert invalid sequences when you need to" or "think twice before forcing validation into your primitive string type" I'd buy in. I also think your choice there to just write the invalid bytes out unchanged also makes a ton of sense. You're right that there's more options than "yolo no validate" or "validate or face death". > And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-) Woof man yeah, that's what I was getting at with you're forcing people to basically implement things twice; it's not the best! This was a really good conversation; before this I was 100% in agreement w/ you on "'required UTF-8' is probably the better thing for a primitive string type" but now I see there's a fair amount of pragmatic issues with that. Definitely a lot to mull over here; thanks :)
- paulddraper 2y ago
  Does . match one bytes or one characters?
  - burntsushi 2y ago
    `.`, which is `(?u:.)` by default, matches the UTF-8 encoding of any Unicode scalar value except for the newline terminator. `(?-u:.)` matches any individual byte except for the newline terminator. You can't use `(?-u:.)` when searching `&str` because it could match invalid UTF-8. `Regex::new` will fail if you try. You can only use it when searching `&[u8]` via `regex::bytes::Regex`.