5 ms·

Things are obviously more complicated than that, lines are a complicated issue for historical reasons. There are two conventions, line termination and line sepa

Things are obviously more complicated than that, lines are a complicated issue for historical reasons. There are two conventions, line termination and line separation. In case of line termination, the newline is part of the line and a string without a newline is not a [complete] line. In case of line separation, the newline is not part of the line but separates two lines. Also the way newlines are encoded is not universal.

fauigerzigerk 3y ago
Why is this relevant when multi-line is disabled?
- danbruc 3y ago
  Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $, the newline at the end is still not part of the content. You have to use \A and \Z if you want to treat all characters as a string instead of one or multiple lines.
  - burntsushi 3y ago
    > Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $ No, you're not, except for this weird corner case where `$` can match before the last `\n` in a string. It's not just any `\n` that non-multiline `$` can match before. It's when it's the last `\n` in the string. See: >>> re.search('cat$', 'cat\n') <re.Match object; span=(0, 3), match='cat'> >>> re.search('cat$', 'cat\n\n') >>> This is weird behavior. I assume this is why RE2 didn't copy this. And it's certainly why I followed RE2 with Rust's regex crate. Non-multiline `$` should only match at the end of the string. It should not be line-aware. In regex engines like Python where it has the behavior above, it is only "partially" line-aware, and only in the sense that it treats the last `\n` as special.
    - danbruc 3y ago
      But that is exactly what it means, the end of the line is before the terminating newline or at the end of the string if there is no terminating newline. Both ^ and $ always match at start or end of lines, \A and \Z match at the start or end of the string. The difference between multi-line and not is whether or not internal newlines end and start lines, it does not change the semantics from end of line to end of string. And if you are not in multi-line mode but have internal newlines, then you might also want single-line/dot-all mode. One could certainly have a debate whether this behavior is too strongly tied to the origins of regular expressions and now does more harm than good, but I am not convinced that this would be an easy and obvious choice to have breaking change.
      - IshKebab 3y ago
        > But that is exactly what it means I think you've kind of missed the point. Sure if `$` in non-multiline mode means "end of line" the behaviour might be reasonable. But the big error is that people DO NOT EXPECT `$` to mean "end of line" in that case. They expect it to mean "end of string". That's clearly the least surprising and most useful behaviour. The bug is not in how they have implemented "end of line" matching in non-multiline mode. It's that they did it at all.
      - burntsushi 3y ago
        re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. And giving it a `string` with multiple new lines doesn't necessarily mean you want to enable multi-line mode. They are orthogonal things. > Both ^ and $ always match at start or end of lines This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` contains two lines and the regex `cat$` says it should match `cat` followed by the "end of a line" according to your definition. Yet it does not match `cat` followed by the end of a line in `cat\n\n`. And it does not do so in Python or in any other regex engine. You're trying to square a circle here. It can't be done. Can you make sense of, historically, why this choice of semantics was made? Sure. I bet you can. But I can still evaluate the choice on its own merits today. And I did when I made the regex crate. > but I am not convinced that this would be an easy and obvious choice to have breaking change. Rust's regex crate, Go's regexp package and RE2 all reject this whacky behavior. As the regex crate maintainer, I don't think I've ever seen anyone complain. Not once. This to me suggests that, at minimum, making `$` and `\z` equivalent in non-multiline mode is a reasonable choice. I would also argue it is the better and more sensible approach. Whether other regex engines should have a breaking change or not to change the meaning of `$` is an entirely different question completely. That is neither here nor there. They absolutely will not be able to make such a change, for many good reasons.

fauigerzigerk 3y ago

Why is this relevant when multi-line is disabled?

danbruc 3y ago
Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $, the newline at the end is still not part of the content. You have to use \A and \Z if you want to treat all characters as a string instead of one or multiple lines.
- burntsushi 3y ago
  > Because even after disabling multi-line you are still dealing with line-based semantics when you use ^ or $ No, you're not, except for this weird corner case where `$` can match before the last `\n` in a string. It's not just any `\n` that non-multiline `$` can match before. It's when it's the last `\n` in the string. See: >>> re.search('cat$', 'cat\n') <re.Match object; span=(0, 3), match='cat'> >>> re.search('cat$', 'cat\n\n') >>> This is weird behavior. I assume this is why RE2 didn't copy this. And it's certainly why I followed RE2 with Rust's regex crate. Non-multiline `$` should only match at the end of the string. It should not be line-aware. In regex engines like Python where it has the behavior above, it is only "partially" line-aware, and only in the sense that it treats the last `\n` as special.
  - danbruc 3y ago
    But that is exactly what it means, the end of the line is before the terminating newline or at the end of the string if there is no terminating newline. Both ^ and $ always match at start or end of lines, \A and \Z match at the start or end of the string. The difference between multi-line and not is whether or not internal newlines end and start lines, it does not change the semantics from end of line to end of string. And if you are not in multi-line mode but have internal newlines, then you might also want single-line/dot-all mode. One could certainly have a debate whether this behavior is too strongly tied to the origins of regular expressions and now does more harm than good, but I am not convinced that this would be an easy and obvious choice to have breaking change.
    - IshKebab 3y ago
      > But that is exactly what it means I think you've kind of missed the point. Sure if `$` in non-multiline mode means "end of line" the behaviour might be reasonable. But the big error is that people DO NOT EXPECT `$` to mean "end of line" in that case. They expect it to mean "end of string". That's clearly the least surprising and most useful behaviour. The bug is not in how they have implemented "end of line" matching in non-multiline mode. It's that they did it at all.
    - burntsushi 3y ago
      re.search does not accept a "line." It accepts a "string." There is no pretext in which re.search is meant to only accept a single line. And giving it a `string` with multiple new lines doesn't necessarily mean you want to enable multi-line mode. They are orthogonal things. > Both ^ and $ always match at start or end of lines This is trivially not true, as I showed in my previous example. The haystack `cat\n\n` contains two lines and the regex `cat$` says it should match `cat` followed by the "end of a line" according to your definition. Yet it does not match `cat` followed by the end of a line in `cat\n\n`. And it does not do so in Python or in any other regex engine. You're trying to square a circle here. It can't be done. Can you make sense of, historically, why this choice of semantics was made? Sure. I bet you can. But I can still evaluate the choice on its own merits today. And I did when I made the regex crate. > but I am not convinced that this would be an easy and obvious choice to have breaking change. Rust's regex crate, Go's regexp package and RE2 all reject this whacky behavior. As the regex crate maintainer, I don't think I've ever seen anyone complain. Not once. This to me suggests that, at minimum, making `$` and `\z` equivalent in non-multiline mode is a reasonable choice. I would also argue it is the better and more sensible approach. Whether other regex engines should have a breaking change or not to change the meaning of `$` is an entirely different question completely. That is neither here nor there. They absolutely will not be able to make such a change, for many good reasons.