4 ms·

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is n

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

al_borland 3y ago
I suppose this is why certain config files are strict about ending with a newline, without it, the last line wouldn’t technically be a line?
deleted 3y ago
[deleted]
masklinn 3y ago
> a line is a sequence of non-newline characters Works for me. How do you square that with your assertion that in your invention of "single-line mode" you implicitly define "line" as matching \n\n?
- danbruc 3y ago
  If you are not in multi-line mode, then a single line is expected and consequently there is at most one newline at the end of the string. You can of course pick an input that violates this, run it against a multi-line string with several newlines in it. cat\n\n will not match cat$ because there is something between cat and the end of the line, it just happens to be a newline but without any special meaning because it is not the last character and you did not say that the input is multi-line.
sltkr 3y ago
Python violates that definition however, by allowing internal newlines in strings. For example /^c[^a]t$/ matches "c\nt\n", but according to POSIX that's not a line. I suspect the real reason for Python's behavior starts with the early decision to include the terminating newline in the string returned by IOBase.readline(). Python's peculiar choice has some minor advantages: you can distinguish between files that do and don't end with a terminating newline (the latter are invalid according to POSIX, but common in practice, especially on Windows), and you can reconstruct the original file by simply concatenating the line strings, which is occasionally useful. The downside of this choice is that as a caller you have to deal with strings that may-or-may-not contain a terminating newline character, which is annoying (I often end up calling rstrip() or strip() on every line returned by readline(), just to get rid of the newlines; read().splitlines() is an option too if you don't mind reading the entire file into memory upfront). My guess is that Python's behavior is just a hack to make re.match() easier to use with readline(), rather than based on any principled belief about what lines are.
- danbruc 3y ago
  Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and match $ accordingly, either at the end of the string or before a terminating newline, if there are additional newlines, you treat them mostly as normal characters, with the exception of dot not matching newlines unless you set the single-line/dot-all flag.
  - sltkr 3y ago
    > Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behavior. > $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. Yes, that is Python's behavior (and PHP's, Java's, etc.). You're just describing it; not motivating why it has to work that way or why it's more correct than the obvious alternative of only matching the end of the string. Subjectively, I find it odd that /^cat$/ matches not just the obvious string "cat" but also the string "cat\n". And I think historically, it didn't. I tried several common tools that predate Python: - awk 'BEGIN { print ("cat\n" ~ /^cat$/) }' prints 0 - in GNU ed, /^M/ does not match any lines - in vim, /^M/ does not match any lines - sed -n '/\n/p' does not print any lines - grep -P '\n' does not match any lines - (I wanted to try `grep -E` too but I don't know how to escape a newline) - perl -e 'print ("cat\n" =~ /^cat$/)' prints 1 So the consensus seems to be that the classic UNIX line-based tools match the regex against the line excluding the newline terminator (which makes sense since it isn't part of the content of that line) and therefore $ only needs to match the end of the string. The odd one out is Perl: it seems to have introduced the idea that $ can match a newline at the end of the string, probably for similar reasons as Python. All of this suggests to me that allowing $ to match both "\n" and "" at the end of the string was a hack designed to make it easier to deal with strings without control characters and string that end with a single newline.
zeehio 3y ago
I'm now amused by the idea of a malicious compliance linter telling me "you have an issue at your code, on line NaN"

al_borland 3y ago

I suppose this is why certain config files are strict about ending with a newline, without it, the last line wouldn’t technically be a line?

deleted 3y ago

[deleted]

masklinn 3y ago

> a line is a sequence of non-newline characters Works for me. How do you square that with your assertion that in your invention of "single-line mode" you implicitly define "line" as matching \n\n?

danbruc 3y ago
If you are not in multi-line mode, then a single line is expected and consequently there is at most one newline at the end of the string. You can of course pick an input that violates this, run it against a multi-line string with several newlines in it. cat\n\n will not match cat$ because there is something between cat and the end of the line, it just happens to be a newline but without any special meaning because it is not the last character and you did not say that the input is multi-line.

sltkr 3y ago

Python violates that definition however, by allowing internal newlines in strings. For example /^c[^a]t$/ matches "c\nt\n", but according to POSIX that's not a line. I suspect the real reason for Python's behavior starts with the early decision to include the terminating newline in the string returned by IOBase.readline(). Python's peculiar choice has some minor advantages: you can distinguish between files that do and don't end with a terminating newline (the latter are invalid according to POSIX, but common in practice, especially on Windows), and you can reconstruct the original file by simply concatenating the line strings, which is occasionally useful. The downside of this choice is that as a caller you have to deal with strings that may-or-may-not contain a terminating newline character, which is annoying (I often end up calling rstrip() or strip() on every line returned by readline(), just to get rid of the newlines; read().splitlines() is an option too if you don't mind reading the entire file into memory upfront). My guess is that Python's behavior is just a hack to make re.match() easier to use with readline(), rather than based on any principled belief about what lines are.

danbruc 3y ago
Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and match $ accordingly, either at the end of the string or before a terminating newline, if there are additional newlines, you treat them mostly as normal characters, with the exception of dot not matching newlines unless you set the single-line/dot-all flag.
- sltkr 3y ago
  > Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behavior. > $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. Yes, that is Python's behavior (and PHP's, Java's, etc.). You're just describing it; not motivating why it has to work that way or why it's more correct than the obvious alternative of only matching the end of the string. Subjectively, I find it odd that /^cat$/ matches not just the obvious string "cat" but also the string "cat\n". And I think historically, it didn't. I tried several common tools that predate Python: - awk 'BEGIN { print ("cat\n" ~ /^cat$/) }' prints 0 - in GNU ed, /^M/ does not match any lines - in vim, /^M/ does not match any lines - sed -n '/\n/p' does not print any lines - grep -P '\n' does not match any lines - (I wanted to try `grep -E` too but I don't know how to escape a newline) - perl -e 'print ("cat\n" =~ /^cat$/)' prints 1 So the consensus seems to be that the classic UNIX line-based tools match the regex against the line excluding the newline terminator (which makes sense since it isn't part of the content of that line) and therefore $ only needs to match the end of the string. The odd one out is Perl: it seems to have introduced the idea that $ can match a newline at the end of the string, probably for similar reasons as Python. All of this suggests to me that allowing $ to match both "\n" and "" at the end of the string was a hack designed to make it easier to deal with strings without control characters and string that end with a single newline.

zeehio 3y ago

I'm now amused by the idea of a malicious compliance linter telling me "you have an issue at your code, on line NaN"