3 ms·

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone rea

> It is start and end of line. You seem to have redefined “line” as “not a line”. > The confusion I’m sure redefining “line” as “nothing like what anyone reasonable would interpret as a line” will help a lot and right clear up the confusion.

danbruc 3y ago
The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.
- al_borland 3y ago
  I suppose this is why certain config files are strict about ending with a newline, without it, the last line wouldn’t technically be a line?
- deleted 3y ago
  [deleted]
- masklinn 3y ago
  > a line is a sequence of non-newline characters Works for me. How do you square that with your assertion that in your invention of "single-line mode" you implicitly define "line" as matching \n\n?
  - danbruc 3y ago
    If you are not in multi-line mode, then a single line is expected and consequently there is at most one newline at the end of the string. You can of course pick an input that violates this, run it against a multi-line string with several newlines in it. cat\n\n will not match cat$ because there is something between cat and the end of the line, it just happens to be a newline but without any special meaning because it is not the last character and you did not say that the input is multi-line.
- sltkr 3y ago
  Python violates that definition however, by allowing internal newlines in strings. For example /^c[^a]t$/ matches "c\nt\n", but according to POSIX that's not a line. I suspect the real reason for Python's behavior starts with the early decision to include the terminating newline in the string returned by IOBase.readline(). Python's peculiar choice has some minor advantages: you can distinguish between files that do and don't end with a terminating newline (the latter are invalid according to POSIX, but common in practice, especially on Windows), and you can reconstruct the original file by simply concatenating the line strings, which is occasionally useful. The downside of this choice is that as a caller you have to deal with strings that may-or-may-not contain a terminating newline character, which is annoying (I often end up calling rstrip() or strip() on every line returned by readline(), just to get rid of the newlines; read().splitlines() is an option too if you don't mind reading the entire file into memory upfront). My guess is that Python's behavior is just a hack to make re.match() easier to use with readline(), rather than based on any principled belief about what lines are.
  - danbruc 3y ago
    Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and match $ accordingly, either at the end of the string or before a terminating newline, if there are additional newlines, you treat them mostly as normal characters, with the exception of dot not matching newlines unless you set the single-line/dot-all flag.
    - sltkr 3y ago
      > Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behavior. > $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. Yes, that is Python's behavior (and PHP's, Java's, etc.). You're just describing it; not motivating why it has to work that way or why it's more correct than the obvious alternative of only matching the end of the string. Subjectively, I find it odd that /^cat$/ matches not just the obvious string "cat" but also the string "cat\n". And I think historically, it didn't. I tried several common tools that predate Python: - awk 'BEGIN { print ("cat\n" ~ /^cat$/) }' prints 0 - in GNU ed, /^M/ does not match any lines - in vim, /^M/ does not match any lines - sed -n '/\n/p' does not print any lines - grep -P '\n' does not match any lines - (I wanted to try `grep -E` too but I don't know how to escape a newline) - perl -e 'print ("cat\n" =~ /^cat$/)' prints 1 So the consensus seems to be that the classic UNIX line-based tools match the regex against the line excluding the newline terminator (which makes sense since it isn't part of the content of that line) and therefore $ only needs to match the end of the string. The odd one out is Perl: it seems to have introduced the idea that $ can match a newline at the end of the string, probably for similar reasons as Python. All of this suggests to me that allowing $ to match both "\n" and "" at the end of the string was a hack designed to make it easier to deal with strings without control characters and string that end with a single newline.
- zeehio 3y ago
  I'm now amused by the idea of a malicious compliance linter telling me "you have an issue at your code, on line NaN"
deleted 3y ago
[deleted]
Bjartr 3y ago
The line delimiter is a newline. If you have a file containing `A\nB\nC` in a file, the file is three lines long. I guess it could be argued that a file containing `A\nB\nC\n` has four lines, with the fourth having zero length. That a regex is applying to an in memory string vs a file doesn't feel to me like it should have different semantics. Digging into the history a little, it looks like regexes were popularized in text editors and other file oriented tooling. In those contexts I imagine it would be far more common to want to discard or ignore the trailing zero length line than to process it like every other line in a file.
- akdev1l 3y ago
  Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.
  - wtetzner 3y ago
    So if you have "A" in a file with no newline, there are no lines in that file?
    - jepler 3y ago
      Yes, that is a file with zero lines that ends with an "incomplete line". Processing of such files by standard line-oriented utilities is undefined in the opengroup spec. So, for instance, the effect of "grep"ping such a file is not defined. Heck, even "cat"ting such a file gives non-ideal results, such as colliding with the regular shell prompt. For this reason, a lot of software projects I work on check and correct this condition whenever creating a commit. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... ("text file")
      - rovr138 3y ago
        > Yes, that is a file with zero lines that ends with an "incomplete line". It's a file with zero complete lines. But it has 1 line, that's incomplete, right? The file starts empty. Anything in it starts "a line". So it's 1 incomplete line. I hate weird states.

danbruc 3y ago

The POSIX definition of a line is a sequence of non-newline characters - possibly zero - followed by a newline. Everything that does not end with a newline is not a [complete] line. So strictly speaking it would even be correct that cat$ does not match cat because there is no terminating newline, it should only match cat\n. But as lines missing a terminating newline is a thing, it seems reasonable to be less strict.

al_borland 3y ago
I suppose this is why certain config files are strict about ending with a newline, without it, the last line wouldn’t technically be a line?
deleted 3y ago
[deleted]
masklinn 3y ago
> a line is a sequence of non-newline characters Works for me. How do you square that with your assertion that in your invention of "single-line mode" you implicitly define "line" as matching \n\n?
- danbruc 3y ago
  If you are not in multi-line mode, then a single line is expected and consequently there is at most one newline at the end of the string. You can of course pick an input that violates this, run it against a multi-line string with several newlines in it. cat\n\n will not match cat$ because there is something between cat and the end of the line, it just happens to be a newline but without any special meaning because it is not the last character and you did not say that the input is multi-line.
sltkr 3y ago
Python violates that definition however, by allowing internal newlines in strings. For example /^c[^a]t$/ matches "c\nt\n", but according to POSIX that's not a line. I suspect the real reason for Python's behavior starts with the early decision to include the terminating newline in the string returned by IOBase.readline(). Python's peculiar choice has some minor advantages: you can distinguish between files that do and don't end with a terminating newline (the latter are invalid according to POSIX, but common in practice, especially on Windows), and you can reconstruct the original file by simply concatenating the line strings, which is occasionally useful. The downside of this choice is that as a caller you have to deal with strings that may-or-may-not contain a terminating newline character, which is annoying (I often end up calling rstrip() or strip() on every line returned by readline(), just to get rid of the newlines; read().splitlines() is an option too if you don't mind reading the entire file into memory upfront). My guess is that Python's behavior is just a hack to make re.match() easier to use with readline(), rather than based on any principled belief about what lines are.
- danbruc 3y ago
  Python's behavior is not a hack, it is the common behavior. $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. But as you said, you can have additional newlines inside of the string which is also the common behavior and not specific to python. Personally I think of this as you just assume that the string is a single line and match $ accordingly, either at the end of the string or before a terminating newline, if there are additional newlines, you treat them mostly as normal characters, with the exception of dot not matching newlines unless you set the single-line/dot-all flag.
  - sltkr 3y ago
    > Python's behavior [..] is the common behavior. The very post we're commenting on shows that that's not true: PHP, Python, Java and .NET (C#) share one behavior (accept "\n" as "$"), and ECMAScript (Javascript), Golang, and Rust share another behavior (do not accept "\n" as $). Let's not argue about which is “the most common”; all of these languages are sufficiently common to say that there is no single common behavior. > $ matches at the end of the string or before the last character if that is a newline, which is logically the same as the end of a single line. Yes, that is Python's behavior (and PHP's, Java's, etc.). You're just describing it; not motivating why it has to work that way or why it's more correct than the obvious alternative of only matching the end of the string. Subjectively, I find it odd that /^cat$/ matches not just the obvious string "cat" but also the string "cat\n". And I think historically, it didn't. I tried several common tools that predate Python: - awk 'BEGIN { print ("cat\n" ~ /^cat$/) }' prints 0 - in GNU ed, /^M/ does not match any lines - in vim, /^M/ does not match any lines - sed -n '/\n/p' does not print any lines - grep -P '\n' does not match any lines - (I wanted to try `grep -E` too but I don't know how to escape a newline) - perl -e 'print ("cat\n" =~ /^cat$/)' prints 1 So the consensus seems to be that the classic UNIX line-based tools match the regex against the line excluding the newline terminator (which makes sense since it isn't part of the content of that line) and therefore $ only needs to match the end of the string. The odd one out is Perl: it seems to have introduced the idea that $ can match a newline at the end of the string, probably for similar reasons as Python. All of this suggests to me that allowing $ to match both "\n" and "" at the end of the string was a hack designed to make it easier to deal with strings without control characters and string that end with a single newline.
zeehio 3y ago
I'm now amused by the idea of a malicious compliance linter telling me "you have an issue at your code, on line NaN"

deleted 3y ago

[deleted]

Bjartr 3y ago

The line delimiter is a newline. If you have a file containing `A\nB\nC` in a file, the file is three lines long. I guess it could be argued that a file containing `A\nB\nC\n` has four lines, with the fourth having zero length. That a regex is applying to an in memory string vs a file doesn't feel to me like it should have different semantics. Digging into the history a little, it looks like regexes were popularized in text editors and other file oriented tooling. In those contexts I imagine it would be far more common to want to discard or ignore the trailing zero length line than to process it like every other line in a file.

akdev1l 3y ago
Technically the “newline” character is actually a line _terminator_. Hence “A\n” is one line, not two. The “\n” is always at the end of a line by definition.
- wtetzner 3y ago
  So if you have "A" in a file with no newline, there are no lines in that file?
  - jepler 3y ago
    Yes, that is a file with zero lines that ends with an "incomplete line". Processing of such files by standard line-oriented utilities is undefined in the opengroup spec. So, for instance, the effect of "grep"ping such a file is not defined. Heck, even "cat"ting such a file gives non-ideal results, such as colliding with the regular shell prompt. For this reason, a lot of software projects I work on check and correct this condition whenever creating a commit. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... ("text file")
    - rovr138 3y ago
      > Yes, that is a file with zero lines that ends with an "incomplete line". It's a file with zero complete lines. But it has 1 line, that's incomplete, right? The file starts empty. Anything in it starts "a line". So it's 1 incomplete line. I hate weird states.