8 ms·

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"?

jamesmunns 3y ago
Same, tho it'd be interesting to see if this behavior holds if the file ends without a trailing newline and your match is on the final newline-less line.
- fooofw 3y ago
  Fortunately, it's pretty simple to test. $ printf 'Line with EOL\nLine without EOL' | grep 'EOL$' Line with EOL Line without EOL $ grep --version | head -n1 grep (GNU grep) 3.8
  - romwell 3y ago
    The line does end with the file, so it's logically consistent. It's not matching the newline character after all.
    - colimbarna 3y ago
      Yes exactly, they match the end of a line, not a newline character. Some examples from documentation: man 7 regex: '$' (matching the null string at the end of a line) pcre2pattern: The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. ... The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.
  - jamesmunns 3y ago
    Thanks! I was AFK and didn't have a grep (or a shell) handy on my phone.
antegamisou 3y ago
Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"? Vim is what did that for me.
wccrawford 3y ago
It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".
- masklinn 3y ago
  It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance: > Matches the start of the string, and in MULTILINE mode also matches immediately after each newline. Or JavaScript: > An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line. \A and \Z are start/end of input regardless of mode… when they’re available, that’s not the case of all engines.
  - eastbound 3y ago
    Probably a vulnerability issue. Programmers would leave multiline mode on by mistake, then validate that some string only contain ^[a-Z]*$… only for the string to have an \n and an SQL injection on the second line.
    - masklinn 3y ago
      > Probably a vulnerability issue. No? It’s a semantics decision.
  - danbruc 3y ago
    It is start and end of line. [1] Usually ^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of the string and at the end of each line (immediately preceding each newline). In single-line [2] mode, the line starts at the start of the string and ends at the end of the line where the end of the line is either the end of the string if there is no terminating newline or just before the final newline if there is a terminating newline. In multi-line mode a new line starts at the start of the string and after each newline and ends before each newline or at the end of the string if the last line has no terminating newline. The confusion is that people think that they are in string-mode if they are not in multi-line mode but they are not, they are in single-line mode, ^ and $ still use the semantics of lines and a terminating newline, if present, is still not part of the content of the line. With \n\n\n in single-line mode the non-greedy ^(\n+?)$ will capture only two of the newlines, the third one will be eaten by the $. If you make it greedy ^(\n+)$ will capture all three newlines. So arguably the implementations that do not match cat\n with cat$ are the broken ones. [1] https://docs.python.org/3/howto/regex.html#more-metacharacters https://docs.python.org/3/howto/regex.html#more-metacharacte... [2] I am using single-line to mean not multi-line for convenience even though single-line already has a different meaning.
kqr 3y ago
I'm the same, but now that I try in Perl, sure enough, $ seems to default to being a positive lookahead assertion for the end of the string. It does not match and consume an EOL character. Only in multiline mode does it match EOL characters, but it does still not appear to consume them. In fact, I cannot construct a regex that captures the last character of one line, then consumes the newline, and then captures the first character of the next line, while using $. The capture group simply ends at $.
- singingfish 3y ago
  To get the newline captured as well you need to add the `/s` modifier too
absoluteunit1 3y ago
I’ve always thought that as well; mostly due to Vim though. ^ - takes you to start of line $ - takes you to end of line
- Izkata 3y ago
  ^ actually takes you to the first non-whitespace character in the line in vim. For start of line you want 0
  - kataklasm 3y ago
    I don't have (n)vi(m) open right now but I think this only applies to prepending spaces. For prepending tabs, 0 will take you to the first non-tab character as well.
    - qu4z-2 3y ago
      Vim takes me to the first character in the line (the first tab), but displays the cursor on the last gridsquare the tab's width covers.
      - Izkata 3y ago
        If you have "set list" to make non-space whitespace visible, it'll go to the leftmost position. I did it long ago along with "set listchars=trail:.,tab:>-" so I can see not only where tabs are, but also their size/alignment without causing the text to shift.
alphazard 3y ago
This must be the "second problem" everyone talks about with regular expressions.
Izkata 3y ago
Same here; when I saw the title I was like "well obviously not, where did you hear that?" In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.
- deleted 3y ago
  [deleted]
- frame_ranger 3y ago
  You couldn’t write a post like this if you didn’t start with a strawman.
  - dbdudbdiddjc 3y ago
    It’s not a straw man, it’s accurate in many contexts.
- michaelt 3y ago
  Take a look at, for example, these stackoverflow answers about a regex to validate and e-mail address: https://stackoverflow.com/a/8829363 https://stackoverflow.com/a/8829363 These people are I think not intending to say a newline character is permitted at the end of an e-mail address. (Of course people using 'grep' would have different expectations for obvious reasons)
  - Izkata 3y ago
    Even disregarding whether or not end-of-string is also an end-of-line or not (see all the other comments below), $ doesn't match the newline, similar to zero-width matches like \b, so the newline wouldn't be included in the matched text either way. I think this series of comments might be clearest: https://news.ycombinator.com/item?id=39764385 https://news.ycombinator.com/item?id=39764385
    - LK5ZJwMwgBbHuVI 3y ago
      Problem is, plenty of software doesn't actually look at the match but rather just validates that there was a match (and then continues to use the input to that match).
notnmeyer 3y ago
i feel like this perspective will be split between folks who use regex in code with strings and more sysadmin folks who are used to consuming lines from files in scripts and at the cli. but yeah seems like a real misunderstanding from “start/end of string” people
cerved 3y ago
In `sed` it's end of string. String is usually end of line, but not if you use stuff like `N`, to manipulate multi-line strings

jamesmunns 3y ago

Same, tho it'd be interesting to see if this behavior holds if the file ends without a trailing newline and your match is on the final newline-less line.

fooofw 3y ago
Fortunately, it's pretty simple to test. $ printf 'Line with EOL\nLine without EOL' | grep 'EOL$' Line with EOL Line without EOL $ grep --version | head -n1 grep (GNU grep) 3.8
- romwell 3y ago
  The line does end with the file, so it's logically consistent. It's not matching the newline character after all.
  - colimbarna 3y ago
    Yes exactly, they match the end of a line, not a newline character. Some examples from documentation: man 7 regex: '$' (matching the null string at the end of a line) pcre2pattern: The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. ... The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.
- jamesmunns 3y ago
  Thanks! I was AFK and didn't have a grep (or a shell) handy on my phone.

antegamisou 3y ago

Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"? Vim is what did that for me.

wccrawford 3y ago

It's kind of driving me nuts that the article says ^ is "start of string" when it's actually "start of line", just like $ is "end of line". \A is apparently "start of string" like \Z is "end of string".

masklinn 3y ago
It’s not start of line though, unless the engine is in multiline mode. Here is the documentation for Python’s re for instance: > Matches the start of the string, and in MULTILINE mode also matches immediately after each newline. Or JavaScript: > An input boundary is the start or end of the string; or, if the m flag is set, the start or end of a line. \A and \Z are start/end of input regardless of mode… when they’re available, that’s not the case of all engines.
- eastbound 3y ago
  Probably a vulnerability issue. Programmers would leave multiline mode on by mistake, then validate that some string only contain ^[a-Z]*$… only for the string to have an \n and an SQL injection on the second line.
  - masklinn 3y ago
    > Probably a vulnerability issue. No? It’s a semantics decision.
- danbruc 3y ago
  It is start and end of line. [1] Usually ^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of the string and at the end of each line (immediately preceding each newline). In single-line [2] mode, the line starts at the start of the string and ends at the end of the line where the end of the line is either the end of the string if there is no terminating newline or just before the final newline if there is a terminating newline. In multi-line mode a new line starts at the start of the string and after each newline and ends before each newline or at the end of the string if the last line has no terminating newline. The confusion is that people think that they are in string-mode if they are not in multi-line mode but they are not, they are in single-line mode, ^ and $ still use the semantics of lines and a terminating newline, if present, is still not part of the content of the line. With \n\n\n in single-line mode the non-greedy ^(\n+?)$ will capture only two of the newlines, the third one will be eaten by the $. If you make it greedy ^(\n+)$ will capture all three newlines. So arguably the implementations that do not match cat\n with cat$ are the broken ones. [1] https://docs.python.org/3/howto/regex.html#more-metacharacters https://docs.python.org/3/howto/regex.html#more-metacharacte... [2] I am using single-line to mean not multi-line for convenience even though single-line already has a different meaning.

kqr 3y ago

I'm the same, but now that I try in Perl, sure enough, $ seems to default to being a positive lookahead assertion for the end of the string. It does not match and consume an EOL character. Only in multiline mode does it match EOL characters, but it does still not appear to consume them. In fact, I cannot construct a regex that captures the last character of one line, then consumes the newline, and then captures the first character of the next line, while using $. The capture group simply ends at $.

singingfish 3y ago
To get the newline captured as well you need to add the `/s` modifier too

absoluteunit1 3y ago

I’ve always thought that as well; mostly due to Vim though. ^ - takes you to start of line $ - takes you to end of line

Izkata 3y ago
^ actually takes you to the first non-whitespace character in the line in vim. For start of line you want 0
- kataklasm 3y ago
  I don't have (n)vi(m) open right now but I think this only applies to prepending spaces. For prepending tabs, 0 will take you to the first non-tab character as well.
  - qu4z-2 3y ago
    Vim takes me to the first character in the line (the first tab), but displays the cursor on the last gridsquare the tab's width covers.
    - Izkata 3y ago
      If you have "set list" to make non-space whitespace visible, it'll go to the leftmost position. I did it long ago along with "set listchars=trail:.,tab:>-" so I can see not only where tabs are, but also their size/alignment without causing the text to shift.

alphazard 3y ago

This must be the "second problem" everyone talks about with regular expressions.

Izkata 3y ago

Same here; when I saw the title I was like "well obviously not, where did you hear that?" In nearly two decades of using regex I think this might be the first time I've heard of $ being end of string. It's always been end of line for me.

deleted 3y ago
[deleted]
frame_ranger 3y ago
You couldn’t write a post like this if you didn’t start with a strawman.
- dbdudbdiddjc 3y ago
  It’s not a straw man, it’s accurate in many contexts.
michaelt 3y ago
Take a look at, for example, these stackoverflow answers about a regex to validate and e-mail address: https://stackoverflow.com/a/8829363 https://stackoverflow.com/a/8829363 These people are I think not intending to say a newline character is permitted at the end of an e-mail address. (Of course people using 'grep' would have different expectations for obvious reasons)
- Izkata 3y ago
  Even disregarding whether or not end-of-string is also an end-of-line or not (see all the other comments below), $ doesn't match the newline, similar to zero-width matches like \b, so the newline wouldn't be included in the matched text either way. I think this series of comments might be clearest: https://news.ycombinator.com/item?id=39764385 https://news.ycombinator.com/item?id=39764385
  - LK5ZJwMwgBbHuVI 3y ago
    Problem is, plenty of software doesn't actually look at the match but rather just validates that there was a match (and then continues to use the input to that match).

notnmeyer 3y ago

i feel like this perspective will be split between folks who use regex in code with strings and more sysadmin folks who are used to consuming lines from files in scripts and at the cli. but yeah seems like a real misunderstanding from “start/end of string” people

cerved 3y ago

In `sed` it's end of string. String is usually end of line, but not if you use stuff like `N`, to manipulate multi-line strings