5 ms·

In my experience `$` does reliably mean end of string for regular expressions, unless you specifically ask for "multiline" mode. Ruby seems to be in multiline

by neilk 2y ago

In my experience `$` does reliably mean end of string for regular expressions, unless you specifically ask for "multiline" mode. Ruby seems to be in multiline mode all the time? $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foobar") else "no"' yes $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar") else "no"' no $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar", re.M) else "no"' yes $ perl -le 'print "foobar" =~ /^[a-z ]+$/ ? "yes" : "no"' yes $ perl -le 'print "foo\nbar" =~ /^[a-z ]+$/ ? "yes" : "no"' no $ perl -le 'print "foo\nbar" =~ /^[a-z ]+$/m ? "yes" : "no"' yes $ node -e 'console.log(/^[a-z ]+$/.test("foobar") ? "yes" : "no")' yes $ node -e 'console.log(/^[a-z ]+$/.test("foo\nbar") ? "yes" : "no")' no $ node -e 'console.log(/^[a-z ]+$/m.test("foo\nbar") ? "yes" : "no")' yes $ ruby -e 'if "foobar" =~ /^[0-9a-z ]+$/i then puts "yes" else puts "no" end' yes $ ruby -e 'if "foo\nbar" =~ /^[0-9a-z ]+$/i then puts "yes" else puts "no" end' yes EDIT: this is documented behavior for Ruby. What other languages call multiline mode is the default; you're supposed to use \A and \Z instead. They do have an `/m` but it only affects the interpretation of `.` https://docs.ruby-lang.org/en/master/Regexp.html#class-Regexp-label-Multiline+Mode https://docs.ruby-lang.org/en/master/Regexp.html#class-Regex...

interroboink 2y ago
Yeah, my takeaway from this was more "the dangers of Ruby" rather than "the dangers of single line regular expressions" (: I think the simplest fix would be to use "\Z" rather than "$", which means "match end of input" rather than "end of line." This is also Perl-compatible. So weird that the "$" default meaning is different in Ruby. I guess one could argue that Ruby's way is better since "$" has a fixed meaning, rather than being context-dependent. > Ruby seems to be in multiline mode all the time? Ruby does have a "/m" for multiline mode, but it just makes "." match newline, rather than changing the meaning of "$", it seems. [1] https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-Anchors https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-An... [2] https://perldoc.perl.org/perlre#Metacharacters https://perldoc.perl.org/perlre#Metacharacters
- neilk 2y ago
  looks like we both updated our answers as we looked up the docs :)
- Borg3 2y ago
  In case of ruby, best would be to actually use result of match for futher computation like this: if !m=/^[a-z0-9 ]+$/match(str) return "Bad Input" end str=m[0]
js2 2y ago
The potential trouble with $ (even in single-line mode) is that it matches the end of a string BOTH with AND without a newline at the end. If you're using it to ensure the string has no newline before doing something with it, this can lead to trouble. $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo\n") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo\n") else "no")' no Even if the newline is not problematic, using \A and \Z makes your intentions clearer to the reader, especially if you add re.X and place comments into the pattern. Asides: 1. Based on syntax, you appear to be testing with python2. 2. With python, re.match is implicitly anchored to the start, so the ^ is redundant. Use re.search or omit the ^.
- medstrom 2y ago
  Correct me if I'm wrong, but if you extract a capture group (^foo$), you would get "foo" without the "\n", right? If so, it is not "matching the end of a string" at all. Just end of line. That's exactly as expected in single-line mode, so it's good. May mismatch your expectations in multi-line mode though.
  - js2 2y ago
    That's right. It all depends on what you're doing with the input string after the match. The point is to be aware of the nuance and to communicate that clearly in the code in cases where it matters.
dwheeler 2y ago
False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulnerabilities caused by thus, let me know.
- sjrd 2y ago
  `$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?<!\r)\n|[\r\u0085\u2028\u2029])` otherwise. I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1]. Edit: also quoting from [2]: > By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence. [1] https://github.com/scala-js/scala-js/blob/eb160f1ef1137949994ba119d3a8d1e0754a09a4/javalib/src/main/scala/java/util/regex/PatternCompiler.scala#L1227 https://github.com/scala-js/scala-js/blob/eb160f1ef113794999... [2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...
  - sjrd 2y ago
    OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]: > If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters. Looks like I have some code to fix. [1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi%2F%2F/java.base/java/util/regex/Pattern.html https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...
brobinson 2y ago
Note that Ruby also has \z which is what you generally want instead of \Z. (\Z allows a trailing newline, \z does not)
- dwheeler 2y ago
  You want \Z in Python, and \z in most other languages, to match on end of string. But in some languages $ really does match end of string. As always, you must check your docs.

by neilk 2y ago

interroboink 2y ago

Yeah, my takeaway from this was more "the dangers of Ruby" rather than "the dangers of single line regular expressions" (: I think the simplest fix would be to use "\Z" rather than "$", which means "match end of input" rather than "end of line." This is also Perl-compatible. So weird that the "$" default meaning is different in Ruby. I guess one could argue that Ruby's way is better since "$" has a fixed meaning, rather than being context-dependent. > Ruby seems to be in multiline mode all the time? Ruby does have a "/m" for multiline mode, but it just makes "." match newline, rather than changing the meaning of "$", it seems. [1] https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-Anchors https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-An... [2] https://perldoc.perl.org/perlre#Metacharacters https://perldoc.perl.org/perlre#Metacharacters

neilk 2y ago
looks like we both updated our answers as we looked up the docs :)
Borg3 2y ago
In case of ruby, best would be to actually use result of match for futher computation like this: if !m=/^[a-z0-9 ]+$/match(str) return "Bad Input" end str=m[0]

js2 2y ago

The potential trouble with $ (even in single-line mode) is that it matches the end of a string BOTH with AND without a newline at the end. If you're using it to ensure the string has no newline before doing something with it, this can lead to trouble. $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo\n") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo") else "no")' yes $ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo\n") else "no")' no Even if the newline is not problematic, using \A and \Z makes your intentions clearer to the reader, especially if you add re.X and place comments into the pattern. Asides: 1. Based on syntax, you appear to be testing with python2. 2. With python, re.match is implicitly anchored to the start, so the ^ is redundant. Use re.search or omit the ^.

medstrom 2y ago
Correct me if I'm wrong, but if you extract a capture group (^foo$), you would get "foo" without the "\n", right? If so, it is not "matching the end of a string" at all. Just end of line. That's exactly as expected in single-line mode, so it's good. May mismatch your expectations in multi-line mode though.
- js2 2y ago
  That's right. It all depends on what you're doing with the input string after the match. The point is to be aware of the nuance and to communicate that clearly in the code in cases where it matters.

dwheeler 2y ago

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulnerabilities caused by thus, let me know.

sjrd 2y ago
`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?<!\r)\n|[\r\u0085\u2028\u2029])` otherwise. I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1]. Edit: also quoting from [2]: > By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence. [1] https://github.com/scala-js/scala-js/blob/eb160f1ef1137949994ba119d3a8d1e0754a09a4/javalib/src/main/scala/java/util/regex/PatternCompiler.scala#L1227 https://github.com/scala-js/scala-js/blob/eb160f1ef113794999... [2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...
- sjrd 2y ago
  OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]: > If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters. Looks like I have some code to fix. [1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi%2F%2F/java.base/java/util/regex/Pattern.html https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...

brobinson 2y ago

Note that Ruby also has \z which is what you generally want instead of \Z. (\Z allows a trailing newline, \z does not)

dwheeler 2y ago
You want \Z in Python, and \z in most other languages, to match on end of string. But in some languages $ really does match end of string. As always, you must check your docs.