4 ms·

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages.

False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages. A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go. I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned. If anyone knows of vulnerabilities caused by thus, let me know.

sjrd 2y ago
`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?<!\r)\n|[\r\u0085\u2028\u2029])` otherwise. I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1]. Edit: also quoting from [2]: > By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence. [1] https://github.com/scala-js/scala-js/blob/eb160f1ef1137949994ba119d3a8d1e0754a09a4/javalib/src/main/scala/java/util/regex/PatternCompiler.scala#L1227 https://github.com/scala-js/scala-js/blob/eb160f1ef113794999... [2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...
- sjrd 2y ago
  OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]: > If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters. Looks like I have some code to fix. [1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi%2F%2F/java.base/java/util/regex/Pattern.html https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...
phyzome 2y ago
Interesting that a trailing newline is accepted. Not as bad as what's in the post, at least. Definitely worth breaking out which languages do which of those, though! Python, for instance, only accepts a trailing newline but not additional chars beyond that. I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.
- dwheeler 2y ago
  Which version of Java (JDK) are you using? Which implementation? If that's true, then I fear the answer for Java may vary. The O'Reilly book on Regular Expressions, and the JDK documentation for version 21, say clearly that $ permits an optional \n at the end. The Java 8 documentation is murky, and maybe Java 8 is different.
thaumaturgy 2y ago
$ php -v PHP 8.2.7 (cli) (built: Jun 9 2023 19:37:27) (NTS) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello world"));' int(1) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld"));' int(0) $ php -r 'var_dump("hello\nworld");' string(11) "hello world" ... $ php -v PHP 7.2.26-1+0~20191218.33+debian8~1.gbpb5a34b (cli) (built: Dec 18 2019 16:09:52) ( NTS ) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello world"));' int(1) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld"));' int(0) $ php -r 'var_dump("hello\nworld");' string(11) "hello world" I'm not sure which version of PHP had the behavior you describe, or whether it misbehaves under more specific conditions, but preg_match() is one of the more commonly-used regex functions, all of which share the same engine. The behavior here seems to be "correct" for at least the last 5 years, for varying interpretations of "correct". edit: https://3v4l.org/N4o8D https://3v4l.org/N4o8D suggests that the behavior here is identical for all versions of PHP from 4.3 to 8.3.6.
xarope 2y ago
yes that's correct. I came from perl and python, and got caught out a few times in Go(lang).
jasonlotito 2y ago
So, you are technically correct when you say PHP accepts a trailing newline, but it doesn't mean it refutes the comment and the context we are discussing. This is easily demonstrated with an example. <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n")); int(1) versus <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld")); int(0) versus <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n\n")); int(0) Which all makes sense, as by default PHP doesn't operate in multiline mode. So, by default, PHP is not going to fall prey to the same problem being discussed here. In addition, the first \n would be apart of the first line it's on, so including it as a part of the string would make sense. More to the point, in this context, $ does mean end of the string in PHP. You can prove otherwise by getting the 2nd and 3rd example above to output a 1 instead of a 0 without going into multiline mode.
- dwheeler 2y ago
  I think we're using different definitions for "end of string". In PHP, the following is considered true: > var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n")); That is clear proof that "$" does NOT just match the end of the string; it also accepts an extra newline at the end of the string. In PHP you need to use \z if you want to match the end of the string, or use the "D" flag when using "$". That definition of "$" is often reasonable when you read files a line-at-a-time from a file, which is why Perl changed its definition. However, PHP is often used for server-side web applications. In this case, you are often NOT reading a line-at-a-time from a file. In such cases, allowing an extra newline at the end could be disastrous. The MediaWiki code (written in PHP) deals with this by adding the "D" flag when it uses "$", but I'm not sure it always uses it, and I doubt all PHP programs use this flag when they should.

sjrd 2y ago

`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?<!\r)\n|[\r\u0085\u2028\u2029])` otherwise. I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1]. Edit: also quoting from [2]: > By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence. [1] https://github.com/scala-js/scala-js/blob/eb160f1ef1137949994ba119d3a8d1e0754a09a4/javalib/src/main/scala/java/util/regex/PatternCompiler.scala#L1227 https://github.com/scala-js/scala-js/blob/eb160f1ef113794999... [2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...

sjrd 2y ago
OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]: > If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters. Looks like I have some code to fix. [1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi%2F%2F/java.base/java/util/regex/Pattern.html https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...

phyzome 2y ago

Interesting that a trailing newline is accepted. Not as bad as what's in the post, at least. Definitely worth breaking out which languages do which of those, though! Python, for instance, only accepts a trailing newline but not additional chars beyond that. I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.

dwheeler 2y ago
Which version of Java (JDK) are you using? Which implementation? If that's true, then I fear the answer for Java may vary. The O'Reilly book on Regular Expressions, and the JDK documentation for version 21, say clearly that $ permits an optional \n at the end. The Java 8 documentation is murky, and maybe Java 8 is different.

thaumaturgy 2y ago

$ php -v PHP 8.2.7 (cli) (built: Jun 9 2023 19:37:27) (NTS) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello world"));' int(1) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld"));' int(0) $ php -r 'var_dump("hello\nworld");' string(11) "hello world" ... $ php -v PHP 7.2.26-1+0~20191218.33+debian8~1.gbpb5a34b (cli) (built: Dec 18 2019 16:09:52) ( NTS ) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello world"));' int(1) $ php -r 'var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld"));' int(0) $ php -r 'var_dump("hello\nworld");' string(11) "hello world" I'm not sure which version of PHP had the behavior you describe, or whether it misbehaves under more specific conditions, but preg_match() is one of the more commonly-used regex functions, all of which share the same engine. The behavior here seems to be "correct" for at least the last 5 years, for varying interpretations of "correct". edit: https://3v4l.org/N4o8D https://3v4l.org/N4o8D suggests that the behavior here is identical for all versions of PHP from 4.3 to 8.3.6.

xarope 2y ago

yes that's correct. I came from perl and python, and got caught out a few times in Go(lang).

jasonlotito 2y ago

So, you are technically correct when you say PHP accepts a trailing newline, but it doesn't mean it refutes the comment and the context we are discussing. This is easily demonstrated with an example. <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n")); int(1) versus <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\nworld")); int(0) versus <?php var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n\n")); int(0) Which all makes sense, as by default PHP doesn't operate in multiline mode. So, by default, PHP is not going to fall prey to the same problem being discussed here. In addition, the first \n would be apart of the first line it's on, so including it as a part of the string would make sense. More to the point, in this context, $ does mean end of the string in PHP. You can prove otherwise by getting the 2nd and 3rd example above to output a 1 instead of a 0 without going into multiline mode.

dwheeler 2y ago
I think we're using different definitions for "end of string". In PHP, the following is considered true: > var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n")); That is clear proof that "$" does NOT just match the end of the string; it also accepts an extra newline at the end of the string. In PHP you need to use \z if you want to match the end of the string, or use the "D" flag when using "$". That definition of "$" is often reasonable when you read files a line-at-a-time from a file, which is why Perl changed its definition. However, PHP is often used for server-side web applications. In this case, you are often NOT reading a line-at-a-time from a file. In such cases, allowing an extra newline at the end could be disastrous. The MediaWiki code (written in PHP) deals with this by adding the "D" flag when it uses "$", but I'm not sure it always uses it, and I doubt all PHP programs use this flag when they should.