18 ms·

Regex character "$" doesn't mean "end-of-string"

437 points by BerislavLopac 3y ago

beardyw 3y ago
Does anyone consider RegEx to be standardised? Moving to a new context is always a relearning exercise in my experience.
- rusk 3y ago
  My understanding is it was standardised for Posix but the variants in popular use have so many variations. I consider sed to be the baseline. If you can do sed you can do anything but it’s seriously limited.
  - susam 3y ago
    POSIX specifies two flavours of regular expressions: basic regular expressions (BRE) and extended regular expressions (ERE). There are subtle differences between the two and ERE supports more features than BRE. For example, what is written as a$bc$\{3\}d in BRE is written as a(bc){3}d in ERE. See https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... for more details. The regular expression engines available in most mainstream languages go well beyond what is specified in POSIX though. An interesting example is named capturing group in Python, e.g., (?P<token>f[o]+).
    - tankenmate 3y ago
      Indeed, and the most common is Perl since it was the source of many of the extensions.
      - rusk 3y ago
        I would hazard that nowadays it’s Java due to its broad permeation of the application space
        account42 3y ago
        If anything it would be ECMAScript (JavaScript dwarfs Java use) or PCRE (the de-facto contiuation of Perl regular expressions written in C but used in many languages).
        rusk 3y ago
        Yes I think you’re right actually. I’m about 10 years off :)
    - jwilk 3y ago
      > what is written as $f..$\1 in BRE is written as (f..)\1 in ERE Oddly, there are no backreferences in POSIX EREs.
      - susam 3y ago
        You are right indeed. Looked at the specification again and indeed there is no back-reference in POSIX ERE. Quoting from <https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/xrat/V4_xbd_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...>: > It was suggested that, in addition to interval expressions, back-references ( '\n' ) should also be added to EREs. This was rejected by the standard developers as likely to decrease consensus. Updated my comment to present a better example that avoids back-references. Thanks!
      - GrumpySloth 3y ago
        That’s because POSIX EREs are actual regular expressions thank god.
    - deleted 3y ago
      [deleted]
  - psd1 3y ago
    No gnu tool can balance brackets, afaics. So you can't do everything in sed. And sed is, by design, useless for matching text that spans lines, so good luck picking out paragraphs with it.
    - rusk 3y ago
      Sorry I meant to write “if you can do it in sed you can do it in anything” thereby implying it is a subset of the more generally available flavours. The issue at hand however is that there isn’t much in the way of standardisation but 95% of sed should work across all of them. Of course you should get more into the specifics of whatever your solution space supports.
    - ykonstant 3y ago
      I am pretty sure even pure Awk can do it; or am I mistaken? I thought there was an even more sophisticated example in the Awk book. Edit: oh, you mean via regex engines available in GNU tools; I am dumb. Hmm... is there no GNU extension with PCRE?
      - colimbarna 3y ago
        "Sed" is the name of a specific tool. It is not defined by the GNU tools, but has existed in some form since 1974, well before Perl. GNU sed and POSIX sed both support BRE and EREs, but not PCREs. Maybe there's some other implementation of sed that supports PCREs but that would really be an extension of that implementation of sed rather than a property of sed. And maybe there's some GNU tool that uses PCREs, but that GNU tool would not be GNU sed, so it would not be a relevant property. Anyway, they probably should have said BREs or EREs rather than "sed"...
      - psd1 3y ago
        Nope. Gnu regex tools also do not support lookahead or alternation. Lookahead is really useful. It's a problem for Linux that it can't move on. The cold dead hand of gnu is firmly around the community's neck.
        3y ago
- telotortium 3y ago
  Languages invented after Perl will generally use some flavor of Perl regex syntax, but there are always some minor differences. The issue of the meaning of `$` and changing it via multi-line mode is usually consistent though.
  - usrusr 3y ago
    I like to think of "whatever browsers do in js" as an updated common baseline. Whatever your regex engine does, describe it as a delta to the js precedent. That thing is just so ubiquitous. I do wonder though what's the highest number of different regex syntaxes I've ever encountered (perhaps written?) within a single line: bash, grep and sed are never not in a "hold my beer" mood!
    - psd1 3y ago
      Reason #2 to use powershell - consistent regex. I've got "hold my beer" commits in .net - I've balanced brackets. I believe that's impossible in sed and grep. If I were going to write a json parser in a script, then a) stop me and b) it's got to be in powershell.
    - layer8 3y ago
      That seems like just a web front-end developer’s perspective.
    - Calzifer 3y ago
      Isn't JavaScripts regex one of the worst modern regex implementations? They seem to improve. Negative lookbehind isn't missing anymore [1]. But still lack the handy \Q and \E to escape stuff [2]. [1] https://stackoverflow.com/a/3950684 https://stackoverflow.com/a/3950684 [2] https://stackoverflow.com/q/6318710 https://stackoverflow.com/q/6318710
    - kstrauser 3y ago
      I’ll go along with that, as long as someone ports pcre to JavaScript and that’s the browser syntax we land on.
    - 3y ago
- wolletd 3y ago
  At some point, I felt like I knew them all. There are probably more regex dialects out there, but I don't encounter them and my set of knowledge works most of the time. I feel it's like driving a rental car. It behaves slightly different than your own car, some features missing, some other features added, but in general, most of the things are pretty similar.
  - stanislavb 3y ago
    What a nice analogy. I’ll borrow it in the future.
- MattHeard 3y ago
  My working assumption has always been to check the docs of your specific regexp parser, and to write some tests (either automated or manually in a REPL) with specific patterns that you are interested in using.
- out-of-ideas 3y ago
  kind of a trick question; there is POSIX and then there is the app you're using and whichever flags are enabled (albeit by default or explicitly defined)
- jasonjayr 3y ago
  The three big ones I know of are POSIX, Perl/PCRE(aka Perl-Compatible Regular Expression), and Go came along and <strike>added</strike> used re2, which is a bit different from the first too. A lot of systems implemented PCRE, including JavaScript, since Perl extended the POSIX system with many useful extensions. IIRC, re2 tries to reign in on some of the performance issues and quirks the original systems had, while implementing the whole thing in Go. edit: Did not realize re2 predated go ...
  - jpgvm 3y ago
    re2 predates Go and was written in C++.
  - foldr 3y ago
    Go's regex implementation is new in the sense that it's not just a binding to the re2 C++ library, but it uses the same non-backtracking algorithm.
  - jerf 3y ago
    POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not. re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having constrained complexity of scanning a string. This does not come for free; you can conceivably end up with a compiled regexp of very large size with an NDFA approach, but most of the time you won't. The result may be generally slower than a PCRE-type approach, but it can also end up safer because you can be confident that there isn't a pathological input string for a given regexp that will go exponential. This is one of those cases where ~99% of the time, it doesn't really matter which you choose, but at the scale of the Entire Programming World, both options need to be available. I've got some security applications where I legitimately prefer the re2 implementation in Go because it is advantageous to be confident that the REs I write have no pathological cases in the arbitrary input they face. PCRE can be necessary in certain high-performance cases, as long as you can be sure you're not going to get that pathological input. RE engines don't quite engender the same emotions as programming languages as a whole, but this is not cheerleading, this is a sober engineering assessment. I use both styles in my code. I've even got one unlucky exe I've been working with lately that has both, because it rather irreducibly has the requirements for both. Professionally annoying, but not actually a problem.
- bregma 3y ago
  The ISO/IEC 14882 C++ standard library <regex> mandates [0] implementations for six de jure standard regex grammars: IEEE Std 1003.1-2008 (POSIX) [1] BRE, ERE, awk, grep, and egrep and ECMA-262 EcmaScript 3 [2]. So, yes, at least someone (me) considers regex to be standardized in several published de jure standards. [0] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3690.pdf#chapter.28 [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html [2] https://262.ecma-international.org/14.0/#sec-regexp-regular-expression-objects
  - pjc50 3y ago
    "At least six different standards" is an XKCD comic, not a standard.
    - riffraff 3y ago
      "The nice thing about standards is that you have so many to choose from." - Andrew Tanenbaum (or Grace Hopper)
  - account42 3y ago
    <regex> is not exactly an example anyone should follow.
    - bregma 3y ago
      You may be prejudiced against C++, but ISO/IEC 14882 is a published international standard that links to recognized regex standards, so answers the question "does anyone consider RegEx standardised?" very much in the affirmative.
- beardyw 3y ago
  And don't get me started about find and replace, what is the symbol to insert the match?
- tonyg 3y ago
  Delightfully, RFC 9485 https://datatracker.ietf.org/doc/rfc9485/ https://datatracker.ietf.org/doc/rfc9485/ "I-Regexp: An Interoperable Regular Expression Format" was published just back in October last year!
ikiris 3y ago
this is mostly due to the different types of regex and less about it being platform dependent. $ was end of string in pcre which is the "old" perl compatible regex. python has its own which has quirks as mentioned, re2 is another option in go for example, and i think rust has its own version as well iirc.
- pjmlp 3y ago
  Indeed, there isn't any kind of universal regexp standard.
  - 7bit 3y ago
    We should create a new RegEx flavour that standardises RegEx for good!
    - jasonjayr 3y ago
      https://xkcd.com/927/ https://xkcd.com/927/
      - rerdavies 3y ago
        https://datatracker.ietf.org/doc/rfc9485/ https://datatracker.ietf.org/doc/rfc9485/ https://xkcd.com/927/ https://xkcd.com/927/
- wolletd 3y ago
  The differences of the various regex "dialects" came to me over the years of using regular expressions for all kinds of stuff. Matching EOL feels natural for every line-based process. What I find way more annoying is escaping characters and writing character groups. Why can't all regex engines support '\d' and '\w' and such? Why, in sed, is an unescaped '.' a regex-dot matching any character, but an unescaped '(' is just a regular bracket?
  - somat 3y ago
    > Why, in sed, is an unescaped '.' a regex-dot matching any character, but an unescaped '(' is just a regular bracket? It is because sed predates the very influential second generation Extended Regular Expression engine and by default uses the first generation Basic Regular Expression engine. So really it is for backwards compatibility. http://man.openbsd.org/re_format#BASIC_REGULAR_EXPRESSIONS http://man.openbsd.org/re_format#BASIC_REGULAR_EXPRESSIONS you can usually pass sed a -r flag to get it to use ERE's Actually I don't really know if BRE's predate ERE's or not. I assume they do based on the name but I might be wrong.
Izmaki 3y ago
The new-line character is an actual character "at the end" of the string though so it makes sense that $ would include the new-line character in multi-line matching.
- IshKebab 3y ago
  Yes and every implementation gets that right. The point was when multi-line matching is disabled and only Javascript, Go and Rust get that right. I'm not too surprised by PHP and Python getting it wrong. Java and C# is a slight surprise though.
  - danbruc 3y ago
    I don't think it is correct to say some get it right and some get it wrong, it is more of an design decision.
    - IshKebab 3y ago
      It's possible to get design decisions wrong. Clearly people expect `$` to only match end-of-string so they did make the wrong decision. It may not have been clear it was the wrong decision at the time.
      - danbruc 3y ago
        Things are obviously more complicated than that, lines are a complicated issue for historical reasons. There are two conventions, line termination and line separation. In case of line termination, the newline is part of the line and a string without a newline is not a [complete] line. In case of line separation, the newline is not part of the line but separates two lines. Also the way newlines are encoded is not universal.
        fauigerzigerk 3y ago
        Why is this relevant when multi-line is disabled?
user2342 3y ago
I'm confused by this blog-post. In the table: what is the reg-ex pattern tested and against which input?
- mnw21cam 3y ago
  The input being matched is "cat\n" and the regex pattern is one of: "cat$" with multiline enabled "cat$" with multiline disabled "cat\z" "cat\Z"
somat 3y ago
Structural regexes as found in the sam editor are an obscure but well engineered regex engine. I am far from an expert but my main takeaway from them is that most regex engines have an implied structure built around "lines" of text. While you can work around this, it is awkward. Structural regexes allow you to explicitly define the structure of a match, that is, you get to tell the engine what a "line" is. http://man.cat-v.org/plan_9/1/sam http://man.cat-v.org/plan_9/1/sam
xlii 3y ago
Regexp was one of the first things I truly internalized years ago when I was discovering Perl (which still lives in a cozy place in my heart due to a lovely “Camel” book). Today most important bit of information is knowledge that implementations differ and I made a habit of pulling reference sheet for a thing I work with. E.g. Emacs Regexp annoyingly doesn’t have word in form of “\w” but uses “\s_-“ (or something no reference sheet on screen) as character class (but Emacs has the best documentation and discoverability - a hill I’m willing to die on) Some utilities require parenthesis escaping and some not. Sometimes this behavior is configurable and sometimes it’s not. I lived through whole confusion, annoyance, denial phase and now I just accept it. Concept is the same everywhere but flavor changes.
- ydant 3y ago
  Exactly the same here, re: Perl. My brain thinks in Perl's regex language and then I have to translate the inconsistent bits to the language I'm using. Especially in the shell - I'm way more likely to just drop a perl into the pipeline instead of trying to remember how sed/grep/awk (GNU or BSD?) prefer their regex.
  - influx 3y ago
    GNU grep supports Perl regexp with -P
    - mwpmaybe 3y ago
      As does git grep!
    - 1letterunixname 3y ago
      Using PCRE2, which doesn't behave exactly the same as Perl or PCRE1. https://pcre.org/current/doc/html/pcre2compat.html https://pcre.org/current/doc/html/pcre2compat.html https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expres... https://stackoverflow.com/questions/70273084/regex-differences-between-pcre-and-pcre2 https://stackoverflow.com/questions/70273084/regex-differenc...
masswerk 3y ago
As for the good old reference implementation (not "Parameter Efficient Reinforcement Learning"): my $string = "cat\n"; /cat$/s -> true /cat\Z/s -> true /cat\z/s -> false
ghusbands 3y ago
> Note: The table of data was gathered from regex101.com, I didn't test using the actual runtimes. Has anyone confirmed this behaviour directly against the runtimes/languages? Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes.
- AtNightWeCode 3y ago
  I fail to add carriage return to the test string on that site. Which I guess would be an issue on Windows.
- zimpenfish 3y ago
  https://go.dev/play/p/Tce1qWjfjOy https://go.dev/play/p/Tce1qWjfjOy matches their results. I've also run that locally against "go1.22.1 darwin/arm64", "go1.21.5 windows/amd64", and "go1.21.0 linux/amd64" with the same result.
- coldtea 3y ago
  >Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes. In what way could newlines at the end of a string "could get lost in transit"?
  - ghusbands 3y ago
    If you write it to a text file by itself and then read it from that text file, each runtime can have a different definition of whether a newline at the end of the file is meaningful or not. Under POSIX, a newline should always be present at the end of a non-empty text file and is not meaningful; not everyone agrees or is aware. There are plenty of other ways, too; bugs happen.
    - coldtea 3y ago
      Ideally no runtime should alter strings passing through ("in transit") from one runtime to another - unless it does some processing on them.
pjc50 3y ago
Special misery case: Visual Studio supports regex search, where '$' matches \n. The end of line character is usually the standard Windows \r\n. Yes, that means if you want to really match the end of line you have to match "\r$". So broken.
- skrebbel 3y ago
  FWIW, and I know this doesn't really address your complaint: I use Windows and I've set all my text editors to use LF exclusively years ago and Things Are Great. No more weird Git autocrlf warnings, no quirks when copying files over to/from people on Macs or Linuxes, etc. Even Notepad supports LF line endings for quite a long time now - to my practical experience, there's little remaining in Windows that makes CRLF "the OS standard line ending". I bet if someday VS Code's Windows build ships with LF default on new installations, people won't even notice. I mean, at some point it did matter what the OS did when you pressed the "Enter" button. But this isn't really the case much anymore. VS Code catches that keypress, and inserts whatever "files.eol" is set to. Sublime does the same. I didn't check, but I assume every other IDE has this setting. Similarly, the HTML spec, which is pretty nuts, makes browsers normalize my enters to LF characters as I type into this textarea here (I can check by reading the `value` property in devtools), but when it's submitted, it converts every LF to a CRLF because that's how HTML forms were once specced back in the day. Again though, what my OS considers to be "the standard newline" is simply not considered at all. Even CMD.EXE batch files support LF. I don't really type newlines all that much outside IDEs and browsers (incl electron apps) and places like MS Word, all of which disregard what the OS does and insert their own thing. Maybe the terminal? I don't even know. I doubt it's very consequential. EDIT: PSA the same holds for backslashes! Do Not Use Backslashes. Don't use "OS specific directory separator constants". It's not 1998, just type "/" - it just works.
  - n_plus_1_acc 3y ago
    I could never get visual studio (not code) to not use \r\n when editing a solution file via the gui
  - divingdragon 3y ago
onion2k 3y ago
I can hear thousands of bad hiring manager's adding 'How do you match the end of a string in a regex?' to their list of 'Ha! You don't know the trick!' questions designed to catch out candidates.
- hoc 3y ago
  "I will hire you anyway, but I will pay you less" Regex, useful in any job...
  - username_my1 3y ago
    regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.
    - ykonstant 3y ago
      Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.
      - thecatspaw 3y ago
        what does gpt say how we should validate email addresses?
        rhd 3y ago
        chatgpt-4: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566e53705 https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566... chatgpt-3.5: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436626cc4 https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436...
Karellen 3y ago
> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"?
- jamesmunns 3y ago
  Same, tho it'd be interesting to see if this behavior holds if the file ends without a trailing newline and your match is on the final newline-less line.
  - fooofw 3y ago
    Fortunately, it's pretty simple to test. $ printf 'Line with EOL\nLine without EOL' | grep 'EOL$' Line with EOL Line without EOL $ grep --version | head -n1 grep (GNU grep) 3.8
    - romwell 3y ago
      The line does end with the file, so it's logically consistent. It's not matching the newline character after all.
      - colimbarna 3y ago
        Yes exactly, they match the end of a line, not a newline character. Some examples from documentation: man 7 regex: '$' (matching the null string at the end of a line) pcre2pattern: The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. ... The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.
hans_castorp 3y ago
Fun fact: in Postgres, 'cat\n' matches 'cat$' when the so called "weird" newline matching is enabled :) https://www.postgresql.org/docs/current/functions-matching.html#POSIX-EMBEDDED-OPTIONS-TABLE https://www.postgresql.org/docs/current/functions-matching.h...
febeling 3y ago
Seriously, just write one unit test for your regex.
- mannykannot 3y ago
  Indeed, one should test any regex one puts any trust in, but the problem is that if you take as a fact something that is actually a false assumption (as the author did here), your test may well fail to find errors which may cause faults when the regex is put to use. This, in a nutshell, is the sort of problem which renders fallacious the notion that you can unit-test your way to correct software.
PuffinBlue 3y ago
This seems like the perfect opportunity to introduce those unfamiliar to Robert Elder. He makes cool YouTube[0] and blog content[1] and has a series on regular expressions[2] and does some quite deep dives into the differing behaviour of the different tools that implement the various versions. His latest on the topic is cool too: https://www.youtube.com/watch?v=ys7yUyyQA-Y https://www.youtube.com/watch?v=ys7yUyyQA-Y He's has quite a lot of content that HN folks might be interested in I think, like the reality and woes of consulting[3] [0] https://www.youtube.com/@RobertElderSoftware https://www.youtube.com/@RobertElderSoftware [1] https://blog.robertelder.org/ https://blog.robertelder.org/ [2] https://blog.robertelder.org/regular-expressions/ https://blog.robertelder.org/regular-expressions/ [3] https://www.youtube.com/watch?v=cK87ktENPrI https://www.youtube.com/watch?v=cK87ktENPrI
- aquariusDue 3y ago
  I'm glad to see someone else that has stumbled over his content. Seconding the recommendation.
- CatchSwitch 3y ago
  He has so many favorite Linux commands lol
  - robertelder 3y ago
    This is my favourite comment.
teknopaul 3y ago
Tldr; $ does not mean end of string in Python.
AtNightWeCode 3y ago
There are many differences between implementations of regex. To name a few. Lookbehind, atomic groups, named capturing groups, recursion, timeouts and my favorite interop problem, unicode.
wruza 3y ago
By default, '$' only matches at the end of the string and immediately before the newline (if any) at the end of the string. The rationale was probably "it should be easier to match input strings" and now it's harder for everyone.
frou_dh 3y ago
Something I found really surprising about Python's regexp implementation is that it doesn't support the typical character classes like [:alnum:] etc. It must be some kind of philosophical objection because there's no way something with as much water under the bridge as Python simply hasn't got around to it.
k3vinw 3y ago
Another poor soul trying to solve one problem using regex and now they have two… ;)
croes 3y ago
Isn't a string with a newline character automatically multiline? The new line is just empty but not the first line anymore.
- Joker_vD 3y ago
  No, it is not. 3.195 Incomplete Line A sequence of one or more non-<newline> characters at the end of the file. 3.206 Line A sequence of zero or more non-<newline> characters plus a terminating <newline> character. courtesy of [0]. See also [1] for rationale on "text file": Text File [...] The definition of "text file" has caused controversy. The only difference between text and binary files is that text files have lines of less than {LINE_MAX} bytes, with no NUL characters, each terminated by a <newline>. The definition allows a file with a single <newline>, or a totally empty file, to be called a text file. If a file ends with an incomplete line it is not strictly a text file by this definition. [...] [0] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... [1] https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd...
  - rerdavies 3y ago
    Applications that write "incomplete lines" - VSCODE - Visual Studio - Notepad C/C++ compilers that process "incomplete lines" All of them. Will my users consider it a bug if my program doesn't process "incomplete lines": Yes. (Yes, I have had this bug logged against code that I wrote). Will I consider it a bug if your program doesn't process "incomplete lines": Yes.
    - Joker_vD 3y ago
      > Applications that write "incomplete lines" Note that all of those three apps come from the Windows world where CRLF was indeed a line separator, and a file ending with CRLF was considered to have an empty line at the end. But yes, you absolutely should accommodate for incomplete lines.
  - 3y ago
perlgeek 3y ago
Raku (formerly Perl 6) has picked ^ and $ for start-of-string and end-of-string, and has introduced ^^ and $$ for start-of-line and end-of-line. No multi line mode is available or necessary. (There's also \h for horizontal and \v for vertical whitespace) That's one of the benefits of a complete rethink/rewrite, you can learn from the fact that the old behavior surprised people.
- Terretta 3y ago
  And this is why this curmudgeon can't use Perl 6[^1]. It randomly shuffles the line noise we learned over decades. It seems so obvious that's the opposite of what they should have defaulted to, that it clearly should have been ^ and $ for lines, and ^^ and $$ for the string, since like ((1)(2)(3)): ^^line1$\n^line2$\n^line3$\n$ [1]: That, and it's not anywhere, while Perl 5 is everywhere.
- richardwhiuk 3y ago
  Think I would have picked exactly the reverse (i.e. ^^ being more "starty" than "^").
  - lcnPylGDnU4H9OF 3y ago
    Reminds me of verbosity flags in some cli utilities. Often, -v is "verbose" and -vv is "very verbose" and -vvv... etc.
- librasteve 3y ago
  Pretty much all the regexen I have written have been predicated on start / end of string (i typically feed lines through the regex) … so picking single ^ and $ for the whole string maintains a degree of backward compatibility (assuming that I am normal)
junshengz99 3y ago
[dead]
wodenokoto 3y ago
> So if you're trying to match a string without a newline at the end, you can't only use $ in Python! My expectation was having multiline mode disabled wouldn't have had this newline-matching behavior, but that isn't the case. A reproducible example would be nice. I don’t understand what it is he cannot do. `re.search('$', 'no new lines')` returns a match.
- iainmerrick 3y ago
  This unexpectedly matches: re.match('^bob$', 'bob\n') I didn't want the trailing newline to be included.
  - wodenokoto 3y ago
    But that string does have a new line at the end.
    - iainmerrick 3y ago
      re.match('^bob$', 'bob') → yes re.match('^bob$', 'bobs') → no Most people would expect 'bob\n' not to match, because I used '$' and it has an extra character at the end, just like 'bobs'. In Python it does match because '\n' is a special case.
      - rerdavies 3y ago
        ... for some arbitrary definition of "most people".
        iainmerrick 3y ago
        That’s true, probably fewer than ten million people in the world would have an opinion one way or the other. :)
danbruc 3y ago
People are confused about strings and lines. A string is a sequence of characters, a line can be two different things. If you consider the newline a line terminator, then a line is a sequence of non-newline characters - possibly zero - plus a newline. If there is no new-line at the end, then it is not a [complete] line. That is what POSIX uses. If you consider the newline a line separator, then a line is a sequence of non-newline characters - possibly zero. In either case, the content of the line ends before the newline, either because the newline terminates the line or because it separates the line from the next. [1] The semantics of ^ and $ is based on lines - whether single-line or multi-line mode. For string based semantics - which you could also think of as entire file if you are dealing with files - use \A and \Z or their equivalents. [1] Both interpretations have their merits. If you transmit text over a serial connection, it is useful to have a newline as line terminator so that you know when you received a complete line. If you put text into text files, it might arguably be easier to look at a newline as a line separator because then you can not have a invalid last line. On the other hand having line terminators in text files allows you to detect incompletely written lines.
Existing4190 3y ago
perlre Metacharacters documentation states: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used) (/m enables multiline mode)
deleted 3y ago
[deleted]
mdavid626 3y ago
Is this a bug?
humanlity 3y ago
Interesting
m0rissette 3y ago
Why isn’t Perl anywhere on that chart when mentioning regex?
- burntsushi 3y ago
  Because they're using regex101 to easily test the semantics of different regex engines and Perl isn't available on regex101. PCRE is though, which is a decent approximation. And indeed, Perl and PCRE behave the same for this particular case.
  - account42 3y ago
    Why isn’t Perl available on regex101 when its all about regex?
    - burntsushi 3y ago
      I dunno. Maybe because nobody has contributed it? Maybe because Perl isn't as widely used as it once was? Maybe because it's hard to compile Perl to WASM? Maybe some other reason?
tyingq 3y ago
Seems odd to leave Perl off the list, given it's regex related. Here's the explanation for $ in the perlre docs: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used)
- toyg 3y ago
  Yeah, omitting what is arguably the language most associated with regexes seems a bit of an oversight. I guess it shows how far off the radar Perl currently is.
  - demondemidi 3y ago
    Perl perfected the simplicity and flexibility of regex syntax from POSIX and it seems every other language after has just made it harder.
  - TillE 3y ago
    PHP uses PCRE, so it more or less serves as a stand-in for Perl in this case.
    - gordonfish 3y ago
      Sort of, though PCRE is not a 100% replacement for regex in Perl proper; the former lacks some features of the latter.
  - gordonfish 3y ago
    > I guess it shows how far off the radar Perl currently is. This is a serious misconception. Perl is far, far from dead. The constant activity of the gargantuan CPAN library more than demonstrates very much the opposite. I would say Perl and its community has done quite well considering it hasn't had the same mountain of corporate funds thrust into it like more highlighted have. Mainstream ain't everything.
    - saghm 3y ago
homakov 3y ago
This led to a few serious bugs in Ruby-based apps. Always use \A\z https://homakov.blogspot.com/2012/05/saferweb-injects-in-various-ruby.html https://homakov.blogspot.com/2012/05/saferweb-injects-in-var... https://sakurity.com/blog/2015/02/28/openuri.html https://sakurity.com/blog/2015/02/28/openuri.html https://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html https://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html
nunez 3y ago
You can also use (?m) to enable multiline processing on PCRE-compatible regexp engines.
Scubabear68 3y ago
In 30 years of developing software I don’t think I ever used multi-line regexp even once.
- thrdbndndn 3y ago
  Definitely not common, but if you are parsing a text file you're going to use it a lot (say, you're writing a JS parser).
  - marcosdumay 3y ago
    You really shouldn't use a lot of regexes for parsing code. They go only on the tokenizer, if they go somewhere at all.
    - thrdbndndn 3y ago
      Agreed, it's more about quick and dirty ad hoc capture than full-fledged parser though (like when you want to extract certain object when scraping).
- Terretta 3y ago
  > In 30 years of developing software I don’t think I ever used multi-line regexp even once. As long as sharing anecdata, in 30 years, it's almost the only way I use it. It's incredible for slicing and dicing repetitious text into structure. You generally want some sort of Practical Extraction and Reporting Language, the core of which is something like a regular expression, generally able to handle the, well, irregularity. Most recent example (I did this last week) was extracting Apple's app store purchases from an OCR of the purchase history available through Apple's Music app's Account page that lets you see all purchases across all digital offerings, but only as a long scrolling dialog box (reading that dialog's contents through accessibility hooks only retrieves the first few pages, unfortunately). Each purchase contains one or more items and each item has one or more vertical lines, and if logos contain text they add arbitrary lines per logo. A good match and sub match multi-line regex folds that mess back into a CSV. In this case, the regex for this was less than an 80 char line of code and worked in the find replace of Sublime Text which has multiline matching, subgroups, and back references. Another way to do this is something like a state match/case machine, but why write a program when you can just write a regular expression?
nebulous1 3y ago
The fact that there are so many different peculiarities in different regex systems has always raised the hairs on the back of my neck. As in when a tool accepts a regex and I have to a trawl the manual to find out exactly what regex is acceptable to it.
silent_cal 3y ago
I think there's a big opportunity to re-write Regex as a SQL-type language. It's too bad I don't feel like trying.
- macintux 3y ago
  I am 99% confident I’ve seen a post on HN about someone creating a much more verbose regex replacement. I have no idea how to find it. Here’s an interesting project for typed regular expressions: https://news.ycombinator.com/item?id=12292389 https://news.ycombinator.com/item?id=12292389
raldi 3y ago
Cmd-F perl no matches
deleted 3y ago
[deleted]
weinzierl 3y ago
The table in the article makes this look complicated, but it really isn't. All the cases in the article can be grouped into two families: - The JS/Go/Rust family, which treats $ like \z and does not support \Z at all - The Java, .NET, PHP, Python family, which treats $ like \Z and may or may not (Python) support \z. \Z does away with \n before the end of the string, while \z treats \n as a regular character. For multiline $ the distinction doesn't matter, because \n is the end. Really the only deviation from the rule is Python's \Z, which is indeed weird.
gorjusborg 3y ago
If you really want to learn regex, you'll have a hard time piecing it all together via blog posts. Brad Freidl's Mastering Regular Expressions is a good book to read if you want to stop being surprised/lost. I'll admit I stopped at the dive into DFA/NFA engine details.
- macintux 3y ago
  Writing a string -> NFA -> DFA grep-like tool is one of my most memorable college projects. Had a lot of fun with that, and decades later I ended up reusing some of the concepts for a work project.
T3RMINATED 3y ago
[dead]
jewel 3y ago
This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.
- jfhufl 3y ago
  Unsure what you mean? $ ruby -e 'x = "25" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "a25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' no Also, you'd want to use something that parameterizes the query with '?' (I use the Sequel gem) instead of just stuffing it into a sql string.
  - jfhufl 3y ago
    Well, learned something today after reading a bit further in the thread: ruby -e 'x = "a\n25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes Good to know.
  - halostatue 3y ago
    You need to make your regex multi-line (`/^\d+$/m`), but that isn't the problem shown. Your query will be searching for `25\n`, not `25` despite your pre-check that it’s a good value. The second line should always be no, which if you use `\A\d+\z`, it will be.
    - jfhufl 3y ago
      Yep, makes sense, thanks!
  - dr-smooth 3y ago
    $ ruby -e 'x = "25\n; delete from people" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes
- 3y ago
SAI_Peregrinus 3y ago
POSIX regexes and Python regexes are different. In general, you need to reference the regex documentation for your implementation, since the syntax is not universal. Per POSIX chapter 9[1]: 9.2 … "The use of regular expressions is generally associated with text processing. REs (BREs and EREs) operate on text strings; that is, zero or more characters followed by an end-of-string delimiter (typically NUL). Some utilities employing regular expressions limit the processing to lines; that is, zero or more characters followed by a <newline>." and 9.3.8 … "A <dollar-sign> ( '$' ) shall be an anchor when used as the last character of an entire BRE. The implementation may treat a <dollar-sign> as an anchor when used as the last character of a subexpression. The <dollar-sign> shall anchor the expression (or optionally subexpression) to the end of the string being matched; the <dollar-sign> can be said to match the end-of-string following the last character." combine to mean that $ may match the end of string OR the end of the line, and it's up to the utility (or mode) to define which. Most of the common utilities (grep, sed, awk, Python, etc) treat it as end of line by default, since they operate on lines by default. THERE IS NO SINGLE UNIVERSAL REGULAR EXPRESSION SYNTAX. You cannot reliably read or write regular expressions without knowing which language & options are being used. [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
lakomen 3y ago
[flagged]
javier_e06 3y ago
I would hold a code review hostage if any file does not end with an empty new line. My reasoning would be if the file is transmitted and gets truncated nobody would know for sure if it does not end a new line. Brownie points if this is code end has a comment that the files ends there. The article calls computer languages platforms but the are computer languages. Bash is not included. Weird. I believe the most common use of regular expressions is the use of grep or egrep with bash or some other shell but, who knows. Maybe I am hanging with the wrong crowd.
vitiral 3y ago
In Lua it's only the start/end of the string > A pattern is a sequence of pattern items. A caret '^' at the beginning of a pattern anchors the match at the beginning of the subject string. A '$' at the end of a pattern anchors the match at the end of the subject string. At other positions, '^' and '$' have no special meaning and represent themselves. https://www.lua.org/manual/5.3/manual.html#6.4.1 https://www.lua.org/manual/5.3/manual.html#6.4.1 Lua's pattern matching is much simpler than regexes though. > Unlike several other scripting languages, Lua does not use POSIX regular expressions (regexp) for pattern matching. The main reason for this is size: A typical implementation of POSIX regexp takes more than 4,000 lines of code. This is bigger than all Lua standard libraries together. In comparison, the implementation of pattern matching in Lua has less than 500 lines. https://www.lua.org/pil/20.1.html https://www.lua.org/pil/20.1.html
- denzquix 3y ago
  > In Lua it's only the start/end of the string There's an additional caveat: if you use the optional "init" parameter to specify an offset into the string to start matching, the ^ anchor will match at that offset, which may or may not be what you expect.
  - vitiral 3y ago
    That is a good point, and something I've actually (personally) used quite a bit when writing parsers
  - Dylan16807 3y ago
    > which may or may not be what you expect I find it hard to imagine any other expectation passing the rubber duck test. "Oh, so you expected the match to always fail, no matter what the string was?"
    - denzquix 3y ago
      Well, it's not a completely outlandish scenario that the value of `init` might come from a variable that is sometimes at the start of the string and sometimes not, and a newcomer might expect `^` to only match when it is. Don't get me wrong, it's certainly far more useful as it is, I'm glad it works this way.
cpeterso 3y ago
$ is the regex’s “the buck stops here” symbol. Here at the end of the line. :)
nurtbo 3y ago
Totally get the desire, but also feels like last two paragraphs are solvable with ``` re.match(text).extract().rstrip(“\n”) ```
deleted 3y ago
[deleted]
menacingly 3y ago
Of course it’s line. How could it be the end of the string when the matter at hand is defining the string?
pksebben 3y ago
Regex would really benefit from a comprehensive industrial standard. It's such a powerful tool that you have to keep relearning whenever you switch contexts.
aftbit 3y ago
Wait, in non-multiline mode, it only matches _one_ trailing newline? And not any other whitespace, including \r or \r\n? That is indeed surprising behavior. Why? Why not just make it end of string like the author expected? >>> import re >>> bool(re.search('abc$', 'abc')) True >>> bool(re.search('abc$', 'abc\n')) True >>> bool(re.search('abc$', 'abc\n\n')) False >>> bool(re.search('abc$', 'abc ')) False >>> bool(re.search('abc$', 'abc\t')) False >>> bool(re.search('abc$', 'abc\r')) False >>> bool(re.search('abc$', 'abc\r\n')) False
mmh0000 3y ago
> So if you're trying to match a string without a newline at the end, you can't only use $ in Python! My expectation was having multiline mode disabled wouldn't have had this newline-matching behavior, but that isn't the case. I would argue this is correct behavior, a "line" isn't a "line" if it doesn't end with \n.[1] > 3.206 Line - A sequence of zero or more non- <newline> characters plus a terminating <newline> character. [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
- remram 3y ago
  Why should it be a "line" though? The whole point is that people pass in a string, they don't care about lines.
shenene 3y ago
[flagged]
librasteve 3y ago
I am surprised that the OP does not include perl5 in their table. In raku (aka perl6) Regexes were reinvented by Larry Wall (the creator of perl which made perlRE the de facto regex standard) Here's what he does with $: (https://docs.raku.org/language/regexes#Start_of_string_and_end_of_string https://docs.raku.org/language/regexes#Start_of_string_and_e...) * The $ anchor only matches at the end of the string * The $$ anchor matches at the end of a logical line. That is, before a newline character, or at the end of the string when the last character is not a newline character.
ary 3y ago
Was any regex documentation unclear on this? Some libraries have modes that change the semantics of ^ and $ but I’ve always found their use to be rather clear. It’s the grouping and look ahead/behind modifiers that I’ve always found hard to understand (at times).
- macintux 3y ago
  This is a feature that seems so painfully obvious in the abstract that I’d wager most have never read the documentation. I’ve been a regex user since the early 90s and I’ve never thought about this.
pmarreck 3y ago
The results did not surprise me. The fact that everyone is in agreement that "cat$" matches "cat" and not "cat\n" if multiline is off did not surprise me. \n is implicitly a multiline-contextual character to me. In other words, if you didn't have any \n, you'd just have an array of lines (without linefeeds), same as if you were reading lines from a file one at a time or splitting a binary on \n. The other results that differ across engines seem to be because people either don't understand regex or because the POSIX description of how to deal with such an input and config was ill-defined.
1letterunixname 3y ago
Ugh. Whenever I hear people talk about regular expressions as a singular language or standard, I die a little inside. PSA: Regex security is particular to each implementation flavor. Please know the nuances of a particular kind and be unambiguously precise.
callwhendone 3y ago
it's end of line right?
smlacy 3y ago
It's easy to get the canonical answer: $ man pcre2syntax Where you'll find the following block under ANCHORS AND SIMPLE ASSERTIONS: $ end of subject also before newline at end of subject also before internal newline in multiline mode So all the cases of "newline at/before end of subject" are covered here. Then, the question becomes "what is a subject?" Is it line-by-line? Are newlines included? What if we want multiline matching? That's where re.MULTILINE comes from, it's not "multiline matching" (sort of) it's "what is the subject of the regular expression that we're matching against"
michaelcampbell 3y ago
...IN PYTHON

437 points by BerislavLopac 3y ago

Does anyone consider RegEx to be standardised? Moving to a new context is always a relearning exercise in my experience.

rusk 3y ago
My understanding is it was standardised for Posix but the variants in popular use have so many variations. I consider sed to be the baseline. If you can do sed you can do anything but it’s seriously limited.
- susam 3y ago
  POSIX specifies two flavours of regular expressions: basic regular expressions (BRE) and extended regular expressions (ERE). There are subtle differences between the two and ERE supports more features than BRE. For example, what is written as a$bc$\{3\}d in BRE is written as a(bc){3}d in ERE. See https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... for more details. The regular expression engines available in most mainstream languages go well beyond what is specified in POSIX though. An interesting example is named capturing group in Python, e.g., (?P<token>f[o]+).
  - tankenmate 3y ago
    Indeed, and the most common is Perl since it was the source of many of the extensions.
    - rusk 3y ago
      I would hazard that nowadays it’s Java due to its broad permeation of the application space
      - account42 3y ago
        If anything it would be ECMAScript (JavaScript dwarfs Java use) or PCRE (the de-facto contiuation of Perl regular expressions written in C but used in many languages).
        rusk 3y ago
        Yes I think you’re right actually. I’m about 10 years off :)
  - jwilk 3y ago
    > what is written as $f..$\1 in BRE is written as (f..)\1 in ERE Oddly, there are no backreferences in POSIX EREs.
    - susam 3y ago
      You are right indeed. Looked at the specification again and indeed there is no back-reference in POSIX ERE. Quoting from <https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/xrat/V4_xbd_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...>: > It was suggested that, in addition to interval expressions, back-references ( '\n' ) should also be added to EREs. This was rejected by the standard developers as likely to decrease consensus. Updated my comment to present a better example that avoids back-references. Thanks!
    - GrumpySloth 3y ago
      That’s because POSIX EREs are actual regular expressions thank god.
  - deleted 3y ago
    [deleted]
- psd1 3y ago
  No gnu tool can balance brackets, afaics. So you can't do everything in sed. And sed is, by design, useless for matching text that spans lines, so good luck picking out paragraphs with it.
  - rusk 3y ago
    Sorry I meant to write “if you can do it in sed you can do it in anything” thereby implying it is a subset of the more generally available flavours. The issue at hand however is that there isn’t much in the way of standardisation but 95% of sed should work across all of them. Of course you should get more into the specifics of whatever your solution space supports.
  - ykonstant 3y ago
    I am pretty sure even pure Awk can do it; or am I mistaken? I thought there was an even more sophisticated example in the Awk book. Edit: oh, you mean via regex engines available in GNU tools; I am dumb. Hmm... is there no GNU extension with PCRE?
    - colimbarna 3y ago
      "Sed" is the name of a specific tool. It is not defined by the GNU tools, but has existed in some form since 1974, well before Perl. GNU sed and POSIX sed both support BRE and EREs, but not PCREs. Maybe there's some other implementation of sed that supports PCREs but that would really be an extension of that implementation of sed rather than a property of sed. And maybe there's some GNU tool that uses PCREs, but that GNU tool would not be GNU sed, so it would not be a relevant property. Anyway, they probably should have said BREs or EREs rather than "sed"...
    - psd1 3y ago
      Nope. Gnu regex tools also do not support lookahead or alternation. Lookahead is really useful. It's a problem for Linux that it can't move on. The cold dead hand of gnu is firmly around the community's neck.
      - 3y ago
telotortium 3y ago
Languages invented after Perl will generally use some flavor of Perl regex syntax, but there are always some minor differences. The issue of the meaning of `$` and changing it via multi-line mode is usually consistent though.
- usrusr 3y ago
  I like to think of "whatever browsers do in js" as an updated common baseline. Whatever your regex engine does, describe it as a delta to the js precedent. That thing is just so ubiquitous. I do wonder though what's the highest number of different regex syntaxes I've ever encountered (perhaps written?) within a single line: bash, grep and sed are never not in a "hold my beer" mood!
  - psd1 3y ago
    Reason #2 to use powershell - consistent regex. I've got "hold my beer" commits in .net - I've balanced brackets. I believe that's impossible in sed and grep. If I were going to write a json parser in a script, then a) stop me and b) it's got to be in powershell.
  - layer8 3y ago
    That seems like just a web front-end developer’s perspective.
  - Calzifer 3y ago
    Isn't JavaScripts regex one of the worst modern regex implementations? They seem to improve. Negative lookbehind isn't missing anymore [1]. But still lack the handy \Q and \E to escape stuff [2]. [1] https://stackoverflow.com/a/3950684 https://stackoverflow.com/a/3950684 [2] https://stackoverflow.com/q/6318710 https://stackoverflow.com/q/6318710
  - kstrauser 3y ago
    I’ll go along with that, as long as someone ports pcre to JavaScript and that’s the browser syntax we land on.
  - 3y ago
wolletd 3y ago
At some point, I felt like I knew them all. There are probably more regex dialects out there, but I don't encounter them and my set of knowledge works most of the time. I feel it's like driving a rental car. It behaves slightly different than your own car, some features missing, some other features added, but in general, most of the things are pretty similar.
- stanislavb 3y ago
  What a nice analogy. I’ll borrow it in the future.
MattHeard 3y ago
My working assumption has always been to check the docs of your specific regexp parser, and to write some tests (either automated or manually in a REPL) with specific patterns that you are interested in using.
out-of-ideas 3y ago
kind of a trick question; there is POSIX and then there is the app you're using and whichever flags are enabled (albeit by default or explicitly defined)
jasonjayr 3y ago
The three big ones I know of are POSIX, Perl/PCRE(aka Perl-Compatible Regular Expression), and Go came along and <strike>added</strike> used re2, which is a bit different from the first too. A lot of systems implemented PCRE, including JavaScript, since Perl extended the POSIX system with many useful extensions. IIRC, re2 tries to reign in on some of the performance issues and quirks the original systems had, while implementing the whole thing in Go. edit: Did not realize re2 predated go ...
- jpgvm 3y ago
  re2 predates Go and was written in C++.
- foldr 3y ago
  Go's regex implementation is new in the sense that it's not just a binding to the re2 C++ library, but it uses the same non-backtracking algorithm.
- jerf 3y ago
  POSIX and PCRE are arguably redundant. They both support backreferences, which puts very significant constraints on their implementations. PCRE is at least functionally a superset of POSIX, whether or not there's some quirky thing POSIX supports that PCRE does not. re2 adds a legitimate option to the menu of using NDFAs, which have the disadvantage of not supporting backreferences, but have the advantage of having constrained complexity of scanning a string. This does not come for free; you can conceivably end up with a compiled regexp of very large size with an NDFA approach, but most of the time you won't. The result may be generally slower than a PCRE-type approach, but it can also end up safer because you can be confident that there isn't a pathological input string for a given regexp that will go exponential. This is one of those cases where ~99% of the time, it doesn't really matter which you choose, but at the scale of the Entire Programming World, both options need to be available. I've got some security applications where I legitimately prefer the re2 implementation in Go because it is advantageous to be confident that the REs I write have no pathological cases in the arbitrary input they face. PCRE can be necessary in certain high-performance cases, as long as you can be sure you're not going to get that pathological input. RE engines don't quite engender the same emotions as programming languages as a whole, but this is not cheerleading, this is a sober engineering assessment. I use both styles in my code. I've even got one unlucky exe I've been working with lately that has both, because it rather irreducibly has the requirements for both. Professionally annoying, but not actually a problem.
bregma 3y ago
The ISO/IEC 14882 C++ standard library <regex> mandates [0] implementations for six de jure standard regex grammars: IEEE Std 1003.1-2008 (POSIX) [1] BRE, ERE, awk, grep, and egrep and ECMA-262 EcmaScript 3 [2]. So, yes, at least someone (me) considers regex to be standardized in several published de jure standards. [0] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3690.pdf#chapter.28 [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html [2] https://262.ecma-international.org/14.0/#sec-regexp-regular-expression-objects
- pjc50 3y ago
  "At least six different standards" is an XKCD comic, not a standard.
  - riffraff 3y ago
    "The nice thing about standards is that you have so many to choose from." - Andrew Tanenbaum (or Grace Hopper)
- account42 3y ago
  <regex> is not exactly an example anyone should follow.
  - bregma 3y ago
    You may be prejudiced against C++, but ISO/IEC 14882 is a published international standard that links to recognized regex standards, so answers the question "does anyone consider RegEx standardised?" very much in the affirmative.
beardyw 3y ago
And don't get me started about find and replace, what is the symbol to insert the match?
tonyg 3y ago
Delightfully, RFC 9485 https://datatracker.ietf.org/doc/rfc9485/ https://datatracker.ietf.org/doc/rfc9485/ "I-Regexp: An Interoperable Regular Expression Format" was published just back in October last year!

ikiris 3y ago

this is mostly due to the different types of regex and less about it being platform dependent. $ was end of string in pcre which is the "old" perl compatible regex. python has its own which has quirks as mentioned, re2 is another option in go for example, and i think rust has its own version as well iirc.

pjmlp 3y ago
Indeed, there isn't any kind of universal regexp standard.
- 7bit 3y ago
  We should create a new RegEx flavour that standardises RegEx for good!
  - jasonjayr 3y ago
    https://xkcd.com/927/ https://xkcd.com/927/
    - rerdavies 3y ago
      https://datatracker.ietf.org/doc/rfc9485/ https://datatracker.ietf.org/doc/rfc9485/ https://xkcd.com/927/ https://xkcd.com/927/
wolletd 3y ago
The differences of the various regex "dialects" came to me over the years of using regular expressions for all kinds of stuff. Matching EOL feels natural for every line-based process. What I find way more annoying is escaping characters and writing character groups. Why can't all regex engines support '\d' and '\w' and such? Why, in sed, is an unescaped '.' a regex-dot matching any character, but an unescaped '(' is just a regular bracket?
- somat 3y ago
  > Why, in sed, is an unescaped '.' a regex-dot matching any character, but an unescaped '(' is just a regular bracket? It is because sed predates the very influential second generation Extended Regular Expression engine and by default uses the first generation Basic Regular Expression engine. So really it is for backwards compatibility. http://man.openbsd.org/re_format#BASIC_REGULAR_EXPRESSIONS http://man.openbsd.org/re_format#BASIC_REGULAR_EXPRESSIONS you can usually pass sed a -r flag to get it to use ERE's Actually I don't really know if BRE's predate ERE's or not. I assume they do based on the name but I might be wrong.

Izmaki 3y ago

The new-line character is an actual character "at the end" of the string though so it makes sense that $ would include the new-line character in multi-line matching.

IshKebab 3y ago
Yes and every implementation gets that right. The point was when multi-line matching is disabled and only Javascript, Go and Rust get that right. I'm not too surprised by PHP and Python getting it wrong. Java and C# is a slight surprise though.
- danbruc 3y ago
  I don't think it is correct to say some get it right and some get it wrong, it is more of an design decision.
  - IshKebab 3y ago
    It's possible to get design decisions wrong. Clearly people expect `$` to only match end-of-string so they did make the wrong decision. It may not have been clear it was the wrong decision at the time.
    - danbruc 3y ago
      Things are obviously more complicated than that, lines are a complicated issue for historical reasons. There are two conventions, line termination and line separation. In case of line termination, the newline is part of the line and a string without a newline is not a [complete] line. In case of line separation, the newline is not part of the line but separates two lines. Also the way newlines are encoded is not universal.
      - fauigerzigerk 3y ago
        Why is this relevant when multi-line is disabled?

user2342 3y ago

I'm confused by this blog-post. In the table: what is the reg-ex pattern tested and against which input?

mnw21cam 3y ago
The input being matched is "cat\n" and the regex pattern is one of: "cat$" with multiline enabled "cat$" with multiline disabled "cat\z" "cat\Z"

somat 3y ago

Structural regexes as found in the sam editor are an obscure but well engineered regex engine. I am far from an expert but my main takeaway from them is that most regex engines have an implied structure built around "lines" of text. While you can work around this, it is awkward. Structural regexes allow you to explicitly define the structure of a match, that is, you get to tell the engine what a "line" is. http://man.cat-v.org/plan_9/1/sam http://man.cat-v.org/plan_9/1/sam

xlii 3y ago

Regexp was one of the first things I truly internalized years ago when I was discovering Perl (which still lives in a cozy place in my heart due to a lovely “Camel” book). Today most important bit of information is knowledge that implementations differ and I made a habit of pulling reference sheet for a thing I work with. E.g. Emacs Regexp annoyingly doesn’t have word in form of “\w” but uses “\s_-“ (or something no reference sheet on screen) as character class (but Emacs has the best documentation and discoverability - a hill I’m willing to die on) Some utilities require parenthesis escaping and some not. Sometimes this behavior is configurable and sometimes it’s not. I lived through whole confusion, annoyance, denial phase and now I just accept it. Concept is the same everywhere but flavor changes.

ydant 3y ago
Exactly the same here, re: Perl. My brain thinks in Perl's regex language and then I have to translate the inconsistent bits to the language I'm using. Especially in the shell - I'm way more likely to just drop a perl into the pipeline instead of trying to remember how sed/grep/awk (GNU or BSD?) prefer their regex.
- influx 3y ago
  GNU grep supports Perl regexp with -P
  - mwpmaybe 3y ago
    As does git grep!
  - 1letterunixname 3y ago
    Using PCRE2, which doesn't behave exactly the same as Perl or PCRE1. https://pcre.org/current/doc/html/pcre2compat.html https://pcre.org/current/doc/html/pcre2compat.html https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expres... https://stackoverflow.com/questions/70273084/regex-differences-between-pcre-and-pcre2 https://stackoverflow.com/questions/70273084/regex-differenc...

masswerk 3y ago

As for the good old reference implementation (not "Parameter Efficient Reinforcement Learning"): my $string = "cat\n"; /cat$/s -> true /cat\Z/s -> true /cat\z/s -> false

ghusbands 3y ago

> Note: The table of data was gathered from regex101.com, I didn't test using the actual runtimes. Has anyone confirmed this behaviour directly against the runtimes/languages? Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes.

AtNightWeCode 3y ago
I fail to add carriage return to the test string on that site. Which I guess would be an issue on Windows.
zimpenfish 3y ago
https://go.dev/play/p/Tce1qWjfjOy https://go.dev/play/p/Tce1qWjfjOy matches their results. I've also run that locally against "go1.22.1 darwin/arm64", "go1.21.5 windows/amd64", and "go1.21.0 linux/amd64" with the same result.
coldtea 3y ago
>Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes. In what way could newlines at the end of a string "could get lost in transit"?
- ghusbands 3y ago
  If you write it to a text file by itself and then read it from that text file, each runtime can have a different definition of whether a newline at the end of the file is meaningful or not. Under POSIX, a newline should always be present at the end of a non-empty text file and is not meaningful; not everyone agrees or is aware. There are plenty of other ways, too; bugs happen.
  - coldtea 3y ago
    Ideally no runtime should alter strings passing through ("in transit") from one runtime to another - unless it does some processing on them.

pjc50 3y ago

Special misery case: Visual Studio supports regex search, where '$' matches \n. The end of line character is usually the standard Windows \r\n. Yes, that means if you want to really match the end of line you have to match "\r$". So broken.

skrebbel 3y ago
FWIW, and I know this doesn't really address your complaint: I use Windows and I've set all my text editors to use LF exclusively years ago and Things Are Great. No more weird Git autocrlf warnings, no quirks when copying files over to/from people on Macs or Linuxes, etc. Even Notepad supports LF line endings for quite a long time now - to my practical experience, there's little remaining in Windows that makes CRLF "the OS standard line ending". I bet if someday VS Code's Windows build ships with LF default on new installations, people won't even notice. I mean, at some point it did matter what the OS did when you pressed the "Enter" button. But this isn't really the case much anymore. VS Code catches that keypress, and inserts whatever "files.eol" is set to. Sublime does the same. I didn't check, but I assume every other IDE has this setting. Similarly, the HTML spec, which is pretty nuts, makes browsers normalize my enters to LF characters as I type into this textarea here (I can check by reading the `value` property in devtools), but when it's submitted, it converts every LF to a CRLF because that's how HTML forms were once specced back in the day. Again though, what my OS considers to be "the standard newline" is simply not considered at all. Even CMD.EXE batch files support LF. I don't really type newlines all that much outside IDEs and browsers (incl electron apps) and places like MS Word, all of which disregard what the OS does and insert their own thing. Maybe the terminal? I don't even know. I doubt it's very consequential. EDIT: PSA the same holds for backslashes! Do Not Use Backslashes. Don't use "OS specific directory separator constants". It's not 1998, just type "/" - it just works.
- n_plus_1_acc 3y ago
  I could never get visual studio (not code) to not use \r\n when editing a solution file via the gui
- divingdragon 3y ago

onion2k 3y ago

I can hear thousands of bad hiring manager's adding 'How do you match the end of a string in a regex?' to their list of 'Ha! You don't know the trick!' questions designed to catch out candidates.

hoc 3y ago
"I will hire you anyway, but I will pay you less" Regex, useful in any job...
- username_my1 3y ago
  regex is useful but chatgpt is amazing at it, so why spend a minute keeping such useless knowledge in mind. if you know where to find something no point in knowing it.
  - ykonstant 3y ago
    Does gpt produce efficient regex? Are there any experts here that can assess the quality and correctness of gpt-generated regex? I wonder how regex responses by gpt are validated if the prompter does not have the knowledge to read the output.
    - thecatspaw 3y ago
      what does gpt say how we should validate email addresses?
      - rhd 3y ago
        chatgpt-4: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566e53705 https://chat.openai.com/share/696f7046-7f43-4331-b12b-538566... chatgpt-3.5: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436626cc4 https://chat.openai.com/share/aaa09ae8-3fd9-4df7-a417-948436...

Karellen 3y ago

> Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string". Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators. Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"?

jamesmunns 3y ago
Same, tho it'd be interesting to see if this behavior holds if the file ends without a trailing newline and your match is on the final newline-less line.
- fooofw 3y ago
  Fortunately, it's pretty simple to test. $ printf 'Line with EOL\nLine without EOL' | grep 'EOL$' Line with EOL Line without EOL $ grep --version | head -n1 grep (GNU grep) 3.8
  - romwell 3y ago
    The line does end with the file, so it's logically consistent. It's not matching the newline character after all.
    - colimbarna 3y ago
      Yes exactly, they match the end of a line, not a newline character. Some examples from documentation: man 7 regex: '$' (matching the null string at the end of a line) pcre2pattern: The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. ... The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

hans_castorp 3y ago

Fun fact: in Postgres, 'cat\n' matches 'cat$' when the so called "weird" newline matching is enabled :) https://www.postgresql.org/docs/current/functions-matching.html#POSIX-EMBEDDED-OPTIONS-TABLE https://www.postgresql.org/docs/current/functions-matching.h...

febeling 3y ago

Seriously, just write one unit test for your regex.

mannykannot 3y ago
Indeed, one should test any regex one puts any trust in, but the problem is that if you take as a fact something that is actually a false assumption (as the author did here), your test may well fail to find errors which may cause faults when the regex is put to use. This, in a nutshell, is the sort of problem which renders fallacious the notion that you can unit-test your way to correct software.

PuffinBlue 3y ago

This seems like the perfect opportunity to introduce those unfamiliar to Robert Elder. He makes cool YouTube[0] and blog content[1] and has a series on regular expressions[2] and does some quite deep dives into the differing behaviour of the different tools that implement the various versions. His latest on the topic is cool too: https://www.youtube.com/watch?v=ys7yUyyQA-Y https://www.youtube.com/watch?v=ys7yUyyQA-Y He's has quite a lot of content that HN folks might be interested in I think, like the reality and woes of consulting[3] [0] https://www.youtube.com/@RobertElderSoftware https://www.youtube.com/@RobertElderSoftware [1] https://blog.robertelder.org/ https://blog.robertelder.org/ [2] https://blog.robertelder.org/regular-expressions/ https://blog.robertelder.org/regular-expressions/ [3] https://www.youtube.com/watch?v=cK87ktENPrI https://www.youtube.com/watch?v=cK87ktENPrI

aquariusDue 3y ago
I'm glad to see someone else that has stumbled over his content. Seconding the recommendation.
CatchSwitch 3y ago
He has so many favorite Linux commands lol
- robertelder 3y ago
  This is my favourite comment.

teknopaul 3y ago

Tldr; $ does not mean end of string in Python.

AtNightWeCode 3y ago

There are many differences between implementations of regex. To name a few. Lookbehind, atomic groups, named capturing groups, recursion, timeouts and my favorite interop problem, unicode.

wruza 3y ago

By default, '$' only matches at the end of the string and immediately before the newline (if any) at the end of the string. The rationale was probably "it should be easier to match input strings" and now it's harder for everyone.

frou_dh 3y ago

Something I found really surprising about Python's regexp implementation is that it doesn't support the typical character classes like [:alnum:] etc. It must be some kind of philosophical objection because there's no way something with as much water under the bridge as Python simply hasn't got around to it.

k3vinw 3y ago

Another poor soul trying to solve one problem using regex and now they have two… ;)

croes 3y ago

Isn't a string with a newline character automatically multiline? The new line is just empty but not the first line anymore.

Joker_vD 3y ago
No, it is not. 3.195 Incomplete Line A sequence of one or more non-<newline> characters at the end of the file. 3.206 Line A sequence of zero or more non-<newline> characters plus a terminating <newline> character. courtesy of [0]. See also [1] for rationale on "text file": Text File [...] The definition of "text file" has caused controversy. The only difference between text and binary files is that text files have lines of less than {LINE_MAX} bytes, with no NUL characters, each terminated by a <newline>. The definition allows a file with a single <newline>, or a totally empty file, to be called a text file. If a file ends with an incomplete line it is not strictly a text file by this definition. [...] [0] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... [1] https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd...
- rerdavies 3y ago
  Applications that write "incomplete lines" - VSCODE - Visual Studio - Notepad C/C++ compilers that process "incomplete lines" All of them. Will my users consider it a bug if my program doesn't process "incomplete lines": Yes. (Yes, I have had this bug logged against code that I wrote). Will I consider it a bug if your program doesn't process "incomplete lines": Yes.
  - Joker_vD 3y ago
    > Applications that write "incomplete lines" Note that all of those three apps come from the Windows world where CRLF was indeed a line separator, and a file ending with CRLF was considered to have an empty line at the end. But yes, you absolutely should accommodate for incomplete lines.
- 3y ago

perlgeek 3y ago

Raku (formerly Perl 6) has picked ^ and $ for start-of-string and end-of-string, and has introduced ^^ and $$ for start-of-line and end-of-line. No multi line mode is available or necessary. (There's also \h for horizontal and \v for vertical whitespace) That's one of the benefits of a complete rethink/rewrite, you can learn from the fact that the old behavior surprised people.

Terretta 3y ago
And this is why this curmudgeon can't use Perl 6[^1]. It randomly shuffles the line noise we learned over decades. It seems so obvious that's the opposite of what they should have defaulted to, that it clearly should have been ^ and $ for lines, and ^^ and $$ for the string, since like ((1)(2)(3)): ^^line1$\n^line2$\n^line3$\n$ [1]: That, and it's not anywhere, while Perl 5 is everywhere.
richardwhiuk 3y ago
Think I would have picked exactly the reverse (i.e. ^^ being more "starty" than "^").
- lcnPylGDnU4H9OF 3y ago
  Reminds me of verbosity flags in some cli utilities. Often, -v is "verbose" and -vv is "very verbose" and -vvv... etc.
librasteve 3y ago
Pretty much all the regexen I have written have been predicated on start / end of string (i typically feed lines through the regex) … so picking single ^ and $ for the whole string maintains a degree of backward compatibility (assuming that I am normal)

junshengz99 3y ago

[dead]

wodenokoto 3y ago

> So if you're trying to match a string without a newline at the end, you can't only use $ in Python! My expectation was having multiline mode disabled wouldn't have had this newline-matching behavior, but that isn't the case. A reproducible example would be nice. I don’t understand what it is he cannot do. `re.search('$', 'no new lines')` returns a match.

iainmerrick 3y ago
This unexpectedly matches: re.match('^bob$', 'bob\n') I didn't want the trailing newline to be included.
- wodenokoto 3y ago
  But that string does have a new line at the end.
  - iainmerrick 3y ago
    re.match('^bob$', 'bob') → yes re.match('^bob$', 'bobs') → no Most people would expect 'bob\n' not to match, because I used '$' and it has an extra character at the end, just like 'bobs'. In Python it does match because '\n' is a special case.
    - rerdavies 3y ago
      ... for some arbitrary definition of "most people".
      - iainmerrick 3y ago
        That’s true, probably fewer than ten million people in the world would have an opinion one way or the other. :)

danbruc 3y ago

People are confused about strings and lines. A string is a sequence of characters, a line can be two different things. If you consider the newline a line terminator, then a line is a sequence of non-newline characters - possibly zero - plus a newline. If there is no new-line at the end, then it is not a [complete] line. That is what POSIX uses. If you consider the newline a line separator, then a line is a sequence of non-newline characters - possibly zero. In either case, the content of the line ends before the newline, either because the newline terminates the line or because it separates the line from the next. [1] The semantics of ^ and $ is based on lines - whether single-line or multi-line mode. For string based semantics - which you could also think of as entire file if you are dealing with files - use \A and \Z or their equivalents. [1] Both interpretations have their merits. If you transmit text over a serial connection, it is useful to have a newline as line terminator so that you know when you received a complete line. If you put text into text files, it might arguably be easier to look at a newline as a line separator because then you can not have a invalid last line. On the other hand having line terminators in text files allows you to detect incompletely written lines.

Existing4190 3y ago

perlre Metacharacters documentation states: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used) (/m enables multiline mode)

deleted 3y ago

[deleted]

mdavid626 3y ago

Is this a bug?

humanlity 3y ago

Interesting

m0rissette 3y ago

Why isn’t Perl anywhere on that chart when mentioning regex?

burntsushi 3y ago
Because they're using regex101 to easily test the semantics of different regex engines and Perl isn't available on regex101. PCRE is though, which is a decent approximation. And indeed, Perl and PCRE behave the same for this particular case.
- account42 3y ago
  Why isn’t Perl available on regex101 when its all about regex?
  - burntsushi 3y ago
    I dunno. Maybe because nobody has contributed it? Maybe because Perl isn't as widely used as it once was? Maybe because it's hard to compile Perl to WASM? Maybe some other reason?

tyingq 3y ago

Seems odd to leave Perl off the list, given it's regex related. Here's the explanation for $ in the perlre docs: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used)

toyg 3y ago
Yeah, omitting what is arguably the language most associated with regexes seems a bit of an oversight. I guess it shows how far off the radar Perl currently is.
- demondemidi 3y ago
  Perl perfected the simplicity and flexibility of regex syntax from POSIX and it seems every other language after has just made it harder.
- TillE 3y ago
  PHP uses PCRE, so it more or less serves as a stand-in for Perl in this case.
  - gordonfish 3y ago
    Sort of, though PCRE is not a 100% replacement for regex in Perl proper; the former lacks some features of the latter.
- gordonfish 3y ago
  > I guess it shows how far off the radar Perl currently is. This is a serious misconception. Perl is far, far from dead. The constant activity of the gargantuan CPAN library more than demonstrates very much the opposite. I would say Perl and its community has done quite well considering it hasn't had the same mountain of corporate funds thrust into it like more highlighted have. Mainstream ain't everything.
  - saghm 3y ago

homakov 3y ago

This led to a few serious bugs in Ruby-based apps. Always use \A\z https://homakov.blogspot.com/2012/05/saferweb-injects-in-various-ruby.html https://homakov.blogspot.com/2012/05/saferweb-injects-in-var... https://sakurity.com/blog/2015/02/28/openuri.html https://sakurity.com/blog/2015/02/28/openuri.html https://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html https://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html

nunez 3y ago

You can also use (?m) to enable multiline processing on PCRE-compatible regexp engines.

Scubabear68 3y ago

In 30 years of developing software I don’t think I ever used multi-line regexp even once.

thrdbndndn 3y ago
Definitely not common, but if you are parsing a text file you're going to use it a lot (say, you're writing a JS parser).
- marcosdumay 3y ago
  You really shouldn't use a lot of regexes for parsing code. They go only on the tokenizer, if they go somewhere at all.
  - thrdbndndn 3y ago
    Agreed, it's more about quick and dirty ad hoc capture than full-fledged parser though (like when you want to extract certain object when scraping).
Terretta 3y ago
> In 30 years of developing software I don’t think I ever used multi-line regexp even once. As long as sharing anecdata, in 30 years, it's almost the only way I use it. It's incredible for slicing and dicing repetitious text into structure. You generally want some sort of Practical Extraction and Reporting Language, the core of which is something like a regular expression, generally able to handle the, well, irregularity. Most recent example (I did this last week) was extracting Apple's app store purchases from an OCR of the purchase history available through Apple's Music app's Account page that lets you see all purchases across all digital offerings, but only as a long scrolling dialog box (reading that dialog's contents through accessibility hooks only retrieves the first few pages, unfortunately). Each purchase contains one or more items and each item has one or more vertical lines, and if logos contain text they add arbitrary lines per logo. A good match and sub match multi-line regex folds that mess back into a CSV. In this case, the regex for this was less than an 80 char line of code and worked in the find replace of Sublime Text which has multiline matching, subgroups, and back references. Another way to do this is something like a state match/case machine, but why write a program when you can just write a regular expression?

nebulous1 3y ago

The fact that there are so many different peculiarities in different regex systems has always raised the hairs on the back of my neck. As in when a tool accepts a regex and I have to a trawl the manual to find out exactly what regex is acceptable to it.

silent_cal 3y ago

I think there's a big opportunity to re-write Regex as a SQL-type language. It's too bad I don't feel like trying.

macintux 3y ago
I am 99% confident I’ve seen a post on HN about someone creating a much more verbose regex replacement. I have no idea how to find it. Here’s an interesting project for typed regular expressions: https://news.ycombinator.com/item?id=12292389 https://news.ycombinator.com/item?id=12292389

raldi 3y ago

Cmd-F perl no matches

deleted 3y ago

[deleted]

weinzierl 3y ago

The table in the article makes this look complicated, but it really isn't. All the cases in the article can be grouped into two families: - The JS/Go/Rust family, which treats $ like \z and does not support \Z at all - The Java, .NET, PHP, Python family, which treats $ like \Z and may or may not (Python) support \z. \Z does away with \n before the end of the string, while \z treats \n as a regular character. For multiline $ the distinction doesn't matter, because \n is the end. Really the only deviation from the rule is Python's \Z, which is indeed weird.

gorjusborg 3y ago

If you really want to learn regex, you'll have a hard time piecing it all together via blog posts. Brad Freidl's Mastering Regular Expressions is a good book to read if you want to stop being surprised/lost. I'll admit I stopped at the dive into DFA/NFA engine details.

macintux 3y ago
Writing a string -> NFA -> DFA grep-like tool is one of my most memorable college projects. Had a lot of fun with that, and decades later I ended up reusing some of the concepts for a work project.

T3RMINATED 3y ago

[dead]

jewel 3y ago

This has security implications! Example exploitable ruby code: unless person_id =~ /^\d+$/ abort "Bad person ID" end sql = "select * from people where person_id = #{person_id}" In addition to injection attacks, this also can bite people when parsing headers, where a bad header is allowed to sneak past a filter.

jfhufl 3y ago
Unsure what you mean? $ ruby -e 'x = "25" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes $ ruby -e 'x = "a25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' no Also, you'd want to use something that parameterizes the query with '?' (I use the Sequel gem) instead of just stuffing it into a sql string.
- jfhufl 3y ago
  Well, learned something today after reading a bit further in the thread: ruby -e 'x = "a\n25\n" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes Good to know.
- halostatue 3y ago
  You need to make your regex multi-line (`/^\d+$/m`), but that isn't the problem shown. Your query will be searching for `25\n`, not `25` despite your pre-check that it’s a good value. The second line should always be no, which if you use `\A\d+\z`, it will be.
  - jfhufl 3y ago
    Yep, makes sense, thanks!
- dr-smooth 3y ago
  $ ruby -e 'x = "25\n; delete from people" ; if x =~ /^\d+$/ ; puts "yes" ; else ; puts "no" ; end' yes
3y ago

SAI_Peregrinus 3y ago

POSIX regexes and Python regexes are different. In general, you need to reference the regex documentation for your implementation, since the syntax is not universal. Per POSIX chapter 9[1]: 9.2 … "The use of regular expressions is generally associated with text processing. REs (BREs and EREs) operate on text strings; that is, zero or more characters followed by an end-of-string delimiter (typically NUL). Some utilities employing regular expressions limit the processing to lines; that is, zero or more characters followed by a <newline>." and 9.3.8 … "A <dollar-sign> ( '$' ) shall be an anchor when used as the last character of an entire BRE. The implementation may treat a <dollar-sign> as an anchor when used as the last character of a subexpression. The <dollar-sign> shall anchor the expression (or optionally subexpression) to the end of the string being matched; the <dollar-sign> can be said to match the end-of-string following the last character." combine to mean that $ may match the end of string OR the end of the line, and it's up to the utility (or mode) to define which. Most of the common utilities (grep, sed, awk, Python, etc) treat it as end of line by default, since they operate on lines by default. THERE IS NO SINGLE UNIVERSAL REGULAR EXPRESSION SYNTAX. You cannot reliably read or write regular expressions without knowing which language & options are being used. [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

lakomen 3y ago

[flagged]

javier_e06 3y ago

I would hold a code review hostage if any file does not end with an empty new line. My reasoning would be if the file is transmitted and gets truncated nobody would know for sure if it does not end a new line. Brownie points if this is code end has a comment that the files ends there. The article calls computer languages platforms but the are computer languages. Bash is not included. Weird. I believe the most common use of regular expressions is the use of grep or egrep with bash or some other shell but, who knows. Maybe I am hanging with the wrong crowd.

vitiral 3y ago

In Lua it's only the start/end of the string > A pattern is a sequence of pattern items. A caret '^' at the beginning of a pattern anchors the match at the beginning of the subject string. A '$' at the end of a pattern anchors the match at the end of the subject string. At other positions, '^' and '$' have no special meaning and represent themselves. https://www.lua.org/manual/5.3/manual.html#6.4.1 https://www.lua.org/manual/5.3/manual.html#6.4.1 Lua's pattern matching is much simpler than regexes though. > Unlike several other scripting languages, Lua does not use POSIX regular expressions (regexp) for pattern matching. The main reason for this is size: A typical implementation of POSIX regexp takes more than 4,000 lines of code. This is bigger than all Lua standard libraries together. In comparison, the implementation of pattern matching in Lua has less than 500 lines. https://www.lua.org/pil/20.1.html https://www.lua.org/pil/20.1.html

denzquix 3y ago
> In Lua it's only the start/end of the string There's an additional caveat: if you use the optional "init" parameter to specify an offset into the string to start matching, the ^ anchor will match at that offset, which may or may not be what you expect.
- vitiral 3y ago
  That is a good point, and something I've actually (personally) used quite a bit when writing parsers
- Dylan16807 3y ago
  > which may or may not be what you expect I find it hard to imagine any other expectation passing the rubber duck test. "Oh, so you expected the match to always fail, no matter what the string was?"
  - denzquix 3y ago
    Well, it's not a completely outlandish scenario that the value of `init` might come from a variable that is sometimes at the start of the string and sometimes not, and a newcomer might expect `^` to only match when it is. Don't get me wrong, it's certainly far more useful as it is, I'm glad it works this way.

cpeterso 3y ago

$ is the regex’s “the buck stops here” symbol. Here at the end of the line. :)

nurtbo 3y ago

Totally get the desire, but also feels like last two paragraphs are solvable with ``` re.match(text).extract().rstrip(“\n”) ```

deleted 3y ago

[deleted]

menacingly 3y ago

Of course it’s line. How could it be the end of the string when the matter at hand is defining the string?

pksebben 3y ago

Regex would really benefit from a comprehensive industrial standard. It's such a powerful tool that you have to keep relearning whenever you switch contexts.

aftbit 3y ago

Wait, in non-multiline mode, it only matches _one_ trailing newline? And not any other whitespace, including \r or \r\n? That is indeed surprising behavior. Why? Why not just make it end of string like the author expected? >>> import re >>> bool(re.search('abc$', 'abc')) True >>> bool(re.search('abc$', 'abc\n')) True >>> bool(re.search('abc$', 'abc\n\n')) False >>> bool(re.search('abc$', 'abc ')) False >>> bool(re.search('abc$', 'abc\t')) False >>> bool(re.search('abc$', 'abc\r')) False >>> bool(re.search('abc$', 'abc\r\n')) False

mmh0000 3y ago

> So if you're trying to match a string without a newline at the end, you can't only use $ in Python! My expectation was having multiline mode disabled wouldn't have had this newline-matching behavior, but that isn't the case. I would argue this is correct behavior, a "line" isn't a "line" if it doesn't end with \n.[1] > 3.206 Line - A sequence of zero or more non- <newline> characters plus a terminating <newline> character. [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

remram 3y ago
Why should it be a "line" though? The whole point is that people pass in a string, they don't care about lines.

shenene 3y ago

[flagged]

librasteve 3y ago

I am surprised that the OP does not include perl5 in their table. In raku (aka perl6) Regexes were reinvented by Larry Wall (the creator of perl which made perlRE the de facto regex standard) Here's what he does with $: (https://docs.raku.org/language/regexes#Start_of_string_and_end_of_string https://docs.raku.org/language/regexes#Start_of_string_and_e...) * The $ anchor only matches at the end of the string * The $$ anchor matches at the end of a logical line. That is, before a newline character, or at the end of the string when the last character is not a newline character.

ary 3y ago

Was any regex documentation unclear on this? Some libraries have modes that change the semantics of ^ and $ but I’ve always found their use to be rather clear. It’s the grouping and look ahead/behind modifiers that I’ve always found hard to understand (at times).

macintux 3y ago
This is a feature that seems so painfully obvious in the abstract that I’d wager most have never read the documentation. I’ve been a regex user since the early 90s and I’ve never thought about this.

pmarreck 3y ago

The results did not surprise me. The fact that everyone is in agreement that "cat$" matches "cat" and not "cat\n" if multiline is off did not surprise me. \n is implicitly a multiline-contextual character to me. In other words, if you didn't have any \n, you'd just have an array of lines (without linefeeds), same as if you were reading lines from a file one at a time or splitting a binary on \n. The other results that differ across engines seem to be because people either don't understand regex or because the POSIX description of how to deal with such an input and config was ill-defined.

1letterunixname 3y ago

Ugh. Whenever I hear people talk about regular expressions as a singular language or standard, I die a little inside. PSA: Regex security is particular to each implementation flavor. Please know the nuances of a particular kind and be unambiguously precise.

callwhendone 3y ago

it's end of line right?

smlacy 3y ago

It's easy to get the canonical answer: $ man pcre2syntax Where you'll find the following block under ANCHORS AND SIMPLE ASSERTIONS: $ end of subject also before newline at end of subject also before internal newline in multiline mode So all the cases of "newline at/before end of subject" are covered here. Then, the question becomes "what is a subject?" Is it line-by-line? Are newlines included? What if we want multiline matching? That's where re.MULTILINE comes from, it's not "multiline matching" (sort of) it's "what is the subject of the regular expression that we're matching against"

michaelcampbell 3y ago

...IN PYTHON