9 ms·
The dangers of single line regular expressions
- SEXMCNIGGA15625 3y ago[flagged]
- SEXMCNIGGA33514 3y ago[dead]
- SEXMCNIGGA39672 3y ago[flagged]
- SEXMCNIGGA25508 3y ago[flagged]
- SEXMCNIGGA39240 3y ago[flagged]
- SEXMCNIGGA24930 3y ago[flagged]
- SEXMCNIGGA5590 3y ago[flagged]
- SEXMCNIGGA19655 3y ago[flagged]
- jlv2 3y agoMore like "the danger of thinking you can trivially validate user-supplied input" before evaluating the string.
- cratermoon 3y agoEven non-trivially validating it can go wrong. See Log4Shell, e.g. The bigger problem here is executing user input.
- wodenokoto 3y agoI think it is a surprise that a partial match return true. But I guess this is why Python has so many ways of matching a pattern against a string (match, find, findall, I think - they are hard to remember)
- roywiggins 3y agoI have to look it up every time.
- sfink 3y agoAlternatively, don't validate and then use the original. Instead, pull out the acceptable input and use that. Even better, compare that to the original and fail validation if they're not identical, but that requires maintaining a higher level of paranoia than may be reasonable to expect.
- Akronymus 3y ago> Alternatively, don't validate and then use the original. Instead, pull out the acceptable input and use that. Parse don't validate https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- sfink 3y agoHeh. I wrote up my comment, and then thought "hey, I bet that's what that 'Parse don't validate' article meant, the one I never quite got around to reading." So I pulled it up — great article! — but then didn't post the link because it uses the type system to record the results of the parse. Whereas here, you'd probably parse from a string into another string. But philosophically I agree, that's exactly the relevant advice.
- Akronymus 3y agoparsing from a string to a string runs the risk of erroneously assigning the original value to the new string. Which kinda defeats the whole parsing, not validating. What would work is having a small object holding a readonly string which parses the original on creation, then becomes immutable.
- neilk 3y agoIn my experience `$` does reliably mean end of string for regular expressions, unless you specifically ask for "multiline" mode. Ruby seems to be in multiline mode all the time? $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foobar") else "no"' yes $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar") else "no"' no $ python -c 'import re; print "yes" if re.match(r"^[a-z ]+$", "foo\nbar", re.M) else "no"' yes $ perl -le 'print "foobar" =~ /^[a-z ]+$/ ? "yes" : "no"' yes $ perl -le 'print "foo\nbar" =~ /^[a-z ]+$/ ? "yes" : "no"' no $ perl -le 'print "foo\nbar" =~ /^[a-z ]+$/m ? "yes" : "no"' yes $ node -e 'console.log(/^[a-z ]+$/.test("foobar") ? "yes" : "no")' yes $ node -e 'console.log(/^[a-z ]+$/.test("foo\nbar") ? "yes" : "no")' no $ node -e 'console.log(/^[a-z ]+$/m.test("foo\nbar") ? "yes" : "no")' yes $ ruby -e 'if "foobar" =~ /^[0-9a-z ]+$/i then puts "yes" else puts "no" end' yes $ ruby -e 'if "foo\nbar" =~ /^[0-9a-z ]+$/i then puts "yes" else puts "no" end' yes EDIT: this is documented behavior for Ruby. What other languages call multiline mode is the default; you're supposed to use \A and \Z instead. They do have an `/m` but it only affects the interpretation of `.` https://docs.ruby-lang.org/en/master/Regexp.html#class-Regexp-label-Multiline+Mode https://docs.ruby-lang.org/en/master/Regexp.html#class-Regex...
- interroboink 3y agoYeah, my takeaway from this was more "the dangers of Ruby" rather than "the dangers of single line regular expressions" (: I think the simplest fix would be to use "\Z" rather than "$", which means "match end of input" rather than "end of line." This is also Perl-compatible. So weird that the "$" default meaning is different in Ruby. I guess one could argue that Ruby's way is better since "$" has a fixed meaning, rather than being context-dependent. > Ruby seems to be in multiline mode all the time? Ruby does have a "/m" for multiline mode, but it just makes "." match newline, rather than changing the meaning of "$", it seems. [1] https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-Anchors https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-An... [2] https://perldoc.perl.org/perlre#Metacharacters https://perldoc.perl.org/perlre#Metacharacters
- neilk 3y ago
- wrsh07 3y agoThis was interesting and new to me, but as other commenters indicate, part of the problem is that we're trying to find the bad thing rather than trying to verify it is the good thing There's a related concept of "failing open vs failing closed" (fail open: fire exit, fail closed: ranch gate) In Jurassic park (amazing book/film to understand system failures), when the power goes out, the fence is functionally an open gate In this case, we shouldn't assume that we can enumerate all possible bad strings (even with a regex)
- tsimionescu 3y agoI don't think this is a good example, because the regex does just that: it doesn't try to filter out bad input, it specifically only accepts known good input. If the regex did what it was meant to do, only allowing strings composed of ascii letters and numbers, and space, than the code would have not been exploitable.
- floxy 3y agoStill seems like that is broken. Shouldn't they be escaping whatever control characters? Like if your user wanted to highlight "Now 75% off". Seems like it is reasonable to want to allow that.
- tsimionescu 3y agoThat's a completely different problem: it may be too closed. But it's definitely not a fail open system. It's a fail closed system with a bug.
- cedws 3y agoI pretty much always consider regex expressions as the wrong solution. They're notoriously hard to get right. There's a whole lot of faulty expressions out there for validating email addresses. I prefer to do less validation and let it fail. If the email address is wrong, whatever service you're using for sending emails will just reject it. If you really do need to validate email addresses, use something somebody else wrote that does it properly. If you're working with some exotic format for which there isn't already an open source library, do what this guy says: parse it, don't try to validate it with regex: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- htek 3y agoHere, you have good advice: "I ... consider regex expressions as the wrong solution. They're notoriously hard to get right." However, the conclusion of "use something somebody else wrote that does it properly", while valid, is asking a lot. As regex is hard to get right, don't assume the code you find on the web or book or via some other means works correctly. My rule is if I didn't write it and can't wrap my head around the code to convince myself it is the right solution, I don't use it. And as I think others have written, there are some interactive online tests for regex expressions that can help.
- cedws 3y agoI think the average developer has a better chance of finding a robust, battle tested library to do what they need than cooking up some regex of their own. Preferably, the library does not use regex at all and checks data more intelligently.
- scarmig 3y agoSometimes valid email addresses will be rejected as invalid, and sometimes invalid email addresses are still successfully delivered. Validation guarantees nothing, and at most it should be a UI cue.
- tsimionescu 3y agoRegex works very well for what it was originally designed: describing/validating regular languages. It can work ok if your language is simple and almost regular. They work very badly for validating non-regulars languages, even when extensions are added Perl-style to support that. And, unfortunately, most structured formats you might care to valdiate are in fact not regular languages at all. Email addresses in particular are surprisingly complicated and far from being regular languages. I don't know how commonly real servers support the full feature set, but even if they just support non-ascii names they quickly become a pain.
- SEXMCNIGGA16803 3y ago[flagged]
- SEXMCNIGGA1172 3y ago[flagged]
- SEXMCNIGGA28062 3y ago[flagged]
- SEXMCNIGGA41422 3y ago[flagged]
- SEXMCNIGGA45936 3y ago[flagged]
- SEXMCNIGGA33243 3y ago[flagged]
- SEXMCNIGGA9176 3y ago[flagged]
- SEXMCNIGGA45308 3y ago[flagged]
- SEXMCNIGGA26727 3y ago[flagged]
- SEXMCNIGGA21359 3y ago[flagged]
- SEXMCNIGGA32308 3y ago[flagged]
- SEXMCNIGGA34645 3y ago[flagged]
- SEXMCNIGGA49464 3y ago[flagged]
- SEXMCNIGGA8974 3y ago[flagged]
- SEXMCNIGGA9569 3y ago[flagged]
- SEXMCNIGGA16062 3y ago[flagged]
- SEXMCNIGGA19034 3y ago[flagged]
- SEXMCNIGGA27030 3y ago[flagged]
- librasteve 3y agoRaku (perl6) was a chance for Larry Wall to fix some of the limitations of the perl regex syntax, as you would expect from the perl heritage, it behaves similarly. ~ > raku -e 'say "foobar" ~~ /^ <[a..z ]> +$/ ?? "yes" !! "no"' yes ~ > raku -e 'say "foo\nbar" ~~ /^ <[a..z ]> +$/ ?? "yes" !! "no"' no ~ > raku -e 'say "foo\nbar" ~~ /^^<[a..z ]>+$$/ ?? "yes" !! "no"' yes - ^^ and $$ are the raku flavour of multiline mode - ~~ the smartmatch operator binds the regex to the matchee and much more - character classes are now <[...]> (plain [...] does what (...) does in math) - perl's triadic x ? y : z becomes x ?? y !! z We can have whitespace in our regexen now (and comments and multiline regexen) my $regex = rx/ \d ** 4 #`(match the year YYYY) '-' \d ** 2 # ...the month MM '-' \d ** 2 /; # ...and the day DD say '2015-12-25'.match($regex); # OUTPUT: «「2015-12-25」»
- btilly 3y agoPerl has supported whitespace and comments in regular expressions since approximately forever. Just use the /x modifier. All that Raku did was make that flag a default. The same thing is available in many other languages. They copied it when they copied from Perl. For example Python's https://docs.python.org/3/library/re.html#flags https://docs.python.org/3/library/re.html#flags documents that re.X, also called re.VERBOSE, does the same exact thing. The fact that people don't use it is because few people care to learn regular expressions well enough to even know that it is an option. One of my favorite examples of astounding people with this was when I was writing a complex stored procedure in PostgreSQL. I read https://www.postgresql.org/docs/current/functions-matching.html https://www.postgresql.org/docs/current/functions-matching.h.... I looked for flags. And yup, there is an x flag. It turns on "extended syntax". Which does the same exact thing. I needed a complex regular expression that I knew my coworkers couldn't have written themselves. So I commented the heck out of it. They couldn't believe that that was even a thing that you could do!
- librasteve 3y agoI was a full time perl coder with plenty of regex back in the day and never appreciated that (sorry) - so maybe making it the default is a good call PS. raku has added quite a lot to the regex facilities we are familiar with, not least a straight line to using them in Grammars with rule and token methods that give you control over handling of whitespace in the target
- SEXMCNIGGA12400 3y ago[flagged]
- SEXMCNIGGA2138 3y ago[flagged]
- SEXMCNIGGA1022 3y ago[flagged]
- SEXMCNIGGA11248 3y ago[flagged]
- SEXMCNIGGA27280 3y ago[flagged]
- SEXMCNIGGA28139 3y ago[dead]
- SEXMCNIGGA29703 3y ago[dead]
- SEXMCNIGGA12951 3y ago[dead]
- SEXMCNIGGA14889 3y ago[dead]
- SEXMCNIGGA21381 3y ago[dead]
- ec109685 3y agoEscape the output based on the context a string is being used in versus trying to sanitize for all use cases on input. This will guarantee that you’re safe no matter how a piece of content is used tomorrow (just need a new escaping function for that content type), and prevent awkward things like not letting users use “unsafe” strings as input. JSX and XHP are example templating systems that understand context and escape appropriately. If a user wants their title to be “hello%0a%3C%25%3D%20File.open%28%27flag.txt%27%29.read%20%25%3E”, so be it. Use input validation / parsing to ensure data types aren’t violated, but not as an output safety mechanism.
- Jerrrry 3y ago>If a user wants their title to be “hello%0a%3C%25%3D%20File.open%28%27flag.txt%27%29.read%20%25%3E”, so be it. that's a good way to horizontally propagate/reflect XSS and other Code As Data vulnerabilities. better to strip the known-bad/problematic characters https://en.wikipedia.org/wiki/Code_as_data https://en.wikipedia.org/wiki/Code_as_data
- ec109685 3y agoThe known problematic characters are different in json, xml, css, html content, html attributes, MySQL, etc. Unless you have output escaping, it is hard to ensure everything gets caught, no matter how the data enters the system.
- tsimionescu 3y agoSure, but there is a common set of safe characters that are guaranteed not to cause problems in any of these: the set described by the regex [a-zA-Z0-9 -]. If you can limit user input to this set, you'll drastically reduce the risk of code injection regardless of the stack below you.
- phyzome 3y agoAnd that's how you end up pissing off users with apostrophes in their names.
- ufmace 3y agoSeems to me this is more about the danger of passing anything derived from user input into the TEMPLATE side of a templating engine. Why in the world would you ever do that?!? Obviously if you pass data into the variable side of the engine, you hardly have to worry about it at all, since it's already going into a place that was designed for handling arbitrary and possibly-hostile input and been battle-tested at doing it correctly in Production for many years. If you pass it into the template side, you're betting that you can be as good as dozens of templating engine writers working for a decade at doing that, in exchange for, well, I can't really think of any possible legitimate advantage for doing that.
- klysm 3y agoWhat if you want to allow users to regex search their documents?
- ses1984 3y agoDo it on the client side? Do it in a sandbox and have aggressive timeouts.
- klysm 3y ago> Do it on the client side? Not practical given a large amount of documents. > Do it in a sandbox and have aggressive timeouts. Sure! I was just replying to this: > the danger of passing anything derived from user input into the TEMPLATE side of a templating engine. Why in the world would you ever do that?!?
- JonChesterfield 3y agoRegular expressions make me sad about our industry. If you read the early papers, you get a very clear language for pattern matching on sequences. They have really nice properties - the compilation to finite automata gives you decidable equality and decidable minimisation. As in you can compile equivalent regex to exactly the same state machine however they were expressed. At some point perl happened and that seems to have sent us down a path to encoding the regular expression in an illegible subset of ascii. The backtracking implementation cost us negation and intersection. What should be linear time matching becomes exponential. Emacs will let you write regex in s-expressions at which point they're much easier to read. Everywhere else has gone with "looks like Perl but has different semantics, which we kind of document, be lucky". I started writing tests to check that regex I'd begrudgingly converted to the perl style behaved the same under different engines and the divergence is rough. Granted I was parsing regex with regex which is possibly a path to insanity but things like a literal [ were a real puzzle to match on different implementations. I don't know that the horrible syntax on semantic beauty is due to perl but it looks likely from a superficial standpoint.
- deleted 3y ago[deleted]
- jacobolus 3y agoIf you read the early papers you get a very limiting mathematical tool of mainly theoretical interest. At some point perl happened and regular expressions became a ubiquitous practical tool saving programmers collectively millions of hours of labor.
- wlesieutre 3y agoSwift's RegexBuilder DSL from a couple years ago gets away from the illegible subset of ASCII. Easy to explode into a lot lines, but I'd rather have a 50 line RegexBuilder implementation than try to keep track of what the equivalent single-line version is doing. Especially if you ever have to come back to it later and understand it again. And if you ever make revisions in RegexBuilder you have useful diffs instead of "the one line that does everything is different than before." https://developer.apple.com/documentation/regexbuilder https://developer.apple.com/documentation/regexbuilder Are there similar tools in any other languages?
- hombre_fatal 3y agoIf this is sufficient for rendering the text as neon: @neon = "Glow With The Flow" erb :'index' What exactly is `@neon = ERB.new(params[:neon]).result(binding)` even supposed to be doing? Why wouldn't it just be: @neon = params[:neon] erb :'index'
- SEXMCNIGGA40759 3y ago[flagged]
- SEXMCNIGGA30416 3y ago[dead]
- SEXMCNIGGA44425 3y ago[dead]
- SEXMCNIGGA20788 3y ago[dead]
- SEXMCNIGGA23391 3y ago[dead]
- SEXMCNIGGA28586 3y ago[dead]
- SEXMCNIGGA42210 3y ago[dead]
- SEXMCNIGGA29895 3y ago[dead]
- SEXMCNIGGA13303 3y ago[dead]
- SEXMCNIGGA14797 3y ago[dead]
- SEXMCNIGGA32201 3y ago[dead]
- SEXMCNIGGA31264 3y ago[dead]
- SEXMCNIGGA18899 3y ago[flagged]
- SEXMCNIGGA38513 3y ago[dead]
- SEXMCNIGGA1743 3y ago[dead]
- SEXMCNIGGA25198 3y ago[dead]
- SEXMCNIGGA43497 3y ago[dead]
- SEXMCNIGGA32583 3y ago[dead]
- ezekg 3y agoRuby 4 should do what every other sane programming language does and require users to opt into multi-line mode via the /m flag. The fact that Ruby has this behavior at all is a major security issue.
- SEXMCNIGGA20391 3y ago[flagged]
- SEXMCNIGGA16843 3y ago[dead]
- SEXMCNIGGA39861 3y ago[flagged]
- SEXMCNIGGA19328 3y ago[flagged]
- SEXMCNIGGA10869 3y ago[dead]
- SEXMCNIGGA504 3y ago[dead]
- SEXMCNIGGA39900 3y ago[dead]
- SEXMCNIGGA36211 3y ago[dead]
- sublinear 3y ago> Hire me for a penetration test When does the blogspam end?
- _wire_ 3y agoAye. "Consider every ambiguity of technology as a personal marketing opportunity." The true topic at hand is that text substitution in scripted services is an eternal hazard of code injection. The point that regex-based input sanitization doesn't work because everyone misunderstands the token semantics for string termination is made to look like a marvelous mitigation, but this teaching on regex is distracting from an unavoidable hazard of scripting. Good news for the contractor: he appears like Jesus to shine the Lord's light on the sin of the fathers while dancing by the moral hazard of the priesthood. Elsewhere another instance of the OP is a service provider pushing business solutions based on the ease of use of scripted service frameworks ("Input sanitization is as simple as a regex!) These hazards are going to get much worse as AI merges the causes of and solutions to these ambiguities into the same semantic mush.
- banish-m4 3y agoI once had to explain this class of security vulnerability to IC5-IC7 senior engineers. 0. There is no universal regex language but many. 1. Perl-like ones (Ruby, Perl, and PCRE1/2) contain additional hidden traps. 2. You must vigorously match untrusted input to assume it to include invalid unicode, control characters, and other oddities. 3. You should replicate frontend and backend validations to ensure they are always exactly consistent and correct, preferably through fuzzing and/or property testing.