15 ms·

Rulex – A new, portable, regular expression language

242 points by thm 4y ago

ffhhj 4y ago
I want to allow users create and share RegEx's in my website, but I'm afraid there could be some abuse like building a very complex one that would get Javascript stuck or crash. Are there any known vulnerabilities and RegEx sanitization?
love2read 4y ago
When would something like this be necessary?
- nyellin 4y ago
  Every time you write a non trivial regex! Regexes often contain complex parts that repeat themselves throughout the regex or can be combined into higher level patterns. This adds support for that.
  - chii 4y ago
    > Regexes often contain complex parts that repeat themselves throughout the regex or can be combined into higher level patterns. sounds like a parser use case.
    - xdfgh1112 4y ago
      One of the coolest things about Raku/Perl6 was how it made grammars a first-class citizen, like how Perl 5 made regexes a first-class citizen.
  - asicsp 4y ago
    If supported, you can use verbose flag and subexpression calls for such cases. From a cursory glance, I found only the numeric `range` support the standout feature I'd want in existing regex support.
  - Diris 4y ago
    That reminds me of that nice article about how to do that kind of thing in python [0] The unreasonable effectiveness of f-strings and re.verbose https://news.ycombinator.com/item?id=31457188 https://news.ycombinator.com/item?id=31457188
  - bmn__ 4y ago
    You make it sound as if regex don't have this ability, but they already do.
- eternityforest 4y ago
  Looks like it would be good if you had to use some ancient regex engine. Otherwise, I find regex is extremely easy, since the problems it solves well are very simple, and one can develop it interactively by trial and error, by looking at the highlights in their editor. I could possibly see actually using this if the playground had a similar "Try it on real text instantly" feature, but I've never had a problem with traditional regex+trial and error.
  - SonOfLilit 4y ago
    Guessing you were downvoted because of the "ancient" part that has nothing to do with it. Also, many people don't agree with the "regex is fine" part.
  - an1sotropy 4y ago
    you may be resigned to trial and error being an essential part of regex development, but the reliance on trial and error should be (1) a signal that something can be improved with how programmers write regexs, and (2) a symptom of a programming ethos that essential things should be difficult. I'm glad that this project (and others like that others have highlighted here) allow us to improve on (1), and I am always encouraging my profession peers in computing to reconsider (2).
- SonOfLilit 4y ago
  The answer is the same as that for more expressive languages with more ergonomic syntax and less footguns vs C.
SonOfLilit 4y ago
Finally someone else trying to fix regex syntax approaches it from the same angle as mine[1]: a portable syntax that transpiles to plain old regex to get bug-for-bug compatibility but a nicer syntax. Instinctively, I think the syntax didn't go far enough and could be made more readable (of course I'd feel that way!), but the implementation and marketing has me beaten! Kudos! [1] https://github.com/SonOfLilit/kleenexp https://github.com/SonOfLilit/kleenexp
- fathyb 4y ago
  I recommend checking Melody out if you're into readable regex alternatives: https://github.com/yoav-lavi/melody https://github.com/yoav-lavi/melody
  - SonOfLilit 4y ago
    Oooh, it has grown. Nice!
  - asicsp 4y ago
    There's also https://github.com/VerbalExpressions https://github.com/VerbalExpressions for various programming languages.
- deleted 4y ago
  [deleted]
bryanrasmussen 4y ago
Ok, despite being more in the now you have two problems camp when discussing regular expressions I find this interesting enough I should schedule learning it. Question - when compiling I suppose it can turn the regex into the most optimized form - so generation of character classes, lazy quantifiers if pertinent?
matthews2 4y ago
See also: Emacs' structured regexp notation, https://www.gnu.org/software/emacs/manual/html_node/elisp/Rx-Notation.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Rx...
- cardanome 4y ago
  Oh, I like that one! Very easy to read. Have you ever worked with it and what was you experience? I can see myself using that or even implement it for other languages.
  - matthews2 4y ago
    It can end up taking a little longer to write (same with all of these alternate syntaxes), but I think it significantly reduces the WTF-factor when you have to try and read it again in weeks/months/years. I think it's worth it for any non-trivial regexp and I miss it whenever I'm using any other language, so I'm glad that this thread is full of alternatives :)
  - natrys 4y ago
    Not OP but I have switched to using rx macro for everything and I never miss regex. I miss rx instead everywhere else. Here are couple of blog posts someone might find useful: https://francismurillo.github.io/2017-03-30-Exploring-Emacs-rx-Macro/ https://francismurillo.github.io/2017-03-30-Exploring-Emacs-... https://karthinks.com/software/bridging-islands-in-emacs-1/ https://karthinks.com/software/bridging-islands-in-emacs-1/
  - rcthompson 4y ago
    I haven't used rx in Elisp specifically, but in my work in R, I've started using the rex package almost exclusively for writing regexes. It's the same idea: you build a regex programmatically instead of as a string, so you no longer need to think about syntax, only semantics, because the syntax is the same as the host language.
- ur-whale 4y ago
  Aside from the parens fetishism common to all lisp-flavored things, it is very nice and readable indeed.
moonchild 4y ago
See also: raku's grammars (https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes).
- asojfdowgh 4y ago
  can't overstate how nice raku is, bit sharper learning curve but still raku: ^ ( \d+ <?{ $/ <= 255 }> ) ** 4 % '.' $ slow raku: ^ ( @(0..255) ) ** 4 % '.' $ rulex: <% ( :( range '0'-'255' ) '.' ){3} :( range '0'-'255' ) %> regex: ^(?:(0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?)\.)((0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?))$ first one running code in the middle of the regex, to see if it matches a mathematical property, the second, running code to generate a [0|1|2|3|...|254|255] equivalent, which also matches as tightly as the rulex/regex code (e.g. no 00 allowed)
  - donatj 4y ago
    Seems like a case where I would just \d+\.\d+\.\d+\.\d+ and validate it in code
    - asojfdowgh 4y ago
      can still do that logic too, without leaving the parsing step, per say /^ (\d+) ** 4 % \. <?{ $/[0].all <= 255 }> $/ or if wanting it really explicit /^ (\d+) \. (\d+) \. (\d+) \. (\d+) <?{ $/.all <= 255 }> $/
  - lbriner 4y ago
    None of those seem much easier to understand than the others. I suspect this is a classic example of "let's take something that is really messy in pcre and compare it other more brief equivalents" but I could easily turn the regex into something more readable and I can find more help for the standard regex syntax than I can for any alternative. As other's have said, it is a minor arguably cosmetic improvement and something that is likely easier just to use a visual helper like regex101 to help you construct.
gajus 4y ago
Got to admire someone who is going to invest time to invent a whole new language just to avoid mastering an existing one. Regex is fine. Been using it nearly daily for the last decade. You get used to it. These days, there are tools that make it incredibly easy to write, test, version and document regex, like https://regex101.com/ https://regex101.com/.
- SonOfLilit 4y ago
  Working with a company that writes hundreds of regexes per day, I can say with confidence: regex syntax is _not_ fine. People with lots of experience make obvious mistakes and lose them in the line noise all the time.
  - xdfgh1112 4y ago
    Do you use the /x modifier?
    - SonOfLilit 4y ago
      The regexes are very simple, there's just thousands of them. /x wouldn't help much.
  - donatj 4y ago
    What could the job possibly be? Based on next to nothing, sounds like you might be better off building custom parsers?
    - chrisshroba 4y ago
      I'd guess any large tech company on average is writing at least hundreds per day, given the crazy amounts of code being produced.
- coldtea 4y ago
  >Got to admire someone who is going to invest time to invent a whole new language just to avoid mastering an existing one. Yes, because any existing language is the global optimum, and we should always strive to reuse it, never invent any new one... I mean, what's the problem with stone cavemen-era cart-wheels? Why reinvent the wheel? Not to mention the implying that the only reason one would try to invent a new language is to "not invest time mastering an existing one". I.e. "It's not that this guy tries to improve regular expression languages - it's that he's dumb/lazy to master the standard regex expressions". Never mind that this new language also transpiles into regular regex, so clearly the person who created it has also "mastered the existing one".
miked85 4y ago
While impressive, I'm not sure in what scenario anyone would choose to use this.
- tiborsaas 4y ago
  Maybe it would be a bit more user friendly to embed it into a product for end users.
raydiatian 4y ago
Saint_Genet 4y ago
To use this you basically have learn regexps anyway, so I can't really see the point.
- bryanrasmussen 4y ago
  well I guess the point would be: 1. if you know regex but for a particular language, this takes care of having to learn the different variants of regex because it will compile down to the version you want. 2. it is a more approachable syntax, so easier to learn. 3. because it is still a regex syntax given that you probably know a bit of regex already you can get better at it by learning this. any others I forgot?
  - benoketamad 4y ago
    Just like sql abstractions. A dead end, useless, unnecessary.
pizza234 4y ago
I'm curious if this does concretely improve regular expressions or not for generic programmers. Overall, to me it seems the classic minor-but-not-radical improvement. The major difference I see is that this is a sort of `/x` modified regex with better readability. This is a significant improvement - the big problem with `/x` is that one then needs to escape/encode whitespace, which typically offsets the improvement. The other significant change is variables, however, when one reaches a certain complexity, IMO they should not use regexes anymore. The other differences (based on the home page presentation) are cosmetic changes of the same semantics. The fundamental assumption is that the main problem of regexes is about syntax rather than semantics. It's true the regexes are somewhat ugly and hard to remember, but, personally, I don't think that's the main problem. It would have probably been nice if regex started with this syntax, but integrating them in a new project is a difficult decision.
- ethbr0 4y ago
  The only problem with regex is that people try to parse HTML with it. And in that sentence, neither 'regex' nor 'HTML' are the root of the problem. IMHO, ignorance should never be a valid feature request.
  - berkes 4y ago
    Are you honestly implying that there are still people who, in all seriousness, use a REGEX to parse HTML? Every programmer I have met, however junior, knows or learns about Zalgo early on. It's like the "foobar" wrt regexps: common knowledge. True, knowing it's a bad idea won't hold people from doing it. I do it. I'll whip up some stupid `:%s/\<?p class="strong"?>/<strong>/g` occasionally in vim, knowing perfectly fine that this regexp won't be solid, but: works for this case. All the while knowing about Zalgo for years (decades?)
    - dmitriid 4y ago
      > Are you honestly implying that there are still people who, in all seriousness, use a REGEX to parse HTML Subsets of it? Yes. See Google's lit-html as an example: https://github.com/lit/lit/blob/main/packages/lit-html/src/lit-html.ts https://github.com/lit/lit/blob/main/packages/lit-html/src/l...
nsajko 4y ago
This is effectively just a nicer syntax than usual for regular expressions, but there are actual areas of semantics where regexp languages are ripe for improvement - most regexp libraries are stuck in the 80s/90s, not keeping up with recent developments in research. For example, most regexp languages basically only have the traditional three operators of alternation, concatenation and closure/iteration. It's not necessary to stop there: the regular languages are closed under various other useful operators. Many of these operators don't even significantly increase the size of the minimal DFA. For example, the complement (as in set/language complement) and reversal/mirror image would be some basic additions that seem like they should be necessary nowadays. Using them can make the regular expression much shorter, nicer and more understandable. For the complement, this should come at no cost for the minimal DFA size. Some other viable operators are: intersection, set difference, merge (AKA shuffle), infiltration (also sometimes known as shuffle), interleaving. Some other possibilites for regexp that could be put to good use more are: * weighted regular expressions: these enable more power in a very elegant way, the idea is that programmers don't want just recognizers, so why limit regexps or finite automata to just that. * JIT. It should be possible to compile regexps with libgccjit or with llvm.
- avgcorrection 4y ago
  Do operators like complement work with Unicode?
  - nsajko 4y ago
    Of course, Unicode is just another alphabet. Specific expressions might be problematic though, depending on the implementation. EDIT: If malicious inputs are something to be worried about, I think a good way to do this would be to check for exceedingly costly blowups of various kinds during/after translation to a finite automaton (so before automaton execution). This would enable having complemented expressions when safe (in lots of cases, at least).
    - joppy 4y ago
      To get something useful and not attackable, the complement operation would need to take the complement inside the set of valid UTF-8 strings rather than the set of all byte-sequences, which is probably doable but I'm not sure how straightforward it would be.
tekknolagi 4y ago
Also notable is Remake, which has Rust bindings: https://github.com/ethanpailes/remake https://github.com/ethanpailes/remake
- nsajko 4y ago
  The owner of that repo had an interesting regexp-related thesis ("Skip Regex"): https://github.com/ethanpailes/thesis https://github.com/ethanpailes/thesis https://www.proquest.com/openview/e7295f20d24f203900f6291d6dc90772/ https://www.proquest.com/openview/e7295f20d24f203900f6291d6d...
gardaani 4y ago
Swift introduces RegexBuilder to make regex easier, but I think it is a bit too verbose: https://developer.apple.com/documentation/regexbuilder?changes=_2_2 https://developer.apple.com/documentation/regexbuilder?chang...
- Someone 4y ago
  For simple regexes, Swift has short literals, and (AFAIK) you can mix and match the DSL and the short literals. https://github.com/apple/swift-evolution/blob/main/proposals/0354-regex-literals.md https://github.com/apple/swift-evolution/blob/main/proposals... gives this example: // A regex for extracting a currency (dollars or pounds) and amount from input // with precisely the form /[$£]\d+\.\d{2}/ let regex = Regex { Capture { /[$£]/ } TryCapture { /\d+/ "." /\d{2}/ } transform: { Amount(twoDecimalPlaces: $0) } } That gives you control over the verbosity. Aside: all of this is nice, but does anybody know whether compilation speed (which can be notoriously bad) improved in the latest Swift beta?
dry_soup 4y ago
I don't understand the syntax of the first example: 'Hello' ' '+ ('world' | 'rulex') To me it looks like there's a + missing after the second single quote
- SonOfLilit 4y ago
  That's "one or more spaces"
- kreetx 4y ago
  The "+" there is most likely the one-or-more quantifier from regular regular expression syntax.
MattPalmer1086 4y ago
Interesting. It's very similar to a regex language I created for byte-oriented regular expressions [0] Similar usability principles: delimitated strings, ignore whitespace, and comments. [0] https://github.com/nishihatapalmer/byteseek/blob/master/syntax.md https://github.com/nishihatapalmer/byteseek/blob/master/synt...
SonOfLilit 4y ago
I've collected the different projects along with a nontrivial syntax example here: https://github.com/SonOfLilit/kleenexp#similar-works https://github.com/SonOfLilit/kleenexp#similar-works - Regular Expressions - very popular, occasionally reads like line noise, backslash for escape (?:What is your (?:name|quest|favourite colour)\?\s?){1,3} - https://github.com/SonOfLilit/kleenexp https://github.com/SonOfLilit/kleenexp - Terse, readable, almost never needs escaping. Python compiler, almost ready VSCode extension, online playground. [1-3 'What is your ' ['name' | 'quest' | 'favourite colour'] '?' [0-1 #space]] - https://github.com/yoav-lavi/melody https://github.com/yoav-lavi/melody - More verbose, supports macros, backslash escapes only for quotes. Rust compiler, babel plugin. Improves with time, getting quite impressive. 1 to 3 of match { "What is your "; either { "name"; "quest"; "favorite color"; } "?"; 0 to 1 of " "; } - https://rulex-rs.github.io/ https://rulex-rs.github.io/ - Very similar to legacy regex syntax, supports macros and number ranges, supports unicode, _amazing_ error messages help convert legacy to new syntax, backslash escapes only for quotes. Rust compiler, as of today no built in way to use outside rust (but they seem to be planning it). ('What is your ' ('name'|'quest'|'favorite colour')'?' [s]){1,3} - https://www.oilshell.org/release/latest/doc/eggex.html https://www.oilshell.org/release/latest/doc/eggex.html Part of a new Unix shell's syntax. Big on composition (macros in kleenexp). Uses backslash for character classes. Production-ready within the shell, not supported elsewhere yet. / 'What is your ' ('name' | 'quest' | 'favorite color') '?' ' '? / - https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes Similar to Eggex, part of Raku (the artist formerly known as Perl 6) - http://verbalexpressions.github.io/ http://verbalexpressions.github.io/ - Embedded DSL, supports 14(!) languages (to some extent? I didn't verify), but don't seem to have syntax for `(a|b)` and `(?:...){3}` const tester = VerEx() .then('What is your ') .either( // this doesn't seem to be implemented yet (?), so I'm proposing a possible syntax VerEx().then('name'), VerEx().then('quest'), VerEx().then('favorite color'), ) .then('?') .maybe(' ') .multiple(3); // not sure this is the correct syntax or how to use it in more complex scenarios, hard to tell from tests and discussions in Issues - There are many more eDSLs, but I will not list them as they are less relevant in my opinion
- draegtun 4y ago
  Also have a look at PARSE in Rebol & Red - http://www.rebol.com/docs/core23/rebolcore-15.html http://www.rebol.com/docs/core23/rebolcore-15.html This is how your example would look like in PARSE rule... 1 3 [{What is your } ["name" | "quest" | "favourite colour"] "?" opt space]
rurban 4y ago
Isn't it strange that the right hand side (the resulting regex) is more readable and precise than its abstract language on the LHS?
- XorNot 4y ago
  This was my first thought too: the example regex's are visually easier to parse then the rulex's. If there's a benefit, I think it needs to be more clearly displayed first - the simple examples are too simple to seem like there's a problem. (also, going as rulex -> regex is a weird way to present things when you're trying to sell "use rulex instead of regex because its simpler").
padjo 4y ago
Feels like if you solve a problem with this you’ll end up with three problems
- yesenadam 4y ago
  How standards proliferate https://xkcd.com/927/ https://xkcd.com/927/
samatman 4y ago
This has an almost uncanny-valley effect, in that yes, this is an improved syntax for regular expression, but that has the effect of highlighting why PEGs are the way forward here. It's great that you can assign variables in Rulex, but they can't reference themselves recursively, so you're still locked out of all the interesting formats. Meanwhile it weakens the "ok but everyone knows how to use regex, it's in the language, and we're just doing a bit of data munging" argument for choosing regex, and you know what? That's a good argument, and it might be the only one.
ThatsAllForNow 4y ago
This might be a really dumb question, but what makes this Portable? Is standard regex portable? I do not think I am understanding what portability is in this case.
- bmn__ 4y ago
  I think the documentation is referring to the point that rulex compile to regex, which are implicitly assumed to be portable across implementations.
- Aloso 4y ago
  When compiling a rulex to a regex, you specify the regex flavor, so the resulting regex is compatible with that regex flavor. This is significant because PCRE, .NET, Ruby, Python, JavaScript, etc. all use slightly different syntax, but you can use the same rulex to target all these different regex engines.
donatj 4y ago
The design of the examples is weird and unintuitive. At the very least should have some sort of clarifying header of “new” and “old” or something. It took me a solid 30 seconds of confusion to realize the regex was the old syntax rather than the new syntax. The arrow points from left to right, and arrows represent progress but that’s backward from their intent. The new syntax should be on the end of the arrow, not the beginning. If this were a regex transcoder and the base were the input syntax and the tip were the output code it could kinda make sense, but best I can tell that is not the case here. The examples read backwards.
- bmn__ 4y ago
  Maybe you missed that rulex compile to regex, so the arrow is pointing in the correct direction under that point of view.
tobyhinloopen 4y ago
Except for ranges and variables, I saw no clear advantage to this. Sure it’s a bit nicer, but it’s also something new you have to learn
Thaxll 4y ago
Lot of work went there to re-invent the weel.
ilyash 4y ago
Here is my take on pattern matching. Tldr: generalize. https://github.com/ngs-lang/ngs/wiki/UPM-Design https://github.com/ngs-lang/ngs/wiki/UPM-Design
oblio 4y ago
Small note to dev: the plural of "regexp" is not "regexp's". It's probably "regexps".
ngvrnd 4y ago
"Now you've got three problems."
flakiness 4y ago
Nice! Can be a Markdown for Regex, meaning nicer (possibly less powerful) replacement of old syntax - If there is enough ecosystem grown.
lozzo 4y ago
This project looks great. Occasionally I need to write a reg expr (JS developer) and I am not that good at it and I always end up googling around. I am thinking it would be much nicer if I could use rulex to generate my reg expr. But is that what it is supposed to be for ? cheers
- Aloso 4y ago
  > But is that what it is supposed to be for? (Author of rulex here) Yes, rulex is compiled to a regex. You can use it with the CLI or the online playground and insert the result into your source code. Eventually, I'd like to have a babel plugin that can transform a rulex`...` literal to the equivalent regex, but we're not there yet.
cphoover 4y ago
Just looked at Rulex briefly... am I the only one that thinks it isn't obviously more readable or easy to understand? Maybe it's because I've taken the time to master regex, but this syntax seems just as opaque as terse regex syntax was to me prior to digging in.
- __david__ 4y ago
  You are not there only one. REs look like gibberish until you understand them. To me Rulex looks like gibberish but the REs they turn into make sense. I don’t think the syntax is inherently more readable, and the whitespace insensitivity + comments feature has been in Perl and Ruby REs for decades.
  - SCLeo 4y ago
    I think the main appeal is its ability to transpile to different languages, so that you can use whitespace insensitiveness + comments in other languages where those features are not supported.
chubot 4y ago
I added this to the Alternative Regex Syntax wiki page with about a dozen similar projects: https://github.com/oilshell/oil/wiki/Alternative-Regex-Syntax https://github.com/oilshell/oil/wiki/Alternative-Regex-Synta... e.g. compare with Melody 3 months ago: https://news.ycombinator.com/item?id=30358554 https://news.ycombinator.com/item?id=30358554 and Oil's Eggex: https://www.oilshell.org/release/latest/doc/eggex.html https://www.oilshell.org/release/latest/doc/eggex.html From a quick glance Rulex looks very similar to Eggex! A difference is that Eggex is embedded in a shell so you can use normal assignment statements to build up subpatterns. And you can also interpolate directly into an 'egrep' or 'awk' command.
- runlevel1 4y ago
  What I'm liking about Rulex is that it's striking a good balance between what's new (mainly readability improvements) and what's familiar. It's close enough to Regex that I could probably get used to it in a few hours. Looking through the items on that list, IMO only Oil's Eggex seems to hit that same sweet spot.
  - chubot 4y ago
    Yes that's my assessment too! Funny thing is that I designed a similar project called "Annex" in Python around 2014. But when I looked back on it after a few years with fresh eyes, it was just too different! It was hard to remember. So Eggex is very similar to the common syntax -- it's actually roughly the same as lex or re2c. I think there's a case to make that we we ALREADY have a good syntax for regexes in lexer generators -- it's just the POSIX/Perl syntax that is bad! The other thing about Annex is that it was a Python library, and it was never worth the extra dependency. (IMO PyPI has a lot of friction.) That is, after writing it, I already knew stdlib re syntax so well it wasn't worth it :) But Eggex is in a shell, which is a context that makes a lot of sense IMO.
pietroppeter 4y ago
Reminds me of https://simple-regex.com/ https://simple-regex.com/ a direct translation of a regex into natural language
TheDesolate0 4y ago
GordonS 4y ago
I don't really get the hate for regular regex syntax. And certainly to my eyes, this doesn't look any better to work with that regex. Take the capturing groups sample, is this really meant to be an improvement?! Rulex: ``` :([s]) :named([.]) ::named ``` Regex: ``` (\s)(?<named>.)\2 ```
- Aloso 4y ago
  Note that rulex replaces the named backreference with a numeric one. Named backreferences are much easier to work with, because you don't have to count capturing groups, which is error-prone and complicates later modifications. An more accurate translation of the above into a (PCRE) regex would be (\s)(?P<named>.)\k<named> However, named backreferences aren't supported in JavaScript. Rulex makes sure that the emitted regex is correct and supported by the target regex flavor.
ricardobeat 4y ago
After looking at all the examples nothing stands out as particularly appealing. It ends up being just a more verbose version of regular expressions, but with less clear semantics, harder to see the underlying constructs. Over the years I've become quite familiar with regexp so maybe I'm just biased, but I'd rather have something like CoffeeScript's block expressions instead, where you can easily group and document each part: https://coffeescript.org/#regexes https://coffeescript.org/#regexes
- point_blank 4y ago
  That was my immediate reaction as well. This doesn't make regular expressions more approachable nor does it appear to bring anything new to the table. > so maybe I'm just biased I read 'transpiled' and immediately dismissed it. I think I have a bigger problem lol ;)
- ZeroGravitas 4y ago
  I think this does that: https://rulex-rs.github.io/docs/language-tour/basics/ https://rulex-rs.github.io/docs/language-tour/basics/ > In Rulex, whitespace is insignificant, except between quotes. This means that we can add spaces and line breaks to make the code look clearer. We can also add comments to explain what the expressions are doing. They start with a # and span until the end of the line:
ZeroGravitas 4y ago
This seems really well thought out. Would a way to list examples be able to be added to this? I guess it might be possible just with comments and a preprocessor but it would be great to have a way to specify a list of matches and non-matches and know that the regex works with them at 'compile' time. Probably works as inline documentation too. And is there a standard file extension and syntax colouring available for editors (yet)?
wsdookadr 4y ago
I had a look at one of their examples https://rulex-rs.github.io/docs/examples/emails/ https://rulex-rs.github.io/docs/examples/emails/ The only part that I can agree to is that it's useful to compose a larger regex from smaller easier-to-understand ones. But I don't see why a new regex engine is required for that. On the other hand I'm probably overlooking some important features that they bring to the table. Maybe I'll come back with more comments if I invest more time looking at Rulex.
skrebbel 4y ago
I really like this. One big benefit of this notation, that I feel is under-highlighted in both this comment page and the Rulex site, is that there are no backslashes in Rulex. A lot of regex usage is inside strings inside another programming language, which typically means double backslash escape messes such as \\\\" and it's just a mess. This makes regexes particularly beginner-unfriendly, because you gotta parse it out at different levels. No backslashes means that you can put a Rulex expression in a string in just about any language, not escape anything, and still get a predictable result. I suspect that a lot of its design actually follows from that requirement. Eg having to quote all character literals/sequences - that's needed because that way, eg a newline can just be "[n]", even inside a character class. (eg ['a'-'z' n]). Also, it looks like you can choose to use either single-quotes or double-quotes. This means that if you use the one style of quotes in your language, you can use the other one in the Rulex expression and still not have to backslash-escape anything.
brokenkebab2 4y ago
Looking at examples, I can't see where's improvement tbh. Both classical regex, and this new kid equally require training to read because they are both terse. Adding whitespaces, and comments doesn't bring novelty - using your PL of choice syntax you can already split regex in smaller chunks for readability.

242 points by thm 4y ago

I want to allow users create and share RegEx's in my website, but I'm afraid there could be some abuse like building a very complex one that would get Javascript stuck or crash. Are there any known vulnerabilities and RegEx sanitization?

love2read 4y ago

When would something like this be necessary?

nyellin 4y ago
Every time you write a non trivial regex! Regexes often contain complex parts that repeat themselves throughout the regex or can be combined into higher level patterns. This adds support for that.
- chii 4y ago
  > Regexes often contain complex parts that repeat themselves throughout the regex or can be combined into higher level patterns. sounds like a parser use case.
  - xdfgh1112 4y ago
    One of the coolest things about Raku/Perl6 was how it made grammars a first-class citizen, like how Perl 5 made regexes a first-class citizen.
- asicsp 4y ago
  If supported, you can use verbose flag and subexpression calls for such cases. From a cursory glance, I found only the numeric `range` support the standout feature I'd want in existing regex support.
- Diris 4y ago
  That reminds me of that nice article about how to do that kind of thing in python [0] The unreasonable effectiveness of f-strings and re.verbose https://news.ycombinator.com/item?id=31457188 https://news.ycombinator.com/item?id=31457188
- bmn__ 4y ago
  You make it sound as if regex don't have this ability, but they already do.
eternityforest 4y ago
Looks like it would be good if you had to use some ancient regex engine. Otherwise, I find regex is extremely easy, since the problems it solves well are very simple, and one can develop it interactively by trial and error, by looking at the highlights in their editor. I could possibly see actually using this if the playground had a similar "Try it on real text instantly" feature, but I've never had a problem with traditional regex+trial and error.
- SonOfLilit 4y ago
  Guessing you were downvoted because of the "ancient" part that has nothing to do with it. Also, many people don't agree with the "regex is fine" part.
- an1sotropy 4y ago
  you may be resigned to trial and error being an essential part of regex development, but the reliance on trial and error should be (1) a signal that something can be improved with how programmers write regexs, and (2) a symptom of a programming ethos that essential things should be difficult. I'm glad that this project (and others like that others have highlighted here) allow us to improve on (1), and I am always encouraging my profession peers in computing to reconsider (2).
SonOfLilit 4y ago
The answer is the same as that for more expressive languages with more ergonomic syntax and less footguns vs C.

SonOfLilit 4y ago

Finally someone else trying to fix regex syntax approaches it from the same angle as mine[1]: a portable syntax that transpiles to plain old regex to get bug-for-bug compatibility but a nicer syntax. Instinctively, I think the syntax didn't go far enough and could be made more readable (of course I'd feel that way!), but the implementation and marketing has me beaten! Kudos! [1] https://github.com/SonOfLilit/kleenexp https://github.com/SonOfLilit/kleenexp

fathyb 4y ago
I recommend checking Melody out if you're into readable regex alternatives: https://github.com/yoav-lavi/melody https://github.com/yoav-lavi/melody
- SonOfLilit 4y ago
  Oooh, it has grown. Nice!
- asicsp 4y ago
  There's also https://github.com/VerbalExpressions https://github.com/VerbalExpressions for various programming languages.
deleted 4y ago
[deleted]

bryanrasmussen 4y ago

Ok, despite being more in the now you have two problems camp when discussing regular expressions I find this interesting enough I should schedule learning it. Question - when compiling I suppose it can turn the regex into the most optimized form - so generation of character classes, lazy quantifiers if pertinent?

matthews2 4y ago

See also: Emacs' structured regexp notation, https://www.gnu.org/software/emacs/manual/html_node/elisp/Rx-Notation.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Rx...

cardanome 4y ago
Oh, I like that one! Very easy to read. Have you ever worked with it and what was you experience? I can see myself using that or even implement it for other languages.
- matthews2 4y ago
  It can end up taking a little longer to write (same with all of these alternate syntaxes), but I think it significantly reduces the WTF-factor when you have to try and read it again in weeks/months/years. I think it's worth it for any non-trivial regexp and I miss it whenever I'm using any other language, so I'm glad that this thread is full of alternatives :)
- natrys 4y ago
  Not OP but I have switched to using rx macro for everything and I never miss regex. I miss rx instead everywhere else. Here are couple of blog posts someone might find useful: https://francismurillo.github.io/2017-03-30-Exploring-Emacs-rx-Macro/ https://francismurillo.github.io/2017-03-30-Exploring-Emacs-... https://karthinks.com/software/bridging-islands-in-emacs-1/ https://karthinks.com/software/bridging-islands-in-emacs-1/
- rcthompson 4y ago
  I haven't used rx in Elisp specifically, but in my work in R, I've started using the rex package almost exclusively for writing regexes. It's the same idea: you build a regex programmatically instead of as a string, so you no longer need to think about syntax, only semantics, because the syntax is the same as the host language.
ur-whale 4y ago
Aside from the parens fetishism common to all lisp-flavored things, it is very nice and readable indeed.

moonchild 4y ago

See also: raku's grammars (https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes).

asojfdowgh 4y ago
can't overstate how nice raku is, bit sharper learning curve but still raku: ^ ( \d+ <?{ $/ <= 255 }> ) ** 4 % '.' $ slow raku: ^ ( @(0..255) ) ** 4 % '.' $ rulex: <% ( :( range '0'-'255' ) '.' ){3} :( range '0'-'255' ) %> regex: ^(?:(0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?)\.)((0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?))$ first one running code in the middle of the regex, to see if it matches a mathematical property, the second, running code to generate a [0|1|2|3|...|254|255] equivalent, which also matches as tightly as the rulex/regex code (e.g. no 00 allowed)
- donatj 4y ago
  Seems like a case where I would just \d+\.\d+\.\d+\.\d+ and validate it in code
  - asojfdowgh 4y ago
    can still do that logic too, without leaving the parsing step, per say /^ (\d+) ** 4 % \. <?{ $/[0].all <= 255 }> $/ or if wanting it really explicit /^ (\d+) \. (\d+) \. (\d+) \. (\d+) <?{ $/.all <= 255 }> $/
- lbriner 4y ago
  None of those seem much easier to understand than the others. I suspect this is a classic example of "let's take something that is really messy in pcre and compare it other more brief equivalents" but I could easily turn the regex into something more readable and I can find more help for the standard regex syntax than I can for any alternative. As other's have said, it is a minor arguably cosmetic improvement and something that is likely easier just to use a visual helper like regex101 to help you construct.

gajus 4y ago

Got to admire someone who is going to invest time to invent a whole new language just to avoid mastering an existing one. Regex is fine. Been using it nearly daily for the last decade. You get used to it. These days, there are tools that make it incredibly easy to write, test, version and document regex, like https://regex101.com/ https://regex101.com/.

SonOfLilit 4y ago
Working with a company that writes hundreds of regexes per day, I can say with confidence: regex syntax is _not_ fine. People with lots of experience make obvious mistakes and lose them in the line noise all the time.
- xdfgh1112 4y ago
  Do you use the /x modifier?
  - SonOfLilit 4y ago
    The regexes are very simple, there's just thousands of them. /x wouldn't help much.
- donatj 4y ago
  What could the job possibly be? Based on next to nothing, sounds like you might be better off building custom parsers?
  - chrisshroba 4y ago
    I'd guess any large tech company on average is writing at least hundreds per day, given the crazy amounts of code being produced.
coldtea 4y ago
>Got to admire someone who is going to invest time to invent a whole new language just to avoid mastering an existing one. Yes, because any existing language is the global optimum, and we should always strive to reuse it, never invent any new one... I mean, what's the problem with stone cavemen-era cart-wheels? Why reinvent the wheel? Not to mention the implying that the only reason one would try to invent a new language is to "not invest time mastering an existing one". I.e. "It's not that this guy tries to improve regular expression languages - it's that he's dumb/lazy to master the standard regex expressions". Never mind that this new language also transpiles into regular regex, so clearly the person who created it has also "mastered the existing one".

miked85 4y ago

While impressive, I'm not sure in what scenario anyone would choose to use this.

tiborsaas 4y ago
Maybe it would be a bit more user friendly to embed it into a product for end users.

raydiatian 4y ago

Saint_Genet 4y ago

To use this you basically have learn regexps anyway, so I can't really see the point.

bryanrasmussen 4y ago
well I guess the point would be: 1. if you know regex but for a particular language, this takes care of having to learn the different variants of regex because it will compile down to the version you want. 2. it is a more approachable syntax, so easier to learn. 3. because it is still a regex syntax given that you probably know a bit of regex already you can get better at it by learning this. any others I forgot?
- benoketamad 4y ago
  Just like sql abstractions. A dead end, useless, unnecessary.

pizza234 4y ago

I'm curious if this does concretely improve regular expressions or not for generic programmers. Overall, to me it seems the classic minor-but-not-radical improvement. The major difference I see is that this is a sort of `/x` modified regex with better readability. This is a significant improvement - the big problem with `/x` is that one then needs to escape/encode whitespace, which typically offsets the improvement. The other significant change is variables, however, when one reaches a certain complexity, IMO they should not use regexes anymore. The other differences (based on the home page presentation) are cosmetic changes of the same semantics. The fundamental assumption is that the main problem of regexes is about syntax rather than semantics. It's true the regexes are somewhat ugly and hard to remember, but, personally, I don't think that's the main problem. It would have probably been nice if regex started with this syntax, but integrating them in a new project is a difficult decision.

ethbr0 4y ago
The only problem with regex is that people try to parse HTML with it. And in that sentence, neither 'regex' nor 'HTML' are the root of the problem. IMHO, ignorance should never be a valid feature request.
- berkes 4y ago
  Are you honestly implying that there are still people who, in all seriousness, use a REGEX to parse HTML? Every programmer I have met, however junior, knows or learns about Zalgo early on. It's like the "foobar" wrt regexps: common knowledge. True, knowing it's a bad idea won't hold people from doing it. I do it. I'll whip up some stupid `:%s/\<?p class="strong"?>/<strong>/g` occasionally in vim, knowing perfectly fine that this regexp won't be solid, but: works for this case. All the while knowing about Zalgo for years (decades?)
  - dmitriid 4y ago
    > Are you honestly implying that there are still people who, in all seriousness, use a REGEX to parse HTML Subsets of it? Yes. See Google's lit-html as an example: https://github.com/lit/lit/blob/main/packages/lit-html/src/lit-html.ts https://github.com/lit/lit/blob/main/packages/lit-html/src/l...

nsajko 4y ago

This is effectively just a nicer syntax than usual for regular expressions, but there are actual areas of semantics where regexp languages are ripe for improvement - most regexp libraries are stuck in the 80s/90s, not keeping up with recent developments in research. For example, most regexp languages basically only have the traditional three operators of alternation, concatenation and closure/iteration. It's not necessary to stop there: the regular languages are closed under various other useful operators. Many of these operators don't even significantly increase the size of the minimal DFA. For example, the complement (as in set/language complement) and reversal/mirror image would be some basic additions that seem like they should be necessary nowadays. Using them can make the regular expression much shorter, nicer and more understandable. For the complement, this should come at no cost for the minimal DFA size. Some other viable operators are: intersection, set difference, merge (AKA shuffle), infiltration (also sometimes known as shuffle), interleaving. Some other possibilites for regexp that could be put to good use more are: * weighted regular expressions: these enable more power in a very elegant way, the idea is that programmers don't want just recognizers, so why limit regexps or finite automata to just that. * JIT. It should be possible to compile regexps with libgccjit or with llvm.

avgcorrection 4y ago
Do operators like complement work with Unicode?
- nsajko 4y ago
  Of course, Unicode is just another alphabet. Specific expressions might be problematic though, depending on the implementation. EDIT: If malicious inputs are something to be worried about, I think a good way to do this would be to check for exceedingly costly blowups of various kinds during/after translation to a finite automaton (so before automaton execution). This would enable having complemented expressions when safe (in lots of cases, at least).
  - joppy 4y ago
    To get something useful and not attackable, the complement operation would need to take the complement inside the set of valid UTF-8 strings rather than the set of all byte-sequences, which is probably doable but I'm not sure how straightforward it would be.

tekknolagi 4y ago

Also notable is Remake, which has Rust bindings: https://github.com/ethanpailes/remake https://github.com/ethanpailes/remake

nsajko 4y ago
The owner of that repo had an interesting regexp-related thesis ("Skip Regex"): https://github.com/ethanpailes/thesis https://github.com/ethanpailes/thesis https://www.proquest.com/openview/e7295f20d24f203900f6291d6dc90772/ https://www.proquest.com/openview/e7295f20d24f203900f6291d6d...

gardaani 4y ago

Swift introduces RegexBuilder to make regex easier, but I think it is a bit too verbose: https://developer.apple.com/documentation/regexbuilder?changes=_2_2 https://developer.apple.com/documentation/regexbuilder?chang...

Someone 4y ago
For simple regexes, Swift has short literals, and (AFAIK) you can mix and match the DSL and the short literals. https://github.com/apple/swift-evolution/blob/main/proposals/0354-regex-literals.md https://github.com/apple/swift-evolution/blob/main/proposals... gives this example: // A regex for extracting a currency (dollars or pounds) and amount from input // with precisely the form /[$£]\d+\.\d{2}/ let regex = Regex { Capture { /[$£]/ } TryCapture { /\d+/ "." /\d{2}/ } transform: { Amount(twoDecimalPlaces: $0) } } That gives you control over the verbosity. Aside: all of this is nice, but does anybody know whether compilation speed (which can be notoriously bad) improved in the latest Swift beta?

dry_soup 4y ago

I don't understand the syntax of the first example: 'Hello' ' '+ ('world' | 'rulex') To me it looks like there's a + missing after the second single quote

SonOfLilit 4y ago
That's "one or more spaces"
kreetx 4y ago
The "+" there is most likely the one-or-more quantifier from regular regular expression syntax.

MattPalmer1086 4y ago

Interesting. It's very similar to a regex language I created for byte-oriented regular expressions [0] Similar usability principles: delimitated strings, ignore whitespace, and comments. [0] https://github.com/nishihatapalmer/byteseek/blob/master/syntax.md https://github.com/nishihatapalmer/byteseek/blob/master/synt...

SonOfLilit 4y ago

I've collected the different projects along with a nontrivial syntax example here: https://github.com/SonOfLilit/kleenexp#similar-works https://github.com/SonOfLilit/kleenexp#similar-works - Regular Expressions - very popular, occasionally reads like line noise, backslash for escape (?:What is your (?:name|quest|favourite colour)\?\s?){1,3} - https://github.com/SonOfLilit/kleenexp https://github.com/SonOfLilit/kleenexp - Terse, readable, almost never needs escaping. Python compiler, almost ready VSCode extension, online playground. [1-3 'What is your ' ['name' | 'quest' | 'favourite colour'] '?' [0-1 #space]] - https://github.com/yoav-lavi/melody https://github.com/yoav-lavi/melody - More verbose, supports macros, backslash escapes only for quotes. Rust compiler, babel plugin. Improves with time, getting quite impressive. 1 to 3 of match { "What is your "; either { "name"; "quest"; "favorite color"; } "?"; 0 to 1 of " "; } - https://rulex-rs.github.io/ https://rulex-rs.github.io/ - Very similar to legacy regex syntax, supports macros and number ranges, supports unicode, _amazing_ error messages help convert legacy to new syntax, backslash escapes only for quotes. Rust compiler, as of today no built in way to use outside rust (but they seem to be planning it). ('What is your ' ('name'|'quest'|'favorite colour')'?' [s]){1,3} - https://www.oilshell.org/release/latest/doc/eggex.html https://www.oilshell.org/release/latest/doc/eggex.html Part of a new Unix shell's syntax. Big on composition (macros in kleenexp). Uses backslash for character classes. Production-ready within the shell, not supported elsewhere yet. / 'What is your ' ('name' | 'quest' | 'favorite color') '?' ' '? / - https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes Similar to Eggex, part of Raku (the artist formerly known as Perl 6) - http://verbalexpressions.github.io/ http://verbalexpressions.github.io/ - Embedded DSL, supports 14(!) languages (to some extent? I didn't verify), but don't seem to have syntax for `(a|b)` and `(?:...){3}` const tester = VerEx() .then('What is your ') .either( // this doesn't seem to be implemented yet (?), so I'm proposing a possible syntax VerEx().then('name'), VerEx().then('quest'), VerEx().then('favorite color'), ) .then('?') .maybe(' ') .multiple(3); // not sure this is the correct syntax or how to use it in more complex scenarios, hard to tell from tests and discussions in Issues - There are many more eDSLs, but I will not list them as they are less relevant in my opinion

draegtun 4y ago
Also have a look at PARSE in Rebol & Red - http://www.rebol.com/docs/core23/rebolcore-15.html http://www.rebol.com/docs/core23/rebolcore-15.html This is how your example would look like in PARSE rule... 1 3 [{What is your } ["name" | "quest" | "favourite colour"] "?" opt space]

rurban 4y ago

Isn't it strange that the right hand side (the resulting regex) is more readable and precise than its abstract language on the LHS?

XorNot 4y ago
This was my first thought too: the example regex's are visually easier to parse then the rulex's. If there's a benefit, I think it needs to be more clearly displayed first - the simple examples are too simple to seem like there's a problem. (also, going as rulex -> regex is a weird way to present things when you're trying to sell "use rulex instead of regex because its simpler").

padjo 4y ago

Feels like if you solve a problem with this you’ll end up with three problems

yesenadam 4y ago
How standards proliferate https://xkcd.com/927/ https://xkcd.com/927/

samatman 4y ago

This has an almost uncanny-valley effect, in that yes, this is an improved syntax for regular expression, but that has the effect of highlighting why PEGs are the way forward here. It's great that you can assign variables in Rulex, but they can't reference themselves recursively, so you're still locked out of all the interesting formats. Meanwhile it weakens the "ok but everyone knows how to use regex, it's in the language, and we're just doing a bit of data munging" argument for choosing regex, and you know what? That's a good argument, and it might be the only one.

ThatsAllForNow 4y ago

This might be a really dumb question, but what makes this Portable? Is standard regex portable? I do not think I am understanding what portability is in this case.

bmn__ 4y ago
I think the documentation is referring to the point that rulex compile to regex, which are implicitly assumed to be portable across implementations.
Aloso 4y ago
When compiling a rulex to a regex, you specify the regex flavor, so the resulting regex is compatible with that regex flavor. This is significant because PCRE, .NET, Ruby, Python, JavaScript, etc. all use slightly different syntax, but you can use the same rulex to target all these different regex engines.

donatj 4y ago

The design of the examples is weird and unintuitive. At the very least should have some sort of clarifying header of “new” and “old” or something. It took me a solid 30 seconds of confusion to realize the regex was the old syntax rather than the new syntax. The arrow points from left to right, and arrows represent progress but that’s backward from their intent. The new syntax should be on the end of the arrow, not the beginning. If this were a regex transcoder and the base were the input syntax and the tip were the output code it could kinda make sense, but best I can tell that is not the case here. The examples read backwards.

bmn__ 4y ago
Maybe you missed that rulex compile to regex, so the arrow is pointing in the correct direction under that point of view.

tobyhinloopen 4y ago

Except for ranges and variables, I saw no clear advantage to this. Sure it’s a bit nicer, but it’s also something new you have to learn

Thaxll 4y ago

Lot of work went there to re-invent the weel.

ilyash 4y ago

Here is my take on pattern matching. Tldr: generalize. https://github.com/ngs-lang/ngs/wiki/UPM-Design https://github.com/ngs-lang/ngs/wiki/UPM-Design

oblio 4y ago

Small note to dev: the plural of "regexp" is not "regexp's". It's probably "regexps".

ngvrnd 4y ago

"Now you've got three problems."

flakiness 4y ago

Nice! Can be a Markdown for Regex, meaning nicer (possibly less powerful) replacement of old syntax - If there is enough ecosystem grown.

lozzo 4y ago

This project looks great. Occasionally I need to write a reg expr (JS developer) and I am not that good at it and I always end up googling around. I am thinking it would be much nicer if I could use rulex to generate my reg expr. But is that what it is supposed to be for ? cheers

Aloso 4y ago
> But is that what it is supposed to be for? (Author of rulex here) Yes, rulex is compiled to a regex. You can use it with the CLI or the online playground and insert the result into your source code. Eventually, I'd like to have a babel plugin that can transform a rulex`...` literal to the equivalent regex, but we're not there yet.

cphoover 4y ago

Just looked at Rulex briefly... am I the only one that thinks it isn't obviously more readable or easy to understand? Maybe it's because I've taken the time to master regex, but this syntax seems just as opaque as terse regex syntax was to me prior to digging in.

__david__ 4y ago
You are not there only one. REs look like gibberish until you understand them. To me Rulex looks like gibberish but the REs they turn into make sense. I don’t think the syntax is inherently more readable, and the whitespace insensitivity + comments feature has been in Perl and Ruby REs for decades.
- SCLeo 4y ago
  I think the main appeal is its ability to transpile to different languages, so that you can use whitespace insensitiveness + comments in other languages where those features are not supported.

chubot 4y ago

I added this to the Alternative Regex Syntax wiki page with about a dozen similar projects: https://github.com/oilshell/oil/wiki/Alternative-Regex-Syntax https://github.com/oilshell/oil/wiki/Alternative-Regex-Synta... e.g. compare with Melody 3 months ago: https://news.ycombinator.com/item?id=30358554 https://news.ycombinator.com/item?id=30358554 and Oil's Eggex: https://www.oilshell.org/release/latest/doc/eggex.html https://www.oilshell.org/release/latest/doc/eggex.html From a quick glance Rulex looks very similar to Eggex! A difference is that Eggex is embedded in a shell so you can use normal assignment statements to build up subpatterns. And you can also interpolate directly into an 'egrep' or 'awk' command.

runlevel1 4y ago
What I'm liking about Rulex is that it's striking a good balance between what's new (mainly readability improvements) and what's familiar. It's close enough to Regex that I could probably get used to it in a few hours. Looking through the items on that list, IMO only Oil's Eggex seems to hit that same sweet spot.
- chubot 4y ago
  Yes that's my assessment too! Funny thing is that I designed a similar project called "Annex" in Python around 2014. But when I looked back on it after a few years with fresh eyes, it was just too different! It was hard to remember. So Eggex is very similar to the common syntax -- it's actually roughly the same as lex or re2c. I think there's a case to make that we we ALREADY have a good syntax for regexes in lexer generators -- it's just the POSIX/Perl syntax that is bad! The other thing about Annex is that it was a Python library, and it was never worth the extra dependency. (IMO PyPI has a lot of friction.) That is, after writing it, I already knew stdlib re syntax so well it wasn't worth it :) But Eggex is in a shell, which is a context that makes a lot of sense IMO.

pietroppeter 4y ago

Reminds me of https://simple-regex.com/ https://simple-regex.com/ a direct translation of a regex into natural language

TheDesolate0 4y ago

GordonS 4y ago

I don't really get the hate for regular regex syntax. And certainly to my eyes, this doesn't look any better to work with that regex. Take the capturing groups sample, is this really meant to be an improvement?! Rulex: ``` :([s]) :named([.]) ::named ``` Regex: ``` (\s)(?<named>.)\2 ```

Aloso 4y ago
Note that rulex replaces the named backreference with a numeric one. Named backreferences are much easier to work with, because you don't have to count capturing groups, which is error-prone and complicates later modifications. An more accurate translation of the above into a (PCRE) regex would be (\s)(?P<named>.)\k<named> However, named backreferences aren't supported in JavaScript. Rulex makes sure that the emitted regex is correct and supported by the target regex flavor.

ricardobeat 4y ago

After looking at all the examples nothing stands out as particularly appealing. It ends up being just a more verbose version of regular expressions, but with less clear semantics, harder to see the underlying constructs. Over the years I've become quite familiar with regexp so maybe I'm just biased, but I'd rather have something like CoffeeScript's block expressions instead, where you can easily group and document each part: https://coffeescript.org/#regexes https://coffeescript.org/#regexes

point_blank 4y ago
That was my immediate reaction as well. This doesn't make regular expressions more approachable nor does it appear to bring anything new to the table. > so maybe I'm just biased I read 'transpiled' and immediately dismissed it. I think I have a bigger problem lol ;)
ZeroGravitas 4y ago
I think this does that: https://rulex-rs.github.io/docs/language-tour/basics/ https://rulex-rs.github.io/docs/language-tour/basics/ > In Rulex, whitespace is insignificant, except between quotes. This means that we can add spaces and line breaks to make the code look clearer. We can also add comments to explain what the expressions are doing. They start with a # and span until the end of the line:

ZeroGravitas 4y ago

This seems really well thought out. Would a way to list examples be able to be added to this? I guess it might be possible just with comments and a preprocessor but it would be great to have a way to specify a list of matches and non-matches and know that the regex works with them at 'compile' time. Probably works as inline documentation too. And is there a standard file extension and syntax colouring available for editors (yet)?

wsdookadr 4y ago

I had a look at one of their examples https://rulex-rs.github.io/docs/examples/emails/ https://rulex-rs.github.io/docs/examples/emails/ The only part that I can agree to is that it's useful to compose a larger regex from smaller easier-to-understand ones. But I don't see why a new regex engine is required for that. On the other hand I'm probably overlooking some important features that they bring to the table. Maybe I'll come back with more comments if I invest more time looking at Rulex.

skrebbel 4y ago

I really like this. One big benefit of this notation, that I feel is under-highlighted in both this comment page and the Rulex site, is that there are no backslashes in Rulex. A lot of regex usage is inside strings inside another programming language, which typically means double backslash escape messes such as \\\\" and it's just a mess. This makes regexes particularly beginner-unfriendly, because you gotta parse it out at different levels. No backslashes means that you can put a Rulex expression in a string in just about any language, not escape anything, and still get a predictable result. I suspect that a lot of its design actually follows from that requirement. Eg having to quote all character literals/sequences - that's needed because that way, eg a newline can just be "[n]", even inside a character class. (eg ['a'-'z' n]). Also, it looks like you can choose to use either single-quotes or double-quotes. This means that if you use the one style of quotes in your language, you can use the other one in the Rulex expression and still not have to backslash-escape anything.

brokenkebab2 4y ago

Looking at examples, I can't see where's improvement tbh. Both classical regex, and this new kid equally require training to read because they are both terse. Adding whitespaces, and comments doesn't bring novelty - using your PL of choice syntax you can already split regex in smaller chunks for readability.