10 ms·
Delighted to discover this kind of breakthrough is based on a gnarly regular expression: https://github.com/Svensson-Lab/pro-hormone-predictor/blob/main/FastaP
by thom 2y ago
Delighted to discover this kind of breakthrough is based on a gnarly regular expression:
https://github.com/Svensson-Lab/pro-hormone-predictor/blob/main/FastaParser.py#L32 https://github.com/Svensson-Lab/pro-hormone-predictor/blob/m...
- mkoubaa 2y agoBeautiful yet terrifying. I'm glad that regex isn't something I'll ever have to touch.
- jagged-chisel 2y agoThe good news is it’s read only- if you’re reading for understanding, you’re doing it wrong :-D
- dleeftink 2y agoIt's not that bad. AutoRegex[0] and regex gen [1] make it more accessible than ever. [0]: https://www.autoregex.xyz/ https://www.autoregex.xyz/ [1]: https://regex-generator.olafneumann.org https://regex-generator.olafneumann.org
- janfoeh 2y agoHeh, I just fed autoregex a regex from one of my projects, and it simply times out. It comforts me to know that billion dollar LLMs have to chew on those just as much as I do.
- dleeftink 2y agoA proper litmus test, what's causing the hang in your case? Regex to me, is pattern finding and abstraction taken to the extreme. I like the challenge
- janfoeh 2y agoGood question. The regex I tried is for extracting amounts in EUR and USD: / (?<=^|[ \t]) (?<currency_prefix_with_space> (?<currency_prefix> €|EUR|\$|USD ) [ \t]? )? (?<number> (?<integral> -? \d{1,3} (?:[\.,]\d{3}|\d*) ) [\.,] (?<fraction> \d{2,3} ) ) (?<currency_postfix_with_space> [ \t]? (?<postfix_or_ending> (?<currency_postfix> €|EUR|\$|USD ) ) | (?<ending> [ \t]|$|\n ) )/x
- dleeftink 2y agoI'd imagine many nested named capturing groups may trip even the best automated system! I do like the solution though. I would've probably approached it differently, trying to first get the 'inverted' match (i.e. ignore anything that isn't a currency-like pattern) and refine from there. A bit like this one I did a while back, to parse garbled strings that may occur after OCR [0]. I imagine the approach does not translate fully, because it's pattern extraction rather than validation. [0]: https://observablehq.com/@dleeftink/never-go-nuts https://observablehq.com/@dleeftink/never-go-nuts
- janfoeh 2y agoThanks for sharing! I have to admit I do not have the necessary brain cycles to spare today, but OCR processing is indeed of interest to me, and I will take a more in-depth look in the upcoming days. The idea of an exclusionary approach sounds interesting as well. I'll have to think about that a bit.
- sgc 2y agoRegex is one of my favorite things to work with. It's surgery. I love having a very specific problem and a very sharp knife. What I hate is chasing down language or library bugs, or undocumented limitations. Of course I would hate it if I had never used it extensively. Its terseness is harder if you only have to pick it up occasionally.
- jpc0 2y agoRemember to get a license for your code https://regexlicensing.org/ https://regexlicensing.org/
- jasonjayr 2y agoOn the list of disasters, they forgot "parsing html": https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
- deleted 2y ago[deleted]
- joseda-hg 2y agoHuh, I tried sharing the Zalgo answer here, but it didn't quite work, never knew HN wouldn't render those kinds of characters
- qingcharles 2y agoIt did, but I kinda went over the top with it once and dang had to tell me off, so it might be they added filtering.
- genewitch 2y agoo, there's your problem. Only half is supposed to go over the top. The rest is supposed to go below.
- sumtechguy 2y agoI think it comes down to the syntax being used to make regex. It is borderline an assembly language level of making them. Better language tools would make it easier for people to grok. Basically instead of using chars like ?^()$ and the like lets use words people understand.
- aranchelk 2y agoIt’s not just the syntax. Regex isn’t directly composable (unless you just mash together strings). They cannot define nested structures, I.e. no recursion. They can’t maintain context. We don’t need better language tools. Better parsers can, and already have, been implemented in libraries. I’d have a look at parser combinators.
- dur-randir 2y ago(?R) is very much recursion in there.
- johnisgood 2y ago> They can’t maintain context. Is true though, or perhaps "state"? I know I had to come up with an algorithm because regexp alone couldn't do what I wanted (not even advanced features like lookahead, lookbehind, etc.)
- davidwritesbugs 2y agoI played with parser combinators in Elm and found it very useful c.f. regexes - the issues were mainly down to the Elm implementation. Not sure how effective they are in Rust or Go
- rstuart4133 2y agoAs @aranchelk said LL and LR grammars are literally that. They allow you to assign names to regexs and compose them. Libraries like python's lrparsing [0] let you assign regex's (aka tokens) to variables, then build grammars by combining them using python expressions. For example: while_statement = Keyword('while') + '(' + expression + ')' + block statement = while_statement | .... block = Token('{') + Repeat(statement + ';') + Token('}') These grammars are more complex, with more rules you have to follow, but also more checking is done when they are compiled. They tend to mostly work once they do compile. So they are what you asked for, but there is no free lunch. On the down side, lrparsing is pure python so it's slower than python's inbuilt regex's. [0] https://lrparsing.sourceforge.net/ https://lrparsing.sourceforge.net/
- badc0ffee 2y agoI'm comfortable with it (having previously been paid to write Perl for several years), but what I don't like is how many other people I have worked with aren't, and just blindly paste in (totally incorrect) regexes from Stack Overflow.
- UltraSane 2y agoWriting one that does exactly what you want is very satisfying. And with Python you can have named groups extracted into a dict and it just feels like magic.
- karolinepauls 2y ago[dead]
- aitchnyu 2y ago[flagged]
- xrisk 2y agoI’m not sure what this brings to the conversation. Are you able to attest to the correctness of what Sonnet claims?
- JKCalhoun 2y agoI think we can be chill about it. For someone on the sidelines, mildly interested in the topic, an AI summary of the regex might be useful — and the person getting downvotes is saving some of us the time.
- SketchySeaBeast 2y agoYes, it might be misinformation, but it's convenient misinformation, is that the idea? And yes, it's true that a source like wikipedia could be too, but that has to withstand scrutiny to remain on the site, no such checks or balances in AI.
- andershaig 2y agoIt's clearly labeled as the output of Sonnet 3.7, not truth. We all need to apply our own critical analysis of anything we read, whether it's claimed to be from an LLM or from Wikipedia. The possibility of inaccuracy isn't a reason to withhold comment.
- SketchySeaBeast 2y agoSo what do we do with that information? If I apply a critical framework around interpreting the LLM output, the answer is to reject it for being both not necessarily true but also knowing that the LLM isn't even trying to be correct, it's strictly trying to produce convincing sentences. What value does a link to a source that's not held to any standard to be informative do? Seems a waste of everyone's time to me.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- choxi 2y agoIs this AGI?
- ch4s3 2y agooh god, can you even imagine?
- ricardonunez 2y agoAfter using a regex that creates AGI researcher will see the world like Neo sees the matrix.
- cropcirclbureau 2y agoAsk Deepseek to generate a regex for detecting python import statements.
- crazygringo 2y agoPlease, if you write regexes like these, comment them... A great way to do it is to split them up by concatenating them across a bunch of lines, and put a brief explanation at the end of each non-obvious part (to the right, on the same line). Plus that also lets you indent within nested parentheses, making it that much more understandable. I'm baffled when I come across a file like this where the code itself is heavily commented, but a gnarly regex is not. Regexes are not strings, they are code -- and with their syntax, they need comments even more.
- loginx 2y agoOne thing I always appreciated about CoffeeScript was its [heregex syntax](https://coffeescript-cookbook.github.io/chapters/regular_expressions/heregexes https://coffeescript-cookbook.github.io/chapters/regular_exp...), which made regexes clear and easy to document. Many CoffeeScript constructs were adopted into JavaScript and TypeScript, but unfortunately, heregexes weren't among them.
- yesbabyyes 2y agoI agree. I'm pretty sure jashkenas got it from Ruby, which got it from Perl, with the `/x` flag (for extended regular expressions). Later languages have added support for the `x` flag, including C# and Rust. There is also a stage 1 proposal for JavaScript: https://github.com/tc39/proposal-regexp-x-mode https://github.com/tc39/proposal-regexp-x-mode
- rtkwe 2y agoregex101.com is a lifesaver for parsing esoteric regexes after you've forgotten what they're supposed to do.
- flobosg 2y agoWelcome to the fascinating world of academic code!
- ge96 2y agoIs that an anti-pattern to put the comment inside the function vs above
- TheRealPomax 2y agoFunction documentation goes inside functions in Python, explicitly making them self-documenting. That can then be automatically picked up by any other code (like tooling for automatically generating API documentation) formalized as PEP 257, https://peps.python.org/pep-0257 https://peps.python.org/pep-0257 Python will literally assign the content of your docstring (a triple-quoted string at the start of a function) to a special double underscored ("dunder") property called `__doc__` that any code can access; not just docs generators, but your own code as well (pretty dang useful for generating on-the-fly help output!). Which also means that if you run into weird behaviour where a function doesn't do what you think it should do, you can just fire up the REPL, import that function, type `print(function_name.__doc__)` and presto, you have the documentation right there, specifically for the exact version you're using. You don't even need to leave your IDE if it comes with an integrated terminal.
- ge96 2y agointeresting TIL too bad we don't have auto doc gen (fastapi what I'm working with atm) edit: not saying it can't be done but our org isn't using auto docs
- TheRealPomax 2y agoYeah, it's one of those "Once you're used to writing doc strings, you have no reason to ever use anything else" but if you don't use them, it can be prohibitively laborious to move whatever alternative you're using over to "PEP-compliant". (lots of fun dunder functions in Python that make metaprogramming a lot easier, but someone needs to tell you about)
- disgruntledphd2 2y agoFluent Python covers a bunch of this stuff.
- nilstycho 2y agoI got lost at (?R). IIUC, this is a feature of the "regex" module that "tries to match the entire regex recursively". Fun.
- deleted 2y ago[deleted]
- hedora 2y agoThat’s only the tip of the iceberg. Here’s “How Perl saved the human genome project” from 1996: https://bioperl.org/articles/How_Perl_saved_human_genome.html https://bioperl.org/articles/How_Perl_saved_human_genome.htm... (And of course all the articles about excel renaming genes, and then people trying to clean up the mess…)
- vlovich123 2y ago> The study would not have been possible without the use of artificial intelligence to weed through dozens of proteins in a class called prohormones. I see we’re being fast and loose with the term “artificial intelligence”?
- cosmie 2y agoCould be that they relied on AI to generate the gnarly regex ¯\_(ツ)_/¯
- ddalex 2y agoAny sufficiently advanced regex is indistinguishable from AI.
- wslh 2y agoEven simple regex ones, since we don't know if they were created by a human or an AI.
- readthenotes1 2y agoI'm trying to think of a funny way to cite Clark's Law, but failing...
- Izkata 2y agoRegexes occasionally get called "black magic", and there is is an inverse of Clarke's Third Law: Any sufficiently advanced magic is indistinguishable from technology.
- xeonmc 2y agoI think you meant “any sufficiently commoditized magic is indistinguishable from technology”, or “any sufficiently analyzed magic is indistinguishable from science.”
- J_Shelby_J 2y ago
- sunrunner 2y agoNeat. I always thought Advent of Code 2015's Day 19 (Medicine for Rudolph) was just an exercise in parsing and validation of production rules against a grammar, and any use of regex in that puzzle solution was purely coincidental ;)
- y33t 2y agoObligatory xkcd: https://xkcd.com/208/ https://xkcd.com/208/
- rendang 2y agoA very early one at that