4 ms·
I'm especially interested in how sequence searching and matching work in libraries like this. Seq has a "match" statement for this task, which implements ACGT c
by hoytech 7y ago
I'm especially interested in how sequence searching and matching work in libraries like this. Seq has a "match" statement for this task, which implements ACGT characters and _ for a single wildcard base and "..." for multiple wildcard bases, and a recursive matching system I haven't quite grokked yet.
Personally I'm more comfortable with a regular expression syntax, so would prefer "." and ".*". Actually, even better than "." is "N" from the IUPAC notation: https://en.wikipedia.org/wiki/Nucleic_acid_notation https://en.wikipedia.org/wiki/Nucleic_acid_notation
The IUPAC notation is nice because it standardises "character classes" for working with nucleic acid sequences. For example, "B" is "[CGT]".
I wrote a module a while ago for searching nucleic acid sequences with regexps: https://metacpan.org/pod/Bio::Regexp https://metacpan.org/pod/Bio::Regexp
When working with sequences there are a bunch of things to think about that aren't really obvious from other types of data (or at least weren't obvious to me!)
Exhaustive search: Often a regexp can match in many ways, and most regexp systems don't provide a way to get a complete list of them all. Fortunately there is the Regexp::Exhaustive perl module which is what I used. The way this module works is pretty awesome. It adds a special "FAIL" directive to the end of a pattern, so that the match can be recorded and artificially failed, triggering the regexp engine's backtracking mechanism to back up and find the next match (if any).
Reverse complements: Because DNA is double-stranded (well, usually... this is biology after all) there is a "complementary" pattern on the first strand that corresponds to the pattern you are interested in on the other strand. You almost always need to search for both. And what's more, DNA is directional so you actually need to search in the reverse direction for this complementary pattern. You can either reverse and complement your sequence (which seq has a special ~ operator for, neat!) and search again, or reverse and complement the search pattern itself, assemble a single combined regexp (Regexp::Assemble module), and do a single scan over the data, which is what Bio::Regexp does.
Circular DNA: Some DNA (plasmids) are actually circular in shape, meaning the start is connected to the end. So a comprehensive search needs to check for cases where the desired patterns span the arbitrary location selected as the "start" in your sequence.
- dalke 7y ago"Often a regexp can match in many ways ..." When I worked in bioinformatics, some 20 years ago, I implemented a SWISS-PROT to regex converter. I ended up having to ask SWISS-PROT was if certain patterns were meant to be greedy or lazy, since the documentation wasn't clear. I've since forgotten the answer, but I have a vague memory that they hadn't really considered that multiple interpretations were possible, so they probably expected greedy matches.
- hoytech 7y agoInteresting, thanks!
- jerven 7y agoParsing the flatfile of Swiss-Prot in detail is the best way to lose your sanity ;) I know because that is part of my day job. It's gotten better but I highly recommend you either use the RDF or XML. The flatfile looks easy but is an endless source of bugs and urgent changes.
- dalke 7y agoI did say it was 20 years ago. ;) Since then I've been working in cheminformatics. Which has its own forms of insanity.
- jerven 7y agoCan I just say that these day's you can search Swiss-Prot with InchiKey's due to our integration with rhea (https://www.uniprot.org/news/2018/12/05/release https://www.uniprot.org/news/2018/12/05/release)
- dekhn 7y agoregular expression matching is not heavily used in biology for a wide range of reasons. Nearly all approaches now are k-mer/probabilistic. Using backtracking is just a path to performance problems.
- hoytech 7y agoWell searching/counting/etc k-mers can be done with regexps, but I certainly agree that regexp-based searching is somewhat niche compared to more generally useful similarity searches like BLAST. One comment though: backtracking is not inherently an issue with regexps. Some implementations will never backtrack, and others that do (like perl's) are nowadays pretty good at avoiding the exponential worst-cases that I think you're referring to, for all but the most pathological cases.