4 ms·

Sure but the nice thing is you can just add an extra catch all to the Grammar. Or better yet create a new Grammar for that specific sequencing machine and overr

Sure but the nice thing is you can just add an extra catch all to the Grammar. Or better yet create a new Grammar for that specific sequencing machine and override the token. For example just adding: rule sequence:sym<seq> { <-[\>]>+ } Would mean you can just grab anything without worrying about if it's correct. Because a tonne of people abuse FASTA doesn't mean you shouldn't at first attempt to enforce something valid. The nice part of that is you know within the parser if you have clean data or something with a badly defined character like * in DNA when it should just be X or N or one of the other confusion characters. Also it's an advent post not some formal public codebase ;P The focus was on showing some of the more obscure features of Perl 6 that aren't like other languages. Python can look as crazy as you like, it's just the majority of people using it fall into the camp of avoiding the crazy. For example here's (https://gist.github.com/MattOates/6195743 https://gist.github.com/MattOates/6195743) some crazy Python I've written. The six line list comprehension is especially readable :P

coriny 12y ago
Sorry, I was just pointing it out in case for anyone copying the code as they are both very common cases - especially dots as gap characters. The Grammar system does look quite nice. Parsing is definitely a Perl forte - though I seriously hope bioinformatics in general will continue to abandon these crazy formats that can't be read easily by humans or machine.
- Ultimatt 12y ago
  Yeah I would rather all the awful formats would go away. But the process of removing them tends to just birth new awful formats into existence :(
  - coriny 12y ago
    I'm increasingly of the opinion, after having to deal with biologist data format hackery for many years (they have no shame!), that every biologist should be taught relational database normalisation. That would make them actually stop and think about the structure of the data they're recording and how to make it computationally tractable. Even more, they should not even start their experiment until they've cleared their data recording structure with a bioinformatician. Then we ban them from using flat files. Related tables or nothing. I've wasted too much time, and they're not competent enough to work with JSON/XML/Excel.
dalke 12y ago
There are a couple of problems with starting with a strict approach. I predict the performance is going to be worse than something which allows (almost) all characters, because of validation overhead. More seriously, I wrote a full grammar based parser engine for Biopython along these lines. The Achilles heel was error recovery. Suppose record #123456 in a database contains a syntax error (I can list dozens I found, like a copyright date of 19995 breaking my four-digit definition, or SWISS-PROT record with the copyright lines in the wrong spot.) How do I have the parser ignore that record, report the problem, and continue parsing with the next record? Error recovery in context free grammars is hard, but doable. Then how do I say that "if the ID is XYZ and the version number is 12 then use this work-around because that release had a bug which has since been fixed"? In practice and unlike a computer language, the format spec ends up being less important than the actual data, so you end up with lots of tweaks to handle vendor errors. P.S.: Wouldn't names = [SQLiteMagician.types.get( type( d ).__name__) for d in data[0]] types = ["NONE" if t == "NoneType" else t for t in names] have been less crazy, and to be preferred? Also, and I'm not sure what that code is trying to do, but sys._getframe() and linecache from the standard library should help, as might this example, which I think does what you want without using the ast module: import sys def show_me(): caller = sys._getframe(1) f_code = caller.f_code names = f_code.co_varnames[:f_code.co_argcount] local_vars = caller.f_locals for i, name in enumerate(names): print repr(name), "=", repr(local_vars[name]) def spam(x, y, z=4): a = "%s %s %s" % (x, y, z) show_me() spam("Hello", 2) This gives as output: 'x' = 'Hello' 'y' = 2 'z' = 4
- dalke 12y ago
  To add some more background for grammar-based bioinformatics parsers, take a look at SRS (Sequence Retrieval System) from the early 1990s, by Etzold and Argos. It was the last generation of purpose-built database systems, before off-the-shelf object-relational systems took over, and used a set of grammars in a declarative language to parse and index flat files.
- Ultimatt 12y ago
  Great thanks! So I think there are two things about your code which are different. You are getting the names and values of the spam function with show_me() ? My version gets the next level up which is the names of the variables passed in to spam. I believe I tried to use some of the same standard functions at first. But this had problems when a function is called over more than one line of source text, and some other limitations. You don't necessarily get back what you expect without properly going into the source and parsing. I'll take another look though since I might have just missed something obvious. My Python isn't /that/ strong. Completely agree with your list comprehension suggestion. The list comprehensions are crazy by design to troll monoglot Python people who hold a false belief about readability of code just being identical to language syntax. If I handed you that code without the comments, and without sensible variable names. Which is frequently what I do get handed to me by people I work with. I've found a myth persists about Python being inherently readable (the opposite of Perl!), and many scientists who don't have a programming background just go with this idea especially as it gets them off the hook of learning to program well. With parsing and grammars I am still more happy having a parse fail and tell me what the garbage data was and where. It's very rare any code would be able to recover from completely arbitrary and unknown problems with a file. With exceptions being thrown you might at least start processing the bit of the file you already have and just set a state that shows the whole thing didn't run. Asking a language to do more for you feels a bit much. Your point about validation overhead I'm not convinced by since you have to check the characters for a record start/end state anyway. Plus it just means garbage data gets even deeper into your program! At some point you have to check it's valid. Fail fast. Checking a small set of match states isn't going to kill performance if the grammar/regex engine is any good. One of the ways I'm getting around this in my Perl 6 libraries is just having a thread that deals with file IO, parsing and object creation. Then a users main program can just deal with the stream of sequence objects as they come in.

coriny 12y ago

Sorry, I was just pointing it out in case for anyone copying the code as they are both very common cases - especially dots as gap characters. The Grammar system does look quite nice. Parsing is definitely a Perl forte - though I seriously hope bioinformatics in general will continue to abandon these crazy formats that can't be read easily by humans or machine.

Ultimatt 12y ago
Yeah I would rather all the awful formats would go away. But the process of removing them tends to just birth new awful formats into existence :(
- coriny 12y ago
  I'm increasingly of the opinion, after having to deal with biologist data format hackery for many years (they have no shame!), that every biologist should be taught relational database normalisation. That would make them actually stop and think about the structure of the data they're recording and how to make it computationally tractable. Even more, they should not even start their experiment until they've cleared their data recording structure with a bioinformatician. Then we ban them from using flat files. Related tables or nothing. I've wasted too much time, and they're not competent enough to work with JSON/XML/Excel.

dalke 12y ago

There are a couple of problems with starting with a strict approach. I predict the performance is going to be worse than something which allows (almost) all characters, because of validation overhead. More seriously, I wrote a full grammar based parser engine for Biopython along these lines. The Achilles heel was error recovery. Suppose record #123456 in a database contains a syntax error (I can list dozens I found, like a copyright date of 19995 breaking my four-digit definition, or SWISS-PROT record with the copyright lines in the wrong spot.) How do I have the parser ignore that record, report the problem, and continue parsing with the next record? Error recovery in context free grammars is hard, but doable. Then how do I say that "if the ID is XYZ and the version number is 12 then use this work-around because that release had a bug which has since been fixed"? In practice and unlike a computer language, the format spec ends up being less important than the actual data, so you end up with lots of tweaks to handle vendor errors. P.S.: Wouldn't names = [SQLiteMagician.types.get( type( d ).__name__) for d in data[0]] types = ["NONE" if t == "NoneType" else t for t in names] have been less crazy, and to be preferred? Also, and I'm not sure what that code is trying to do, but sys._getframe() and linecache from the standard library should help, as might this example, which I think does what you want without using the ast module: import sys def show_me(): caller = sys._getframe(1) f_code = caller.f_code names = f_code.co_varnames[:f_code.co_argcount] local_vars = caller.f_locals for i, name in enumerate(names): print repr(name), "=", repr(local_vars[name]) def spam(x, y, z=4): a = "%s %s %s" % (x, y, z) show_me() spam("Hello", 2) This gives as output: 'x' = 'Hello' 'y' = 2 'z' = 4

dalke 12y ago
To add some more background for grammar-based bioinformatics parsers, take a look at SRS (Sequence Retrieval System) from the early 1990s, by Etzold and Argos. It was the last generation of purpose-built database systems, before off-the-shelf object-relational systems took over, and used a set of grammars in a declarative language to parse and index flat files.
Ultimatt 12y ago
Great thanks! So I think there are two things about your code which are different. You are getting the names and values of the spam function with show_me() ? My version gets the next level up which is the names of the variables passed in to spam. I believe I tried to use some of the same standard functions at first. But this had problems when a function is called over more than one line of source text, and some other limitations. You don't necessarily get back what you expect without properly going into the source and parsing. I'll take another look though since I might have just missed something obvious. My Python isn't /that/ strong. Completely agree with your list comprehension suggestion. The list comprehensions are crazy by design to troll monoglot Python people who hold a false belief about readability of code just being identical to language syntax. If I handed you that code without the comments, and without sensible variable names. Which is frequently what I do get handed to me by people I work with. I've found a myth persists about Python being inherently readable (the opposite of Perl!), and many scientists who don't have a programming background just go with this idea especially as it gets them off the hook of learning to program well. With parsing and grammars I am still more happy having a parse fail and tell me what the garbage data was and where. It's very rare any code would be able to recover from completely arbitrary and unknown problems with a file. With exceptions being thrown you might at least start processing the bit of the file you already have and just set a state that shows the whole thing didn't run. Asking a language to do more for you feels a bit much. Your point about validation overhead I'm not convinced by since you have to check the characters for a record start/end state anyway. Plus it just means garbage data gets even deeper into your program! At some point you have to check it's valid. Fail fast. Checking a small set of match states isn't going to kill performance if the grammar/regex engine is any good. One of the ways I'm getting around this in my Perl 6 libraries is just having a thread that deals with file IO, parsing and object creation. Then a users main program can just deal with the stream of sequence objects as they come in.