3 ms·

Great thanks! So I think there are two things about your code which are different. You are getting the names and values of the spam function with show_me() ? My

Great thanks! So I think there are two things about your code which are different. You are getting the names and values of the spam function with show_me() ? My version gets the next level up which is the names of the variables passed in to spam. I believe I tried to use some of the same standard functions at first. But this had problems when a function is called over more than one line of source text, and some other limitations. You don't necessarily get back what you expect without properly going into the source and parsing. I'll take another look though since I might have just missed something obvious. My Python isn't /that/ strong. Completely agree with your list comprehension suggestion. The list comprehensions are crazy by design to troll monoglot Python people who hold a false belief about readability of code just being identical to language syntax. If I handed you that code without the comments, and without sensible variable names. Which is frequently what I do get handed to me by people I work with. I've found a myth persists about Python being inherently readable (the opposite of Perl!), and many scientists who don't have a programming background just go with this idea especially as it gets them off the hook of learning to program well. With parsing and grammars I am still more happy having a parse fail and tell me what the garbage data was and where. It's very rare any code would be able to recover from completely arbitrary and unknown problems with a file. With exceptions being thrown you might at least start processing the bit of the file you already have and just set a state that shows the whole thing didn't run. Asking a language to do more for you feels a bit much. Your point about validation overhead I'm not convinced by since you have to check the characters for a record start/end state anyway. Plus it just means garbage data gets even deeper into your program! At some point you have to check it's valid. Fail fast. Checking a small set of match states isn't going to kill performance if the grammar/regex engine is any good. One of the ways I'm getting around this in my Perl 6 libraries is just having a thread that deals with file IO, parsing and object creation. Then a users main program can just deal with the stream of sequence objects as they come in.

dalke 12y ago
Getting the variable names used for the caller is of course much harder, and meaningless for a call like f(1+g(b)). py.test, which made a more serious attempt at handling the f() case, gave up on that approach, preferring to rewrite the original code behind the scenes rather than infer it backwards. So you're a troll. Thank you for letting me know. By trolling the uninformed you seemingly have no meaningful point other than to point your finger and laugh at the views of others. This action runs counter to your stated goal of wanting scientists to be better programmers. What good do you think it will do to the world to troll people using your novice Python skills? You do realize that when someone searches for "Matt Oates", "bioinformatics" and "Python" they are likely to find this page in the future, and learn about your love of trolling, right? I do not want to live in a troll world, just like I don't want to live in a world where people use false and undercutting statements disguised as joviality. Instead, I teach scientists how to be better at the programming they need to do good science. Your last paragraph tells me that you have very little experience in writing parsers. In a record oriented format, and with hand-written parsers, resynchronization to the record boundary is easy. In Perl's it's almost trivial, because nearly all record-oriented formats have a boundary which can be handled with $/ ; FASTA is an exception but still easy to handle. On the other hand, my point is that it's much harder to resynchronize with a context-free grammar. Here's an example of a failing case with my strict parser. HAS2_CHICK was the only record in SWISS-PROT 39 with a DT line of: DT 30-MAY-2000 (REL. 39, Created) This failed because the spec said that the "REL." was only supposed to be a "Rel." My parser failed. Every other parser basically said (by lack of full validation) that they weren't interested in that field, and ignored it. You can certainly condemn the entire SWISS-PROT 39 release as "garbage" for having a single record which didn't match the spec. Effectively everyone who wants the data will disagree with you. You can also train your parser to ignore those details, and retrain it for every release, though you'll find it increasingly difficult to accept the format as it changes across 5 different database releases. Here's another example. Some of the GENBANK records included a qualifier of "/primer", which wasn't in the specification. My parsers failed, because it validated the qualifier names. And you know what? No one really cares about that failure, so I had to tweak my parser to allow the non-standard name. The same with EMBL: the ID line is supposed to have the molecule type (one of "DNA|RNA|circular DNA|circular RNA"). In actuality? Record S40706 had a type of 'XXX'... and so again, my validating parser stopped, when in reality almost no one cares. As these are further examples of simple differences causing a failure within a record, your bringing up "completely arbitrary and unknown problems with a file" suggests to me that you have little real desire to engage in my point, and rather prefer to troll me by diverting the topic. In case I am wrong, please assume that Perl's $/ or a similarly simple method is sufficient to determine the record boundary and resynchronize. The underlying issue is that a database format spec changes after almost every release, so a validating parser ends up having to handle a family of formats, not one. Which is much easier for a line-oriented parser to do, since those formats are written with line-oriented hand-written parsers in mind, and usually updated in such a way as to not break those parsers. When you find out that your nice, elegant, validating context-free parser breaks after nearly every release, due to details that are irrelevant to what you care about, then you might conclude that the rest of the world is a bunch of ignorant fools who can't program; or reject the idea of a validating parser for this sort of data and switch to a line-oriented one; or provide a recovery mechanism to determine what to do with the records which couldn't be parsed, and continue if requested. I do the latter now. (It's also difficult to use a CFG-based parser that doesn't know about record boundaries to parse files that don't fit into memory even though each record does. I don't know if that's still an issue in bioinformatics these days, but it certainly is in cheminformatics. In any case, a mix of simple record extraction and a CFG-based grammar suffice, excepting for records containing an entire genome, which is outside the scope of this sort of task, and for PIR, which had header and footer components to the file.) Regarding performance, don't trust your beliefs or mine. Measure it. Empiricism trumps faith. Try benchmarking a hand-written, line-oriented FASTA parser against your grammar. I wrote one just now in Python. The regex which used "[^>\n].*\n" was about 2% faster than checking the [specific sequence letters]. A 2% here and a 2% there and those validations quickly incur noticeable overhead. By comparison, my line-oriented parser took about 1/2 the time of the regex-based parser. Of course, Perl has a different runtime environment and will have different conclusions. Then again, there's already a history in Perl of variable stringency levels for improved performance. Swissknife (doi: 10.1093/bioinformatics/15.9.771) was written as a lazy parser because Perl5 wasn't fast enough for full validation and parsing of each record. Partial evaluation gave a 4x performance improvement when full validation wasn't needed.
- raiph 12y ago
  > my point is that it's much harder to resynchronize with a context-free grammar. So, don't use a context free grammar. [1] Use something else, eg a PEG. [2] Not only are Perl 6 grammars PEGs, not CFGs, but individual rules are closures (of arbitrary code) as well as input matchers so one can write arbitrary code to handle anything that happens during a parse. [3] As a very trivial example, in a grammar I wrote for parsing a CPAN control file, I added a rule for handling unexpected field types. It prints a warning to stderr and then keeps on going. [4] Imo Perl 6 is, among its many strengths, an innovative parsing tool with at least three big problems right now: * The grammar engine is slow. [5] * Cat is likely a post Perl 6.0.0 feature. (Among other things, Cat is for enabling practical parsing of arbitrarily long and arbitrarily buffered data streams without having to tweak userland code.) [6] * Macro, slang and parse tweaking features are still evolving. [7] [1] http://en.wikipedia.org/wiki/Context-free_grammar http://en.wikipedia.org/wiki/Context-free_grammar [2] http://en.wikipedia.org/wiki/Parsing_expression_grammar http://en.wikipedia.org/wiki/Parsing_expression_grammar [3] http://en.wikipedia.org/wiki/Perl_6_rules http://en.wikipedia.org/wiki/Perl_6_rules [4] https://gist.github.com/raiph/63508a5f4ade000acc7f https://gist.github.com/raiph/63508a5f4ade000acc7f See the `data-item:dst_GENERIC-MATCH` rule; note the embedded `{ warn ... }` closure. [5] http://www.reddit.com/r/perl6/comments/2ec2hd/rakudo_perl_6_and_moarvm_performance_advances_in/ http://www.reddit.com/r/perl6/comments/2ec2hd/rakudo_perl_6_... [6] http://perlcabal.org/syn/S03.html http://perlcabal.org/syn/S03.html which includes: "The Cat type allows you to have an infinitely extensible string. ... Then a Regex can be used against it as if it were an ordinary string ... The Cat interface allows the regex to match element boundaries with the <,> assertion ... Strings, arrays, lists, sequences, captures, and tree nodes can all be pattern matched by regexes or by signatures more or less interchangeably." [7] http://perl6advent.wordpress.com/2014/12/17/day-17-three-strange-loops/ http://perl6advent.wordpress.com/2014/12/17/day-17-three-str...

dalke 12y ago

Getting the variable names used for the caller is of course much harder, and meaningless for a call like f(1+g(b)). py.test, which made a more serious attempt at handling the f() case, gave up on that approach, preferring to rewrite the original code behind the scenes rather than infer it backwards. So you're a troll. Thank you for letting me know. By trolling the uninformed you seemingly have no meaningful point other than to point your finger and laugh at the views of others. This action runs counter to your stated goal of wanting scientists to be better programmers. What good do you think it will do to the world to troll people using your novice Python skills? You do realize that when someone searches for "Matt Oates", "bioinformatics" and "Python" they are likely to find this page in the future, and learn about your love of trolling, right? I do not want to live in a troll world, just like I don't want to live in a world where people use false and undercutting statements disguised as joviality. Instead, I teach scientists how to be better at the programming they need to do good science. Your last paragraph tells me that you have very little experience in writing parsers. In a record oriented format, and with hand-written parsers, resynchronization to the record boundary is easy. In Perl's it's almost trivial, because nearly all record-oriented formats have a boundary which can be handled with $/ ; FASTA is an exception but still easy to handle. On the other hand, my point is that it's much harder to resynchronize with a context-free grammar. Here's an example of a failing case with my strict parser. HAS2_CHICK was the only record in SWISS-PROT 39 with a DT line of: DT 30-MAY-2000 (REL. 39, Created) This failed because the spec said that the "REL." was only supposed to be a "Rel." My parser failed. Every other parser basically said (by lack of full validation) that they weren't interested in that field, and ignored it. You can certainly condemn the entire SWISS-PROT 39 release as "garbage" for having a single record which didn't match the spec. Effectively everyone who wants the data will disagree with you. You can also train your parser to ignore those details, and retrain it for every release, though you'll find it increasingly difficult to accept the format as it changes across 5 different database releases. Here's another example. Some of the GENBANK records included a qualifier of "/primer", which wasn't in the specification. My parsers failed, because it validated the qualifier names. And you know what? No one really cares about that failure, so I had to tweak my parser to allow the non-standard name. The same with EMBL: the ID line is supposed to have the molecule type (one of "DNA|RNA|circular DNA|circular RNA"). In actuality? Record S40706 had a type of 'XXX'... and so again, my validating parser stopped, when in reality almost no one cares. As these are further examples of simple differences causing a failure within a record, your bringing up "completely arbitrary and unknown problems with a file" suggests to me that you have little real desire to engage in my point, and rather prefer to troll me by diverting the topic. In case I am wrong, please assume that Perl's $/ or a similarly simple method is sufficient to determine the record boundary and resynchronize. The underlying issue is that a database format spec changes after almost every release, so a validating parser ends up having to handle a family of formats, not one. Which is much easier for a line-oriented parser to do, since those formats are written with line-oriented hand-written parsers in mind, and usually updated in such a way as to not break those parsers. When you find out that your nice, elegant, validating context-free parser breaks after nearly every release, due to details that are irrelevant to what you care about, then you might conclude that the rest of the world is a bunch of ignorant fools who can't program; or reject the idea of a validating parser for this sort of data and switch to a line-oriented one; or provide a recovery mechanism to determine what to do with the records which couldn't be parsed, and continue if requested. I do the latter now. (It's also difficult to use a CFG-based parser that doesn't know about record boundaries to parse files that don't fit into memory even though each record does. I don't know if that's still an issue in bioinformatics these days, but it certainly is in cheminformatics. In any case, a mix of simple record extraction and a CFG-based grammar suffice, excepting for records containing an entire genome, which is outside the scope of this sort of task, and for PIR, which had header and footer components to the file.) Regarding performance, don't trust your beliefs or mine. Measure it. Empiricism trumps faith. Try benchmarking a hand-written, line-oriented FASTA parser against your grammar. I wrote one just now in Python. The regex which used "[^>\n].*\n" was about 2% faster than checking the [specific sequence letters]. A 2% here and a 2% there and those validations quickly incur noticeable overhead. By comparison, my line-oriented parser took about 1/2 the time of the regex-based parser. Of course, Perl has a different runtime environment and will have different conclusions. Then again, there's already a history in Perl of variable stringency levels for improved performance. Swissknife (doi: 10.1093/bioinformatics/15.9.771) was written as a lazy parser because Perl5 wasn't fast enough for full validation and parsing of each record. Partial evaluation gave a 4x performance improvement when full validation wasn't needed.

raiph 12y ago
> my point is that it's much harder to resynchronize with a context-free grammar. So, don't use a context free grammar. [1] Use something else, eg a PEG. [2] Not only are Perl 6 grammars PEGs, not CFGs, but individual rules are closures (of arbitrary code) as well as input matchers so one can write arbitrary code to handle anything that happens during a parse. [3] As a very trivial example, in a grammar I wrote for parsing a CPAN control file, I added a rule for handling unexpected field types. It prints a warning to stderr and then keeps on going. [4] Imo Perl 6 is, among its many strengths, an innovative parsing tool with at least three big problems right now: * The grammar engine is slow. [5] * Cat is likely a post Perl 6.0.0 feature. (Among other things, Cat is for enabling practical parsing of arbitrarily long and arbitrarily buffered data streams without having to tweak userland code.) [6] * Macro, slang and parse tweaking features are still evolving. [7] [1] http://en.wikipedia.org/wiki/Context-free_grammar http://en.wikipedia.org/wiki/Context-free_grammar [2] http://en.wikipedia.org/wiki/Parsing_expression_grammar http://en.wikipedia.org/wiki/Parsing_expression_grammar [3] http://en.wikipedia.org/wiki/Perl_6_rules http://en.wikipedia.org/wiki/Perl_6_rules [4] https://gist.github.com/raiph/63508a5f4ade000acc7f https://gist.github.com/raiph/63508a5f4ade000acc7f See the `data-item:dst_GENERIC-MATCH` rule; note the embedded `{ warn ... }` closure. [5] http://www.reddit.com/r/perl6/comments/2ec2hd/rakudo_perl_6_and_moarvm_performance_advances_in/ http://www.reddit.com/r/perl6/comments/2ec2hd/rakudo_perl_6_... [6] http://perlcabal.org/syn/S03.html http://perlcabal.org/syn/S03.html which includes: "The Cat type allows you to have an infinitely extensible string. ... Then a Regex can be used against it as if it were an ordinary string ... The Cat interface allows the regex to match element boundaries with the <,> assertion ... Strings, arrays, lists, sequences, captures, and tree nodes can all be pattern matched by regexes or by signatures more or less interchangeably." [7] http://perl6advent.wordpress.com/2014/12/17/day-17-three-strange-loops/ http://perl6advent.wordpress.com/2014/12/17/day-17-three-str...