8 ms·

How Lisp-family languages facilitate building bioinformatics applications

118 points by abrax3141 10y ago

arca_vorago 10y ago
What an awesome paper! It was during my time as a sysadmin at a DNA lab I became really enamoured with lisp (and emacs), and functional, homoiconic languages as a whole, for many of the reasons listed. Great to read them all and more articulated so well. BioLisp would be a great project.
yomly 10y ago
Surprised that Racket didn't get a mention in the paper, especially given the authors' interest in building DSLs EDIT: (I consider Racket separate from Scheme)
cixin 10y ago
Using lisp always seems attractive. However, in Bioinformatics, the reality is that many users are still using Perl. It would be get to have them migrate to, modestly more readable languages. Outside of scripting, alignment and other performance sensitive code is usually written in C (or C++). As with many academic projects, the code quality unfortunately is quite low.
- pjmlp 10y ago
  Java and .NET are also used a lot.
  - collyw 10y ago
    Python more (though I have had to refactor completely incomprehensible Python written by bioinformacians).
    - pjmlp 10y ago
      Depends on the company. We have some customers with zero lines of Python code, but they do use R and Tableau. All the software used to talk to the devices and do ETL processing or graphical analysis is then written in Java or .NET, depending on the department and their set of OSes.
      - deleted 10y ago
        [deleted]
- throwanem 10y ago
  > many users are still using Perl. It would be [hard] to have them migrate I spent a year as a staff member of a genomics institute, and researchers occasionally came to me for help getting the local sui generis tooling stack set up and working sanely. (Well, I say "stack"; jwz's bookcase-from-mashed-potatoes metaphor hastens to mind.) Having seen the miseries they went through, I don't think it would necessarily have to be all that hard to make migration look good, especially to a language whose syntax is other than wildly irregular and whose performance is other than usually abysmal.
- yomly 10y ago
  >modestly more readable languages. This is definitely all relative, I would argue lisp is easier to read given that the philosophy of its syntax is that there is no syntax... Though if you've only ever read classical languages I can see how lisp would seem alien
  - throwanem 10y ago
    Lisp someone else wrote is harder to read than Perl you wrote, but any Lisp is easier to read than Perl someone else wrote.
    - wtbob 10y ago
      My experience with Perl was that me-a-week-ago counts as someone else. That's not flippant — it's an accurate reflection of my experience that Perl tends to be write-only. There are some people who do write beautiful Perl, but they are few and far between (again, in my experience).
- jghn 10y ago
  I've been in the field since 2001. I have seen exactly two perl files in that time
  - jhbadger 10y ago
    It really depends on the subfield. Sequence-based genomics is very highly Perl-dependent even today. BioPerl is pretty much the standard library for that even though BioPython, etc. are beginning to take over. Other subfields such as differential sequence abundance which are more about the statistics rather than the raw sequence tend to be R-centered thanks to Bioconductor.
    - jghn 10y ago
      I think by your definitions there are sub-subfields then ;) In my neck of the woods sequence based stuff has java & c doing a lot of the heavy lifting.
- acomjean 10y ago
  I work in a lab (academic/genetics). We have a public facing website so when we develop tools they go there. So we're using: perl, java, R, python, php apart from the stuff we compile from c. Stringy code thats hard to decipher can be written in any language I have discovered. My take, Perl is kinda on the way out, though it was used extensively. BioPerl is a nice package. Biologist like R, it pretty quick and behaves like they aren't programming. It can make graphs nicely. Python seems to be the go to compromise. BioPerl is a really nice package. But people start to want to use it for big things and to get it performing adequately requires a lot. There is confusion from the researchers that scares them away: between python 2/3 numby, pypy. BioPython packages are pretty excellent. If it needs to be fast and crunch large sets of data (fairly common) the tool is in C or C++. We should start using Rust.. We use Php (Silex) to deliver a front end and some quick database lookup and display. Its replacing perl for this.
lispm 10y ago
Ira Kalet (RIP, 1944-2015) wrote a book using Lisp for the examples: Principles of Biomedical Informatics, Second Edition, 2013 https://www.amazon.com/Principles-Biomedical-Informatics-Second-Kalet/dp/0124160190/ https://www.amazon.com/Principles-Biomedical-Informatics-Sec...
torrent-of-ions 10y ago
When I started working in bioinformatics I was excited to use Lisp for my work. I wrote a considerable amount of library code and programs at first. But I always found resistance when it came to other people using my programs and even more if the program was to be published and released for general use. There is a large amount of really, really bad Perl code still in use in biology. Trust me when I say to those outside of the field, I've seen things you people wouldn't believe. I try to discourage Perl use between it makes it so easy for biologists to write terrible code. I've found a compromise in Python. People will happily use it because it's trendy and ubiquitous. And I think it makes it easier to write OK code. But I miss using Lisp. The style of programming when you have a REPL is perfect for this kind of thing. The ability to quickly write prototypes which can later be improved is an advantage which cannot be stressed enough. C is fantastic when you have a good idea of the algorithms and data structures you want to implement and are concerned with implementing them efficiently. But that's rarely the case when writing bioinformatics code. I need to know if the damn thing works first before I spend time making it work fast. And why would you not use a language where both things are equally easy? I'm happy to see this paper because it means that I have a way to justify my usage of Lisp in the future. Big thanks to the authors.
- open_bear 10y ago
  I think that IPython is one of the best tools for scientists. Jupyter Notebooks that allow reproducible results, a lot of data-wrangling libraries, visualizations, distributed computing. I wish more scientists knew of IPython.
  - svwobwbou 10y ago
    yes - and the thing that makes python approachable is the syntax and standard formatting. I love LISP, I'm not confused or annoyed by the parens, but many people are. LISP would be more popular if there was a good (browser-based) hybrid structural editor with the keyboard-navigation familiarity of a text editor, in a notebook format where the LISP code is displayed like Python (the parens hidden, and handled automatically, but serialized as normal LISP code behind the scenes). This is not that hard to do, and we're nearly there: * http://gorilla-repl.org/ http://gorilla-repl.org/ * https://github.com/kovasb/session https://github.com/kovasb/session (Also my own early abandoned experiments: http://celeriac.net/iiiiioiooooo/ http://celeriac.net/iiiiioiooooo/ http://celeriac.net/iiiiioiooooo/public/ http://celeriac.net/iiiiioiooooo/public/ http://celeriac.net/iiiiioiooooo-dom/ http://celeriac.net/iiiiioiooooo-dom/ http://celeriac.net/io/public/ http://celeriac.net/io/public/)
vt100 10y ago
How about Hy? It's a Lisp that runs on the Python interpreter, and interoperates with Python. It's not extremely mature yet, but it is really cool!
- goatlover 10y ago
  It's pretty wild using numpy or pandas in Lisp. Hy is definitely cool.
agumonkey 10y ago
I asked Bioinformatics MOOC teachers about non python/java language in the field, naming common lisp or haskell. I got a large NOPE from them, saying it was mostly for performance reason. I specifically asked about the paradigm too, because the MOOC spent a large amount of time talking about control flags and temporary variables, that to me obscured the overall logic.. but alas.
mybrid 10y ago
Usability matters. Functional versus procedural programming is first and foremost a usability problem for developers. The two are interchangeable as far as being Turing complete. I just spent a week at a Jakolb Nielson's education course on usability. I asked the executive vice-president at the company if they had branched into software development itself, sadly the answer is still no. Functional programming for the majority of developers is not usable. The article mentions they are are targeting DSL, domain specific languages. Which is fine. For example, Haskell has seems to have found a boutique community in async, MQ world. But, from a usability perspective one is severely limiting ones pool of developers choosing a functional language. It is fascinating from a human nature perspective that people who think in functional programming are brains are wired differently from the procedural programmer. There is very visceral reaction when either camp is asked to program in the manner they find least usable. Ultimately I think compilers will get to a development stage where independent of functional programming or procedural programming approaches the same optimized code gets implemented under the hood. Postgres was originally written in LISP. Ultimately Stonebraker had to make the call that if Postgres was going to get adopted widely, LISP had to go and it was rewritten in C/C++ before being open sourced. As a point of research you all might want to look into the Postgres experience.
- DonaldFisk 10y ago
  The original paper is here: http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf There are other cases where an original Lisp implementation has been rewritten: ViaWeb and Reddit. I think the reason is that the teams which took over the projects were unfamiliar with Lisp. My experience is different, though I should mention that I'm a sole developer. I'm roughly equally experienced in C and Lisp, but find I'm far more productive in Lisp. My first language was FORTRAN. Unlike C, Lisp is memory safe and you don't have to manage the memory yourself, interactive, strongly typed, you never need to write a parser, the built in symbol and list types are incredibly useful, and there are fewer "gotchas". Lisp is multi-paradigm rather than purely functional, so like the Algol derivatives it has loops and destructive assignment.
swuecho 10y ago
Not try to be negative. But Perl is still used more widely than Lisp in bioinformatics. or lisp can be neglected in bioinformatics compared to Perl. Check the status of BioPython, BioPerl. BioLisp? not even exists.
- DonaldFisk 10y ago
  It does exist. It's now called BioBIKE. https://en.wikipedia.org/wiki/BioBIKE https://en.wikipedia.org/wiki/BioBIKE http://biobike.csbc.vcu.edu http://biobike.csbc.vcu.edu
jxy 10y ago
on average, the Lisp programs ran significantly faster than the C/C++ programs and much faster than the Java programs (mean runtimes were 41 s for Lisp versus 165 s for C/C++). The only thing I can say is that their C/C++ code has serious problems.
- lucidguppy 10y ago
  Perhaps they didn't optimize.
jonathanyc 10y ago
As a Lisp fan myself, I wish they would show an example of how WITH-GENES or MAP-GENES encouraged encapsulation vs. objects or passing closures.
- abrax3141 10y ago
  Bunch of examples here: http://nar.oxfordjournals.org/content/37/suppl_2/W28.full-text-lowres.pdf http://nar.oxfordjournals.org/content/37/suppl_2/W28.full-te...
- abrax3141 10y ago
  The section of the paper where this is mentioned offers a novel (at least to me) argument in support of homoiconicity, to wit, that you build DSLs directly into the base language, which (it is asserted) makes the DSL development process more flexible (or something), and that this is important for 'living' domains, where you're trying to work out the domain semantics, like biology. So, you're not really so much building a specialized language for X, as slowly turning (almost in the sense of wood turning, as on a lathe) Lisp into X-Lang.
  - Volt 10y ago
    A concrete example.
dangom 10y ago
"Gat [3] compared the run times, development times and memory usage of 16 programs written by 14 programmers in Lisp, C/C ++ and Java. Development times for the Lisp programs ranged from 2 to 8.5 h, compared with 2 to 25 h for C/C ++ and 4 to 63 h for Java (programmer experience alone does not account for the differences). The Lisp programs were also significantly shorter than the other programs." Could selection bias be skewing these results? Proficient Lisp programmers can certainly create shorter and faster programs with Lisp. Who would ever contest that? Average programmers, on the other hand, can probably develop in a similar amount of time and write faster programs with C++ (given the amount of libraries and information available - in comparison to Lisp - and the ubiquity of tooling).
chmaynard 10y ago
This is obviously an opinion piece disguised as a scholarly article. I'm not sure what the hidden agenda is, and the HN comments don't clarify matters much. For example, this nonsense statement reveals the author's bias: "Clojure is a rising star language in the modern software development community." Huh?
- swuecho 10y ago
  in my opinion, better to publish as a blog post.

118 points by abrax3141 10y ago

What an awesome paper! It was during my time as a sysadmin at a DNA lab I became really enamoured with lisp (and emacs), and functional, homoiconic languages as a whole, for many of the reasons listed. Great to read them all and more articulated so well. BioLisp would be a great project.

yomly 10y ago

Surprised that Racket didn't get a mention in the paper, especially given the authors' interest in building DSLs EDIT: (I consider Racket separate from Scheme)

cixin 10y ago

Using lisp always seems attractive. However, in Bioinformatics, the reality is that many users are still using Perl. It would be get to have them migrate to, modestly more readable languages. Outside of scripting, alignment and other performance sensitive code is usually written in C (or C++). As with many academic projects, the code quality unfortunately is quite low.

pjmlp 10y ago
Java and .NET are also used a lot.
- collyw 10y ago
  Python more (though I have had to refactor completely incomprehensible Python written by bioinformacians).
  - pjmlp 10y ago
    Depends on the company. We have some customers with zero lines of Python code, but they do use R and Tableau. All the software used to talk to the devices and do ETL processing or graphical analysis is then written in Java or .NET, depending on the department and their set of OSes.
    - deleted 10y ago
      [deleted]
throwanem 10y ago
> many users are still using Perl. It would be [hard] to have them migrate I spent a year as a staff member of a genomics institute, and researchers occasionally came to me for help getting the local sui generis tooling stack set up and working sanely. (Well, I say "stack"; jwz's bookcase-from-mashed-potatoes metaphor hastens to mind.) Having seen the miseries they went through, I don't think it would necessarily have to be all that hard to make migration look good, especially to a language whose syntax is other than wildly irregular and whose performance is other than usually abysmal.
yomly 10y ago
>modestly more readable languages. This is definitely all relative, I would argue lisp is easier to read given that the philosophy of its syntax is that there is no syntax... Though if you've only ever read classical languages I can see how lisp would seem alien
- throwanem 10y ago
  Lisp someone else wrote is harder to read than Perl you wrote, but any Lisp is easier to read than Perl someone else wrote.
  - wtbob 10y ago
    My experience with Perl was that me-a-week-ago counts as someone else. That's not flippant — it's an accurate reflection of my experience that Perl tends to be write-only. There are some people who do write beautiful Perl, but they are few and far between (again, in my experience).
jghn 10y ago
I've been in the field since 2001. I have seen exactly two perl files in that time
- jhbadger 10y ago
  It really depends on the subfield. Sequence-based genomics is very highly Perl-dependent even today. BioPerl is pretty much the standard library for that even though BioPython, etc. are beginning to take over. Other subfields such as differential sequence abundance which are more about the statistics rather than the raw sequence tend to be R-centered thanks to Bioconductor.
  - jghn 10y ago
    I think by your definitions there are sub-subfields then ;) In my neck of the woods sequence based stuff has java & c doing a lot of the heavy lifting.
acomjean 10y ago
I work in a lab (academic/genetics). We have a public facing website so when we develop tools they go there. So we're using: perl, java, R, python, php apart from the stuff we compile from c. Stringy code thats hard to decipher can be written in any language I have discovered. My take, Perl is kinda on the way out, though it was used extensively. BioPerl is a nice package. Biologist like R, it pretty quick and behaves like they aren't programming. It can make graphs nicely. Python seems to be the go to compromise. BioPerl is a really nice package. But people start to want to use it for big things and to get it performing adequately requires a lot. There is confusion from the researchers that scares them away: between python 2/3 numby, pypy. BioPython packages are pretty excellent. If it needs to be fast and crunch large sets of data (fairly common) the tool is in C or C++. We should start using Rust.. We use Php (Silex) to deliver a front end and some quick database lookup and display. Its replacing perl for this.

lispm 10y ago

Ira Kalet (RIP, 1944-2015) wrote a book using Lisp for the examples: Principles of Biomedical Informatics, Second Edition, 2013 https://www.amazon.com/Principles-Biomedical-Informatics-Second-Kalet/dp/0124160190/ https://www.amazon.com/Principles-Biomedical-Informatics-Sec...

torrent-of-ions 10y ago

When I started working in bioinformatics I was excited to use Lisp for my work. I wrote a considerable amount of library code and programs at first. But I always found resistance when it came to other people using my programs and even more if the program was to be published and released for general use. There is a large amount of really, really bad Perl code still in use in biology. Trust me when I say to those outside of the field, I've seen things you people wouldn't believe. I try to discourage Perl use between it makes it so easy for biologists to write terrible code. I've found a compromise in Python. People will happily use it because it's trendy and ubiquitous. And I think it makes it easier to write OK code. But I miss using Lisp. The style of programming when you have a REPL is perfect for this kind of thing. The ability to quickly write prototypes which can later be improved is an advantage which cannot be stressed enough. C is fantastic when you have a good idea of the algorithms and data structures you want to implement and are concerned with implementing them efficiently. But that's rarely the case when writing bioinformatics code. I need to know if the damn thing works first before I spend time making it work fast. And why would you not use a language where both things are equally easy? I'm happy to see this paper because it means that I have a way to justify my usage of Lisp in the future. Big thanks to the authors.

open_bear 10y ago
I think that IPython is one of the best tools for scientists. Jupyter Notebooks that allow reproducible results, a lot of data-wrangling libraries, visualizations, distributed computing. I wish more scientists knew of IPython.
- svwobwbou 10y ago
  yes - and the thing that makes python approachable is the syntax and standard formatting. I love LISP, I'm not confused or annoyed by the parens, but many people are. LISP would be more popular if there was a good (browser-based) hybrid structural editor with the keyboard-navigation familiarity of a text editor, in a notebook format where the LISP code is displayed like Python (the parens hidden, and handled automatically, but serialized as normal LISP code behind the scenes). This is not that hard to do, and we're nearly there: * http://gorilla-repl.org/ http://gorilla-repl.org/ * https://github.com/kovasb/session https://github.com/kovasb/session (Also my own early abandoned experiments: http://celeriac.net/iiiiioiooooo/ http://celeriac.net/iiiiioiooooo/ http://celeriac.net/iiiiioiooooo/public/ http://celeriac.net/iiiiioiooooo/public/ http://celeriac.net/iiiiioiooooo-dom/ http://celeriac.net/iiiiioiooooo-dom/ http://celeriac.net/io/public/ http://celeriac.net/io/public/)

vt100 10y ago

How about Hy? It's a Lisp that runs on the Python interpreter, and interoperates with Python. It's not extremely mature yet, but it is really cool!

goatlover 10y ago
It's pretty wild using numpy or pandas in Lisp. Hy is definitely cool.

agumonkey 10y ago

I asked Bioinformatics MOOC teachers about non python/java language in the field, naming common lisp or haskell. I got a large NOPE from them, saying it was mostly for performance reason. I specifically asked about the paradigm too, because the MOOC spent a large amount of time talking about control flags and temporary variables, that to me obscured the overall logic.. but alas.

mybrid 10y ago

Usability matters. Functional versus procedural programming is first and foremost a usability problem for developers. The two are interchangeable as far as being Turing complete. I just spent a week at a Jakolb Nielson's education course on usability. I asked the executive vice-president at the company if they had branched into software development itself, sadly the answer is still no. Functional programming for the majority of developers is not usable. The article mentions they are are targeting DSL, domain specific languages. Which is fine. For example, Haskell has seems to have found a boutique community in async, MQ world. But, from a usability perspective one is severely limiting ones pool of developers choosing a functional language. It is fascinating from a human nature perspective that people who think in functional programming are brains are wired differently from the procedural programmer. There is very visceral reaction when either camp is asked to program in the manner they find least usable. Ultimately I think compilers will get to a development stage where independent of functional programming or procedural programming approaches the same optimized code gets implemented under the hood. Postgres was originally written in LISP. Ultimately Stonebraker had to make the call that if Postgres was going to get adopted widely, LISP had to go and it was rewritten in C/C++ before being open sourced. As a point of research you all might want to look into the Postgres experience.

DonaldFisk 10y ago
The original paper is here: http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf There are other cases where an original Lisp implementation has been rewritten: ViaWeb and Reddit. I think the reason is that the teams which took over the projects were unfamiliar with Lisp. My experience is different, though I should mention that I'm a sole developer. I'm roughly equally experienced in C and Lisp, but find I'm far more productive in Lisp. My first language was FORTRAN. Unlike C, Lisp is memory safe and you don't have to manage the memory yourself, interactive, strongly typed, you never need to write a parser, the built in symbol and list types are incredibly useful, and there are fewer "gotchas". Lisp is multi-paradigm rather than purely functional, so like the Algol derivatives it has loops and destructive assignment.

swuecho 10y ago

Not try to be negative. But Perl is still used more widely than Lisp in bioinformatics. or lisp can be neglected in bioinformatics compared to Perl. Check the status of BioPython, BioPerl. BioLisp? not even exists.

DonaldFisk 10y ago
It does exist. It's now called BioBIKE. https://en.wikipedia.org/wiki/BioBIKE https://en.wikipedia.org/wiki/BioBIKE http://biobike.csbc.vcu.edu http://biobike.csbc.vcu.edu

jxy 10y ago

on average, the Lisp programs ran significantly faster than the C/C++ programs and much faster than the Java programs (mean runtimes were 41 s for Lisp versus 165 s for C/C++). The only thing I can say is that their C/C++ code has serious problems.

lucidguppy 10y ago
Perhaps they didn't optimize.

jonathanyc 10y ago

As a Lisp fan myself, I wish they would show an example of how WITH-GENES or MAP-GENES encouraged encapsulation vs. objects or passing closures.

abrax3141 10y ago
Bunch of examples here: http://nar.oxfordjournals.org/content/37/suppl_2/W28.full-text-lowres.pdf http://nar.oxfordjournals.org/content/37/suppl_2/W28.full-te...
abrax3141 10y ago
The section of the paper where this is mentioned offers a novel (at least to me) argument in support of homoiconicity, to wit, that you build DSLs directly into the base language, which (it is asserted) makes the DSL development process more flexible (or something), and that this is important for 'living' domains, where you're trying to work out the domain semantics, like biology. So, you're not really so much building a specialized language for X, as slowly turning (almost in the sense of wood turning, as on a lathe) Lisp into X-Lang.
- Volt 10y ago
  A concrete example.

dangom 10y ago

"Gat [3] compared the run times, development times and memory usage of 16 programs written by 14 programmers in Lisp, C/C ++ and Java. Development times for the Lisp programs ranged from 2 to 8.5 h, compared with 2 to 25 h for C/C ++ and 4 to 63 h for Java (programmer experience alone does not account for the differences). The Lisp programs were also significantly shorter than the other programs." Could selection bias be skewing these results? Proficient Lisp programmers can certainly create shorter and faster programs with Lisp. Who would ever contest that? Average programmers, on the other hand, can probably develop in a similar amount of time and write faster programs with C++ (given the amount of libraries and information available - in comparison to Lisp - and the ubiquity of tooling).

chmaynard 10y ago

This is obviously an opinion piece disguised as a scholarly article. I'm not sure what the hidden agenda is, and the HN comments don't clarify matters much. For example, this nonsense statement reveals the author's bias: "Clojure is a rising star language in the modern software development community." Huh?

swuecho 10y ago
in my opinion, better to publish as a blog post.