4 ms·

So I'm not really an ask me anything type guy, but I work on a bioinformatics team at an academic cancer research center. We have genomic sequencers and a soft

by tom_b 15y ago

So I'm not really an ask me anything type guy, but I work on a bioinformatics team at an academic cancer research center. We have genomic sequencers and a software stack running 24/7/365 plus your normal collection of IT and small dev projects here. If you have questions about what it's like to be a hacker in this type of environment, post them here and I'll share what my experience is like. BTW, I completely wish I knew more stats and bioinformatics, so I probably should purchase the Yegge book collection myself . . .

awj 15y ago
I guess my biggest question is: in what way can programmers outside that specific discipline pitch in and help out? Are there relevant open source projects that could benefit from someone with a lot of programming experience and a slightly-better-than-layman science background?
- tom_b 15y ago
  My gut feeling is that there is a large need for good query/visualization tools of the datasets the sequencers produce. At least, I think if researchers could "play" interactively with data they would be pretty excited. But I am most definitely non-expert in this area, so take that with a grain of salt. I sometimes think about whether or not tools developed with column stores (e.g., the programming language J or something like KDB+) would actually be cool for data exploration. For visualization, check out: http://genome.ucsc.edu/ http://genome.ucsc.edu/ Also, a huge list of projects is at: http://en.wikipedia.org/wiki/Genome_browser http://en.wikipedia.org/wiki/Genome_browser In the genomics analysis space, it seems that I hear these three tools mentioned for sequence alignment are tophat, BWA, and MapSplice. http://bio-bwa.sourceforge.net/ http://bio-bwa.sourceforge.net/ http://www.netlab.uky.edu/p/bioinfo/MapSplice http://www.netlab.uky.edu/p/bioinfo/MapSplice http://tophat.cbcb.umd.edu/ http://tophat.cbcb.umd.edu/ These are actively maintained projects that I think are mostly developed inside of various academic research groups. There is also The Cancer Genome Atlas project at: http://cancergenome.nih.gov/ http://cancergenome.nih.gov/ You can probably find research groups via TCGA that might appreciate some one-off development or support, but it might not be exciting from a tech viewpoint. There is a ton of EMR (electronic medical record) data out there in free text. If you have skills or interest in things like Lucene/Solr, I would bet that almost any research hospital might appreciate your time and skills. And, if you talk to the right group, want to hire you . . .
- hirenj 15y ago
  Bioinformatician here: First of all, I think it's got to be clear that a lone bioinformatician, or even a group of them isn't going to go about changing the world. Essentially, when you sign up for this, you're still a cog in a machine, albeit a slightly more altruistic machine. Here's my pet peeve in bioinformatics - If there's one thing that's poorly suited to science, it's the building of computational infrastructure. We're talking basic stuff like databases, tools etc. Sure, anyone can knock out a bit of code for a basic database, but the big problem is that there's no incentive to have decent code, or maintain it so that it lasts any longer than the person is in the lab, or has funding. So, what will be great is if existing resources are cleaned up - data is normalised and pulled out so that it is actually accessible for doing some kind of analysis on it. If you want to do bigger work, do something actually novel, or that has any biological relevance there's no getting around collecting your own data (e.g. sequencing the crap out of a bunch of things). I'm in the process of trying to get funding now for a project of mine to make that very leap now. I'm sure someone working on next gen sequencing (the new hotness) can pipe up with the big problems to be solved there.
  - enjalot 15y ago
    I know a professor who has a big grant to sequence a whole bunch of animals, and will also do a high res CT scan of each specimen. He plans to make a comprehensive site where scientists and school children can access the data they are interested and learn more. He even has money for a dev position for 4 years, I'm just worried that he gets someone who slaps together a proprietary and incompatible site when this would be a perfect chance to experiment with implementing some standard data access APIs. I've heard bioinformatics people complain about the lack of standards and fragmented nature that comes from various small groups of scientists doing it on their own. If anyone is interested pm me and I'll put you in touch, he is in South Carolina so he can't offer the salary and other perks of the Bay, but it's a real chance to put good development energy into science.
  - nkassis 15y ago
    You don't always have to get new data to make a splash. I particularly like this project (it's in neuroscience but similar things probably exist in biology) http://neurosynth.org/ http://neurosynth.org/ They are doing meta-analysis of the neuroscience literature.
- nkassis 15y ago
  I fit the mold you are talking about. I started working for a neuroimaging lab about 1 year and a half ago. Basically I'm not a scientist at all, did Math as an undergrad but almost no stats or any of that type of vodoo stuff (topology, abstract algebra etc real math stuff is what I did ;p). So basically I had no clue what Neuroimaging meant or did other than people get shoved in scanner, huge magnets turn and they see inside you ;p But what I found is there are plenty of computer science problems in a field of Neuroimaging (and neuroscience) that a programmer can help with. (processing, image analysis, data mining, storage, Visualization whatever) Most labs don't really have people who's primary job is programming. Thus there a lots of tools that are just hack jobs long forgotten that no one is maintaining but everyone depends on. Those things can be helped out by good programming practice and with real programmers behind them. What sucks is getting funding for these people but Open Source can help here by pooling multiple people from many labs into common projects. Also, if you do work with scientist, most of them will talk to you for hours about the science of what they do. You can usually ask the most stupid question and they will be happy to answer it. I've found that most people I work with are open and even more if the work you do helps them achieve their scientific goals. So in the end if you need to learn some science stuff, they will usually be helpful. (BTW my project is in my profile, will be open source soon, waiting for some political approval process).
- davi 15y ago
  There is a new field of extracting wiring diagrams from brains at the level of individual neurons. This is what I work on. http://openconnectomeproject.org http://openconnectomeproject.org has some introductory material. (This is not my site, it is by some people at JHU who picked up our data set and are working on it.) Massive image data sets, lots of need to develop workflow. You can browse the image data here: http://openconnectomeproject.org/catmaid/?pid=4&zp=40635&yp=239236&xp=316396&sid0=4&s0=1 http://openconnectomeproject.org/catmaid/?pid=4&zp=40635... Concretely, look at the plugins being developed by the Fiji project and pitch in, especially on the electron-microscopy-centric plugins: http://pacific.mpi-cbg.de/wiki/index.php/Category:Plugins http://pacific.mpi-cbg.de/wiki/index.php/Category:Plugins edit: also think about contributing to the CATMAID project, which is the software serving the browsable data set above, and perhaps will someday enable crowd-sourced markup: http://fly.mpi-cbg.de/~saalfeld/catmaid/ http://fly.mpi-cbg.de/~saalfeld/catmaid/
CBgrad 15y ago
I'm a CS guy (programmer with a heavy interest in Biology) going to get a Master's in Computational Biology (from CMU). What's it like to be a hacker in that sort of environment? :P Really, anything you could tell me about what you do now, the state of the industry, and your experience would be really valuable.
- tom_b 15y ago
  This is my first bioinformatics job and my first return to an academic environment since finishing a MS in CS and doing the standard industry thing. This group is mostly a core services group rather than a pure research support group. I came in to do traditional data warehouse work, loading data from separate databases into one spot to make reports across those different datasets easier. That's most of my days now - simple data integration and reporting. Oddly enough, we are an Oracle shop because our university has a site license. But I do a little Ruby/Rails/Sinatra for a project that is reporting + some app-like enhancements. Tons of SQL for manipulating data, but it is usually simple SQL, especially compared to stuff I wrote previously in a financial corp. A significant amount of my day is ferreting requirements out of users. I wish I was better at this. There is a much larger amount of grunt work happening in the data management space than I expected. We deal with a huge amount of protected health information (PHI), which can be frustrating since you're usually working around data policies that require somewhat careful interpretation. Obviously, the genomic datasets are huge, so dealing with storage and clusters for data processing is often a discussion point around here. It is an academic/research place, so it is a little more relaxed than industry. Expectations (for software) are a little low - we're talking about users who have, for the most part, kept data in spreadsheets or little Access style databases for years. And they often haven't had the budget to hire dedicated software guys, so sometimes tools are suffering from bit-rot. Good hackers seem to be appreciated. It is below your standard industry salary, but not terribly so. I try to avoid being sucked into low-value work since "the reward for work well-done is more work." Just like grad school, if you want to work on a specific topic or research area, make sure you get a job in a group doing that work. Once you're established, it seems easy to get some space to do some exploratory hacking.
  - 15y ago
david927 15y ago
We're creating a group here: http://digitalauteur.wikispaces.com http://digitalauteur.wikispaces.com
cing 15y ago
I worked at a bioinformatics co-op job for 8 months about 4 years ago. I had fun, but my main impression was that the field is a major clusterf*ck of databases and data management. I think you could take this list as evidence for that: http://en.wikipedia.org/wiki/Bioinformatics_workflow_management_systems http://en.wikipedia.org/wiki/Bioinformatics_workflow_managem...
- tom_b 15y ago
  This is definitely true. I deal with unstructured data, and every flavor of data storage weekly (XML, Access, Oracle, Postgres, MySQL, a few MS SQL Server instances, Excel, and plain text). <edit> oh, and forget about stuff being normalized or making any sense . . .
AndrewO 15y ago
How would you describe the attention to craftsmanship of software in scientific computing? The most illuminating thing about Climategate to me was the state of the software: is this kind of code widespread? Is there interest in improvement? If I went to work for a research team and suggested pairing, code reviews, version control, continuous integration, or other accepted good practices from my experience in software development how would that be received? Also, the dilemma in my mind is whether I can stand going back to grad school in my late 20s for a career I don't really know much about. I'm not sure if you can speak to that experience, but would you say it's been rewarding?
- tom_b 15y ago
  I haven't had much luck getting people on my team interested in code reviews. The day I started we put a ton of stuff in version control though. In general, you should interview a team when you're trying to get a job and trust your gut on how much those practices you mention will be accepted. It's a little harder here, because for years the model was assign a dev to a project and that dev will own the entire project from start to finish. We still don't do a great job working as a team. Craftmanship can take a back-seat to schedule. Lots of operational stuff (for grants and administrative purposes) gets pushed off to the last minute. And for research, the focus here is on the results much less than the process. The researchers don't care if you do it in a bash shell script or a clojure jewel as long as it's done - kind of like a startup. So if you can do it fast in a maintainable and cool way, all the better. We are suffering the effects right now of some bad QC code. A non-insignificant amount of data had to be re-analyzed because of some code bugs. I would say interest is currently much higher in improvement. :-) I did grad school (MS in CS), went to industry, and then came back. Returning was mainly lucky timing and personal network effects. But I was late 20's when I did grad school, so we're not too different. It was completely worth it for me. Some PIs are like typical academics - they have MD/PhD (usually both) and can be somewhat dictatorial. But some are great and want your thoughts and expertise. You could consider simply getting a job doing typical IT work in an organization like this. You're probably right that eventually you'll want some grad school (we are walking around in credential heaven here), but you could always try it out first.
- PaulHoule 15y ago
jgmmo 15y ago
Lets say I want to get in this field. I have a BBA in Econ -- but no upper level math -- would I be better off getting a BA in Math and then going off to grad school (in CS or Bio, or both..)? Or would it be a better use of time to take certain maths and then just head into a masters program?
- tom_b 15y ago
  My co-workers have very diverse backgrounds - PhDs in bioinformatics all the way to folks with a BA in history. They all seem to be hackers. So I would say, if you're already a hacker, you can probably find a way in somewhere given enough time regardless of your current degree level. If you know you are going to head off to grad school, don't do another undergrad before. Many CS programs take on students who need a little remediation in math or CS background. I would guess that would be much more difficult for a MS program in biology. You might even find that a MS in bioinformatics is the appropriate choice. There seem to be several of these programs floating around, they are multi-disciplinary to begin with, and if you can crank out a good GRE score, you can probably get in one without too much effort. You will probably find it easier to break into the academic research world with at least a MS. But, I hear lots of buzz about commercial companies wanting to get into the sequencing business. Maybe you could just give yourself a crash course in python or ruby and a little biostatistics and sneak into one of those? Especially if you can find someone in your network already working in the field. I recruited a friend to work with me who had a BS in history and a MBA (I teased him). But he was working in the software field already and I knew he would pick up the simple tech we were using to get work done.

by tom_b 15y ago

awj 15y ago

I guess my biggest question is: in what way can programmers outside that specific discipline pitch in and help out? Are there relevant open source projects that could benefit from someone with a lot of programming experience and a slightly-better-than-layman science background?

tom_b 15y ago
My gut feeling is that there is a large need for good query/visualization tools of the datasets the sequencers produce. At least, I think if researchers could "play" interactively with data they would be pretty excited. But I am most definitely non-expert in this area, so take that with a grain of salt. I sometimes think about whether or not tools developed with column stores (e.g., the programming language J or something like KDB+) would actually be cool for data exploration. For visualization, check out: http://genome.ucsc.edu/ http://genome.ucsc.edu/ Also, a huge list of projects is at: http://en.wikipedia.org/wiki/Genome_browser http://en.wikipedia.org/wiki/Genome_browser In the genomics analysis space, it seems that I hear these three tools mentioned for sequence alignment are tophat, BWA, and MapSplice. http://bio-bwa.sourceforge.net/ http://bio-bwa.sourceforge.net/ http://www.netlab.uky.edu/p/bioinfo/MapSplice http://www.netlab.uky.edu/p/bioinfo/MapSplice http://tophat.cbcb.umd.edu/ http://tophat.cbcb.umd.edu/ These are actively maintained projects that I think are mostly developed inside of various academic research groups. There is also The Cancer Genome Atlas project at: http://cancergenome.nih.gov/ http://cancergenome.nih.gov/ You can probably find research groups via TCGA that might appreciate some one-off development or support, but it might not be exciting from a tech viewpoint. There is a ton of EMR (electronic medical record) data out there in free text. If you have skills or interest in things like Lucene/Solr, I would bet that almost any research hospital might appreciate your time and skills. And, if you talk to the right group, want to hire you . . .
hirenj 15y ago
Bioinformatician here: First of all, I think it's got to be clear that a lone bioinformatician, or even a group of them isn't going to go about changing the world. Essentially, when you sign up for this, you're still a cog in a machine, albeit a slightly more altruistic machine. Here's my pet peeve in bioinformatics - If there's one thing that's poorly suited to science, it's the building of computational infrastructure. We're talking basic stuff like databases, tools etc. Sure, anyone can knock out a bit of code for a basic database, but the big problem is that there's no incentive to have decent code, or maintain it so that it lasts any longer than the person is in the lab, or has funding. So, what will be great is if existing resources are cleaned up - data is normalised and pulled out so that it is actually accessible for doing some kind of analysis on it. If you want to do bigger work, do something actually novel, or that has any biological relevance there's no getting around collecting your own data (e.g. sequencing the crap out of a bunch of things). I'm in the process of trying to get funding now for a project of mine to make that very leap now. I'm sure someone working on next gen sequencing (the new hotness) can pipe up with the big problems to be solved there.
- enjalot 15y ago
  I know a professor who has a big grant to sequence a whole bunch of animals, and will also do a high res CT scan of each specimen. He plans to make a comprehensive site where scientists and school children can access the data they are interested and learn more. He even has money for a dev position for 4 years, I'm just worried that he gets someone who slaps together a proprietary and incompatible site when this would be a perfect chance to experiment with implementing some standard data access APIs. I've heard bioinformatics people complain about the lack of standards and fragmented nature that comes from various small groups of scientists doing it on their own. If anyone is interested pm me and I'll put you in touch, he is in South Carolina so he can't offer the salary and other perks of the Bay, but it's a real chance to put good development energy into science.
- nkassis 15y ago
  You don't always have to get new data to make a splash. I particularly like this project (it's in neuroscience but similar things probably exist in biology) http://neurosynth.org/ http://neurosynth.org/ They are doing meta-analysis of the neuroscience literature.
nkassis 15y ago
I fit the mold you are talking about. I started working for a neuroimaging lab about 1 year and a half ago. Basically I'm not a scientist at all, did Math as an undergrad but almost no stats or any of that type of vodoo stuff (topology, abstract algebra etc real math stuff is what I did ;p). So basically I had no clue what Neuroimaging meant or did other than people get shoved in scanner, huge magnets turn and they see inside you ;p But what I found is there are plenty of computer science problems in a field of Neuroimaging (and neuroscience) that a programmer can help with. (processing, image analysis, data mining, storage, Visualization whatever) Most labs don't really have people who's primary job is programming. Thus there a lots of tools that are just hack jobs long forgotten that no one is maintaining but everyone depends on. Those things can be helped out by good programming practice and with real programmers behind them. What sucks is getting funding for these people but Open Source can help here by pooling multiple people from many labs into common projects. Also, if you do work with scientist, most of them will talk to you for hours about the science of what they do. You can usually ask the most stupid question and they will be happy to answer it. I've found that most people I work with are open and even more if the work you do helps them achieve their scientific goals. So in the end if you need to learn some science stuff, they will usually be helpful. (BTW my project is in my profile, will be open source soon, waiting for some political approval process).
davi 15y ago
There is a new field of extracting wiring diagrams from brains at the level of individual neurons. This is what I work on. http://openconnectomeproject.org http://openconnectomeproject.org has some introductory material. (This is not my site, it is by some people at JHU who picked up our data set and are working on it.) Massive image data sets, lots of need to develop workflow. You can browse the image data here: http://openconnectomeproject.org/catmaid/?pid=4&zp=40635&yp=239236&xp=316396&sid0=4&s0=1 http://openconnectomeproject.org/catmaid/?pid=4&zp=40635... Concretely, look at the plugins being developed by the Fiji project and pitch in, especially on the electron-microscopy-centric plugins: http://pacific.mpi-cbg.de/wiki/index.php/Category:Plugins http://pacific.mpi-cbg.de/wiki/index.php/Category:Plugins edit: also think about contributing to the CATMAID project, which is the software serving the browsable data set above, and perhaps will someday enable crowd-sourced markup: http://fly.mpi-cbg.de/~saalfeld/catmaid/ http://fly.mpi-cbg.de/~saalfeld/catmaid/

CBgrad 15y ago

I'm a CS guy (programmer with a heavy interest in Biology) going to get a Master's in Computational Biology (from CMU). What's it like to be a hacker in that sort of environment? :P Really, anything you could tell me about what you do now, the state of the industry, and your experience would be really valuable.

tom_b 15y ago
This is my first bioinformatics job and my first return to an academic environment since finishing a MS in CS and doing the standard industry thing. This group is mostly a core services group rather than a pure research support group. I came in to do traditional data warehouse work, loading data from separate databases into one spot to make reports across those different datasets easier. That's most of my days now - simple data integration and reporting. Oddly enough, we are an Oracle shop because our university has a site license. But I do a little Ruby/Rails/Sinatra for a project that is reporting + some app-like enhancements. Tons of SQL for manipulating data, but it is usually simple SQL, especially compared to stuff I wrote previously in a financial corp. A significant amount of my day is ferreting requirements out of users. I wish I was better at this. There is a much larger amount of grunt work happening in the data management space than I expected. We deal with a huge amount of protected health information (PHI), which can be frustrating since you're usually working around data policies that require somewhat careful interpretation. Obviously, the genomic datasets are huge, so dealing with storage and clusters for data processing is often a discussion point around here. It is an academic/research place, so it is a little more relaxed than industry. Expectations (for software) are a little low - we're talking about users who have, for the most part, kept data in spreadsheets or little Access style databases for years. And they often haven't had the budget to hire dedicated software guys, so sometimes tools are suffering from bit-rot. Good hackers seem to be appreciated. It is below your standard industry salary, but not terribly so. I try to avoid being sucked into low-value work since "the reward for work well-done is more work." Just like grad school, if you want to work on a specific topic or research area, make sure you get a job in a group doing that work. Once you're established, it seems easy to get some space to do some exploratory hacking.
- 15y ago

david927 15y ago

We're creating a group here: http://digitalauteur.wikispaces.com http://digitalauteur.wikispaces.com

cing 15y ago

I worked at a bioinformatics co-op job for 8 months about 4 years ago. I had fun, but my main impression was that the field is a major clusterf*ck of databases and data management. I think you could take this list as evidence for that: http://en.wikipedia.org/wiki/Bioinformatics_workflow_management_systems http://en.wikipedia.org/wiki/Bioinformatics_workflow_managem...

tom_b 15y ago
This is definitely true. I deal with unstructured data, and every flavor of data storage weekly (XML, Access, Oracle, Postgres, MySQL, a few MS SQL Server instances, Excel, and plain text). <edit> oh, and forget about stuff being normalized or making any sense . . .

AndrewO 15y ago

How would you describe the attention to craftsmanship of software in scientific computing? The most illuminating thing about Climategate to me was the state of the software: is this kind of code widespread? Is there interest in improvement? If I went to work for a research team and suggested pairing, code reviews, version control, continuous integration, or other accepted good practices from my experience in software development how would that be received? Also, the dilemma in my mind is whether I can stand going back to grad school in my late 20s for a career I don't really know much about. I'm not sure if you can speak to that experience, but would you say it's been rewarding?

tom_b 15y ago
I haven't had much luck getting people on my team interested in code reviews. The day I started we put a ton of stuff in version control though. In general, you should interview a team when you're trying to get a job and trust your gut on how much those practices you mention will be accepted. It's a little harder here, because for years the model was assign a dev to a project and that dev will own the entire project from start to finish. We still don't do a great job working as a team. Craftmanship can take a back-seat to schedule. Lots of operational stuff (for grants and administrative purposes) gets pushed off to the last minute. And for research, the focus here is on the results much less than the process. The researchers don't care if you do it in a bash shell script or a clojure jewel as long as it's done - kind of like a startup. So if you can do it fast in a maintainable and cool way, all the better. We are suffering the effects right now of some bad QC code. A non-insignificant amount of data had to be re-analyzed because of some code bugs. I would say interest is currently much higher in improvement. :-) I did grad school (MS in CS), went to industry, and then came back. Returning was mainly lucky timing and personal network effects. But I was late 20's when I did grad school, so we're not too different. It was completely worth it for me. Some PIs are like typical academics - they have MD/PhD (usually both) and can be somewhat dictatorial. But some are great and want your thoughts and expertise. You could consider simply getting a job doing typical IT work in an organization like this. You're probably right that eventually you'll want some grad school (we are walking around in credential heaven here), but you could always try it out first.
PaulHoule 15y ago

jgmmo 15y ago

Lets say I want to get in this field. I have a BBA in Econ -- but no upper level math -- would I be better off getting a BA in Math and then going off to grad school (in CS or Bio, or both..)? Or would it be a better use of time to take certain maths and then just head into a masters program?

tom_b 15y ago
My co-workers have very diverse backgrounds - PhDs in bioinformatics all the way to folks with a BA in history. They all seem to be hackers. So I would say, if you're already a hacker, you can probably find a way in somewhere given enough time regardless of your current degree level. If you know you are going to head off to grad school, don't do another undergrad before. Many CS programs take on students who need a little remediation in math or CS background. I would guess that would be much more difficult for a MS program in biology. You might even find that a MS in bioinformatics is the appropriate choice. There seem to be several of these programs floating around, they are multi-disciplinary to begin with, and if you can crank out a good GRE score, you can probably get in one without too much effort. You will probably find it easier to break into the academic research world with at least a MS. But, I hear lots of buzz about commercial companies wanting to get into the sequencing business. Maybe you could just give yourself a crash course in python or ruby and a little biostatistics and sneak into one of those? Especially if you can find someone in your network already working in the field. I recruited a friend to work with me who had a BS in history and a MBA (I teased him). But he was working in the software field already and I knew he would pick up the simple tech we were using to get work done.