5 ms·

There are also many people in biology research who do these kinds of repetitive tasks on daily basis for hours without realizing that much of it could easily be

There are also many people in biology research who do these kinds of repetitive tasks on daily basis for hours without realizing that much of it could easily be automated. While we're sharing horror stories, a shining example from my experience is a friend who spends days analyzing western blots from experiments: it comes down to identifying a set of 10 rectangular black patches on clear white background and measuring their average brightness. The standard solution is this software package that lets you open each .jpg file one by one (there is a folder with hundreds), and you draw little rectangles and it tells you the average brightness inside. You write the number down inside an excel spreadsheet one by one, alt-tabbing back and forth. After a brief facepalm moment I opened MATLAB, wrote ~30 lines of code and processed all images in the folder automatically, dumping all average brightnesses into a giant matrix in 1 minute of runtime. Several days of work saved that could go to curing cancer instead of filling spreadsheets.

tomdtpink 14y ago
Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.
DeepDuh 14y ago
Did you share your solution? Imagine them reapplying your method to some testcases, yielding the same results for days of work within minutes.
- swah 14y ago
  Thing is the programming in this case is trivial; but if there's no tick on those guys that what they are doing is easily automatable; they won't go looking for a solution anyway.
  - nzmsv 14y ago
    It's only trivial for someone who has used MATLAB, taken a machine learning course, or otherwise has similar experience. I'm sure the biology people know this is automatable; it's the "easily" part that's the problem.
    - DeepDuh 14y ago
      Exactly. It's probably not even that common for biology groups to have a programmer doing work for them, so they don't often get that chance. That's why we need more interdisciplinary research groups / teams.
    - karpathy 14y ago
      Actually, I don't think I agree. It's quite fascinating but the friend I mentioned is not very good at telling the easily automatable tasks apart from those that aren't. On a later occasion the friend came to me again with a different problem to solve, but this time it involved spotting dead cells in a mix of cells, images taken with a microscope. They had to count the ratio of dead cells. The difference involved properties of texture and size that could be easily explained and I could do it with near perfect accuracy with little training, and yet it would be much much harder to automate this task. The cells were tightly packed, there were scratches and noise on the image, brightness and contrast variations, distortions,... It would involve training a model, computing features, cross-validations, etc. That did not seem obvious at all to them and it took a while to explain. I'm not even sure if they got it. After all, both tasks are very repetitive and boring, what's the difference? To address your point, you're right in that I wouldn't expect my friend to did what I did if they only read a "learn how to program in x days!", or even several similar books over few months, or if they knew how to write simple scripts in python or something. But what I would at least hope for is that they realize that this kind of task is very easily automated, and have a very basic understanding of what it would involve. Perhaps labs should come with computer scientist hacker floater technicians?
    - swah 14y ago
      Sorry, I didn't mean trivial like "my mom can do it" - but like karpathy below understood.
sanswork 14y ago
At one of the first computer conferences I went to(HAL2001) there was a talk on "wetware hacking". The guy giving the talk was a neuroscientist I believe and he was literally begging the programmers in attendance to give his field a try since they had so many programable tasks and no one able or willing to do them. Unfortunately he fell short of giving advice on paths into the field if you aren't already in academia. The problems these people are solving though are definitely things that myself and a lot of my friends would find interesting problemsets. I think its just a matter of making it accessible.
actua11y 14y ago
These "wet" labs usually do not have many people who are enthusiastic about computers in the programming, text-only sense. Usually there is one or two people that know some stuff and they do all the technical stuff for everyone else. It's not uncommon to find people who are actually quite computer phobic in these labs. The general interest in computers is prefering Mac's to PC's and that's about as far as it goes. If you start talking up some programming solution to the wrong person you may actually alienate them. The problem extends to much of the software that is written too. There are some command line suites that do batch processing, e.g. for sequences, but much of the software is point and click. And that's the problem. No one expects the typical lab user of these programs to be versed in using the command line and doing batch jobs. Everything is mouse-driven. Even if the back end is powerful, the front end is always a slow GUI. This just slows everything down. Another shining example of this sort of thing is audio engineering. You have programs that are designed as GUI's that look like physical pieces of gear. You move the mouse around to adjust turn knobs, switches and sliders and adjust settings. Extremely cumbersome. But this is how they expect users to be comfortable. Then companies started selling physical consoles with real knobs, switches and sliders to control the software. It's comical. The touch screen trend is not going to help people get comfortable with the command line. And the key to realy fast computing is using a command line and batch processing. There is no way around it. When you are pointing and clicking, you are just sending a signal that began as a text instruction in the code anyway. Why not just start with the text command and skip the point and click?
- blt 14y ago
  You can't turn two knobs at once in a live performance with a mouse.
- bitwize 14y ago
  It's comical. Only to someone who's never mixed a song. Professional audio engineers do their work by FEELING the position of the equipment, making adjustments, and LISTENING to the results. And they expect the feedback to be instant -- just like playing a musical instrument. Now put them in front of a command line and imagine if they had to type out their intents, listen to the results, and then make tweaks to their typed-in textual program to effect some change in the audio output. You'd end up with a bored, pissed, unproductive audio engineer. The sliders and switches made digital audio professionals MORE productive, not less -- even more so when they became controls on a physical console with digital outputs, as it put controlling the digital gear back in the analog realm a musician is already familiar with. I find it also interesting that lab environments and music are where plenty of programming takes place by people who are not occupational programmers -- and much of it is visual, in environments like LabVIEW and Max/MSP. This is the future of programming. It is why we should listen to those who say we are too bound to a textual representation for our programs and that in order for the craft to evolve, a richer representation must be embraced.
VMG 14y ago
I work as the IT guy at a biological institute (fixing Windows boxes). I don't know for sure that there are repetitive tasks that the graduates do there, I am pretty certain that there are some. But there simply is no structure in place that would allow me to work for them. I could do it pro bono, but honestly I don't have the time for it. On the other hand they simply don't hire programmers to automate tasks, that's just not part of their budget.
- mkopinsky 14y ago
  And of course, the reason it's not part of their budget is because they are wasting money on boring stuff. Have lunch with some of them. Schmooze with them the next time they call you for a hardware/software fix. Point out a single, simple way they could do something more easily by scripting it. Then recommend that the next time they want to someone to do some research, they instead hire an undergrad who can write code. Undergrads are cheap, and will get them thinking about what they can automate.

tomdtpink 14y ago

Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.Bring out the sock puppets.

DeepDuh 14y ago

Did you share your solution? Imagine them reapplying your method to some testcases, yielding the same results for days of work within minutes.

swah 14y ago
Thing is the programming in this case is trivial; but if there's no tick on those guys that what they are doing is easily automatable; they won't go looking for a solution anyway.
- nzmsv 14y ago
  It's only trivial for someone who has used MATLAB, taken a machine learning course, or otherwise has similar experience. I'm sure the biology people know this is automatable; it's the "easily" part that's the problem.
  - DeepDuh 14y ago
    Exactly. It's probably not even that common for biology groups to have a programmer doing work for them, so they don't often get that chance. That's why we need more interdisciplinary research groups / teams.
  - karpathy 14y ago
    Actually, I don't think I agree. It's quite fascinating but the friend I mentioned is not very good at telling the easily automatable tasks apart from those that aren't. On a later occasion the friend came to me again with a different problem to solve, but this time it involved spotting dead cells in a mix of cells, images taken with a microscope. They had to count the ratio of dead cells. The difference involved properties of texture and size that could be easily explained and I could do it with near perfect accuracy with little training, and yet it would be much much harder to automate this task. The cells were tightly packed, there were scratches and noise on the image, brightness and contrast variations, distortions,... It would involve training a model, computing features, cross-validations, etc. That did not seem obvious at all to them and it took a while to explain. I'm not even sure if they got it. After all, both tasks are very repetitive and boring, what's the difference? To address your point, you're right in that I wouldn't expect my friend to did what I did if they only read a "learn how to program in x days!", or even several similar books over few months, or if they knew how to write simple scripts in python or something. But what I would at least hope for is that they realize that this kind of task is very easily automated, and have a very basic understanding of what it would involve. Perhaps labs should come with computer scientist hacker floater technicians?
  - swah 14y ago
    Sorry, I didn't mean trivial like "my mom can do it" - but like karpathy below understood.

sanswork 14y ago

At one of the first computer conferences I went to(HAL2001) there was a talk on "wetware hacking". The guy giving the talk was a neuroscientist I believe and he was literally begging the programmers in attendance to give his field a try since they had so many programable tasks and no one able or willing to do them. Unfortunately he fell short of giving advice on paths into the field if you aren't already in academia. The problems these people are solving though are definitely things that myself and a lot of my friends would find interesting problemsets. I think its just a matter of making it accessible.

actua11y 14y ago

These "wet" labs usually do not have many people who are enthusiastic about computers in the programming, text-only sense. Usually there is one or two people that know some stuff and they do all the technical stuff for everyone else. It's not uncommon to find people who are actually quite computer phobic in these labs. The general interest in computers is prefering Mac's to PC's and that's about as far as it goes. If you start talking up some programming solution to the wrong person you may actually alienate them. The problem extends to much of the software that is written too. There are some command line suites that do batch processing, e.g. for sequences, but much of the software is point and click. And that's the problem. No one expects the typical lab user of these programs to be versed in using the command line and doing batch jobs. Everything is mouse-driven. Even if the back end is powerful, the front end is always a slow GUI. This just slows everything down. Another shining example of this sort of thing is audio engineering. You have programs that are designed as GUI's that look like physical pieces of gear. You move the mouse around to adjust turn knobs, switches and sliders and adjust settings. Extremely cumbersome. But this is how they expect users to be comfortable. Then companies started selling physical consoles with real knobs, switches and sliders to control the software. It's comical. The touch screen trend is not going to help people get comfortable with the command line. And the key to realy fast computing is using a command line and batch processing. There is no way around it. When you are pointing and clicking, you are just sending a signal that began as a text instruction in the code anyway. Why not just start with the text command and skip the point and click?

blt 14y ago
You can't turn two knobs at once in a live performance with a mouse.
bitwize 14y ago
It's comical. Only to someone who's never mixed a song. Professional audio engineers do their work by FEELING the position of the equipment, making adjustments, and LISTENING to the results. And they expect the feedback to be instant -- just like playing a musical instrument. Now put them in front of a command line and imagine if they had to type out their intents, listen to the results, and then make tweaks to their typed-in textual program to effect some change in the audio output. You'd end up with a bored, pissed, unproductive audio engineer. The sliders and switches made digital audio professionals MORE productive, not less -- even more so when they became controls on a physical console with digital outputs, as it put controlling the digital gear back in the analog realm a musician is already familiar with. I find it also interesting that lab environments and music are where plenty of programming takes place by people who are not occupational programmers -- and much of it is visual, in environments like LabVIEW and Max/MSP. This is the future of programming. It is why we should listen to those who say we are too bound to a textual representation for our programs and that in order for the craft to evolve, a richer representation must be embraced.

VMG 14y ago

I work as the IT guy at a biological institute (fixing Windows boxes). I don't know for sure that there are repetitive tasks that the graduates do there, I am pretty certain that there are some. But there simply is no structure in place that would allow me to work for them. I could do it pro bono, but honestly I don't have the time for it. On the other hand they simply don't hire programmers to automate tasks, that's just not part of their budget.

mkopinsky 14y ago
And of course, the reason it's not part of their budget is because they are wasting money on boring stuff. Have lunch with some of them. Schmooze with them the next time they call you for a hardware/software fix. Point out a single, simple way they could do something more easily by scripting it. Then recommend that the next time they want to someone to do some research, they instead hire an undergrad who can write code. Undergrads are cheap, and will get them thinking about what they can automate.