5 ms·
It seems like there's good opportunity for skilled data scientists and engineers to make a real difference here. I do think that laypersons (to both medicine an
by danso 10y ago
It seems like there's good opportunity for skilled data scientists and engineers to make a real difference here. I do think that laypersons (to both medicine and engineering) think that practitioners in medicine and biology have mastered such mundane things like data pipelines, because you have to be so smart to be in medicine/biology, but my limited experience has been more along the lines of what Neil Saunders describes as the inspiration for his coding+bioinformatics blog:
https://nsaunders.wordpress.com/about-2/about/ https://nsaunders.wordpress.com/about-2/about/
> You may be wondering about the title of this blog.
> Early in my bioinformatics career, I gave a talk to my department. It was fairly basic stuff – how to identify genes in a genome sequence, standalone BLAST, annotation, data munging with Perl and so on. Come question time, a member of the audience raised her hand and said:
> “It strikes me that what you’re doing is rather desperate. Wouldn’t you be better off doing some experiments?”
> It was one of the few times in my life when my jaw literally dropped and swung uselessly on its hinges. Ultimately though, her question did make a great blog title.
edit: To add an anecdote that I believe I read on HN; regarding the topic of the huge datamine of DNA and other health data provided by the U.S. government, a commenter said that the reason it was all on FTP was because professors couldn't download large datasets via their web browser, or some such technical hiccup.
I won't say that putting data on the Web makes it automatically more accessible, but data discovery through FTP requires a bit of scripting skill that I imagine the average biomedical scientist does not have.
- brandonb 10y agoI agree--the impact of a great engineer working in healthcare is very high, particularly if you partner with medical experts. We're a small startup that has partnered with UCSF Cardiology to detect abnormal heart rhythms, and other conditions, using deep learning on Apple Watch heart rate data: https://wsj.com/articles/new-study-seeks-to-use-deep-learning-to-detect-heart-disease-1458240739 https://blog.cardiogr.am/three-challenges-for-artificial-intelligence-in-medicine-dfb9993ae750 https://a16z.com/2016/10/20/cardiogram/ We have about 10B sensor data points so far. If you're a machine learning engineer and interested in working on this type of problem, feel free to email me: brandon@cardiogr.am.
- daveguy 10y agoI'm curious. What is your definition of "machine learning engineer"? Are you talking mostly feature engineering or something deeper. If so, what?
- brandonb 10y agoIn our case, we're applying deep learning to sensor data, so much of the day-to-day work of a machine learning engineer is experimenting with new neural architectures rather than feature engineering by hand. For example, we're using or interested in techniques like: * semi-supervised sequence learning (we have a paper in a NIPS workshop next week on applying sequence autoencoders to health data, for example) * deep generative models * variational RNNs From a day-to-day perspective, we use tools like Tensorflow and Keras, similar to most AI research labs. In general, we try to act as a software startup that happens to work in healthcare, rather than as what you might think of as a traditional biotech or medical device startup. Does that help answer your question?
- searine 10y ago>It seems like there's good opportunity for skilled data scientists and engineers to make a real difference here. The logistics of dealing with big data is a solved problem in biology. We don't need data fast, and a laptop can run most analyses now, so it's just storage really. The hard part is having familiarity with all the esoteric metrics used to compare genomes and all the unfamiliar ways your results can be biased. An off-the-shelf corporate data scientist, while a critical thinker, lacks the domain specific knowledge to be able to ask good questions.
- danso 10y ago> An off-the-shelf corporate data scientist, while a critical thinker, lacks the domain specific knowledge to be able to ask good questions. Yeah, I think that's the other side of the coin here (for biomedicine and most other fields). Even if better engineering is what is most greatly needed, it's not just about just being purely good at code and logistics. As a corollary, if you do have the domain knowledge and are in a position to act, you don't have to be an ace programmer to make a huge difference.
- dekhn 10y agoVery specifically, with actual knowledge: NCBI insisted on FTP (and still does) because a single prominent professor downloaded a file from web and IE truncated it at 2GB (known bug). Said professor threw a shitfit. This IE bug hasn't been a problem for ~10 years.
- jerven 10y agoNot just that. FTP sites have tendency to not fail. HTTP downloads are really hard to keep up at 2gb+ over someones wifi. Dropped connections are quite normal and doing the correct resume work is possible but not trivial in practice. So for files FTP is much better. Especially if the other side is just a script.
- dekhn 10y agoNone of this is true. FTP is just bytes over TCP, it's just as prone to problems. HTTP recovery is far simpler because of ranged GETs. FTP is definitely not better.
- jerven 10y agoYeah, but in practice ranged GETs and HTTP servers are more likely to be down than the FTP servers.
- dekhn 10y agoThis is just false. What evidence do you have? HTTP was architected for availability (load balancing across many servers). Also, FTP has two TCP connections per copy, doubling the chance of abrubt termination. I've moved petabytes over the internet using FTP and HTTP. HTTP wins hands-down.
- jerven 10y agoPractical experience in such an institute providing such data. FTP servers for files just run, http servers tend to do more things than just serving files. This could be separated out of course but in _practice_ they aren't. Setting up the http server right and not touching them is the hard part, not the actual http protocol part. i.e. I do accept that http is a much better protocol than FTP but that social and organisational reasons lead to FTP being more stable and dependable in the field for large file download than http servers.
- a_bonobo 10y ago>edit: To add an anecdote that I believe I read on HN; regarding the topic of the huge datamine of DNA and other health data provided by the U.S. government, a commenter said that the reason it was all on FTP was because professors couldn't download large datasets via their web browser, or some such technical hiccup. All data on the sequence read archive (SRA), the main worldwide database for sequencing reads from NCBI, is accessible via FTP and via ascp. FTP is slow, ascp is blazing fast.