4 ms·

Let me make a statement and let you can judge it. (its below and stated as "STATEMENT") BACKGROUND I have been working for 15 years in industry doing hardcore

Let me make a statement and let you can judge it. (its below and stated as "STATEMENT") BACKGROUND I have been working for 15 years in industry doing hardcore ML. I have the fortunate drive and background that I was able to get my masters degree while working full time from an R1 school. No watered-down online degree, no certificate. I would drive twice a week to class for 4 years and did a full thesis which was published. Since then, I have published 6 papers, all peer reviewed. I even did a sabbatical with another research lab of which I was invited to come. After 15 years, I decided to go back and get my Phd, all while continue to work full time. My thought was that it would be easy to get a phd with all my technical experience and math chops ive developed over the last 15 years. I essentially have been doing math 5 days a week for 15 years. Here's what happened... Coursework was a breeze. I barely put any time into it and I easily can get a B+. This is really helpful because I am working 40-50 hours a week at my full time job nd managing my family. I passed my candidacy exam on the first try with little issue (this is rare for my department). The biggest hangup I have about the phd process is what my advisor wants me to do when writing papers. He is the youngest full professor in the department and is from a well known and well respected graduate research university. But, the way he has me slant my papers is absurd. Results which I feel are very important to the assessment of the reader to decide if they should use the method, he has me remove because the results are "too subtle." He is constantly beating on me to think about "the casual reviewer." Students in the lab produce papers which are very brittle and overfit to the test data. His lab uses the same dataset paper after paper. My advisor was so proud of a method his top student produced that he offered his code for my workplace to use. It didn't work as well as a much simpler method we used. Eventually we gave the student our data so there could be a fairest shake at getting students method to work. The student never got the method work to work nearly as well as in his published paper despite telling my company and I over and over that "it will work". The student is now at amazon lab 126. STATEMENT: Academia is peer reviewed driven but the peers are other academics and so the system of innovation is dead; academics have very little understanding of what actually works in practice. Great example: its of no surprise that Google has such a hard time using ML on MRI datasets. The groups working on this are made up of Phds from my grad lab! TL;DR - worked for 15 years, went back for phd, here's what i hear: "think of the casual reviewer" "fiddle with your net so that your results are better than X" "you have to sell your results so that its clear your method has merit" "can you get me results that are 1% better? use the tricks from blog Y" "As long as your results are 1% better, you are fine" Edit 1:boasts are given to avoid "your experience doesn't count because you X" strawmans, where X={are lazy,are a young student, are inexperienced, went to an easy school, are in an easy program, naive to the peer review process}

Der_Einzige 6y ago
You're getting downvoted because you're right and grad-students / ML professors don't want to admit that they can do a lot better in this regard.
CrazyStat 6y ago
It's unfortunate that this is getting downvoted. Perhaps it comes across as a little boastful, but it matches my experience in academia. When I was doing my PhD (Statistics) I spent several weeks running simulation studies to compare our new statistical model to 2-3 existing models that did similar things, across a wide range of data sets both real and generated with varying parameters. It was an unsupervised learning problem so there wasn't a "correct" answer to compare to on the real data sets, but on the simulated data we outperformed the existing models over about 60% of the parameter space that we tested, including the part of the parameter space that I thought was most useful/likely in practice. Rather than present all the simulation results and have a nuanced discussion of where we did better and where we did worse, my advisor had me remove the 40% of the parameter space where we did worse from the results, so it looked like we always did as good or better. The scientific contribution would undoubtably have been better if we had presented the results where we did worse and discussed why we believed that to be the case, but getting the paper past reviewers and into a top journal (as we eventually did) was considered more important.
- scott_s 6y ago
  I find this sad as well, particularly since, in my experience, explaining that nuance makes for excellent papers. I have found reviewers actually respond positively to being upfront about limitations, along with discussions about why. Because that is not the norm, it ends up being novel. But, because it requires nuance, it depends on an ability to write well.
devalgo 6y ago
Of course you have the other side of the coin where those same ML students take a pre-trained VGG model, fine tune it on a couple thousand pictures of hotdogs or whatever and raise millions in VC money for their "AI" company.

Der_Einzige 6y ago

You're getting downvoted because you're right and grad-students / ML professors don't want to admit that they can do a lot better in this regard.

CrazyStat 6y ago

It's unfortunate that this is getting downvoted. Perhaps it comes across as a little boastful, but it matches my experience in academia. When I was doing my PhD (Statistics) I spent several weeks running simulation studies to compare our new statistical model to 2-3 existing models that did similar things, across a wide range of data sets both real and generated with varying parameters. It was an unsupervised learning problem so there wasn't a "correct" answer to compare to on the real data sets, but on the simulated data we outperformed the existing models over about 60% of the parameter space that we tested, including the part of the parameter space that I thought was most useful/likely in practice. Rather than present all the simulation results and have a nuanced discussion of where we did better and where we did worse, my advisor had me remove the 40% of the parameter space where we did worse from the results, so it looked like we always did as good or better. The scientific contribution would undoubtably have been better if we had presented the results where we did worse and discussed why we believed that to be the case, but getting the paper past reviewers and into a top journal (as we eventually did) was considered more important.

scott_s 6y ago
I find this sad as well, particularly since, in my experience, explaining that nuance makes for excellent papers. I have found reviewers actually respond positively to being upfront about limitations, along with discussions about why. Because that is not the norm, it ends up being novel. But, because it requires nuance, it depends on an ability to write well.

devalgo 6y ago

Of course you have the other side of the coin where those same ML students take a pre-trained VGG model, fine tune it on a couple thousand pictures of hotdogs or whatever and raise millions in VC money for their "AI" company.