7 ms·

This is not just isolated to machine learning, but most technical fields (at least one other to my knowledge). I used to work in microscopy image analysis and

This is not just isolated to machine learning, but most technical fields (at least one other to my knowledge). I used to work in microscopy image analysis and the papers often would obfuscate the fact that they were not exactly doing anything new by using what looks like fancy math and some trendy names. One of the most outrageous examples is this "high profile" paper that says it does compressive sensing with superresolution microscopy - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ except I don't think they do; the math when you remove the bullshit sounds more like deconvolution than anything else (and the results are only as good). Yet, it got reviewed and accepted by Nature Methods, and is cited by 360 papers already. Why? Apparently no one in this field knows what compressive sensing really means. At least one professor in the field when I confronted him, just said he doesn't have time to go through compressive sensing literature first before evaluating this paper. What's the root cause? Frankly I'd argue the majority of professors nowadays aren't smart in innovation but smart in hustling. Because hustlers are who become professors in today's academic climate. They are able to publish good papers still if the field isn't mature, but if the field is saturated, you have to be really smart to make meaningful progress, and these hustlers are not. So they just try to find some way to wrap meaningless progress in fancy math and shove it in papers. The papers also go to reviewers who are similar hustlers (not every paper can be reviewed by Hinton ) so they either don't notice the problem or they do but let it slide because it's just their colleague (yay for journals asking for "suggested reviewers" to the authors itself).

deepnotderp 7y ago
I'm somewhat familiar with compressed sensing, but I only glanced through the linked paper for like a minute, but it looks like you're right. They seem to be taking the overlapped returns and doing some sort of weird sparse basis deconvolution. Specifically "Using compressed sensing, we analyzed a simulated image with 100 molecules randomly distributed in a 4 µm × 4 µm region (Fig. 1a). Although the images of individual molecules completely overlapped, we could identify almost all of the molecules. " Unless that is a close enough approximation to a random projection? I'm not that familiar with STORM.
- ramraj07 7y ago
  My understanding is that you need to apply masks at the sampling stage, and there are legitimate researchers trying to play with that concept using single pixel cameras. Never heard of anyone else try compressed sensing with regular CCD or CMOS sensor array cameras.
  - deepnotderp 7y ago
    Yeah exactly. I'm not familiar enough with STORM to say if it already provides something close enough to a random projection simply by the acquisition process.
Iv 7y ago
On my first internship I worked at a lab and at one point a researcher asked all the staff to be nice enough to read his draft to spot mistakes. I obliged as well, not understanding he meant typos. I asked him about an equation I did not understand in his paper, because I could not link some terms to the rest of the text. He answered that he did just copy it from another paper, that it only made sense in the original publication and that he was not really understanding it either but just enough to know it had to be there. On another internship I worked for a researcher who bragged about having published almost 10 papers on one of his algorithmic discoveries without ever revealing totally (which allowed him to start a for-profit company with it) You know, the whole publication + review + reproduction thing really helped science become a more solid process, but we need something more elaborate now. Probably some kind of reputation system that would not just be the number of citations by friends and colleagues.
- sevensor 7y ago
  I had this experience with multi-objective evolutionary algorithms. I worked with a researcher who was hung up on the idea of "rotationally invariant search operators." This idea made little sense in the first place. To the extent that it meant anything at all, the properties he ascribed to the search operators were more or less the opposite of their actual behavior.
  - polynomial 7y ago
    https://etda.libraries.psu.edu/files/final_submissions/11634 https://etda.libraries.psu.edu/files/final_submissions/11634 Interesting, to say the least, though I can't say I have time to read all 150 pages at this very moment.
    - sevensor 7y ago
      Chapter 6 is the only place you'll find me talking about rotational invariance, and unfortunately I had to recite the B.S. party line there. If you read any of it, read chapter 5 and chapter 2.
- 7y ago
godelski 7y ago
I'm not so sure that it is hustling, though I'm sure that exists. I'm a firm believer of Hanlon's Razor [0], and we shouldn't rush to attribute malice. I do think there is a problem that our breadth of knowledge, as humans, is far larger than what one person can understand. There's famous examples of revolutions in science being claimed as mundane results. Topologists said Nash was just applying topology to economics and it was nothing new (to them). Mathematicians saw Einstein's results as unsurprising because of the tensor analysis. (Some of these are over exaggerated and there's definitely a post hoc superiority complex in play). But if we just take this at face value, is any of this bad? I would argue no, because it still takes someone to connect the dots between fields and push studies to think in those ways. But one thing is for sure, as we gain more knowledge it is more likely that someone else independently discovers something that was already discovered. It is also more likely that some of these are rediscoveries of ideas that were not useful at the time. I think there is a way to solve this though, but I'm not sure we can (yet). We need some good way to check research in a cross-disciplinary manner. Not only that, but in a highly technical way. [0] https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
- btrettel 7y ago
  > There's famous examples of revolutions in science being claimed as mundane results. Topologists said Nash was just applying topology to economics and it was nothing new (to them). Mathematicians saw Einstein's results as unsurprising because of the tensor analysis. (Some of these are over exaggerated and there's definitely a post hoc superiority complex in play). Do you have any references for these examples?
  - godelski 7y ago
    I could not find any through a quick search. Maybe it is convoluted search terms. I got a degree in physics and was pretty close to the math department. This is just stuff that I've heard from both departments and seen in a few documentaries. I had seen a direct source on Nash (IIRC someone upset about the Nobel), but that was years ago and I can't find it easily. I did try to give a bunch of side notes to downplay and justify it in ways that may be conceited, but also reasonably human. I don't know if these stories I heard were a single person (I'd be surprised if there weren't at least one person!) or a larger group. I wouldn't be surprised if a ridiculous voice got amplified (can't think of any times that happened in modern times.....)
btrettel 7y ago
In my view many if not most professors are "careerists" in the sense that they put advancement of their career over accuracy. I wouldn't quite say that you need to be really smart to make progress in most fields, though being smart helps. If you can do a lot of work often you can get a great result without being a supergenius. Ultimately it's much easier to fool reviewers than to do good research. I think most researchers are somewhat innocent in the sense that they produce bad research without knowing it's bad. Many of these folks seem apathetic to the quality of research, as you found in the professor who said they don't have time to go through the literature. As far as I'm concerned, if you don't have time to go through the literature (at a reasonable level, i.e., more than is typical now), you don't have time to do research at all.
trenchgun 7y ago
PSA: Use https://pubpeer.com/ https://pubpeer.com/ and https://retractionwatch.com/ https://retractionwatch.com/ There is also a browser plugin for pubpeer and a zotero plugin for retraction watch database: https://chrome.google.com/webstore/detail/pubpeer/fmcdfigcdfkdghdklblbbpacikcchbbh?hl=en https://chrome.google.com/webstore/detail/pubpeer/fmcdfigcdf... https://www.zotero.org/blog/retracted-item-notifications/ https://www.zotero.org/blog/retracted-item-notifications/
- btrettel 7y ago
  I'm really disappointed that PubPeer is used so little outside of bio/medical/social science fields. I'm not sure why this is. There seems to be essentially negligible interest in spotting mistakes in the literature in CS/physics/math/engineering (outside of materials science, just glacing at the site). (For the record: I have posted a comment on PubPeer myself so I'm not a hypocrite.)
3xblah 7y ago
"... yay for journal asking for "suggested reviewers" to the authors [themselves])" That is common practice at a journal like Nature when there are a limited number of investigators in the field with the reputation and experience "necessary" to evaluate the work. Of course the chosen panel of reviewers is supposed to remain anonymous. How would you improve the system?
- ramraj07 7y ago
  >Of course the chosen panel of reviewers is supposed to remain anonymous. If only! My professors were quite adept at figuring out who the reviewers were, especially if they're someone from their "cabal", because every investigator has their own grammar and language style and whether professors are good at science or not, they are most definitely really good at English (gotta write those grants perfect!). Typical practice nowadays, is that these journals would choose two reviewers from the suggested list, and get one other by themselves just to try to be unbiased. Hence, the fate of the paper would reside on that single reviewer (my experience has been that the "suggested" reviewers will never reject the paper unless it's absolute shit). Perhaps the journals should check the citation history between the suggested reviewer and the author to make sure they are not mutually citing each other and stroking their backs.
foobarbecue 7y ago
You really hit the nail on the head there about hustlers VS innovators. This is also epidemic in my fields - volcanology, planetary science, and robotics.
qmmmur 7y ago
No one in academia has time except for a few at the top. We're constantly swamped with tasks that aren't research and we're not paid to research. What do people expect?
- chrisseaton 7y ago
  > we're not paid to research Isn't it in your job description, and what your performance is assessed against?
  - qmmmur 7y ago
    It depends, it is more complicated than having it as a metric in your contract and being paid to meet some demand. Generally what happens is you are expected to research outside all the other responsibilities that are not research which consume your contracted hours. So undergrad teaching, administration, grant applications etc take the majority of your day alongside managing the welfare of students you are responsible for. Don't forget meetings, committees, staff training days, postgraduate supervision and development of up to date course materials. I wouldn't even bother trying to fit in a personal life while trying to stay on top of the field. When you're graded against whatever metric your institution uses they will decide whether or not to pile on more undergraduate teaching and administrative responsibilities (basically seen as the shit jobs) which in turn reduces your time available to research. That is problematic in and of itself as developing successful undergraduate systems requires people who know the field well and can teach relevant materials born from practice. What you end up seeing is young researchers who go into Lecturer positions end up deciding the next 30 years of their life on the first evaluation rather than being invested in as researchers who could contribute to their own projects or to teams. To compound the issue university administration has exploded in the last 10-20 years creating more middle management, more cost and more downward pressure on researchers to adhere to the demands of people less qualified than themselves. The same administrators are also the ones calling for mass casualisation of teaching roles to save money without seeing the long term downsides this has for building institutional knowledge and departments which can generate high quality research. Luckily I am in an institute that mandates at least 1 day per week is a research day and you are allowed to work from home and be uncontactable on that day if you wish but other people have it so much worse.
itronitron 7y ago
A fair amount of techies suffer from 'math sickness' which is most often spread by hot papers at conferences that apply some difficult to grasp mathematical approach to a new problem domain.

deepnotderp 7y ago

I'm somewhat familiar with compressed sensing, but I only glanced through the linked paper for like a minute, but it looks like you're right. They seem to be taking the overlapped returns and doing some sort of weird sparse basis deconvolution. Specifically "Using compressed sensing, we analyzed a simulated image with 100 molecules randomly distributed in a 4 µm × 4 µm region (Fig. 1a). Although the images of individual molecules completely overlapped, we could identify almost all of the molecules. " Unless that is a close enough approximation to a random projection? I'm not that familiar with STORM.

ramraj07 7y ago
My understanding is that you need to apply masks at the sampling stage, and there are legitimate researchers trying to play with that concept using single pixel cameras. Never heard of anyone else try compressed sensing with regular CCD or CMOS sensor array cameras.
- deepnotderp 7y ago
  Yeah exactly. I'm not familiar enough with STORM to say if it already provides something close enough to a random projection simply by the acquisition process.

Iv 7y ago

On my first internship I worked at a lab and at one point a researcher asked all the staff to be nice enough to read his draft to spot mistakes. I obliged as well, not understanding he meant typos. I asked him about an equation I did not understand in his paper, because I could not link some terms to the rest of the text. He answered that he did just copy it from another paper, that it only made sense in the original publication and that he was not really understanding it either but just enough to know it had to be there. On another internship I worked for a researcher who bragged about having published almost 10 papers on one of his algorithmic discoveries without ever revealing totally (which allowed him to start a for-profit company with it) You know, the whole publication + review + reproduction thing really helped science become a more solid process, but we need something more elaborate now. Probably some kind of reputation system that would not just be the number of citations by friends and colleagues.

sevensor 7y ago
I had this experience with multi-objective evolutionary algorithms. I worked with a researcher who was hung up on the idea of "rotationally invariant search operators." This idea made little sense in the first place. To the extent that it meant anything at all, the properties he ascribed to the search operators were more or less the opposite of their actual behavior.
- polynomial 7y ago
  https://etda.libraries.psu.edu/files/final_submissions/11634 https://etda.libraries.psu.edu/files/final_submissions/11634 Interesting, to say the least, though I can't say I have time to read all 150 pages at this very moment.
  - sevensor 7y ago
    Chapter 6 is the only place you'll find me talking about rotational invariance, and unfortunately I had to recite the B.S. party line there. If you read any of it, read chapter 5 and chapter 2.
7y ago

godelski 7y ago

I'm not so sure that it is hustling, though I'm sure that exists. I'm a firm believer of Hanlon's Razor [0], and we shouldn't rush to attribute malice. I do think there is a problem that our breadth of knowledge, as humans, is far larger than what one person can understand. There's famous examples of revolutions in science being claimed as mundane results. Topologists said Nash was just applying topology to economics and it was nothing new (to them). Mathematicians saw Einstein's results as unsurprising because of the tensor analysis. (Some of these are over exaggerated and there's definitely a post hoc superiority complex in play). But if we just take this at face value, is any of this bad? I would argue no, because it still takes someone to connect the dots between fields and push studies to think in those ways. But one thing is for sure, as we gain more knowledge it is more likely that someone else independently discovers something that was already discovered. It is also more likely that some of these are rediscoveries of ideas that were not useful at the time. I think there is a way to solve this though, but I'm not sure we can (yet). We need some good way to check research in a cross-disciplinary manner. Not only that, but in a highly technical way. [0] https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor

btrettel 7y ago
> There's famous examples of revolutions in science being claimed as mundane results. Topologists said Nash was just applying topology to economics and it was nothing new (to them). Mathematicians saw Einstein's results as unsurprising because of the tensor analysis. (Some of these are over exaggerated and there's definitely a post hoc superiority complex in play). Do you have any references for these examples?
- godelski 7y ago
  I could not find any through a quick search. Maybe it is convoluted search terms. I got a degree in physics and was pretty close to the math department. This is just stuff that I've heard from both departments and seen in a few documentaries. I had seen a direct source on Nash (IIRC someone upset about the Nobel), but that was years ago and I can't find it easily. I did try to give a bunch of side notes to downplay and justify it in ways that may be conceited, but also reasonably human. I don't know if these stories I heard were a single person (I'd be surprised if there weren't at least one person!) or a larger group. I wouldn't be surprised if a ridiculous voice got amplified (can't think of any times that happened in modern times.....)

btrettel 7y ago

In my view many if not most professors are "careerists" in the sense that they put advancement of their career over accuracy. I wouldn't quite say that you need to be really smart to make progress in most fields, though being smart helps. If you can do a lot of work often you can get a great result without being a supergenius. Ultimately it's much easier to fool reviewers than to do good research. I think most researchers are somewhat innocent in the sense that they produce bad research without knowing it's bad. Many of these folks seem apathetic to the quality of research, as you found in the professor who said they don't have time to go through the literature. As far as I'm concerned, if you don't have time to go through the literature (at a reasonable level, i.e., more than is typical now), you don't have time to do research at all.

trenchgun 7y ago

PSA: Use https://pubpeer.com/ https://pubpeer.com/ and https://retractionwatch.com/ https://retractionwatch.com/ There is also a browser plugin for pubpeer and a zotero plugin for retraction watch database: https://chrome.google.com/webstore/detail/pubpeer/fmcdfigcdfkdghdklblbbpacikcchbbh?hl=en https://chrome.google.com/webstore/detail/pubpeer/fmcdfigcdf... https://www.zotero.org/blog/retracted-item-notifications/ https://www.zotero.org/blog/retracted-item-notifications/

btrettel 7y ago
I'm really disappointed that PubPeer is used so little outside of bio/medical/social science fields. I'm not sure why this is. There seems to be essentially negligible interest in spotting mistakes in the literature in CS/physics/math/engineering (outside of materials science, just glacing at the site). (For the record: I have posted a comment on PubPeer myself so I'm not a hypocrite.)

3xblah 7y ago

"... yay for journal asking for "suggested reviewers" to the authors [themselves])" That is common practice at a journal like Nature when there are a limited number of investigators in the field with the reputation and experience "necessary" to evaluate the work. Of course the chosen panel of reviewers is supposed to remain anonymous. How would you improve the system?

ramraj07 7y ago
>Of course the chosen panel of reviewers is supposed to remain anonymous. If only! My professors were quite adept at figuring out who the reviewers were, especially if they're someone from their "cabal", because every investigator has their own grammar and language style and whether professors are good at science or not, they are most definitely really good at English (gotta write those grants perfect!). Typical practice nowadays, is that these journals would choose two reviewers from the suggested list, and get one other by themselves just to try to be unbiased. Hence, the fate of the paper would reside on that single reviewer (my experience has been that the "suggested" reviewers will never reject the paper unless it's absolute shit). Perhaps the journals should check the citation history between the suggested reviewer and the author to make sure they are not mutually citing each other and stroking their backs.

foobarbecue 7y ago

You really hit the nail on the head there about hustlers VS innovators. This is also epidemic in my fields - volcanology, planetary science, and robotics.

qmmmur 7y ago

No one in academia has time except for a few at the top. We're constantly swamped with tasks that aren't research and we're not paid to research. What do people expect?

chrisseaton 7y ago
> we're not paid to research Isn't it in your job description, and what your performance is assessed against?
- qmmmur 7y ago
  It depends, it is more complicated than having it as a metric in your contract and being paid to meet some demand. Generally what happens is you are expected to research outside all the other responsibilities that are not research which consume your contracted hours. So undergrad teaching, administration, grant applications etc take the majority of your day alongside managing the welfare of students you are responsible for. Don't forget meetings, committees, staff training days, postgraduate supervision and development of up to date course materials. I wouldn't even bother trying to fit in a personal life while trying to stay on top of the field. When you're graded against whatever metric your institution uses they will decide whether or not to pile on more undergraduate teaching and administrative responsibilities (basically seen as the shit jobs) which in turn reduces your time available to research. That is problematic in and of itself as developing successful undergraduate systems requires people who know the field well and can teach relevant materials born from practice. What you end up seeing is young researchers who go into Lecturer positions end up deciding the next 30 years of their life on the first evaluation rather than being invested in as researchers who could contribute to their own projects or to teams. To compound the issue university administration has exploded in the last 10-20 years creating more middle management, more cost and more downward pressure on researchers to adhere to the demands of people less qualified than themselves. The same administrators are also the ones calling for mass casualisation of teaching roles to save money without seeing the long term downsides this has for building institutional knowledge and departments which can generate high quality research. Luckily I am in an institute that mandates at least 1 day per week is a research day and you are allowed to work from home and be uncontactable on that day if you wish but other people have it so much worse.

itronitron 7y ago

A fair amount of techies suffer from 'math sickness' which is most often spread by hot papers at conferences that apply some difficult to grasp mathematical approach to a new problem domain.