3 ms·

Just to add another data point: I asked an Ivy League professor about hype and they said the exact opposite of your Ivy League professor. Anyway, that's certai

Just to add another data point: I asked an Ivy League professor about hype and they said the exact opposite of your Ivy League professor. Anyway, that's certainly the hype part that your anecdote refers to. Now how about the hokum? Did your Ivy League professor draw a computer on a cardboard box and put a kid inside there? Or did they use it for a bona fide machine learning research project? I think those are fair questions as there's at least one story of a Media Lab project that misrepresented its technology and did not really perform the tasks which it claimed to have performed.

xiaolingxiao 7y ago
Hmm it's a bit more subtle than that. There are all kinds of ways to misrepresent progress, the simplest way is to pick bogus metrics. For example, all of machine-learning based dialog system work uses metrics that were designed for machine translation ( basically n-gram overlaps ), this is an example of a metric that you can improve on but is meaningless. So the resulting system is very much the dog-and-pony show variety. Or people who use simulations to train and test ML systems, and say the metrics get better but do not address the sim2real gap. > I asked an Ivy League professor about hype and they said the exact opposite of your Ivy League professor The context matters, I have heard the opposite response as well, from a professor I was working for, while he was doing just that: packaging narratives w/o regard to how well the methods actually work. The honest take came at a wine-and-cheese party from a professor I was not working for. There is a real conflict of interest here, and if people stand to gain by withholding information, they will do Just that.
dkonofalski 7y ago
Just curious but what's the exact opposite of those statements? That he hasn't had to convince anyone of their work in order to get money? Or that it isn't based on that but on results? As someone that doesn't live in that world, I have no idea what the "exact opposite" of that really is.
- mattkrause 7y ago
  For research funding, the sweet spot is the flashiest idea that seems extremely likely to work. It’s hard to get grants to try something really out there, especially from the NIH or NSF; DARPA is a bit more adventurous. Pilot data helps with this, but those experiments cost money too, so one approach is to make the project sound almost, but not quite, obvious in light of existing work. At the same time, it’s also really hard to get funding to confirm something that everybody “knows”, even when the underlying data is actually not very convincing. These experiments need some hype.

xiaolingxiao 7y ago

Hmm it's a bit more subtle than that. There are all kinds of ways to misrepresent progress, the simplest way is to pick bogus metrics. For example, all of machine-learning based dialog system work uses metrics that were designed for machine translation ( basically n-gram overlaps ), this is an example of a metric that you can improve on but is meaningless. So the resulting system is very much the dog-and-pony show variety. Or people who use simulations to train and test ML systems, and say the metrics get better but do not address the sim2real gap. > I asked an Ivy League professor about hype and they said the exact opposite of your Ivy League professor The context matters, I have heard the opposite response as well, from a professor I was working for, while he was doing just that: packaging narratives w/o regard to how well the methods actually work. The honest take came at a wine-and-cheese party from a professor I was not working for. There is a real conflict of interest here, and if people stand to gain by withholding information, they will do Just that.

dkonofalski 7y ago

Just curious but what's the exact opposite of those statements? That he hasn't had to convince anyone of their work in order to get money? Or that it isn't based on that but on results? As someone that doesn't live in that world, I have no idea what the "exact opposite" of that really is.

mattkrause 7y ago
For research funding, the sweet spot is the flashiest idea that seems extremely likely to work. It’s hard to get grants to try something really out there, especially from the NIH or NSF; DARPA is a bit more adventurous. Pilot data helps with this, but those experiments cost money too, so one approach is to make the project sound almost, but not quite, obvious in light of existing work. At the same time, it’s also really hard to get funding to confirm something that everybody “knows”, even when the underlying data is actually not very convincing. These experiments need some hype.