3 ms·
I suspect this is also why large industrial ML labs have mixed productivity. My observation has been that people decide that a problem is important because some
by lumost 3y ago
I suspect this is also why large industrial ML labs have mixed productivity. My observation has been that people decide that a problem is important because someone high up thought it was important. The researchers using the methodology that shows the largest offline impact are rewarded - not those with the most rigorous finding.
This creates a strong incentive to fudge the results and never ship anything. I knew of one scientist who figured out you could just fake the entire thing and skip the work part. He coasted for 5 years doing this.
- golergka 3y agoWhat happened after 5 years? Did he get caught? Or just founded his own unicorn ai startup
- lumost 3y agoI think around year 4 a manager made the mistake of believing the research and pushing hard on it. This in turn became a tracked goal with a multi-year timeline. The individual left before things could fall on them too heavily.
- mmmmpancakes 3y agohow do you work 4 years on a project before having a tracked goal. absurd. sounds like management's fault.
- dchftcs 3y agoThe goals before the 4 years were probably based on results claimed, not making a product. Research management is difficult and efficiently running a lab requires some degree of trust that researchers don't lie. I think the bigger part of management's fault here is that they should have spent some time on verifying the research before dedicating more resources.
- lumost 3y agoAlso setting up mechanisms to verify results and share results. Even a basic mechanism of “check in your code” would have served the purpose of identifying fake work. A code review mechanism can help stop irresponsible work.
- fshbbdssbbgdd 3y ago“Check in your code” can identify certain kinds of blatant fraud, but it only does so much. For example, you can p-hack by evaluating your result in many different ways and then commit the version that gives the best number. It takes a lot of discipline to prevent this at an organization level. You can even have an emergent group form of this, where people collectively p-hack by individually evaluating their work and randomly finding lottery tickets. I’ve seen this phenomenon extend from research all the way to shipped product changes.
- coldtea 3y agoIncidentally that's how the best research with the most impact had been made in the past. Letting brillians minds play around and supporting their tinkering.
- ct0 3y agoSame goes for politics. Grant money available on partisan issues is quite the motivator too. I was doing research on a researcher and publisher at rutgers: https://gunviolenceresearchcenter.rutgers.edu/mike-anestis-phd/ https://gunviolenceresearchcenter.rutgers.edu/mike-anestis-p... and was surprised to learn how much funding him and his team has received. Literally millions, much more than small market research firms.
- pyuser583 3y agoI remember a thing a while back where Texas cut family planning funding. Around the same time maternal deaths skyrockets. Tons of research, sponsored by pro-choice medical groups Turns out the rise in maternal deaths preceded cuts in funding. Speaking to my relatives in Texas, a bunch of ERs shut down around the same time - which could explain the deaths. Abortion is a hit button issue - family planning is to a lesser degree. But plain old ERs closing - that’s not political. Just a statistic.