5 ms·

I think we are 80% in agreement but still using words differently. > when I get grant money I need to account for how every dollar is spent Yes I know, but th

I think we are 80% in agreement but still using words differently. > when I get grant money I need to account for how every dollar is spent Yes I know, but that's not what I mean by accountability. Again: nobody is upset with academics because of expenses scandals or taking too many expensive flights. Well, except maybe for climatologists who supposedly take more flights than the average academic, but that's due to the perception of hypocrisy rather than concern over cost. People are getting upset because when they download and read papers, the papers turn out to be bad and there are no visible consequences for that. Even just getting a clearly fraudulent paper retracted is reported to be a nightmare, according to people who search for scientific fraud as a hobby like Elizabeth Bik. And I've read endless reams of terrible papers that were useless or outright deceptive, I tried reporting a few and nobody ever cared. Now, you're arguing that there is accountability of the following form: > It's accountability in the form of: if you didn't do what you promised you'd do, then you don't get any more money This is true given that scientists are promising the NSF to publish papers, not strictly speaking to do research, and therefore by implication promising to come up with interesting claims, not necessarily true claims. But that's not what we want. This is an inevitable problem with government funding of research. The buyer, the government, cannot really check if the claims they're buying from scientists are true, so they need proxies like did it get published, did it get cited, etc. But those aren't the same things. Corporate research doesn't have this problem because the corporate will try to apply the research at some point and if it was fraudulent they will discover it at that point, and of course they're strongly incentivized to ensure it never gets to that point in the first place. In theory the government could write grants in such a way that money is awarded independent of what claims end up being made, instead awarding money for the quality of work done. That's what you're arguing for here. And indeed corporate labs write contracts in this exact way. Scientists get a salary in a corporate lab, they don't have to write grants. They do have to convince their management chain that the research is worth funding, but there are many different ways to do that which don't involve continually publishing astonishing claims in scientific journals. You're asking me to propose how science should work instead but, indeed, you already know my answer: eliminate the NSF completely, and stop subsidizing student loans. All science should be funded by companies. They have already solved the problems you're treating as novel / intractable above. Scientists are awarded salaries and promotions by firms on a more flexible basis than the NSF. Importantly, they are rewarded for doing research not producing claims. Companies can do this because they have management structures sufficiently well staffed to closely monitor what scientists are doing. That means if a firm is truly committed to research then the scientists will get paid even if their programme has some dry years. Plus there's a huge body of law handling fraud and corruption in the workplace. At the same time, firms are incentivized to eliminate the research that is probably always going to be nearly useless. Outside of firms selling books or self help courses I doubt many would subsidize sociology or gender studies for example, and it's also unclear that would be a loss. Your argument about who it would or wouldn't be good for seems a bit contradictory and I struggled to follow it. You're arguing it would be both bad for Google and Microsoft yet also make them stronger. I disagree with both possibilities: I think they would hardly notice the difference and it wouldn't affect how powerful they are. Having worked for one of those companies and also worked at a startup where we often read research papers in a certain subfield of CS with views to maybe applying them, my view is that even in the relatively good field of computer science, most academic output is useless and has no impact. These firms do not rely heavily on government funded research: - The web was very briefly funded for a couple of years as a side project of CERN, but then R&D was taken over by the private sector where it remained ever since. Page & Brin never even finished their PhD before moving their research into the private sector! It's hardly a mystery where the next Google will come from - probably the same place the previous one did, a garage in Silicon Valley. - What government funded tech was Microsoft built on? The internet? Microsoft is still with us in spite of the internet, not because of it! Or are you going back to military computers in World War 2? Military R&D is different, governments can fund that semi-effectively because they actually use the outputs. - Neural networks were a backwater until Jeff Dean resurrected the field using the resources of the private sector, academia has been chasing to catch up ever since. There are a lot of other examples. The DARPA Grand Challenge is not an example of what I'm talking about because: 1. DARPA is military research and therefore structured differently to how the NSF does things. The very structure of it as a Grand Challenge is a clue here: the output of the programme was cars (not) going round a track, not papers and citations. 2. I'm not arguing for prosecution of researchers who end up with null results!

ModernMech 5y ago
I'll try not to do another wall of text since we're mostly in agreement, but I will make a couple final comments: > Your argument about who it would or wouldn't be good for seems a bit contradictory and I struggled to follow it. You're arguing it would be both bad for Google and Microsoft yet also make them stronger. What I meant is, if e.g. Page and Brin in 1998 had no access to government funding and research because it was privatized by e.g. AOL, there wouldn't be a Google today. But if we were to privatize all research, Google of today would certainly like that insofar as it strengthens their market position (jut like the AOL of 1998 would like the situation), but it also means they have to start funding more research because now they can't get any from the public. > - The web was very briefly funded - What government funded tech was Microsoft built on? - Neural networks were a backwater But the point is that it all started with government funding, so we need to be very careful about the consequences of privatizing it all. Today, ideas start out funded by the government, they gain legs in academia, move out into corporations, and are productized and disseminated to the public in the form of consumer goods. This is the progress pipeline, and it's proven extremely effective and enduring at driving innovation. You want to cut out the beginning of the process because you think corporations can handle that part, but I don't think you've really demonstrated that. Can you point to any tech product out there that is exclusively built on in-house, private research? I certainly can't think of one. For example, you bring up the origin of Page & Brin. Yes, they never finished their Ph.D., but the fact is they did meet in grad school while they were doing NSF funded work. Brin was at Stanford on an NSF fellowship. They built the first prototype of Google on an NSF grant. They were mentored by academics who also were funded by the NSF as professors and graduate students themselves. You take that funding away, and maybe these two people never meet, maybe they never learn what they need to get that spark of insight. So I agree with you that the next Google will come from the same place the previous one did - a government-funded research lab in Silicon Valley. The garage is where they moved their operation only after they had already used a lot of NSF money to get their start. > 1. DARPA is military research and therefore structured differently to how the NSF does things. The very structure of it as a Grand Challenge is a clue here: the output of the programme was cars (not) going round a track, not papers and citations. The processes of getting grants from NSF and DARPA are very similar, and in most cases the deliverable is a paper. The Grand Challenges are the exception of DARPA funding, not the rule. > Military R&D is different, governments can fund that semi-effectively because they actually use the outputs. Yes and no. DARPA would like to use the fruits of its funded research, but it funds projects on a very long timescale, so what it funds may or may not be used in the long term. Sometimes the research is not to strengthen the military per se, but to strengthen American interests though creating domestic tech sectors. e.g. I'm sure the military would like to use autonomous vehicles, but what's even better is for America to have its own domestic autonomous car sector that can produce those vehicles. > most academic output is useless and has no impact. You've tried to make the case that we should optimize toward useful research, and companies are better at identifying useful research because they have a profit motive, but I still think it's difficult to say today what research will be important 30-40 years down the line. DARPA recognizes that it's very hard to tell how useful research will be ahead of time, and that corporations don't like to engage in foundational research when there is no obvious short-term path to profit. This was the entire point of the Grand Challenge series, and it worked out well -- they wanted to bootstrap the autonomous car industry, so they paid researchers to get them rolling and now look where we are. If the government hadn't gotten involved, there probably wouldn't be an autonomous car sector in the US today. There are plenty of cases in our history where some technology seemed useless initially turned out to be bigger than anyone could have imagined. We need to be careful not squelch those ideas too quickly because they don't return an immediate profit. Things like the Internet and neural networks come to mind. A lot of people, particularly large corporations, thought the Internet was a toy when it first was introduced. Neural networks seemed like a dead end and then found new life. But the fact is they started in academia. The Deepmind arcade paper and essentially the entire deep reinforcement learning field today is based on decades-old research funded by the UK government. What if that research was locked away in a UK corporation? Would Deepmind even exist? That research was a toy for 30 years, until it wasn't. The whole point of DARPA and other government funding agencies is that they don't know what the winners are ahead of time, and I don't think corporations can know this either. (if they could, why didn't they do more to fund RL research 30 years ago?). Therefore we shouldn't try to optimize for obvious winners because we'll miss out on non-obvious winners, which bring the biggest upsides. This means we have to fund losers and research that ends up not being useful, and we should be okay with that, because things have turned out pretty well over all. > 2. I'm not arguing for prosecution of researchers who end up with null results! Sorry I thought you were with this: We will therefore prosecute you for research fraud and failure to meet the terms of your contract. I guess you mean failing to meet the terms of your contract and fraudulently representing that. But it still doesn't address the incentive to commit fraud because if you fail to meet your objectives, you're still not going to get published and therefore won't get the next grant, so your career is still derailed. It just means people will try to hide the fraud better. After I typed all this I realized I failed at my pledge to not give you a wall of text. Oops!

ModernMech 5y ago

I'll try not to do another wall of text since we're mostly in agreement, but I will make a couple final comments: > Your argument about who it would or wouldn't be good for seems a bit contradictory and I struggled to follow it. You're arguing it would be both bad for Google and Microsoft yet also make them stronger. What I meant is, if e.g. Page and Brin in 1998 had no access to government funding and research because it was privatized by e.g. AOL, there wouldn't be a Google today. But if we were to privatize all research, Google of today would certainly like that insofar as it strengthens their market position (jut like the AOL of 1998 would like the situation), but it also means they have to start funding more research because now they can't get any from the public. > - The web was very briefly funded - What government funded tech was Microsoft built on? - Neural networks were a backwater But the point is that it all started with government funding, so we need to be very careful about the consequences of privatizing it all. Today, ideas start out funded by the government, they gain legs in academia, move out into corporations, and are productized and disseminated to the public in the form of consumer goods. This is the progress pipeline, and it's proven extremely effective and enduring at driving innovation. You want to cut out the beginning of the process because you think corporations can handle that part, but I don't think you've really demonstrated that. Can you point to any tech product out there that is exclusively built on in-house, private research? I certainly can't think of one. For example, you bring up the origin of Page & Brin. Yes, they never finished their Ph.D., but the fact is they did meet in grad school while they were doing NSF funded work. Brin was at Stanford on an NSF fellowship. They built the first prototype of Google on an NSF grant. They were mentored by academics who also were funded by the NSF as professors and graduate students themselves. You take that funding away, and maybe these two people never meet, maybe they never learn what they need to get that spark of insight. So I agree with you that the next Google will come from the same place the previous one did - a government-funded research lab in Silicon Valley. The garage is where they moved their operation only after they had already used a lot of NSF money to get their start. > 1. DARPA is military research and therefore structured differently to how the NSF does things. The very structure of it as a Grand Challenge is a clue here: the output of the programme was cars (not) going round a track, not papers and citations. The processes of getting grants from NSF and DARPA are very similar, and in most cases the deliverable is a paper. The Grand Challenges are the exception of DARPA funding, not the rule. > Military R&D is different, governments can fund that semi-effectively because they actually use the outputs. Yes and no. DARPA would like to use the fruits of its funded research, but it funds projects on a very long timescale, so what it funds may or may not be used in the long term. Sometimes the research is not to strengthen the military per se, but to strengthen American interests though creating domestic tech sectors. e.g. I'm sure the military would like to use autonomous vehicles, but what's even better is for America to have its own domestic autonomous car sector that can produce those vehicles. > most academic output is useless and has no impact. You've tried to make the case that we should optimize toward useful research, and companies are better at identifying useful research because they have a profit motive, but I still think it's difficult to say today what research will be important 30-40 years down the line. DARPA recognizes that it's very hard to tell how useful research will be ahead of time, and that corporations don't like to engage in foundational research when there is no obvious short-term path to profit. This was the entire point of the Grand Challenge series, and it worked out well -- they wanted to bootstrap the autonomous car industry, so they paid researchers to get them rolling and now look where we are. If the government hadn't gotten involved, there probably wouldn't be an autonomous car sector in the US today. There are plenty of cases in our history where some technology seemed useless initially turned out to be bigger than anyone could have imagined. We need to be careful not squelch those ideas too quickly because they don't return an immediate profit. Things like the Internet and neural networks come to mind. A lot of people, particularly large corporations, thought the Internet was a toy when it first was introduced. Neural networks seemed like a dead end and then found new life. But the fact is they started in academia. The Deepmind arcade paper and essentially the entire deep reinforcement learning field today is based on decades-old research funded by the UK government. What if that research was locked away in a UK corporation? Would Deepmind even exist? That research was a toy for 30 years, until it wasn't. The whole point of DARPA and other government funding agencies is that they don't know what the winners are ahead of time, and I don't think corporations can know this either. (if they could, why didn't they do more to fund RL research 30 years ago?). Therefore we shouldn't try to optimize for obvious winners because we'll miss out on non-obvious winners, which bring the biggest upsides. This means we have to fund losers and research that ends up not being useful, and we should be okay with that, because things have turned out pretty well over all. > 2. I'm not arguing for prosecution of researchers who end up with null results! Sorry I thought you were with this: We will therefore prosecute you for research fraud and failure to meet the terms of your contract. I guess you mean failing to meet the terms of your contract and fraudulently representing that. But it still doesn't address the incentive to commit fraud because if you fail to meet your objectives, you're still not going to get published and therefore won't get the next grant, so your career is still derailed. It just means people will try to hide the fraud better. After I typed all this I realized I failed at my pledge to not give you a wall of text. Oops!