11 ms·

Why Does DARPA Work?

347 points by MKais 6y ago

fastball 6y ago
I'd like to write a comment but I generally prefer to have read the entire article before I do... see you in a week or so. Jokes aside, I agree with the author (I think) that DARPA has been a surprisingly effective org in an age of frequent failures of other orgs with similarly lofty goals. DARPA is effectively what things like the SoftBank Vision Fund should've been (wanted to be?). It would certainly be interesting to see what it would look like to have a privately run clone of DARPA if you injected as much cash as the Vision Fund did. Per the article, about $400M/y is spent on actual R&D by DARPA, where as Vision Fund has injected that amount into single companies many times over.
- deleted 6y ago
  [deleted]
- jariel 6y ago
  Vision Fund is there to throw huge amounts of money at a business in order to ramp it up quickly and capture large, global markets. Think Uber: disrupting the Taxi business (or just regulations ...). Once something like Uber starts to work, a 'Vision Fund' takes this fledgling thing and backs it with billions to conquer the world. It's not about moon-shots, it's about market power and speed on a global scale. Like hyper-supercharged 'Round C or D' - instead of doing an initial public offering for cash, you take on massive cash from Vision. Wether or not it will work is something else, but there's logic there.
  - fastball 6y ago
    Oh yeah, absolutely. I just kinda wish we had that level of private interest/investment with similarly lofty goals. Not just "eat the world".
dnautics 6y ago
Does DARPA work(anymore)? The first example of how it works was it's "badass program managers", but all of the program managers I've met have been, to put it lightly, idiots. One of the worst was the PM in charge of the entire subject research division. It turned out, they had a graduate thesis that was entirely observing an instrument artifact (showing bad jugement). Not too long after I interacted with them, there was a minor scandal in the community because postdoc who struggled with reproducing the effect and complained about the results was abused and railroaded by the PI in charge of the project. Shortly thereafter the PM didn't seem to be a PM anymore.
- throwawaygh 6y ago
  ...so your best anecdote about why DARPA doesn't work is that they fire incompetent PMs?
  - dnautics 6y ago
    Yes, after being promoted to the head of the division with several people speculating that they were being groomed for even higher posts. And it took a scandal to get rid of them. Person should never had been a PM in the first place. You never hear people say "it's proof that Congress works because they ejected that pedophile (sending lewd messages to the pages) from their seat".
    - throwawaygh 6y ago
      Hate to break it to you, but "an incompetent person rising through the ranks quickly before crashing and burning" is something that happens in pretty literally every large org. Any 60 year old organization with a large budget is going to have multiple instances of this happening.
      - dnautics 6y ago
        Perhaps I have a flair for the dramatic, but consider my statement that I haven't met a competent pm in the division I was in (n~5). The incentives are not that well-aligned. Think about it this way. If you're really smart, why do you become a DARPA PM, and not a PI? Yes there are a few good reasons, but the population it applies to have largely been selected out by the postdoc phase (where I would say you have had sufficient experience with crushing scientific and engineering failure onself and watching others to be effective) and the pool of candidates therefore is vanishing.
dsukhin 6y ago
The irony is palpable: > I would rather this be read by a few people motivated to take action than by a broad audience who will find it merely interesting. In that vein, if you find yourself wanting to share this on Twitter or Hacker News, consider instead sharing it with one or two friends who will take action on it. Thank you for indulging me! I'm glad of course it was shared here. As a distillation - I think the author's theme lies with enabling more _researchers_ rather than business people to take moonshots and do foundational knowledge building and discovery that redefines a field and then focus on commercialization from a birds eye view by technically capable visionaries. This is the SBIR [1] model (also a US Gov requirement to fund small business research for any federal agency with >$100M in funding), the Bell Labs model (which yeilded amazing foundational work like UNIX and the transistor and was a direct result of AT&T's monopoly and excess resources), and perhaps even the YC model (though that one is obviously focused on a shorter horizon and more on commercializing existing tech and more rarely on foundational research). I've personally thought about this problem a lot and done this at a small scale and would love to expand upon it. https://augmentedlabs.org https://augmentedlabs.org Would love to hear others experiences and thoughts. [1] https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Research https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Re...
xVedun 6y ago
It seems that this would be a much better way of funding research in a general sense, since that seems to be the general process of their problem solving. The only thing is that to apply this to other fields, the barrier for getting people that are equally motivated and intelligent about analyzing if the research is 'going' somewhere is high
riazrizvi 6y ago
IMO this analysis gets caught up looking for answers in process when the real problem that DARPA solves is a political-economic one. ATT didn't invent the internet not for lack of process, they were extremely innovative. They held back on innovations because they were making huge sums of money overcharging long distance mainframe-mainframe data link rates in the 50's-70's. DARPA succeeded as a trust buster. They have the resources and legality to make whatever they want, regardless of patents (because National Security/patent law). They can make technology a reality, then develop political support with working prototypes. They build things to show how the Government is getting screwed over by some giant defense corp, because of a lack of competition in certain types of contracts. I think they are becoming less relevant because the corporate-political landscape is becoming more trust-based.
- siege_engineer 6y ago
  Author here. Good point about DARPA serving a political-economic role. It's a different lens on the coordination role. Is the corporate-political landscape becoming more trust-based than the Military-Industrial Complex that Eisenhower warned about?
  - riazrizvi 6y ago
    I don't understand the comparison in reference to the question of 'Why Does DARPA Work?' or my comment. Military-Industrial Complex is where defense industry profit motives drive unwarranted political influence that shapes defense policy, foreign policy. Military-Industrial Complex drives defense spending up. Trust is about the erosion of market competition, which results in decrease product value-price ratio. In the context of defense product, it means that the products that the consumer (here military/govt) can buy from the market become increasingly expensive and weak. Which creates an opportunity for consumers to do-it-better-yourself-and-for-less, which to me, explains DARPA.
    - TheColorYellow 6y ago
      To a certain extent the MIC as you define it would explain the erosion of the market wouldn't it? Also I've read your comments a few times now and I'm not sure if I'm completely understanding your view of what is becoming more "trust-based". Are you saying that DARPA is more trusted by the consumer (the federal government) than the market entities (the contractors)? If so, isn't that exactly what government entities of all types are created for? These agencies exist as part of the government working for the government because it's difficult to have an effective private market created for these similar purposes. I think it makes the question about "what makes DARPA successful" so much more interesting from if we acknowledge the limitations of the sector. Many agencies, entities, or initiatives of various types have failed when given similar mandates and similar market conditions. That's even if you believe DARPA is truly as effective as is claimed at achieving it's mandate of innovation in the Defense and National Security sector...
currymj 6y ago
A bill, the Endless Frontiers Act, has been introduced in both houses of Congress to make a section of the National Science Foundation that works more like DARPA, while massively increasing its funding. The idea is to extend the DARPA model to many more non-military areas, just as suggested in this article. https://www.sciencemag.org/news/2020/05/us-lawmakers-unveil-bold-100-billion-plan-remake-nsf https://www.sciencemag.org/news/2020/05/us-lawmakers-unveil-... Certainly a timely article. I don't know whether it's a good thing or not. The DARPA model has certainly been productive, but it isn't suited for every research topic or subdiscipline. As discussed in the article, currently NSF is the most open to basic science of funding agencies, and gives grantees the most latitude in what they work on. That is a valuable thing to have in the ecosystem. If Congress expects the majority of publicly-funded research even from the NSF to be on short-term grants for specific visions and technologies, it will rule out working on a lot of important things.
- gh02t 6y ago
  There is also already ARPA-E, which is a similar mandate to DARPA (i.e., risky projects with high payoff) for energy technology. How effective it has been thus far is a matter of debate due to a variety of factors, but I like the basic idea. https://en.wikipedia.org/wiki/ARPA-E https://en.wikipedia.org/wiki/ARPA-E
- siege_engineer 6y ago
  Author here. I actually wrote a discussion of the Endless Frontier Act: https://benjaminreinhardt.com/the-endless-frontier-act/ https://benjaminreinhardt.com/the-endless-frontier-act/ I agree that the ARPA model is not suitable for all (or even the majority of research) and the act doesn't inspire confidence that shifting the NSF towards a DARPA-like model would do well.
  - lazyjeff 6y ago
    Just wanted to say, thanks for writing the article about DARPA, and this analysis of the endless frontiers act. From reading both articles, obviously you're very pro-DARPA and anti-NSF. I have two thoughts that might erode your thesis a bit, that 1) I think you're overestimating the amount of time that faculty spend on writing grants. We like to complain about it, but I've tracked my time to the minute over the past 7 years, and grant writing (both the proposals and the reports) is about 2% of my work time. 2) You might be misunderstanding indirect costs like many people, where you say "Universities can take more than half of grant money as administrative overhead". Indirect cost math is funny, but in order for administrative overhead to be more than half of the grant, the indirect cost rate would have to be over 100%. I can explain more if you're interested. Obviously we're both biased by our job, but it's still useful to read your perspective.
thoraway1010 6y ago
Small anecdote, but good experience with DARPA and SBIR both. One thing I liked - they seemed pretty low overhead / low hassle and outcome / objective focused. That contrasts VERY strong with most govt contracting which is under allowable cost or cost reimbursement setups where the absolute most important criteria is to bill enough costs to draw contract in the right cost buckets (which can be super annoying if local agencies require super complicated budget mod processes). In the cost based contracts, the focus really focuses on the accounting for the costs and other compliance related items. Ie, did you send someone to a conference with govt money, how can you prove you didn't use the govt money in this way or that way etc etc. Bam, welcome to personal activity reports with fund codes that no one understands (ie, major universities have an insane number of codes), and all the nightmares that follow including a fair bit of rule pending that even normally ethical folks find themselves being asked to to get through the paperwork. Seriously, you deliver the product at 50% of cost? You will get a nasty note from the head of your agency saying make sure you draw full contract because agency budget depends on the indirect portion of this award (ie, 30% to overhead) and even the govt agency supervising (who also budgeted based on a cut of full contract) AND other folks for whom leftover money makes life difficult (harder to close contract etc) AND because a lot of govt funding is on the repeat what we did last year model so drawing down everything avoids a cut next year when you may really need it. You determine it would be cheaper to do x vs y and that required a budget mod? Wait 1-2 months for commission approval if you even bother trying to fight it through (changes < 10% often ignored thankfully). I don't know how the accounting for SBIRs etc work, but somehow those projects always seemed more results oriented (so a LOT more fun to work on, focus is on getting a solution going).
- tia4tia 6y ago
  SBIR? Totally corrupt. Please don't ask. Can not, will not, must not tell. I can tell you one thing. If not the best but the second best or third best proposal is accepted, IT IS a problem for the US in the long run. Side node: there is money on China! Fell free to down vote me down. Again Please don't ask. Can not, will not, must not tell.
  - tia4tia 6y ago
mncharity 6y ago
If it were possible, a closer reading of history might provide additional lessons. For instance, ARPA took a hit in the early '90s under Bush. Some silver linings, but that's a transition which might illuminate process tradeoffs. There was (at least back then) an ongoing meta discussion about how to do better. So it might be useful to explore not just the organizational designs which were realized, but also the space of things considered. For instance, before Bush hit, there was discussion of tiny "fix that!" grants. Like there's one person who is an outlier in understanding how to do X, and their book just isn't getting finished. So rather than society waiting years on diffusion and reinvention (which is what ended up happening), it might be worth paying someone to sit outside their office, and stand on their desk, and be a forcing factor on making the book happen. At least back then, with a failed attempt at a commerce ARPA clone, it was thought important to have a clear metric to prioritize projects. "What's better for DoD?", rather than the far less tractable "what's better for the commercial economy?". Perhaps I missed it skimming, but a major issue has been the death valley between research and commercial impact. And attempts to address that making things even worse (ie, researchers encouraged to think commercialization, sacrifice impact by holding things close, and then commercialization generally fails, so there's no offsetting benefit). And there's the unfortunate pipeline from research to patent to unsuccessful startup to dominant company having yet more anticompetitive ammo. I wonder if it might be fruitful to broaden focus to the research pipeline rooted in ARPA? Because a successful clone would presumably again face this difficulty. And there might be some other design point that is less ARPA-like, but does impact better. How well ARPA works comes and goes. It's not a stable equilibrium. So instead of asking how to create a successful clone, perhaps one might ask how to create something viable in the vicinity of success, and separately, how to increase time spent less distant from success? The difference between an old-school autonomous PM, rolodex and checkbook in hand, showing up on your doorstep and saying "I've heard you interested in doing X - what would you need?", and say NSF exploratory grants of "groups with the following characteristics, may submit grants addressing the following issues, with a deadline of mumble, and the following logistics"... is really really big.
empath75 6y ago
When you say that ‘DARPA’ “works”, what precisely do you mean by that, and how do you measure success? It seems to me that it primarily works by throwing large amounts of money at academics and then claiming credit for anything they invent, whether or not it contributes to whatever DARPA is trying to accomplish.
darpa_commentor 6y ago
DARPA is effective, but this document paints a rosy picture that is far from the truth. I've been a part of DARPA projects both commerical, university and on the government side. The real process is: 1. PM gets picked due to knowing someone or being a former employee. 2. The biggest test of a program isn't if it's doable or a good idea, but if its able to be transitioned to another government agency with deeper pockets. 3. Most contracts are lost before you begin writing, as people have insider information about what the PM wants. This is done through just talking with each other (remember that most of the PMs come from the same companies), and not through any other formal process. 4. DARPA has some really cool stuff, but fails to transition it well enough (leading to 2.) DARPA is not without it's problems, but has a better track record then NSF (NIH has them beat). What is funny is that you quickly realize how much bunk there is in scientific research and how many papers are not replicated.
- dnautics 6y ago
  this aligns with my experience. NIH has a better track record than DARPA, which has a better track record than NSF and the worst is the DOE. The DOE knows this and is trying to cargo-cult DARPA through stuff like ARPA-E. Predictable results have ensued.
  - raziel2701 6y ago
    What does a better track record mean? More papers? Products? Social impact? Why is DOE the worst?
  - N1H1L 6y ago
    Ummm.. The DOE run National labs have pioneered fundamental science. A lot of modern materials science technique development originally happened at some DOE National Lab. Judging basic science by productivity is the absolute worst metric to choose. DOE Labs are going down that path, and are becoming glorified university research groups all in the name of productivity.
- tia4tia 6y ago
  100% on track. You will get downvoted!
ycombonator 6y ago
Put some BLM, antifa and communists in there. They will burn down the place is an afternoon.
- IggleSniggle 6y ago
  I'm not sure where this comment is even coming from. Whether you agree or disagree with their agenda, these are all inherently movements of civil-disobedience, seeking to up-end a cultural status quo. Upending the cultural status quo is inherently short-term destabilizing even if it improves long-term state stability. That said, BLM is clearly about finding a pareto improvement to individual well-being in the context of systemic racism, Antifa is as anti-government as the so called "Patriot" movement, and Communists value fairness over maximal well-being. Maybe I'm dense but I don't see how these three collectives are at all inherently related. Organizations working to defend the State (like DARPA) are dedicated to both the short and long term survivability of the State. Any activist movement, right or left, is definitionally not aligned with the interests of the short-term status quo, even if they may well produce innovations that promote long-term stability. That said, those goals don't have an impact on the output of an R&D organization, except as they allow or disallow diversity of thought within an R&D organization. Any ideology, "left", "right", or "center", sufficiently outspoken enough, can be used to suppress the diversity of thought necessary to recognize/embrace true innovations when they are produced.
scotty79 6y ago
My more top level view is that DARPA and such are just side channels to sneakily funnel public money into research. In theory free market should take care of the economy. It's really good at optimizing manufacturing processes and getting stuff as cheaply as possible into hands of as many people as possible. However there's one thing essential to the economy that free market sucks at. It's research. And that's not because companies are bad at doing research. It's because companies are bad at funding research. Research is inherently risky and no sane capitalist will invest in anything beyond tinker level research because he will loose. And he's not supposed to loose. In theory you could fund research overtly with country budget, but no one is going to support that. Why spend money on eggheads playing with useless stuff if people are hungry and streets are dirty? What nobody opposes is giving more money to the military. And what anyone can't oppose is military spending money in whichever way they please. So they can spend it on research. Most will be wasted, very few will have actual military potential. The rest can be graciously dumped into the economy for companies to tinker, optimize making of, manufacture, market and sell. IMHO military is the core of USA success (or even success of global capitalism), not through might, but through ability to get plenty of public money and ability to allocate it into things that would never get funded in any other way.
- alibaba_x 6y ago
  Great comment. I agree that companies would much rather do the exploiting and leave exploration to the government.
- hutzlibu 6y ago
  Well, the public does support money going to universities. And I believe there are quite some people, who opposes more money going to the military.
golemiprague 6y ago
> Only 5–10 out of every 100 programs successfully produce transformative research, while only 10% of projects are terminated early How is this different to the regular startup ecosystem in principal? It doesn't look so special to me
walczyk 6y ago
Darpa is incredible because it's run by scientists and not managers. A lot of small tech is also run like that, it's so refreshing.
godelmachine 6y ago
There's an organization closely resembling DARPA but with exclusive focus on computing. Wish more gets written on it → https://en.wikipedia.org/wiki/Intelligence_Advanced_Research_Projects_Activity https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...
- Invictus0 6y ago
  IARPA is specifically mentioned in the article.
- nl 6y ago
  I've done some work around the edges of IARPA. It's not exclusivly computing BTW. It's pretty much DARPA with smaller budgets. Some pretty interesting projects - the one about increasing human intelligence with magnets is pretty far out!
  - hutzlibu 6y ago
    "the one about increasing human intelligence with magnets is pretty far out" Sounds very esoteric to me. Is there a link somewhere? I would like to know, if there is substance to it, or that someone was able to talk quite good.
    - nl 6y ago
      It's actually both magnetic (transcranial magnetic stimulation) and electro-magentic stimulation (transcranial direct current stimulation): https://www.theatlantic.com/magazine/archive/2014/09/prepare-to-be-shocked/375072/ https://www.theatlantic.com/magazine/archive/2014/09/prepare... There's a reddit dedicated to people trying it on themselves: https://www.reddit.com/r/tDCS/ https://www.reddit.com/r/tDCS/ I wouldn't say the results are exactly conclusive, but there did seem to be some results showing that pain can be reduced in some circumstances. And while "magnets" sounds like some kind of crystal therapy or something, ignoring those aspects it's actually exactly the kind of way-out things that *ARPA should be funding.
      - 6y ago
maxander 6y ago
A big part of the "DARPA formula" that doesn't seem to get much treatment here is consistency. DARPA is going to spend a few billion this year, and it spent a few billion last year, and it spent a few billion twenty years ago, and you can feel confident they'll spend a few billion next year. That means that if an academic researcher or similar can position their lab in an area that DARPA likes, they have a good shot at reliable funding for the length of a career. With all the economic uncertainties around research, this is tremendously appealing and will cause a lot of smart people to re-orient their entire research programs around DARPA-friendly subjects [0]. A funding organization that might disappear in a few years, by contrast, will get plenty of grant applications, but won't attract nearly the same level of researcher devotion. (As a side note, I want to pull out a quote I thought was really nice, hidden a ways in: "DARPA funds wacky things that go nowhere. DARPA programs have a 5—10% success rate and have included things like jetpacks, earthworm robots, creating fusion with sound waves, spider-man wall climbing, and bomb detecting bees. You can’t cut off just one tail of a distribution.") [0] Which may, incidentally, go a ways towards explaining why DARPA heads describe their projects as "idea-limited"
- Aperocky 6y ago
  creating fusion with sound waves (or shock waves, which is essentially what sound becomes at high enough energy) is one of the crazier ideas I've heard..
  - eximius 6y ago
    It's literally one of the ways they make nuclear bombs so it isn't that crazy.
    - Aperocky 6y ago
      Well yeah, if your shock waves for fusion are created by nuclear fission chain reaction that technology has been quite ready and mature since the 1960s.
  - stochtastic 6y ago
    Sonofusion [0] was a neat idea, and some of the relevant experiments (at least the cavitation part) can be done on a bench top with a hobbyist level budget. Last I checked, the research had been somewhat tarnished by academic fraud. The physics of jet formation during bubble collapse near a solid boundary remains fascinating [1]. [0] https://en.wikipedia.org/wiki/Bubble_fusion https://en.wikipedia.org/wiki/Bubble_fusion [1] https://authors.library.caltech.edu/48933/1/On%20the%20Mechanism%20of%20Cavitation%20Damage%20by%20Nonhemispherical%20Cavities%20Collapsing%20in%20Contact%20With%20a%20Solid%20Boundary.pdf https://authors.library.caltech.edu/48933/1/On%20the%20Mecha...
ilaksh 6y ago
I think it comes down to the fact that the military has a lot of money and the projects have goals that are somewhat directed but also long term. Don't think it's more complex than that really.
twarge 6y ago
I would say the key difference between DARPA funding and other research functing is that DARPA PMs have a crystal clear map of the leading edge in a field that's ready for advancement and can target it perfectly with both money and real competition.
b20000 6y ago
it works because there are no google bros
- semi_good 6y ago
  This is correct. The privately run version of this setup Alphabets Project X was wrecked by lousy hiring decisions. Who the heck recruits people like this? “ As the Times reported, DeVaul, whose title is “director of rapid evaluation and mad science,” told a young female job candidate during a 2013 interview that he was in a polyamorous relationship. Later, when he saw the woman at Burning Man while she was still waiting to hear back about the job, DeVaul asked her if she would take off her shirt for a back massage.” https://slate.com/technology/2018/10/google-x-sexual-harassment-allegations-employment.html https://slate.com/technology/2018/10/google-x-sexual-harassm...
killjoywashere 6y ago
It's worth viewing DARPA from their customer's perspective. What DARPA brings to the DoD in a crisis is a stable of highly competent engineers and scientists who want to push the envelope and are smart enough to not let a crisis go to waste. I'm on the demand side right now and it's been awesome to see all these crack folks swarm down on my requirements. If you think the DoD is behind the power curve, um, well, I have seen things that don't officially exist show up, obviously already engineered, so fast you can calculate from the shipping label how long it took someone to get off the phone and get it onto a plane. So, yes, there are absolutely the long game 5-year efforts (which are often really sprouts of 20 or 30 year efforts). But it's a bit like YC: the alumni network is amazing.
- davidhowlett 6y ago
  What does the power curve mean in this context?

347 points by MKais 6y ago

I'd like to write a comment but I generally prefer to have read the entire article before I do... see you in a week or so. Jokes aside, I agree with the author (I think) that DARPA has been a surprisingly effective org in an age of frequent failures of other orgs with similarly lofty goals. DARPA is effectively what things like the SoftBank Vision Fund should've been (wanted to be?). It would certainly be interesting to see what it would look like to have a privately run clone of DARPA if you injected as much cash as the Vision Fund did. Per the article, about $400M/y is spent on actual R&D by DARPA, where as Vision Fund has injected that amount into single companies many times over.

deleted 6y ago
[deleted]
jariel 6y ago
Vision Fund is there to throw huge amounts of money at a business in order to ramp it up quickly and capture large, global markets. Think Uber: disrupting the Taxi business (or just regulations ...). Once something like Uber starts to work, a 'Vision Fund' takes this fledgling thing and backs it with billions to conquer the world. It's not about moon-shots, it's about market power and speed on a global scale. Like hyper-supercharged 'Round C or D' - instead of doing an initial public offering for cash, you take on massive cash from Vision. Wether or not it will work is something else, but there's logic there.
- fastball 6y ago
  Oh yeah, absolutely. I just kinda wish we had that level of private interest/investment with similarly lofty goals. Not just "eat the world".

dnautics 6y ago

Does DARPA work(anymore)? The first example of how it works was it's "badass program managers", but all of the program managers I've met have been, to put it lightly, idiots. One of the worst was the PM in charge of the entire subject research division. It turned out, they had a graduate thesis that was entirely observing an instrument artifact (showing bad jugement). Not too long after I interacted with them, there was a minor scandal in the community because postdoc who struggled with reproducing the effect and complained about the results was abused and railroaded by the PI in charge of the project. Shortly thereafter the PM didn't seem to be a PM anymore.

throwawaygh 6y ago
...so your best anecdote about why DARPA doesn't work is that they fire incompetent PMs?
- dnautics 6y ago
  Yes, after being promoted to the head of the division with several people speculating that they were being groomed for even higher posts. And it took a scandal to get rid of them. Person should never had been a PM in the first place. You never hear people say "it's proof that Congress works because they ejected that pedophile (sending lewd messages to the pages) from their seat".
  - throwawaygh 6y ago
    Hate to break it to you, but "an incompetent person rising through the ranks quickly before crashing and burning" is something that happens in pretty literally every large org. Any 60 year old organization with a large budget is going to have multiple instances of this happening.
    - dnautics 6y ago
      Perhaps I have a flair for the dramatic, but consider my statement that I haven't met a competent pm in the division I was in (n~5). The incentives are not that well-aligned. Think about it this way. If you're really smart, why do you become a DARPA PM, and not a PI? Yes there are a few good reasons, but the population it applies to have largely been selected out by the postdoc phase (where I would say you have had sufficient experience with crushing scientific and engineering failure onself and watching others to be effective) and the pool of candidates therefore is vanishing.

dsukhin 6y ago

The irony is palpable: > I would rather this be read by a few people motivated to take action than by a broad audience who will find it merely interesting. In that vein, if you find yourself wanting to share this on Twitter or Hacker News, consider instead sharing it with one or two friends who will take action on it. Thank you for indulging me! I'm glad of course it was shared here. As a distillation - I think the author's theme lies with enabling more _researchers_ rather than business people to take moonshots and do foundational knowledge building and discovery that redefines a field and then focus on commercialization from a birds eye view by technically capable visionaries. This is the SBIR [1] model (also a US Gov requirement to fund small business research for any federal agency with >$100M in funding), the Bell Labs model (which yeilded amazing foundational work like UNIX and the transistor and was a direct result of AT&T's monopoly and excess resources), and perhaps even the YC model (though that one is obviously focused on a shorter horizon and more on commercializing existing tech and more rarely on foundational research). I've personally thought about this problem a lot and done this at a small scale and would love to expand upon it. https://augmentedlabs.org https://augmentedlabs.org Would love to hear others experiences and thoughts. [1] https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Research https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Re...

xVedun 6y ago

It seems that this would be a much better way of funding research in a general sense, since that seems to be the general process of their problem solving. The only thing is that to apply this to other fields, the barrier for getting people that are equally motivated and intelligent about analyzing if the research is 'going' somewhere is high

riazrizvi 6y ago

IMO this analysis gets caught up looking for answers in process when the real problem that DARPA solves is a political-economic one. ATT didn't invent the internet not for lack of process, they were extremely innovative. They held back on innovations because they were making huge sums of money overcharging long distance mainframe-mainframe data link rates in the 50's-70's. DARPA succeeded as a trust buster. They have the resources and legality to make whatever they want, regardless of patents (because National Security/patent law). They can make technology a reality, then develop political support with working prototypes. They build things to show how the Government is getting screwed over by some giant defense corp, because of a lack of competition in certain types of contracts. I think they are becoming less relevant because the corporate-political landscape is becoming more trust-based.

siege_engineer 6y ago
Author here. Good point about DARPA serving a political-economic role. It's a different lens on the coordination role. Is the corporate-political landscape becoming more trust-based than the Military-Industrial Complex that Eisenhower warned about?
- riazrizvi 6y ago
  I don't understand the comparison in reference to the question of 'Why Does DARPA Work?' or my comment. Military-Industrial Complex is where defense industry profit motives drive unwarranted political influence that shapes defense policy, foreign policy. Military-Industrial Complex drives defense spending up. Trust is about the erosion of market competition, which results in decrease product value-price ratio. In the context of defense product, it means that the products that the consumer (here military/govt) can buy from the market become increasingly expensive and weak. Which creates an opportunity for consumers to do-it-better-yourself-and-for-less, which to me, explains DARPA.
  - TheColorYellow 6y ago
    To a certain extent the MIC as you define it would explain the erosion of the market wouldn't it? Also I've read your comments a few times now and I'm not sure if I'm completely understanding your view of what is becoming more "trust-based". Are you saying that DARPA is more trusted by the consumer (the federal government) than the market entities (the contractors)? If so, isn't that exactly what government entities of all types are created for? These agencies exist as part of the government working for the government because it's difficult to have an effective private market created for these similar purposes. I think it makes the question about "what makes DARPA successful" so much more interesting from if we acknowledge the limitations of the sector. Many agencies, entities, or initiatives of various types have failed when given similar mandates and similar market conditions. That's even if you believe DARPA is truly as effective as is claimed at achieving it's mandate of innovation in the Defense and National Security sector...

currymj 6y ago

A bill, the Endless Frontiers Act, has been introduced in both houses of Congress to make a section of the National Science Foundation that works more like DARPA, while massively increasing its funding. The idea is to extend the DARPA model to many more non-military areas, just as suggested in this article. https://www.sciencemag.org/news/2020/05/us-lawmakers-unveil-bold-100-billion-plan-remake-nsf https://www.sciencemag.org/news/2020/05/us-lawmakers-unveil-... Certainly a timely article. I don't know whether it's a good thing or not. The DARPA model has certainly been productive, but it isn't suited for every research topic or subdiscipline. As discussed in the article, currently NSF is the most open to basic science of funding agencies, and gives grantees the most latitude in what they work on. That is a valuable thing to have in the ecosystem. If Congress expects the majority of publicly-funded research even from the NSF to be on short-term grants for specific visions and technologies, it will rule out working on a lot of important things.

gh02t 6y ago
There is also already ARPA-E, which is a similar mandate to DARPA (i.e., risky projects with high payoff) for energy technology. How effective it has been thus far is a matter of debate due to a variety of factors, but I like the basic idea. https://en.wikipedia.org/wiki/ARPA-E https://en.wikipedia.org/wiki/ARPA-E
siege_engineer 6y ago
Author here. I actually wrote a discussion of the Endless Frontier Act: https://benjaminreinhardt.com/the-endless-frontier-act/ https://benjaminreinhardt.com/the-endless-frontier-act/ I agree that the ARPA model is not suitable for all (or even the majority of research) and the act doesn't inspire confidence that shifting the NSF towards a DARPA-like model would do well.
- lazyjeff 6y ago
  Just wanted to say, thanks for writing the article about DARPA, and this analysis of the endless frontiers act. From reading both articles, obviously you're very pro-DARPA and anti-NSF. I have two thoughts that might erode your thesis a bit, that 1) I think you're overestimating the amount of time that faculty spend on writing grants. We like to complain about it, but I've tracked my time to the minute over the past 7 years, and grant writing (both the proposals and the reports) is about 2% of my work time. 2) You might be misunderstanding indirect costs like many people, where you say "Universities can take more than half of grant money as administrative overhead". Indirect cost math is funny, but in order for administrative overhead to be more than half of the grant, the indirect cost rate would have to be over 100%. I can explain more if you're interested. Obviously we're both biased by our job, but it's still useful to read your perspective.

thoraway1010 6y ago

Small anecdote, but good experience with DARPA and SBIR both. One thing I liked - they seemed pretty low overhead / low hassle and outcome / objective focused. That contrasts VERY strong with most govt contracting which is under allowable cost or cost reimbursement setups where the absolute most important criteria is to bill enough costs to draw contract in the right cost buckets (which can be super annoying if local agencies require super complicated budget mod processes). In the cost based contracts, the focus really focuses on the accounting for the costs and other compliance related items. Ie, did you send someone to a conference with govt money, how can you prove you didn't use the govt money in this way or that way etc etc. Bam, welcome to personal activity reports with fund codes that no one understands (ie, major universities have an insane number of codes), and all the nightmares that follow including a fair bit of rule pending that even normally ethical folks find themselves being asked to to get through the paperwork. Seriously, you deliver the product at 50% of cost? You will get a nasty note from the head of your agency saying make sure you draw full contract because agency budget depends on the indirect portion of this award (ie, 30% to overhead) and even the govt agency supervising (who also budgeted based on a cut of full contract) AND other folks for whom leftover money makes life difficult (harder to close contract etc) AND because a lot of govt funding is on the repeat what we did last year model so drawing down everything avoids a cut next year when you may really need it. You determine it would be cheaper to do x vs y and that required a budget mod? Wait 1-2 months for commission approval if you even bother trying to fight it through (changes < 10% often ignored thankfully). I don't know how the accounting for SBIRs etc work, but somehow those projects always seemed more results oriented (so a LOT more fun to work on, focus is on getting a solution going).

tia4tia 6y ago
SBIR? Totally corrupt. Please don't ask. Can not, will not, must not tell. I can tell you one thing. If not the best but the second best or third best proposal is accepted, IT IS a problem for the US in the long run. Side node: there is money on China! Fell free to down vote me down. Again Please don't ask. Can not, will not, must not tell.
- tia4tia 6y ago

mncharity 6y ago

If it were possible, a closer reading of history might provide additional lessons. For instance, ARPA took a hit in the early '90s under Bush. Some silver linings, but that's a transition which might illuminate process tradeoffs. There was (at least back then) an ongoing meta discussion about how to do better. So it might be useful to explore not just the organizational designs which were realized, but also the space of things considered. For instance, before Bush hit, there was discussion of tiny "fix that!" grants. Like there's one person who is an outlier in understanding how to do X, and their book just isn't getting finished. So rather than society waiting years on diffusion and reinvention (which is what ended up happening), it might be worth paying someone to sit outside their office, and stand on their desk, and be a forcing factor on making the book happen. At least back then, with a failed attempt at a commerce ARPA clone, it was thought important to have a clear metric to prioritize projects. "What's better for DoD?", rather than the far less tractable "what's better for the commercial economy?". Perhaps I missed it skimming, but a major issue has been the death valley between research and commercial impact. And attempts to address that making things even worse (ie, researchers encouraged to think commercialization, sacrifice impact by holding things close, and then commercialization generally fails, so there's no offsetting benefit). And there's the unfortunate pipeline from research to patent to unsuccessful startup to dominant company having yet more anticompetitive ammo. I wonder if it might be fruitful to broaden focus to the research pipeline rooted in ARPA? Because a successful clone would presumably again face this difficulty. And there might be some other design point that is less ARPA-like, but does impact better. How well ARPA works comes and goes. It's not a stable equilibrium. So instead of asking how to create a successful clone, perhaps one might ask how to create something viable in the vicinity of success, and separately, how to increase time spent less distant from success? The difference between an old-school autonomous PM, rolodex and checkbook in hand, showing up on your doorstep and saying "I've heard you interested in doing X - what would you need?", and say NSF exploratory grants of "groups with the following characteristics, may submit grants addressing the following issues, with a deadline of mumble, and the following logistics"... is really really big.

empath75 6y ago

When you say that ‘DARPA’ “works”, what precisely do you mean by that, and how do you measure success? It seems to me that it primarily works by throwing large amounts of money at academics and then claiming credit for anything they invent, whether or not it contributes to whatever DARPA is trying to accomplish.

darpa_commentor 6y ago

DARPA is effective, but this document paints a rosy picture that is far from the truth. I've been a part of DARPA projects both commerical, university and on the government side. The real process is: 1. PM gets picked due to knowing someone or being a former employee. 2. The biggest test of a program isn't if it's doable or a good idea, but if its able to be transitioned to another government agency with deeper pockets. 3. Most contracts are lost before you begin writing, as people have insider information about what the PM wants. This is done through just talking with each other (remember that most of the PMs come from the same companies), and not through any other formal process. 4. DARPA has some really cool stuff, but fails to transition it well enough (leading to 2.) DARPA is not without it's problems, but has a better track record then NSF (NIH has them beat). What is funny is that you quickly realize how much bunk there is in scientific research and how many papers are not replicated.

dnautics 6y ago
this aligns with my experience. NIH has a better track record than DARPA, which has a better track record than NSF and the worst is the DOE. The DOE knows this and is trying to cargo-cult DARPA through stuff like ARPA-E. Predictable results have ensued.
- raziel2701 6y ago
  What does a better track record mean? More papers? Products? Social impact? Why is DOE the worst?
- N1H1L 6y ago
  Ummm.. The DOE run National labs have pioneered fundamental science. A lot of modern materials science technique development originally happened at some DOE National Lab. Judging basic science by productivity is the absolute worst metric to choose. DOE Labs are going down that path, and are becoming glorified university research groups all in the name of productivity.
tia4tia 6y ago
100% on track. You will get downvoted!

ycombonator 6y ago

Put some BLM, antifa and communists in there. They will burn down the place is an afternoon.

IggleSniggle 6y ago
I'm not sure where this comment is even coming from. Whether you agree or disagree with their agenda, these are all inherently movements of civil-disobedience, seeking to up-end a cultural status quo. Upending the cultural status quo is inherently short-term destabilizing even if it improves long-term state stability. That said, BLM is clearly about finding a pareto improvement to individual well-being in the context of systemic racism, Antifa is as anti-government as the so called "Patriot" movement, and Communists value fairness over maximal well-being. Maybe I'm dense but I don't see how these three collectives are at all inherently related. Organizations working to defend the State (like DARPA) are dedicated to both the short and long term survivability of the State. Any activist movement, right or left, is definitionally not aligned with the interests of the short-term status quo, even if they may well produce innovations that promote long-term stability. That said, those goals don't have an impact on the output of an R&D organization, except as they allow or disallow diversity of thought within an R&D organization. Any ideology, "left", "right", or "center", sufficiently outspoken enough, can be used to suppress the diversity of thought necessary to recognize/embrace true innovations when they are produced.

scotty79 6y ago

My more top level view is that DARPA and such are just side channels to sneakily funnel public money into research. In theory free market should take care of the economy. It's really good at optimizing manufacturing processes and getting stuff as cheaply as possible into hands of as many people as possible. However there's one thing essential to the economy that free market sucks at. It's research. And that's not because companies are bad at doing research. It's because companies are bad at funding research. Research is inherently risky and no sane capitalist will invest in anything beyond tinker level research because he will loose. And he's not supposed to loose. In theory you could fund research overtly with country budget, but no one is going to support that. Why spend money on eggheads playing with useless stuff if people are hungry and streets are dirty? What nobody opposes is giving more money to the military. And what anyone can't oppose is military spending money in whichever way they please. So they can spend it on research. Most will be wasted, very few will have actual military potential. The rest can be graciously dumped into the economy for companies to tinker, optimize making of, manufacture, market and sell. IMHO military is the core of USA success (or even success of global capitalism), not through might, but through ability to get plenty of public money and ability to allocate it into things that would never get funded in any other way.

alibaba_x 6y ago
Great comment. I agree that companies would much rather do the exploiting and leave exploration to the government.
hutzlibu 6y ago
Well, the public does support money going to universities. And I believe there are quite some people, who opposes more money going to the military.

golemiprague 6y ago

> Only 5–10 out of every 100 programs successfully produce transformative research, while only 10% of projects are terminated early How is this different to the regular startup ecosystem in principal? It doesn't look so special to me

walczyk 6y ago

Darpa is incredible because it's run by scientists and not managers. A lot of small tech is also run like that, it's so refreshing.

godelmachine 6y ago

There's an organization closely resembling DARPA but with exclusive focus on computing. Wish more gets written on it → https://en.wikipedia.org/wiki/Intelligence_Advanced_Research_Projects_Activity https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...

Invictus0 6y ago
IARPA is specifically mentioned in the article.
nl 6y ago
I've done some work around the edges of IARPA. It's not exclusivly computing BTW. It's pretty much DARPA with smaller budgets. Some pretty interesting projects - the one about increasing human intelligence with magnets is pretty far out!
- hutzlibu 6y ago
  "the one about increasing human intelligence with magnets is pretty far out" Sounds very esoteric to me. Is there a link somewhere? I would like to know, if there is substance to it, or that someone was able to talk quite good.
  - nl 6y ago
    It's actually both magnetic (transcranial magnetic stimulation) and electro-magentic stimulation (transcranial direct current stimulation): https://www.theatlantic.com/magazine/archive/2014/09/prepare-to-be-shocked/375072/ https://www.theatlantic.com/magazine/archive/2014/09/prepare... There's a reddit dedicated to people trying it on themselves: https://www.reddit.com/r/tDCS/ https://www.reddit.com/r/tDCS/ I wouldn't say the results are exactly conclusive, but there did seem to be some results showing that pain can be reduced in some circumstances. And while "magnets" sounds like some kind of crystal therapy or something, ignoring those aspects it's actually exactly the kind of way-out things that *ARPA should be funding.
    - 6y ago

maxander 6y ago

A big part of the "DARPA formula" that doesn't seem to get much treatment here is consistency. DARPA is going to spend a few billion this year, and it spent a few billion last year, and it spent a few billion twenty years ago, and you can feel confident they'll spend a few billion next year. That means that if an academic researcher or similar can position their lab in an area that DARPA likes, they have a good shot at reliable funding for the length of a career. With all the economic uncertainties around research, this is tremendously appealing and will cause a lot of smart people to re-orient their entire research programs around DARPA-friendly subjects [0]. A funding organization that might disappear in a few years, by contrast, will get plenty of grant applications, but won't attract nearly the same level of researcher devotion. (As a side note, I want to pull out a quote I thought was really nice, hidden a ways in: "DARPA funds wacky things that go nowhere. DARPA programs have a 5—10% success rate and have included things like jetpacks, earthworm robots, creating fusion with sound waves, spider-man wall climbing, and bomb detecting bees. You can’t cut off just one tail of a distribution.") [0] Which may, incidentally, go a ways towards explaining why DARPA heads describe their projects as "idea-limited"

Aperocky 6y ago
creating fusion with sound waves (or shock waves, which is essentially what sound becomes at high enough energy) is one of the crazier ideas I've heard..
- eximius 6y ago
  It's literally one of the ways they make nuclear bombs so it isn't that crazy.
  - Aperocky 6y ago
    Well yeah, if your shock waves for fusion are created by nuclear fission chain reaction that technology has been quite ready and mature since the 1960s.
- stochtastic 6y ago
  Sonofusion [0] was a neat idea, and some of the relevant experiments (at least the cavitation part) can be done on a bench top with a hobbyist level budget. Last I checked, the research had been somewhat tarnished by academic fraud. The physics of jet formation during bubble collapse near a solid boundary remains fascinating [1]. [0] https://en.wikipedia.org/wiki/Bubble_fusion https://en.wikipedia.org/wiki/Bubble_fusion [1] https://authors.library.caltech.edu/48933/1/On%20the%20Mechanism%20of%20Cavitation%20Damage%20by%20Nonhemispherical%20Cavities%20Collapsing%20in%20Contact%20With%20a%20Solid%20Boundary.pdf https://authors.library.caltech.edu/48933/1/On%20the%20Mecha...

ilaksh 6y ago

I think it comes down to the fact that the military has a lot of money and the projects have goals that are somewhat directed but also long term. Don't think it's more complex than that really.

twarge 6y ago

I would say the key difference between DARPA funding and other research functing is that DARPA PMs have a crystal clear map of the leading edge in a field that's ready for advancement and can target it perfectly with both money and real competition.

b20000 6y ago

it works because there are no google bros

semi_good 6y ago
This is correct. The privately run version of this setup Alphabets Project X was wrecked by lousy hiring decisions. Who the heck recruits people like this? “ As the Times reported, DeVaul, whose title is “director of rapid evaluation and mad science,” told a young female job candidate during a 2013 interview that he was in a polyamorous relationship. Later, when he saw the woman at Burning Man while she was still waiting to hear back about the job, DeVaul asked her if she would take off her shirt for a back massage.” https://slate.com/technology/2018/10/google-x-sexual-harassment-allegations-employment.html https://slate.com/technology/2018/10/google-x-sexual-harassm...

killjoywashere 6y ago

It's worth viewing DARPA from their customer's perspective. What DARPA brings to the DoD in a crisis is a stable of highly competent engineers and scientists who want to push the envelope and are smart enough to not let a crisis go to waste. I'm on the demand side right now and it's been awesome to see all these crack folks swarm down on my requirements. If you think the DoD is behind the power curve, um, well, I have seen things that don't officially exist show up, obviously already engineered, so fast you can calculate from the shipping label how long it took someone to get off the phone and get it onto a plane. So, yes, there are absolutely the long game 5-year efforts (which are often really sprouts of 20 or 30 year efforts). But it's a bit like YC: the alumni network is amazing.

davidhowlett 6y ago
What does the power curve mean in this context?