10 ms·

DARPA Has Open-Sourced 'Dark Web' Search Tech

188 points by aburan28 12y ago

farresito 12y ago
This is the link of the project: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html
- phy6 12y ago
  Of additional interest would be the other DARPA programs participating in the open catalog, of which MEMEX is but one. http://www.darpa.mil/opencatalog/ http://www.darpa.mil/opencatalog/ There are many interesting 'pieces of the puzzle' here that you can glue together to make something awesome. If anyone cares about what success means in our case, it's the transition of technology to other agencies and companies (not necessarily military!). We want the research funded here to live beyond the program instead of dying when the program ends (truly satisfying from a Software Engineering perspective, where many R&D deliverables are either too specialized or never see the light of day)
Drdrdrq 12y ago
Yeeeeeah... There's much more to google than just search. Also, there is much more to search engine than just seaech (seo spam countermeasures come to mind). Good luck replacing Google on its home turf.
- XJOKOLAT 12y ago
  However, the idea is interesting. I once used Chrome. I now use Firefox. Nothing lasts forever. Then again, what's to stop Google adopting the same technology?
alexmobile 12y ago
Thanks for posting - this is amazing... I was actually working on a far from finished article "How a Search Engine Startup company could compete with Google" http://bitexperts.com/Question/Detail/42/how-a-search-engine-startup-company-could-compete-with-google http://bitexperts.com/Question/Detail/42/how-a-search-engine... and then this announcement came across. Will be looking at what kind of crawlers they would release. Hopefully some modern ones based on a WebKit / Chromium core, that exposes DOM model and suitable for navigating all these AJAX script fueled modern web interfaces. Also very interested to see what kind of Machine Learning / classifiers they are using. When working on our search engine, we were using purely statistical classifiers like all variations of Bayes, SVM (Support Vector Machines) and decision trees C4.5 glued together with some custom algos. We did not use neural nets at all. Nowadays, neural nets have a new name - "deep learning" and seem to be everywhere. Really really interesting in terms of what people would build with an Open Source Search Engine. Watch out Google :)
- pixelmonkey 12y ago
  One of the supported projects is splash, which is basically WebKit-as-a-service. It takes an interesting approach to crawling where it renders the page using WebKit, and then exposes the "rendered DOM" -- so that your crawling code doesn't need to actually use JavaScript for information extraction. See: https://github.com/scrapinghub/splash https://github.com/scrapinghub/splash People often use Scrapy + Splash together in the Python community for crawling more dynamic websites. A team I collaborated with is also working on a project to make Scrapy usable in a "cluster context", it's called scrapy-cluster. The idea is scrapy workers running across machines and a single crawling queue (in the current prototype, powered by Redis) in between them all. https://github.com/istresearch/scrapy-cluster https://github.com/istresearch/scrapy-cluster
  - alexmobile 12y ago
    Thanks! I've briefly looked at Splash and related projects like ScrapingHub, etc - looks like this niche is live and kicking... The distributed scrapy-cluster is the way to go, if you need to crawl anything of decent size ( maybe even Amazon - 300+ MM webpages, j/k :) I see a lot of Python based projects recently, even in Bitcoin niche, we even have a local Toronto based Python meetup. Looks like Python dev community is active. I have a domain name PYFORUM.com - would it be good idea to launch a forum site? With Bitcoin tipping built-in? So instead of saying "Thanks" people would be able to send $0.25 in Bitcoin to those who helped them in the forums or made them laugh? What are the most established Python forums out there? Thanks!
    - pki 12y ago
      Launch a forum actually using python... Even the largest 'python forum' is on phpbb..
  - WalterGR 12y ago
    It takes an interesting approach to crawling where it renders the page using WebKit, and then exposes the "rendered DOM" -- so that your crawling code doesn't need to actually use JavaScript for information extraction. It is an interesting approach. There's evidence that Google crawls the web that way, though I don't know if it's been confirmed by the company. GoogleBot indexes content rendered by Javascript - even content delivered by an AJAX request. They've announced they are going to start penalizing sites that don't work well on mobile. I don't know the specifics of that (and they probably haven't shared them) but I do know that I've received automated email from Google Webmaster Tools and/or AdSense about one of my sites not working great on mobile: small UI elements grouped too closely together, content that's too wide, etc.
amelius 12y ago
In order to replace Google, just implement their patents and then wait until they expire. At least, that is how it should work.
- CaveTech 12y ago
  Except you'll be 20 years behind their current version.
  - amelius 12y ago
    You are assuming that search will keep on growing beyond an asymptotic limit. It could very well be that we will soon reach a point where search is good enough for most people.
    - learc83 12y ago
      The problem is that search is a constant battle between Google and spammers/SEO. It's a never ending arms race that ensures there won't be a perfect solution. And in this arms race Google's massive resources gives them a huge advantage. I think Google will probably maintain their dominance until they inevitably get caught doing something egregious enough that the government decides to step in and bust them up.
    - gwern 12y ago
      You're assuming a sort of static model of search where it makes sense to talk about it growing to an asymptotic limit. But it's a dynamic and adversarial environment: the corpus evolves to destroy the quality of search results (spam/SEO). A 20-yo version of Google will be way worse than Google was 20 years ago, because when it goes live it'll be attacked by 19 years of spam innovation (if I can use that word in this context).
      - lotsofmangos 12y ago
        flying spam - http://files.tested.com/photos/2012/10/09/40850-blade-runner-blimp.jpg http://files.tested.com/photos/2012/10/09/40850-blade-runner...
downandout 12y ago
The idea that a single company can perfect search - even one employing a significant percentage of the world's PhDs - is absurd. However, Google will likely continue to dominate search even in the face of superior alternatives because of their position in the marketplace and in the minds of consumers, just as Microsoft continues to dominate the OS space long after the introduction of viable free alternatives. This is why, if you have ever tried to pitch a search startup to VC's, you have almost certainly received a less-than-polite "no" along with your coconut water. These tools will be interesting to use for a handful of techies like myself, but Google will remain safely ensconced in its fortress of cash for at least the next several decades regardless of this and whatever else comes along that may enable the creation of superior search engines.
- afarrell 12y ago
  > long after the introduction of viable free alternatives Maybe Ubuntu (or mint?) has learned UX design and found a way to sand down all its' rough edges in the 3 years since I switched to OSX, but I doubt it. Running linux requires the time and patience of a knowledge person. Linux is far too expensive for most people.
  - learc83 12y ago
    Elementary OS [1] attempts to do that with a design first philosophy. However, it was still too much of a hassle to get it working with 3 monitors last time I tried it. I've used various linux distros for years, and there's always something that doesn't work right. Whether it's the speakers not turning off when I plug my headphones in through my thinkpad dock in Arch, or monitor position settings changing every time I reboot my desktop in Elementary OS, the lack of polish really shows. I understand why this happens, and I don't fault the people making on the various distros. I've even dove in and fixed a few problems myself. But nevertheless, it remains a problem, so for now I've just decided on windows for day to day use with a VM running linux when I need it. [1] http://elementary.io/ http://elementary.io/
  - beagle3 12y ago
    While OSX is probably easier to use still, I've been using Ubuntu on the desktop since 2005, and every time I have to do something on Win8 I cry in desparation. Win7 was frustrating, but workable. Win8.1 is a horrible mess, as was Win8 before it; The Metro/Old dichotomy is intellectually painful, distracting, and damaging to any sane workflow.
pixelmonkey 12y ago
BTW, I'm one of the co-authors of streamparse, one of the DARPA-supported projects that is being developed by my company, Parse.ly. It lets you integrate Apache Storm cleanly with Python. I just gave a talk about streamparse at PyCon US (https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ) a few days ago, it was entitled "streamparse: defeat the Python GIL with Apache Storm". I'm glad to answer any questions about it.
- look_lookatme 12y ago
  How does one get their commercial project supported by DARPA?
  - pixelmonkey 12y ago
    Hmm, not sure I could answer that question, as in this case, DARPA is supporting our open source projects, not our commercial projects. Or is that what you are asking? That said, FastCompany covered the story of how we got involved with DARPA here: http://www.fastcompany.com/3040363/the-future-of-search-brought-to-you-by-the-pentagon http://www.fastcompany.com/3040363/the-future-of-search-brou...
    - look_lookatme 12y ago
      I looked at the MEMEX page and say a bunch of companies represented and was genuinely curious... thanks for sharing the link.
  - phy6 12y ago
    Send your proposal in response to the Broad Agency Announcements (BAA) that the agency puts out.
- strgrd 12y ago
  With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. > ...DARPA said Memex wasn’t about destroying the privacy protections offered by Tor, even though it wanted to help uncover criminals’ identities. “None of them [Tor, the Navy, Memex partners] want child exploitation and child pornography to be accessible, especially on Tor. We’re funding those groups for testing...” Doesn't this sound like the same "protect the kids" line embedded in every press release for not-so-subtle government spy programs? $1 million is a lot of money, and I'm sure being able to name drop DARPA in any conversation about your company carries its own cache -- surely you feel pressured to design your optimizations to fit DARPA's needs. Does it feel weird to write code that's being used to track people? Or is that off base?
dublinben 12y ago
It looks like we can add Forbes to the list of publications that don't understand what the "dark web" is. It's too bad that this article starts right off with gross misinformation.
- ryanlol 12y ago
  To be honest, "dark web" seems to lack a clear, well defined meaning and should be avoided.
elorant 12y ago
Once again someone confuses "deep web" for "dark web".
WalterGR 12y ago
This is where the "Memex" name comes from: http://en.wikipedia.org/wiki/Memex http://en.wikipedia.org/wiki/Memex A memex is a "hypothetical proto-hypertext system that Vannevar Bush described in his 1945 The Atlantic Monthly article 'As We May Think'." I'm rather disappointed that DARPA chose it as the name for a project that, according to Wired according to Wikipedia, "aims to shine a light on the dark web and uncover patterns and relationships in online data to help law enforcement and others track illegal activity".
- nl 12y ago
  It's explicitly designed to work outside that single sphere: Memex seeks to develop software that advances online search capabilities far beyond the current state of the art. The goal is to invent better methods for interacting with and sharing information, so users can quickly and thoroughly organize and search subsets of information relevant to their individual interests.... Memex would ultimately apply to any public domain content; initially, DARPA plans to develop Memex to address a key Defense Department mission: fighting human trafficking. http://www.darpa.mil/Our_Work/I2O/Programs/Memex.aspx http://www.darpa.mil/Our_Work/I2O/Programs/Memex.aspx
  - WalterGR 12y ago
    No, DARPA plans to expand Memex so it works outside that single sphere. I'm not reading anything that suggests it's designed to work outside that sphere. But we could go back and forth on the distinction between "seeks to" and "designed to" all night, so I'll leave it at that. :)
    - nl 12y ago
      I agree, but it is worth noting that DARPA isn't known for their lack of ambition, nor for their lack of long term commitment. If they "seek to do" something that generally means that is what they are aiming for.
    - phy6 12y ago
      You're both right. The trend with these I2O programs (and other departments) is to design a system of loosely coupled pieces that are tailored to a specific problem domain, in this case human trafficking, and then generalize in the next couple years later. By the third or fourth year you could expect to see multiple engagements with corporate industry and Gov't/DOD/LE/etc to appl these tools to their data, and thus generalize and polish the final product. The end goal is adoption of the technology and successful transition to some partner. The good programs tend to run for 3-4 years if they can justify their existence to the DARPA leadership. This is also roughly the tenure of a program manager, although a program manager may have more than one program going at a time.
anigbrowl 12y ago
This is a nice surprise! I wrote to ask about public access to this in February and got no reply. I had forgotten about it until I got a lengthy reply from someone at DARPA in mid-march saying that they were considering it - and sure enough, here it is: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html
mirimir 12y ago
This is wonderful news. Information (including Tor onion services) wants to be freely findable ;) Obscurity should never be the major aspect of OPSEC. Even with the sorry state of Tor onion services, sites can authenticate users at both network level, using the "stealth" authorization protocol, and application level. Of course, prudent users will use end-to-end encryption for sensitive information. And they'll consider carefully before sharing. All of the "hidden markets", by the way, are little more than honeypots in waiting, enticing customers to pwn themselves. Edit: See Roger Dingledine's reply in the "Clarification of Tor's involvement with DARPA's Memex" thread on tor-talk <https://lists.torproject.org/pipermail/tor-talk/2015-April/037538.html> https://lists.torproject.org/pipermail/tor-talk/2015-April/0....
schoen 12y ago
Roger Dingledine was asked about the Tor Project's involvement with the things mentioned in this article. His comment is here: https://lists.torproject.org/pipermail/tor-talk/2015-April/037538.html https://lists.torproject.org/pipermail/tor-talk/2015-April/0...
kbwt 12y ago
So, browsing through these projects I see a lot of crawlers. Where is the search tech? Other than that one project which seems to be a JS-frontend for Lucene/Solr, I can't find anything having to do with Information Retrieval. Unless I'm missing something, the article and the previous one it links to seem to be a bit overblown. "Could The U.S. Military's New Search Engine Replace Google?" -> Not if the U.S. Military's New Search Engine is the vanilla Lucene/Solr. Link: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html

188 points by aburan28 12y ago

This is the link of the project: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html

phy6 12y ago
Of additional interest would be the other DARPA programs participating in the open catalog, of which MEMEX is but one. http://www.darpa.mil/opencatalog/ http://www.darpa.mil/opencatalog/ There are many interesting 'pieces of the puzzle' here that you can glue together to make something awesome. If anyone cares about what success means in our case, it's the transition of technology to other agencies and companies (not necessarily military!). We want the research funded here to live beyond the program instead of dying when the program ends (truly satisfying from a Software Engineering perspective, where many R&D deliverables are either too specialized or never see the light of day)

Drdrdrq 12y ago

Yeeeeeah... There's much more to google than just search. Also, there is much more to search engine than just seaech (seo spam countermeasures come to mind). Good luck replacing Google on its home turf.

XJOKOLAT 12y ago
However, the idea is interesting. I once used Chrome. I now use Firefox. Nothing lasts forever. Then again, what's to stop Google adopting the same technology?

alexmobile 12y ago

Thanks for posting - this is amazing... I was actually working on a far from finished article "How a Search Engine Startup company could compete with Google" http://bitexperts.com/Question/Detail/42/how-a-search-engine-startup-company-could-compete-with-google http://bitexperts.com/Question/Detail/42/how-a-search-engine... and then this announcement came across. Will be looking at what kind of crawlers they would release. Hopefully some modern ones based on a WebKit / Chromium core, that exposes DOM model and suitable for navigating all these AJAX script fueled modern web interfaces. Also very interested to see what kind of Machine Learning / classifiers they are using. When working on our search engine, we were using purely statistical classifiers like all variations of Bayes, SVM (Support Vector Machines) and decision trees C4.5 glued together with some custom algos. We did not use neural nets at all. Nowadays, neural nets have a new name - "deep learning" and seem to be everywhere. Really really interesting in terms of what people would build with an Open Source Search Engine. Watch out Google :)

pixelmonkey 12y ago
One of the supported projects is splash, which is basically WebKit-as-a-service. It takes an interesting approach to crawling where it renders the page using WebKit, and then exposes the "rendered DOM" -- so that your crawling code doesn't need to actually use JavaScript for information extraction. See: https://github.com/scrapinghub/splash https://github.com/scrapinghub/splash People often use Scrapy + Splash together in the Python community for crawling more dynamic websites. A team I collaborated with is also working on a project to make Scrapy usable in a "cluster context", it's called scrapy-cluster. The idea is scrapy workers running across machines and a single crawling queue (in the current prototype, powered by Redis) in between them all. https://github.com/istresearch/scrapy-cluster https://github.com/istresearch/scrapy-cluster
- alexmobile 12y ago
  Thanks! I've briefly looked at Splash and related projects like ScrapingHub, etc - looks like this niche is live and kicking... The distributed scrapy-cluster is the way to go, if you need to crawl anything of decent size ( maybe even Amazon - 300+ MM webpages, j/k :) I see a lot of Python based projects recently, even in Bitcoin niche, we even have a local Toronto based Python meetup. Looks like Python dev community is active. I have a domain name PYFORUM.com - would it be good idea to launch a forum site? With Bitcoin tipping built-in? So instead of saying "Thanks" people would be able to send $0.25 in Bitcoin to those who helped them in the forums or made them laugh? What are the most established Python forums out there? Thanks!
  - pki 12y ago
    Launch a forum actually using python... Even the largest 'python forum' is on phpbb..
- WalterGR 12y ago
  It takes an interesting approach to crawling where it renders the page using WebKit, and then exposes the "rendered DOM" -- so that your crawling code doesn't need to actually use JavaScript for information extraction. It is an interesting approach. There's evidence that Google crawls the web that way, though I don't know if it's been confirmed by the company. GoogleBot indexes content rendered by Javascript - even content delivered by an AJAX request. They've announced they are going to start penalizing sites that don't work well on mobile. I don't know the specifics of that (and they probably haven't shared them) but I do know that I've received automated email from Google Webmaster Tools and/or AdSense about one of my sites not working great on mobile: small UI elements grouped too closely together, content that's too wide, etc.

amelius 12y ago

In order to replace Google, just implement their patents and then wait until they expire. At least, that is how it should work.

CaveTech 12y ago
Except you'll be 20 years behind their current version.
- amelius 12y ago
  You are assuming that search will keep on growing beyond an asymptotic limit. It could very well be that we will soon reach a point where search is good enough for most people.
  - learc83 12y ago
    The problem is that search is a constant battle between Google and spammers/SEO. It's a never ending arms race that ensures there won't be a perfect solution. And in this arms race Google's massive resources gives them a huge advantage. I think Google will probably maintain their dominance until they inevitably get caught doing something egregious enough that the government decides to step in and bust them up.
  - gwern 12y ago
    You're assuming a sort of static model of search where it makes sense to talk about it growing to an asymptotic limit. But it's a dynamic and adversarial environment: the corpus evolves to destroy the quality of search results (spam/SEO). A 20-yo version of Google will be way worse than Google was 20 years ago, because when it goes live it'll be attacked by 19 years of spam innovation (if I can use that word in this context).
    - lotsofmangos 12y ago
      flying spam - http://files.tested.com/photos/2012/10/09/40850-blade-runner-blimp.jpg http://files.tested.com/photos/2012/10/09/40850-blade-runner...

downandout 12y ago

The idea that a single company can perfect search - even one employing a significant percentage of the world's PhDs - is absurd. However, Google will likely continue to dominate search even in the face of superior alternatives because of their position in the marketplace and in the minds of consumers, just as Microsoft continues to dominate the OS space long after the introduction of viable free alternatives. This is why, if you have ever tried to pitch a search startup to VC's, you have almost certainly received a less-than-polite "no" along with your coconut water. These tools will be interesting to use for a handful of techies like myself, but Google will remain safely ensconced in its fortress of cash for at least the next several decades regardless of this and whatever else comes along that may enable the creation of superior search engines.

afarrell 12y ago
> long after the introduction of viable free alternatives Maybe Ubuntu (or mint?) has learned UX design and found a way to sand down all its' rough edges in the 3 years since I switched to OSX, but I doubt it. Running linux requires the time and patience of a knowledge person. Linux is far too expensive for most people.
- learc83 12y ago
  Elementary OS [1] attempts to do that with a design first philosophy. However, it was still too much of a hassle to get it working with 3 monitors last time I tried it. I've used various linux distros for years, and there's always something that doesn't work right. Whether it's the speakers not turning off when I plug my headphones in through my thinkpad dock in Arch, or monitor position settings changing every time I reboot my desktop in Elementary OS, the lack of polish really shows. I understand why this happens, and I don't fault the people making on the various distros. I've even dove in and fixed a few problems myself. But nevertheless, it remains a problem, so for now I've just decided on windows for day to day use with a VM running linux when I need it. [1] http://elementary.io/ http://elementary.io/
- beagle3 12y ago
  While OSX is probably easier to use still, I've been using Ubuntu on the desktop since 2005, and every time I have to do something on Win8 I cry in desparation. Win7 was frustrating, but workable. Win8.1 is a horrible mess, as was Win8 before it; The Metro/Old dichotomy is intellectually painful, distracting, and damaging to any sane workflow.

pixelmonkey 12y ago

BTW, I'm one of the co-authors of streamparse, one of the DARPA-supported projects that is being developed by my company, Parse.ly. It lets you integrate Apache Storm cleanly with Python. I just gave a talk about streamparse at PyCon US (https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ) a few days ago, it was entitled "streamparse: defeat the Python GIL with Apache Storm". I'm glad to answer any questions about it.

look_lookatme 12y ago
How does one get their commercial project supported by DARPA?
- pixelmonkey 12y ago
  Hmm, not sure I could answer that question, as in this case, DARPA is supporting our open source projects, not our commercial projects. Or is that what you are asking? That said, FastCompany covered the story of how we got involved with DARPA here: http://www.fastcompany.com/3040363/the-future-of-search-brought-to-you-by-the-pentagon http://www.fastcompany.com/3040363/the-future-of-search-brou...
  - look_lookatme 12y ago
    I looked at the MEMEX page and say a bunch of companies represented and was genuinely curious... thanks for sharing the link.
- phy6 12y ago
  Send your proposal in response to the Broad Agency Announcements (BAA) that the agency puts out.
strgrd 12y ago
With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. > ...DARPA said Memex wasn’t about destroying the privacy protections offered by Tor, even though it wanted to help uncover criminals’ identities. “None of them [Tor, the Navy, Memex partners] want child exploitation and child pornography to be accessible, especially on Tor. We’re funding those groups for testing...” Doesn't this sound like the same "protect the kids" line embedded in every press release for not-so-subtle government spy programs? $1 million is a lot of money, and I'm sure being able to name drop DARPA in any conversation about your company carries its own cache -- surely you feel pressured to design your optimizations to fit DARPA's needs. Does it feel weird to write code that's being used to track people? Or is that off base?

dublinben 12y ago

It looks like we can add Forbes to the list of publications that don't understand what the "dark web" is. It's too bad that this article starts right off with gross misinformation.

ryanlol 12y ago
To be honest, "dark web" seems to lack a clear, well defined meaning and should be avoided.

elorant 12y ago

Once again someone confuses "deep web" for "dark web".

WalterGR 12y ago

This is where the "Memex" name comes from: http://en.wikipedia.org/wiki/Memex http://en.wikipedia.org/wiki/Memex A memex is a "hypothetical proto-hypertext system that Vannevar Bush described in his 1945 The Atlantic Monthly article 'As We May Think'." I'm rather disappointed that DARPA chose it as the name for a project that, according to Wired according to Wikipedia, "aims to shine a light on the dark web and uncover patterns and relationships in online data to help law enforcement and others track illegal activity".

nl 12y ago
It's explicitly designed to work outside that single sphere: Memex seeks to develop software that advances online search capabilities far beyond the current state of the art. The goal is to invent better methods for interacting with and sharing information, so users can quickly and thoroughly organize and search subsets of information relevant to their individual interests.... Memex would ultimately apply to any public domain content; initially, DARPA plans to develop Memex to address a key Defense Department mission: fighting human trafficking. http://www.darpa.mil/Our_Work/I2O/Programs/Memex.aspx http://www.darpa.mil/Our_Work/I2O/Programs/Memex.aspx
- WalterGR 12y ago
  No, DARPA plans to expand Memex so it works outside that single sphere. I'm not reading anything that suggests it's designed to work outside that sphere. But we could go back and forth on the distinction between "seeks to" and "designed to" all night, so I'll leave it at that. :)
  - nl 12y ago
    I agree, but it is worth noting that DARPA isn't known for their lack of ambition, nor for their lack of long term commitment. If they "seek to do" something that generally means that is what they are aiming for.
  - phy6 12y ago
    You're both right. The trend with these I2O programs (and other departments) is to design a system of loosely coupled pieces that are tailored to a specific problem domain, in this case human trafficking, and then generalize in the next couple years later. By the third or fourth year you could expect to see multiple engagements with corporate industry and Gov't/DOD/LE/etc to appl these tools to their data, and thus generalize and polish the final product. The end goal is adoption of the technology and successful transition to some partner. The good programs tend to run for 3-4 years if they can justify their existence to the DARPA leadership. This is also roughly the tenure of a program manager, although a program manager may have more than one program going at a time.

anigbrowl 12y ago

This is a nice surprise! I wrote to ask about public access to this in February and got no reply. I had forgotten about it until I got a lengthy reply from someone at DARPA in mid-march saying that they were considering it - and sure enough, here it is: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html

mirimir 12y ago

This is wonderful news. Information (including Tor onion services) wants to be freely findable ;) Obscurity should never be the major aspect of OPSEC. Even with the sorry state of Tor onion services, sites can authenticate users at both network level, using the "stealth" authorization protocol, and application level. Of course, prudent users will use end-to-end encryption for sensitive information. And they'll consider carefully before sharing. All of the "hidden markets", by the way, are little more than honeypots in waiting, enticing customers to pwn themselves. Edit: See Roger Dingledine's reply in the "Clarification of Tor's involvement with DARPA's Memex" thread on tor-talk <https://lists.torproject.org/pipermail/tor-talk/2015-April/037538.html> https://lists.torproject.org/pipermail/tor-talk/2015-April/0....

schoen 12y ago

Roger Dingledine was asked about the Tor Project's involvement with the things mentioned in this article. His comment is here: https://lists.torproject.org/pipermail/tor-talk/2015-April/037538.html https://lists.torproject.org/pipermail/tor-talk/2015-April/0...

kbwt 12y ago

So, browsing through these projects I see a lot of crawlers. Where is the search tech? Other than that one project which seems to be a JS-frontend for Lucene/Solr, I can't find anything having to do with Information Retrieval. Unless I'm missing something, the article and the previous one it links to seem to be a bit overblown. "Could The U.S. Military's New Search Engine Replace Google?" -> Not if the U.S. Military's New Search Engine is the vanilla Lucene/Solr. Link: http://www.darpa.mil/opencatalog/MEMEX.html http://www.darpa.mil/opencatalog/MEMEX.html