4 ms·

Who would have thought a story about reading thousands of pages of archived documents could be so riveting? For anyone interesting in how this task has changed

Who would have thought a story about reading thousands of pages of archived documents could be so riveting? For anyone interesting in how this task has changed, there are similar meta-stories about, as but one example, the process of analyzing the Panama Papers. The investigative collaboration sharing access to them digitized millions of pages, and ran (among other things) named entity recognition and clustering / graph discovery algorithms on them to find the signal in that particular heap of noise. If you’re looking for a project exploring / learning some new technology, I can only encourage you to try to find an angle that explores some public data or document set. That could yield something far more interesting than yet another RSS or HN client.

adamson 8y ago
Sure, but how do you find a project like that to enter at a useful stage? I feel like with the Panama Papers, any semi-public plea for help sorting through that data would have thwarted the journalists
- ALittleLight 8y ago
  The CIA discloses sets of documents from time to time as do other government agencies. The CIA recently (past few years) released a huge collection of PDFs regarding their investigation into psychic phenomenon. I only read a few documents at random of these (bizarrely the CIA seemed or seems to have some success with psychics) but I'd be interested if someone could synthesize them into something meaningful or explain exactly what the CIA was doing, to what extent they believe in psychics, and what evidence they really have. https://www.cia.gov/library/readingroom/collection/stargate https://www.cia.gov/library/readingroom/collection/stargate
  - rurban 8y ago
    In short, the Russians had a project to transmit psychic messages on submarines. This was of vital military importance if it would have worked. So the US tried that also, and then they also tried to use LSD (for interrogation, and other drugs) and subconscious hallucinogenic orders (for agents). This worked better, but of course was also shut down (MKULTRA). They still rely on the equally unscientific lie detector tests, which works even worse than Scientology methods. But very similar.
- matt4077 8y ago
  Data.gov, the public datasets on BigQuery, thebInterhet Archive, etc. The Twitter API is also relatively open, so is Flickr. There’s a lot of data to play around with, even if not all is as scandalous as the Panama Papers (but who knows...)

adamson 8y ago

Sure, but how do you find a project like that to enter at a useful stage? I feel like with the Panama Papers, any semi-public plea for help sorting through that data would have thwarted the journalists

ALittleLight 8y ago
The CIA discloses sets of documents from time to time as do other government agencies. The CIA recently (past few years) released a huge collection of PDFs regarding their investigation into psychic phenomenon. I only read a few documents at random of these (bizarrely the CIA seemed or seems to have some success with psychics) but I'd be interested if someone could synthesize them into something meaningful or explain exactly what the CIA was doing, to what extent they believe in psychics, and what evidence they really have. https://www.cia.gov/library/readingroom/collection/stargate https://www.cia.gov/library/readingroom/collection/stargate
- rurban 8y ago
  In short, the Russians had a project to transmit psychic messages on submarines. This was of vital military importance if it would have worked. So the US tried that also, and then they also tried to use LSD (for interrogation, and other drugs) and subconscious hallucinogenic orders (for agents). This worked better, but of course was also shut down (MKULTRA). They still rely on the equally unscientific lie detector tests, which works even worse than Scientology methods. But very similar.
matt4077 8y ago
Data.gov, the public datasets on BigQuery, thebInterhet Archive, etc. The Twitter API is also relatively open, so is Flickr. There’s a lot of data to play around with, even if not all is as scandalous as the Panama Papers (but who knows...)