7 ms·

We've got thousands of pages of contracts with Booz Allen: Help us analyze

105 points by morisy 13y ago

scottshea 13y ago
Now this is investigative journalism!
stfu 13y ago
Is there already some decent crowd sourcing text analysis software out there? Some simplified version of Nvivo with a wiki-like user management would do wonders for something like that. 1. Button "I commit to analyze 2 / 12 /22 pages". 2. You get a very short tutorial on how to do it. 3. User is assigned relevant pages, starts tagging the documents for specific content and highlights + comments lines worth a closer look. I would be definitely up for that. With enough traction one could just compare inter-coder reliability and establishing a high quality analyzing process (i.e. getting the same document analyzed by multiple users).
- solstice 13y ago
  How about django-public-project? The following is from their description at http://django-public-project.readthedocs.org/en/latest/ http://django-public-project.readthedocs.org/en/latest/: Django Public Project (DPP) is a specialised content management system written in Python/Django for building an information website around big projects. It is open source (BSD licence) and can be used by civic groups to monitor the progress of publicly funded projects but also from within governmental institutions itself to bring transparency in the proceeding of a project, inform the interested public about what is going on and thus enhance the acceptance of the project by the different stakeholders. It is currently being used by the investigative comittee of the Berlin city parliament in order to investigate the massive clusterfk that is the city's new and very disfuntional airport (BER). The source code can be found at: https://github.com/holgerd77/django-public-project https://github.com/holgerd77/django-public-project Regarding participation of ordinary citizens, the docs say: On DPP websites users can comment on nearly everything and connect their comments to various elements of the project. Comments are then shown alongside every connected element. All admin users having activated the “Receive new comment emails” in the admin user settings receive an email after someone sent a comment and can then activate the comment to be shown on the website. http://django-public-project.readthedocs.org/en/latest/users.html#user-comments http://django-public-project.readthedocs.org/en/latest/users...
patdennis 13y ago
People get excited about things like this and say things like "this is REAL investigative journalism". Or "I wish mainstream journalists would start doing this". Look, I read documents like this all day long. So do investigative journalists. It's just that very few people care about the stuff we find until it turns into an international spy thriller. Just because you don't care about it on the state/local level doesn't mean we don't put in the work. Edit: You can look at some of the great investigative journalism you're probably not paying attention to here: http://www.ire.org/blog/extra-extra/ http://www.ire.org/blog/extra-extra/
- rhizome 13y ago
  I'm not sure what your point is, is someone saying that you're lazy?
  - patdennis 13y ago
    I was just trying to preempt the inevitable "investigative journalism is dead" comments.
    - rhizome 13y ago
      Judging from the comments so far, it may not be so inevitable. However, there is still the law of averages, I guess.
- deleted 13y ago
  [deleted]
- morisy 13y ago
  "I wish mainstream journalists would start doing this". Spin it however you want, fewer and fewer mainstream journalists are able to focus on this. I work with a dozen or so newsrooms a year on FOIA, CAR, and other investigative reporting training, and in most newsrooms only 1 or 2 hands go up when I ask who has filed a public records request in the past year. MuckRock was built to change that: To make it easier to add document-requesting and analysis to the daily news cycle, to help non-journalists get interested, excited, and able to request and share documents, and to enable large scale-FOIA projects like request BAH files from dozens of agencies. And we've found that lots of people do care about it at the state/local level, but news organizations are a) terrible about marketing their work. b) assume readers aren't interested in the nuts and bolts. c) let that work die in obscurity, content to try to win awards with it rather than to truly engage the public. So yes, this is a flashy dump of documents, but we've helped hundreds of others do meaningful, smaller scale stories, and whatever way you cut it, I don't think there's enough groups doing that. I also keep my own list of interesting, public-records driven stories: http://pinboard.in/u:morisy/t:to_foi/ http://pinboard.in/u:morisy/t:to_foi/ Encourage anyone who is interested to reverse engineer them and try those requests in their city, state, and country.
rhizome 13y ago
They should be using plagiarism-detection software to strain out the boilerplate commonalities and focus on the differences (as well as a single copy of the commonalities).
rdl 13y ago
This is the kind of stuff I wish Wikileaks had stuck to -- mostly or completely legal, beneficial to national security, etc. They are still challenging powerful entities and supporting change, but without becoming the story themselves.
- camus 13y ago
  who made the "press" shift its attention on wikileaks rather than on the docs they leaked ? your favorite DC spin doctors. So please , dont blame the messenger , or you are as stupid as the government think you are. And they do think you are stupid , trust me.
  - rdl 13y ago
    Releasing 250k classified documents is a news story itself, so it was inevitable that the story would shift to wikileaks. (shifting to Julian Assange was some DC spin, though, but he sure didn't help things.)
    - dantheman 13y ago
      But they didn't release the cables at first, they released an encrypted file. Then a journalist who didn't like wikileaks that had access to the file published the password in a book. Since the files were then available to those with resources to know about they password, they felt it was reasonable to release them completely so people could see if they were mentioned in the documents which were at that point for all intents and purposes public.
- mpyne 13y ago
  Likewise. I really liked WikiLeaks at the beginning but it became clear after some time that there was a definite agenda behind their actions. The impression I've had for years is that they will oppose the U.S. for no better reason than that it's the U.S. Which is, if you think about it, just like opposing any maligned group for what they are instead of what they do. In other words if we start from the assumption that the U.S. should be a "bastion of the free world" then it would make sense to tirelessly note instances where the U.S., corporations, states, etc. fall from that ideal. But diplomatic cables describing what kind of women Berlusconi likes, while interesting, are hardly indicators of the collapse of a free society. Those types of things were released because they were embarrassing, nothing more. And while I'm certainly glad that Assange is able to stroke his ego by thumbing his nose in Uncle Sam's face, I don't actually hate my country; I want it to be better.
morisy 13y ago
A good way to search through the text of the documents is DocumentCloud, which also offers some analysis tools: https://www.documentcloud.org/search/group:muckrock https://www.documentcloud.org/search/group:muckrock Modify the search terms as desired (i.e., "Booz")
Spooky23 13y ago
What exactly is the point of this exercise? Booz Allen is a big consulting company. They have lots of contracts and SOWs, most of which are for really lame things. You are not going to find a contract that says: "provide 3 big brother spies to read everyone's mail". Of you're trying to prove that that government spends lots of money for consultants... Well, we all know that already. Also, unless you really know what you are talking about, you are not going to be able to spot most of the petty, obvious or really offensive contracting practices. The Feds have their own language re: procurement.
- nitrogen 13y ago
  A while back there was a request posted to fbo.gov for an online sockpuppet management system. It's possible that some important details can be gleaned from these contracts as well.
- rdl 13y ago
  Based on the ones I've seen in other contexts, the key is to be able to aggregate. If you do, you can often find really interesting stuff from unclassified contracts around the periphery, then presumably FOIA more, but knowing how the system works helps a lot.
patrickg_zill 13y ago
http://taxanalysts.com/www/features.nsf/Articles/557540D8F539A26685257B94004CD0D1?OpenDocument http://taxanalysts.com/www/features.nsf/Articles/557540D8F53... I think this might provide some interesting background.
jashkenas 13y ago
If anyone wants to search through MuckRock's recent documents via the DocumentCloud interface, they're also available over here: http://www.documentcloud.org/public/search/group:muckrock http://www.documentcloud.org/public/search/group:muckrock Edit: Whoops -- I see that Michael's already commented along the same lines. To expand a bit, you'll be able to do boolean searches among the documents, scan through excerpts around your search query, look at extracted entities (people, places, company names), and plot any dates mentioned in the documents on a timeline.

105 points by morisy 13y ago

Now this is investigative journalism!

stfu 13y ago

Is there already some decent crowd sourcing text analysis software out there? Some simplified version of Nvivo with a wiki-like user management would do wonders for something like that. 1. Button "I commit to analyze 2 / 12 /22 pages". 2. You get a very short tutorial on how to do it. 3. User is assigned relevant pages, starts tagging the documents for specific content and highlights + comments lines worth a closer look. I would be definitely up for that. With enough traction one could just compare inter-coder reliability and establishing a high quality analyzing process (i.e. getting the same document analyzed by multiple users).

solstice 13y ago
How about django-public-project? The following is from their description at http://django-public-project.readthedocs.org/en/latest/ http://django-public-project.readthedocs.org/en/latest/: Django Public Project (DPP) is a specialised content management system written in Python/Django for building an information website around big projects. It is open source (BSD licence) and can be used by civic groups to monitor the progress of publicly funded projects but also from within governmental institutions itself to bring transparency in the proceeding of a project, inform the interested public about what is going on and thus enhance the acceptance of the project by the different stakeholders. It is currently being used by the investigative comittee of the Berlin city parliament in order to investigate the massive clusterfk that is the city's new and very disfuntional airport (BER). The source code can be found at: https://github.com/holgerd77/django-public-project https://github.com/holgerd77/django-public-project Regarding participation of ordinary citizens, the docs say: On DPP websites users can comment on nearly everything and connect their comments to various elements of the project. Comments are then shown alongside every connected element. All admin users having activated the “Receive new comment emails” in the admin user settings receive an email after someone sent a comment and can then activate the comment to be shown on the website. http://django-public-project.readthedocs.org/en/latest/users.html#user-comments http://django-public-project.readthedocs.org/en/latest/users...

patdennis 13y ago

People get excited about things like this and say things like "this is REAL investigative journalism". Or "I wish mainstream journalists would start doing this". Look, I read documents like this all day long. So do investigative journalists. It's just that very few people care about the stuff we find until it turns into an international spy thriller. Just because you don't care about it on the state/local level doesn't mean we don't put in the work. Edit: You can look at some of the great investigative journalism you're probably not paying attention to here: http://www.ire.org/blog/extra-extra/ http://www.ire.org/blog/extra-extra/

rhizome 13y ago
I'm not sure what your point is, is someone saying that you're lazy?
- patdennis 13y ago
  I was just trying to preempt the inevitable "investigative journalism is dead" comments.
  - rhizome 13y ago
    Judging from the comments so far, it may not be so inevitable. However, there is still the law of averages, I guess.
deleted 13y ago
[deleted]
morisy 13y ago
"I wish mainstream journalists would start doing this". Spin it however you want, fewer and fewer mainstream journalists are able to focus on this. I work with a dozen or so newsrooms a year on FOIA, CAR, and other investigative reporting training, and in most newsrooms only 1 or 2 hands go up when I ask who has filed a public records request in the past year. MuckRock was built to change that: To make it easier to add document-requesting and analysis to the daily news cycle, to help non-journalists get interested, excited, and able to request and share documents, and to enable large scale-FOIA projects like request BAH files from dozens of agencies. And we've found that lots of people do care about it at the state/local level, but news organizations are a) terrible about marketing their work. b) assume readers aren't interested in the nuts and bolts. c) let that work die in obscurity, content to try to win awards with it rather than to truly engage the public. So yes, this is a flashy dump of documents, but we've helped hundreds of others do meaningful, smaller scale stories, and whatever way you cut it, I don't think there's enough groups doing that. I also keep my own list of interesting, public-records driven stories: http://pinboard.in/u:morisy/t:to_foi/ http://pinboard.in/u:morisy/t:to_foi/ Encourage anyone who is interested to reverse engineer them and try those requests in their city, state, and country.

rhizome 13y ago

They should be using plagiarism-detection software to strain out the boilerplate commonalities and focus on the differences (as well as a single copy of the commonalities).

rdl 13y ago

This is the kind of stuff I wish Wikileaks had stuck to -- mostly or completely legal, beneficial to national security, etc. They are still challenging powerful entities and supporting change, but without becoming the story themselves.

camus 13y ago
who made the "press" shift its attention on wikileaks rather than on the docs they leaked ? your favorite DC spin doctors. So please , dont blame the messenger , or you are as stupid as the government think you are. And they do think you are stupid , trust me.
- rdl 13y ago
  Releasing 250k classified documents is a news story itself, so it was inevitable that the story would shift to wikileaks. (shifting to Julian Assange was some DC spin, though, but he sure didn't help things.)
  - dantheman 13y ago
    But they didn't release the cables at first, they released an encrypted file. Then a journalist who didn't like wikileaks that had access to the file published the password in a book. Since the files were then available to those with resources to know about they password, they felt it was reasonable to release them completely so people could see if they were mentioned in the documents which were at that point for all intents and purposes public.
mpyne 13y ago
Likewise. I really liked WikiLeaks at the beginning but it became clear after some time that there was a definite agenda behind their actions. The impression I've had for years is that they will oppose the U.S. for no better reason than that it's the U.S. Which is, if you think about it, just like opposing any maligned group for what they are instead of what they do. In other words if we start from the assumption that the U.S. should be a "bastion of the free world" then it would make sense to tirelessly note instances where the U.S., corporations, states, etc. fall from that ideal. But diplomatic cables describing what kind of women Berlusconi likes, while interesting, are hardly indicators of the collapse of a free society. Those types of things were released because they were embarrassing, nothing more. And while I'm certainly glad that Assange is able to stroke his ego by thumbing his nose in Uncle Sam's face, I don't actually hate my country; I want it to be better.

morisy 13y ago

A good way to search through the text of the documents is DocumentCloud, which also offers some analysis tools: https://www.documentcloud.org/search/group:muckrock https://www.documentcloud.org/search/group:muckrock Modify the search terms as desired (i.e., "Booz")

Spooky23 13y ago

What exactly is the point of this exercise? Booz Allen is a big consulting company. They have lots of contracts and SOWs, most of which are for really lame things. You are not going to find a contract that says: "provide 3 big brother spies to read everyone's mail". Of you're trying to prove that that government spends lots of money for consultants... Well, we all know that already. Also, unless you really know what you are talking about, you are not going to be able to spot most of the petty, obvious or really offensive contracting practices. The Feds have their own language re: procurement.

nitrogen 13y ago
A while back there was a request posted to fbo.gov for an online sockpuppet management system. It's possible that some important details can be gleaned from these contracts as well.
rdl 13y ago
Based on the ones I've seen in other contexts, the key is to be able to aggregate. If you do, you can often find really interesting stuff from unclassified contracts around the periphery, then presumably FOIA more, but knowing how the system works helps a lot.

patrickg_zill 13y ago

http://taxanalysts.com/www/features.nsf/Articles/557540D8F539A26685257B94004CD0D1?OpenDocument http://taxanalysts.com/www/features.nsf/Articles/557540D8F53... I think this might provide some interesting background.

jashkenas 13y ago

If anyone wants to search through MuckRock's recent documents via the DocumentCloud interface, they're also available over here: http://www.documentcloud.org/public/search/group:muckrock http://www.documentcloud.org/public/search/group:muckrock Edit: Whoops -- I see that Michael's already commented along the same lines. To expand a bit, you'll be able to do boolean searches among the documents, scan through excerpts around your search query, look at extracted entities (people, places, company names), and plot any dates mentioned in the documents on a timeline.