4 ms·
The Strangest FOIA Redactions
- lykr0n 8y agoThe person who can make an amazing, open source search engine (like the Google Appliance) that can take in Exchange Servers, File Shares, and any other data source and turn around and be able to do full text search on the data would become very rich- or very well liked. Some of the examples here show that even in the government, information is not really centrally cataloged or searchable. One hand doesn't know about the other. Maybe some of it is malice, but I have to think part of it is just not having the ability to search Email, File Shares, Sharepoint, et cetera quickly and easily.
- marklyon 8y agoLike Relativity, Nuix or any of the existing eDiscovery processing tools?
- deleted 8y ago[deleted]
- sansnomme 8y agoElasticsearch?
- lykr0n 8y agoYou need to add data to Elasticsearch for it to be useful. I wrote a search engine for Confluence using Elasticsearch- which is amazing, but you need to add components to get it functional. I'm thinking a "index this site" or "Index this fileshare" and go.
- runlevel1 8y agoAny chance of open sourcing it? Confluence's search is appallingly bad, and tags barely help.
- lykr0n 8y agoMaybe. I've PoC'ed it, but it's really rough. The biggest issue is UI. The indexer and query engine is primitive, but it works. I've been meaning to re-write it. If I do it after work I can opensource it. Look for a Show HN Post or I'll fire off an email directly too you in about a week.
- sansnomme 8y agoThink you can write a blog post on general purpose indexing with Elasticsearch? (Also, thoughts on Mayan EDMS?)
- logon_carrier 8y agoNice mention of Mayan. It is a fantastic tool and the secret weapon in our arsenal! I'm the IT manager at a legal firm, these are our use case scenarios for Mayan: - Defeat document dumps (https://en.wikipedia.org/wiki/Document_dump https://en.wikipedia.org/wiki/Document_dump). We just scan everything and use Mayan's full text search. - We added an .eml importer to upload dumped emails from Exchange servers. The OCR and the indexing sort all the emails by keywords which we then compile as evidence in cases. - We uploaded tons of historical cases, judicial opinions, laws, amendments, etc. We now have a superb, free, self-hosted, electronic legal research library. It is not documented, but we tried it and versions > 3.0 of Mayan added search operators (AND, OR, NOT). This puts it on par with dedicated full text search engines like Elasticsearch. We have been talking about adding something like a Mayan web scrapper to get PDFs from government sites automatically and bypass the FOIA requests for published files since we'll have them indexed preemptively. Final note: Mayan scales beautifully but after > 500K documents start using Postgres connection pooling or you'll kill your DB server.
- techslave 8y agothat is a solved problem. the person who can do “AI” on the results to save all the human postprocessing hours stands to make a fortune.
- btrettel 8y agoI filed a FOI request to the UK Ministry of Defence for a classified report in my field. About a month later they gave me a copy. The only redactions were names of various researchers, including the names of the authors of the report that I requested. The odd part was that I already knew the authors' names and gave them in the request in the first place. Must have been some policy that is followed regardless of whether it makes sense.
- jrootabega 8y agoWhat if you knew the names of some of them but took a guess at the rest? Including the names would confirm your guess.
- taneq 8y agoGood point, the same way if someone phones you out of the blue and asks you to confirm your name and date of birth (or other personally identifiable information), you don't tell them anything.
- deleted 8y ago[deleted]
- heelix 8y agoI was amazed at how quickly they processed my FOIA requests. Asked customs for my travel so I could match up some dates on my heavily stamped passport. Within a month or so, they sent me a decade of travel all cleanly printed out. I'd already sent out the guestimate which is what inspired the request - but it was nice to sync up what I thought it was to what they thought it was.
- j16sdiz 8y agoNot surprising. Searching travel record is what they do in daily operation (applying visa, border control, etc..).
- Turukawa 8y agoI've filed over 1,600 FOI requests for commercial ratepayer data in the UK over the last 4 years, and I've seen some truly pathetic reasons for rejection of my requests. The worst are the ones claiming that such data would cause terrorism.
- bpchaps 8y agoNeat - one of my requests is listed! https://www.eff.org/deeplinks/2019/03/foilies-2019#seattle https://www.eff.org/deeplinks/2019/03/foilies-2019#seattle