4 ms·

>Does anyone have a file listing for this? Is it just a PGDATA dump, or are there more interesting things? I see a lot of text files (docx, doc, rtf, txt), and

>Does anyone have a file listing for this? Is it just a PGDATA dump, or are there more interesting things? I see a lot of text files (docx, doc, rtf, txt), and two DB dumps. Not sure how/where to start looking. Opening files at random yields things like collective bargaining agreements and legal briefs. Any suggestions for munging through all of this?

encoderer 11y ago
There should be tools for crowdsourcing this sort of thing. It's only going to become more common. Does such a tool already exist? Something to help split up a corpus so people can collaborate on it.
- mutagen 11y ago
  Yes, there are tools exploring a couple different approaches. It's a tough problem if you want solid results, verification to avoid forged documents inserted alongside authentic ones, building tools to facilitate finding the important stuff in all the minutia of day to day exchanges. Some stuff isn't easily fully crowdsourced because the domain knowledge to find the smoking gun among the jargon of a particular field. A good tool will at least let the average citizen distill information into a form an expert can quickly digest large numbers of documents though. Here's a linkdump: https://www.documentcloud.org/home https://www.documentcloud.org/home https://civic.mit.edu/blog/shidash/effective-approaches-and-experiments-in-leak-processing https://civic.mit.edu/blog/shidash/effective-approaches-and-... https://github.com/crowdata/crowdata https://github.com/crowdata/crowdata http://towcenter.org/research/guide-to-crowdsourcing/ http://towcenter.org/research/guide-to-crowdsourcing/ https://en.wikipedia.org/wiki/List_of_crowdsourcing_projects https://en.wikipedia.org/wiki/List_of_crowdsourcing_projects (too broad, search page for roots of terms like 'transcribe' or 'document' or the like.
- jordigh 11y ago
  > There should be tools for crowdsourcing this sort of thing. It's only going to become more common. That's why Wikileaks originally decided to call itself "wiki", but they gave up on that idea very quickly.

encoderer 11y ago

There should be tools for crowdsourcing this sort of thing. It's only going to become more common. Does such a tool already exist? Something to help split up a corpus so people can collaborate on it.

mutagen 11y ago
Yes, there are tools exploring a couple different approaches. It's a tough problem if you want solid results, verification to avoid forged documents inserted alongside authentic ones, building tools to facilitate finding the important stuff in all the minutia of day to day exchanges. Some stuff isn't easily fully crowdsourced because the domain knowledge to find the smoking gun among the jargon of a particular field. A good tool will at least let the average citizen distill information into a form an expert can quickly digest large numbers of documents though. Here's a linkdump: https://www.documentcloud.org/home https://www.documentcloud.org/home https://civic.mit.edu/blog/shidash/effective-approaches-and-experiments-in-leak-processing https://civic.mit.edu/blog/shidash/effective-approaches-and-... https://github.com/crowdata/crowdata https://github.com/crowdata/crowdata http://towcenter.org/research/guide-to-crowdsourcing/ http://towcenter.org/research/guide-to-crowdsourcing/ https://en.wikipedia.org/wiki/List_of_crowdsourcing_projects https://en.wikipedia.org/wiki/List_of_crowdsourcing_projects (too broad, search page for roots of terms like 'transcribe' or 'document' or the like.
jordigh 11y ago
> There should be tools for crowdsourcing this sort of thing. It's only going to become more common. That's why Wikileaks originally decided to call itself "wiki", but they gave up on that idea very quickly.