7 ms·
I had some sort of violent dopamine release just reading the headline. I'm working on a project to make (EU-) law more accessible. So if anybody here knows goo
by matt4077 12y ago
I had some sort of violent dopamine release just reading the headline.
I'm working on a project to make (EU-) law more accessible. So if anybody here knows good methods to visualise/summarise long legal texts (30-300 pages) you could do something for humanity by posting a reply.
(Word clouds just don't cut it in these cases.)
- miket 12y agoCheck out http://casetext.com http://casetext.com. Their "ReCites" extraction is a pretty useful way of summarizing a legal case text.
- matt4077 12y agoExcellent! It seem to heavily rely on the community & moderation – something we're exploring with as well. Kinda like (rap)genius.
- mjohn 12y agoIs there a link to your project? I've recently been thinking of trying something similar for UK law.
- mikecb 12y agoHaven't looked at EU law in a year or two, but europa and eulex used to provide useful and simple summaries of both legislation and court of justice caselaw.
- alexwestin 12y agoContact Kasian Franks kasian.franks@gmail.com he has a good summarization system I'm using.
- bane 12y agoA classic summarization method is: 1. Split your text into sentences 2. Remove stopwords (keeping a copy of the original sentence) 3. For the remaining words, calculate the synthetic TF-IDF score of all the words in the sentence (tf-idf each word then sum them). 4. Keep the n-highest scoring sentences (in the order they appear), or all the sentences with synthetic TF-IDF scores above some threshold. There's your summary.
- riazrizvi 12y agoI find it works better with additional step 2.5) Lemmatize remaining words, using for example Python's NLTK library.
- bane 12y agoOh yeah, good idea. That should generally improve the results. There's a bit of art to it for sure that can improve the results. You may also have to do some pronoun substitution in the summarized sentences (and then decide to do that before or after calculating the synthetic score) so they make more sense. I find it's hard to tell how well all this will work until you just do it. Not every kind of text works equally well and the only proof that it's really working is "does the summary make sense or not?" It's also possible that for long structured documents like laws or contracts, that you don't want to summarize the whole thing, but treat major sections like different documents and do intra-document summarization to maintain understandability. Here's one that does something kinda like what I was writing about above. http://textcompactor.com/ http://textcompactor.com/ note I think the correct measure is not TF-IDF, but TF-ISF (Inverse sentence frequency) for single document summarization, but I might be wrong.
- ccleve 12y agoWhen you lemmatize, what are you doing exactly? Are you merely reducing words to a more common form, thereby reducing IDF? For example, are you reducing "walking" or "walked" to "walk", and then using the IDF of walk?
- bane 12y ago
- cafebeen 12y agoThis might be helpful: http://www.cs.ubc.ca/labs/imager/tr/2014/Overview/ http://www.cs.ubc.ca/labs/imager/tr/2014/Overview/ It includes a number of the data mining techniques others have mentioned and implements some nice design guidelines from visual analytics.
- ntoshev 12y agoWhat is the user supposed to learn from the visualization? Is it high level understanding what the text is about? Is it some support to make the text easier to read and understand (like syntax highlighting is for programmers)? Is it something in between, something else?