2 ms·
[Statistical parsing](http://nlp.stanford.edu/courses/lsa354/ http://nlp.stanford.edu/courses/lsa354/) is often used in Natural Language Processing, where can't
by mrcactu5 13y ago
[Statistical parsing](http://nlp.stanford.edu/courses/lsa354/ http://nlp.stanford.edu/courses/lsa354/) is often used in Natural Language Processing, where can't decide which how to arrange sentences into trees, so they pick one at random among acceptable choices.
I've gotten very frustrated writing rules and regexes and exceptions. This method seems really great when scraping data which is structured but a little bit noisy.
- tlarkworthy 13y agoI got sick of writing rules too, hence the diversion! yeah so you could extend that tree decision processes this. You could add a cost function on the trees, which includes long range communication between nodes (e.g. the results section of a document includes at least one word commonly associated with results). So you could go to higher levels of abstractions. However, I say limiting yourself to even a very abstract parse tree structure does kinda exclude the spatial layout of words on a page and the semantic meaning behind the layout (tables, web pages, code formatting!). MIPs can fully express tree logic so you lose nothing.
- mrcactu5 13y agoLooks like you were trying to complete data found with Optical Character Recognition by maximum-entropy. This is awesome. What are some other use cases for this? With APIs much of the data I use has already been formatted in JSON or CSV already, so I don't get to scrape much anymore.