3 ms·
It turns out this problem even shows up for relatively small datasets, things that would easily fit into memory on any consumer-level laptop from a couple years
by bane 12y ago
It turns out this problem even shows up for relatively small datasets, things that would easily fit into memory on any consumer-level laptop from a couple years ago. I did some contract work for the NYPD a few years ago and the primary data sources I had to deal with was a block of free text describing a 911 call. All they really wanted was some frequency counts on specific terms of interest, sounds easy enough.
Oh, but their data warehouse thing that nobody knew what it was called, inserted a random space every so often and removed random spaces every so often making the text completely unreliable. Compiled with all the normal misspellings, fat fingered acronyms, slang and other nonstandard ways of writing things it turned out to be a huge job.
"Bicycle stolen at 0930 from residence on the 300 block of E 10th St, East Village"
would get turned into "Bicycle stol en a t093 0 from r es iden ceon the 300 block of E 10 thst, Ea st Vil lage."
Good luck with that.
- arthurjj 12y agoHow did you end up dealing with the issue? My first idea would be to remove all the spaces and parse that
- marktangotango 12y agoNothing related to analyzing free form text should ever be considered 'easy'. The wrinkle you describe was just the cherry on top.
- ithayer 12y agoProbably way beyond the scope of that project, but actually this is a pretty interesting and well-studied problem called segmentation, where you train a system to insert the correct breaks. This is used in natural language problems like OCR or machine translation. Maybe google "machine translation segmentation" or OCR for references to papers on the topic, but methods for doing this are really successful (especially on english).
- adkatrit 12y agoi bet you'd have better luck just removing all spaces and then resorting to a text segmentation algorithm. Peter Norvig has some great papers on how to do this effectively. I've wrapped his code with a Tornado web service: https://github.com/adkatrit/text-segmentation-server https://github.com/adkatrit/text-segmentation-server