4 ms·

What was your project, and what did you end up using?

by beering 10y ago

bpchaps 10y ago
unstructured blob: I'm working with data from a FOIA request for ~5yrs worth of parking tickets in Chicago. About 18m rows in total. The end goal is to be able to say whether a parking ticket was valid or not. It's been ongoing for about two years now, but when I have chunks of time I find myself getting back into the old code or just rewriting chunks. Part of the reason it's taking so long is that a personal requirement of mine is to treat every ticket on the same footing, so throwing rows out during cleanups etc isn't something I'm necessarily comfortable with. It's kinda frustrating but it's a hopeful goal. Since ticketers don't record lat/long and only [badly typo'd] addresses, I'm stuck doing the address to lat/long translation myself. It's not easy by any means since there are many, many many edge cases. The rate limits on common street address resolution tends to be pretty low (~10k/day) and the paid APIs are ungodly expensive. SM has been a great tool to help things go a bit forward, but when an address is missing, so is its lat/long which in many cases means rows get thrown out - which I'm not comfortable with. Interpolation using shapefile data would work, but the number of edge cases makes that difficult. I believe the data I have right now uses a combination of census data, OSM data and shapefiles from data.cityofchicago.org. Someone in the local data civics group put together a cleaned up copy of that where I was originally using the sources separately. I can pull it if you're curious. To complicate things, having missing data adds noise and false positives when correcting typos. There don't really seem to be many address sources that are consistent with address types - eg, "pl" vs "plaza" vs "place". I threw together some pretty decent python code with recursive difflib to clean typos using other typo'd addresses as a path to the correct address and it does pretty well to correct much of that. For example, in a few steps 'nrragansitt' becomes 'narragansett'.
- nl 10y ago
  Given that you have a fixed area the tickets can be in, and you know every possible address, it seems like you could investigate string distance measures. It would solve your pl vs plaza vs place problem.
  - bpchaps 10y ago
    That's what using difflib does. Plaza vs pl vs place is just one example of many, though. There are about 1k different street names in chicago and about 18k different street names in the data. Many are missing street type and there are many street names which have different street types. So that tends to open up cans of worms. That, and street names that have 'street' in their names. Also, 'avenue' is a street name which screws things up left and right. It's not a simple problem, and none of the popular machine learning libraries which use string distance for this sort of problem have been very effective.
    - maxerickson 10y ago
      Have you tried https://github.com/openvenues/libpostal https://github.com/openvenues/libpostal?
      - bpchaps 10y ago
        Yep - it's actually been pretty invaluable in getting this project to where it is now. :) It definitely has its own flaws, but those are easy to work around. Edit - looks like they have a similar problem from dealing with hand typed data. From their road map: Ambiguous token classification (coming soon): e.g. "dr" => "doctor" or "drive" for an English address depending on the context. Multiclass logistic regression trained on OSM addresses, where abbreviations are discouraged, giving us many examples of fully qualified addresses on which to train

What was your project, and what did you end up using?

by beering 10y ago

What was your project, and what did you end up using?

unstructured blob: I'm working with data from a FOIA request for ~5yrs worth of parking tickets in Chicago. About 18m rows in total. The end goal is to be able to say whether a parking ticket was valid or not. It's been ongoing for about two years now, but when I have chunks of time I find myself getting back into the old code or just rewriting chunks. Part of the reason it's taking so long is that a personal requirement of mine is to treat every ticket on the same footing, so throwing rows out during cleanups etc isn't something I'm necessarily comfortable with. It's kinda frustrating but it's a hopeful goal. Since ticketers don't record lat/long and only [badly typo'd] addresses, I'm stuck doing the address to lat/long translation myself. It's not easy by any means since there are many, many many edge cases. The rate limits on common street address resolution tends to be pretty low (~10k/day) and the paid APIs are ungodly expensive. SM has been a great tool to help things go a bit forward, but when an address is missing, so is its lat/long which in many cases means rows get thrown out - which I'm not comfortable with. Interpolation using shapefile data would work, but the number of edge cases makes that difficult. I believe the data I have right now uses a combination of census data, OSM data and shapefiles from data.cityofchicago.org. Someone in the local data civics group put together a cleaned up copy of that where I was originally using the sources separately. I can pull it if you're curious. To complicate things, having missing data adds noise and false positives when correcting typos. There don't really seem to be many address sources that are consistent with address types - eg, "pl" vs "plaza" vs "place". I threw together some pretty decent python code with recursive difflib to clean typos using other typo'd addresses as a path to the correct address and it does pretty well to correct much of that. For example, in a few steps 'nrragansitt' becomes 'narragansett'.

nl 10y ago
Given that you have a fixed area the tickets can be in, and you know every possible address, it seems like you could investigate string distance measures. It would solve your pl vs plaza vs place problem.
- bpchaps 10y ago
  That's what using difflib does. Plaza vs pl vs place is just one example of many, though. There are about 1k different street names in chicago and about 18k different street names in the data. Many are missing street type and there are many street names which have different street types. So that tends to open up cans of worms. That, and street names that have 'street' in their names. Also, 'avenue' is a street name which screws things up left and right. It's not a simple problem, and none of the popular machine learning libraries which use string distance for this sort of problem have been very effective.
  - maxerickson 10y ago
    Have you tried https://github.com/openvenues/libpostal https://github.com/openvenues/libpostal?
    - bpchaps 10y ago
      Yep - it's actually been pretty invaluable in getting this project to where it is now. :) It definitely has its own flaws, but those are easy to work around. Edit - looks like they have a similar problem from dealing with hand typed data. From their road map: Ambiguous token classification (coming soon): e.g. "dr" => "doctor" or "drive" for an English address depending on the context. Multiclass logistic regression trained on OSM addresses, where abbreviations are discouraged, giving us many examples of fully qualified addresses on which to train