4 ms·

I did the same years ago. We were providing realtime suburb data for a fleet of trains. Each train received a GPS coordinate once per minute, we took this and d

I did the same years ago. We were providing realtime suburb data for a fleet of trains. Each train received a GPS coordinate once per minute, we took this and displayed the suburb. So 1440 updates per day per train. For the fleet it was going to be over $100 a day in API costs. We were going to not display suburb data because of cost. In the end I found a creative commons placename database (geonames.org). For placenames with >500 people it's ~10MB of data and that covers the entire planet (surprisingly small). I then wrote a KD-Tree based library to look it up the nearest point in this table extremely efficiently (log(N) time). I'll admit i haven't updated or maintained it. The server running it has been chugging along well though >5years later. https://github.com/AReallyGoodName/OfflineReverseGeocode https://github.com/AReallyGoodName/OfflineReverseGeocode

nerdponx 7y ago
We were going to not display suburb data because of cost. In the end I found a creative commons placename database (geonames.org). For placenames with >500 people it's ~10MB of data and that covers the entire planet (surprisingly small). I then wrote a KD-Tree based library to look it up the nearest point in this table extremely efficiently (log(N) time). At a previous employer I tried to convince my managers to let me do this for months. They always balked. Their loss.
saila 7y ago
Have you used PostGIS at all? It seems like a good choice for this kind of lookup. https://postgis.net/ https://postgis.net/
- AnotherGoodName 7y ago
  PostGIS would be worth it if there was a lot more to do on that project and full respect to that project. I remember looking at it and it felt like it was a sledgehammer for that particular job. You still had to write up a lot of code to import the dataset you had and then extract it again. Not to mention maintenance since you now have a whole dependency chain installed on your server. This was a single engineer project. The above solution is a copy paste-able set of classes with zero dependencies that will output the nearest place. Sometimes a single purpose solution is perfect and i'm really not kidding when i say i haven't maintained or even looked at it in over 5 years yet it's still running fine as part of a larger application.
eruci 7y ago
I built a similar open service at https://3geonames.org/api https://3geonames.org/api . It supports XML and Json output and is entirely free. Eg: https://api.3geonames.org/51.4647,0.0079.json https://api.3geonames.org/51.4647,0.0079.json
BubRoss 7y ago
Why wouldn't you cache the results instead of making 1440 queries per day?
- AnotherGoodName 7y ago
  You still have the same problem. The input is a GPS coordinate. You can just return the same value as the last X minutes but that obviously sucks since you could be X minutes off in the real time suburb of a fast moving train. You could find the nearest to the cache but then you might as well just have a list of places you're finding the nearest again. Which is what i did.
  - BubRoss 7y ago
    I didn't realize the train coordinates were what the API was for. Were these public trains? How does google have their gps coordinates and no one else does?
    - AnotherGoodName 7y ago
      We installed hardware on each train to monitor them. So from each moving train we received a coordinate every minute to our servers.
      - BubRoss 7y ago
        Are you saying that you did that to switch off of google's API?
        AnotherGoodName 7y ago
        No. The only part of Googles API we considered was reverse geocoding. The train data was always ours.
        BubRoss 7y ago
        Right... but that would mean that what you are querying google for doesn't change. The coordinates of the trains would even only be along their route. Once you have done a reverse lookup on enough points along the route, you should have a pretty dense idea of where the neighborhoods change, right? I am not sure if I'm missing some big, but it really seems like there wouldn't be an ongoing need for querying google's APIs in this case. Finding the boundaries of the neighborhoods and especially labeling stops should be easy enough to do if it would save so much money.

nerdponx 7y ago

We were going to not display suburb data because of cost. In the end I found a creative commons placename database (geonames.org). For placenames with >500 people it's ~10MB of data and that covers the entire planet (surprisingly small). I then wrote a KD-Tree based library to look it up the nearest point in this table extremely efficiently (log(N) time). At a previous employer I tried to convince my managers to let me do this for months. They always balked. Their loss.

saila 7y ago

Have you used PostGIS at all? It seems like a good choice for this kind of lookup. https://postgis.net/ https://postgis.net/

AnotherGoodName 7y ago
PostGIS would be worth it if there was a lot more to do on that project and full respect to that project. I remember looking at it and it felt like it was a sledgehammer for that particular job. You still had to write up a lot of code to import the dataset you had and then extract it again. Not to mention maintenance since you now have a whole dependency chain installed on your server. This was a single engineer project. The above solution is a copy paste-able set of classes with zero dependencies that will output the nearest place. Sometimes a single purpose solution is perfect and i'm really not kidding when i say i haven't maintained or even looked at it in over 5 years yet it's still running fine as part of a larger application.

eruci 7y ago

I built a similar open service at https://3geonames.org/api https://3geonames.org/api . It supports XML and Json output and is entirely free. Eg: https://api.3geonames.org/51.4647,0.0079.json https://api.3geonames.org/51.4647,0.0079.json

BubRoss 7y ago

Why wouldn't you cache the results instead of making 1440 queries per day?

AnotherGoodName 7y ago
You still have the same problem. The input is a GPS coordinate. You can just return the same value as the last X minutes but that obviously sucks since you could be X minutes off in the real time suburb of a fast moving train. You could find the nearest to the cache but then you might as well just have a list of places you're finding the nearest again. Which is what i did.
- BubRoss 7y ago
  I didn't realize the train coordinates were what the API was for. Were these public trains? How does google have their gps coordinates and no one else does?
  - AnotherGoodName 7y ago
    We installed hardware on each train to monitor them. So from each moving train we received a coordinate every minute to our servers.
    - BubRoss 7y ago
      Are you saying that you did that to switch off of google's API?
      - AnotherGoodName 7y ago
        No. The only part of Googles API we considered was reverse geocoding. The train data was always ours.
        BubRoss 7y ago
        Right... but that would mean that what you are querying google for doesn't change. The coordinates of the trains would even only be along their route. Once you have done a reverse lookup on enough points along the route, you should have a pretty dense idea of where the neighborhoods change, right? I am not sure if I'm missing some big, but it really seems like there wouldn't be an ongoing need for querying google's APIs in this case. Finding the boundaries of the neighborhoods and especially labeling stops should be easy enough to do if it would save so much money.