5 ms·

GeoGuessing with Deep Learning

109 points by healeycodes 6y ago

wp381640 6y ago
I started working on a similar project - my approach was to train a model based on all of the geoguessr contest sets. Geoguessr gives you both the street view _and_ the correct answer when you guess wrong. They're all hand-curated so there are only so many points that you need to identify (the largest sets are 100k+ points) so the idea would be to run through it thousands of times with wrongs answers but use the correct answer to train the model It's only a matter of time before the game has cheat bots and that problem, much in the same way online chess has that issue
- luguenth 6y ago
  What would be the advantage to train a model on the whole dataset, you would be running it against later? Wouldn't that overfit the network? And for the cheating part; geoGuessr actually exposes the right answer in their API. You you could just use that to pinpoint the exact location automated.
  - ajmurmann 6y ago
    > What would be the advantage to train a model on the whole dataset, you would be running it against later? Wouldn't that overfit the network? The opposite, it would almost be cheating since the model would just have memorized all the correct solutions. You'd usually want to train on a subset of the data and evaluate against the remainder to protect against overfitting. So to that point I agree with your evaluation. But after that you'd use it with real data in the wild for your actual use case. With geogussr you'd know already what all the "real-world" situations are and overfitting wouldn't matter as long as you retrain wherever they add new sets.
totetsu 6y ago
This could be useful for finding new places to go that are look similar to your holiday snaps. Also if I could do this with a picture of food I want to eat.. and it guess the probable location of a restaurant serving it, that would be great.
jibolso 6y ago
Nicely done. It will be interesting to see the results when you combine multiple modalities.
tobr 6y ago
GeoGuessr is a lot of fun. I played it a lot a few years ago, but only recently discovered that there’s a community of very serious players, and got interested in it again. For example check this video from yesterday where Tom Davies aka GeoWizard[1] gets a pretty good score on An Urban World: https://youtube.com/watch?v=TVt1GKBZMzc https://youtube.com/watch?v=TVt1GKBZMzc 1: https://en.wikipedia.org/wiki/GeoWizard https://en.wikipedia.org/wiki/GeoWizard
- herbstein 6y ago
  Tom Davis is very good at the game. I also recommend checking out his "straight line mission" videos. He tries crossing Wales (and more recently Norway) in a single straight line. Deviating by only a few feet across the trips. He manages to make it quite interesting to "just" watch a man walking in fields.
  - herbstein 6y ago
    I thought a link was in order here. First video of his first "mission". Not everything he does is strictly legal, but it's never malicious. https://youtu.be/M7w986ni7_g https://youtu.be/M7w986ni7_g
- sllabres 6y ago
  The linked article https://somerandomstuff1.wordpress.com/2019/02/08/geoguessr-the-top-tips-tricks-and-techniques/ https://somerandomstuff1.wordpress.com/2019/02/08/geoguessr-... is interesting -- at least for me, a very casual player ;).
  - lastofthemojito 6y ago
    I'm a fairly casual player too, and I guess I have my favorite aspects of the game - I've worked with foreign languages before so language identification is fun for me, and I'm a car enthusiast so I like informing guesses based on the cars I see. One thing this article raised that I'd never considered was the direction that satellite dishes face. Definitely going to keep my eye out for that!
chanind 6y ago
I wondered if this could be useful for open source journalism stuff like bellingcat does. Essentially combing through tons of photos online to figure out where / when they were taken and Put together an understanding of an incident
bvm 6y ago
You know how you get off a plane and everything about a place 3000 miles away feels different? Not just smells and heat, but the light seems different. The surfaces and materials are different. The uh...(actual) atmosphere seems different. Is for example, the atmosphere seeming different a real physical phenomenon that could be detected by ML?
- qayxc 6y ago
  Sure, why not? Chemical composition, temperature, and pressure of the atmosphere are just physical parameters (e.g. elements of a feature set) like any other (e.g. colour and intensity of pixels in an image). Everything that can be measured can be used as an input for ML algorithms. The things that "feel" different are the result of the "post-processing" your brain applies to the inputs it receives (olfactory, temperature, light intensity and -spectral composition, etc.). The sensation is based on physical phenomena, though it might be interesting how much of it remains if we were to take the knowledge about the change of location away, e.g. would you feel the same way if you didn't know you were 3000 miles away? In other words, does knowing about the change in location sharpen your senses such that you subconsciously look for changes in the environment?
- TheAdamAndChe 6y ago
  I've never traveled like this, merely drove cross-country and didn't experience this. Does it really feel so different?
hyper_reality 6y ago
Interesting post, but I disagree with the conclusion that with a bit of work AI could take down the best Geoguessrs. The best players already take into account all the metadata like camera quality and distinctive features of the Google car. Furthermore, they would surely have an edge in urban locations where proximate locations are visible in street-signs. I could be proven wrong though, and it would be super interesting to reverse engineer AI guesses that turn out to be surprisingly correct based off little info. To see just how good the top players are at instantly recognising a country using these clues, then check out https://www.youtube.com/watch?v=zEmoAYpTJuA https://www.youtube.com/watch?v=zEmoAYpTJuA . Or maybe don't if you don't want the game spoiled!
- tobr 6y ago
  > I disagree with the conclusion that with a bit of work AI could take down the best Geoguessrs. The best players already take into account all the metadata like camera quality and distinctive features of the Google car. Isn’t this exactly what an AI might do even better, though? There might be statistical variations in image quality between cameras and processing that are imperceptible to a human but easy for machine learning to pick up?
  - hyper_reality 6y ago
    True, but less than half of world countries are covered by street view and it's possible for a human to learn the distinctive metadata features of every single one, together with all the common roadside features like poles, road markings, and vegetation. So I don't know if the level of granularity where statistical variations are noticed is required to succeed at least in determining the country. But in terms of picking the closest point within a country, perhaps with a large enough dataset the AI would be able to distinguish based on local weather conditions when the Google car was driving through certain regions. And this would trump the player's ability to read place names and signage.
lastofthemojito 6y ago
I'm a GeoGuessr fan, and like the author, experienced a bit of a GeoGuessr renaissance when I had a baby. I guess it's the sort of slow, quiet game that lends itself to being played between baby care tasks. In any case, the criteria discussed here is the sort of thing that all too often makes me skeptical of deep learning results. There are the things that I consider "real" discriminators when playing the game - architectural styles, building materials, road quality, driving direction, alphabets/languages on signage, flags, flora, topography, makes/types of vehicles, skin color and clothing style of humans, etc. Then there are the "artificial" things - glimpses at the street view vehicle, copyright notices, image quality, knowledge of which countries do or don't have Google Street View coverage, etc. Obviously by taking what I'm calling the "artificial" criteria into account a deep learning approach would score better today than if only the "real" criteria were considered. But I feel like if tomorrow, GeoGuessr swapped Apple Maps Look Around or Bing Maps StreetSide in place of Google Street View (or if Google released vastly updated imagery), the deep learning approach would fall apart, but humans who have built knowledge of the "real" features to look for would continue to do just about as well.
- alkonaut 6y ago
  Coming from 64 degrees north I always react when I see satellite dishes tilted up and realized a satellite dish is an exact indication - to the degree! - of latitude. Some climates can be deceiving depending on local conditions (Gulf Stream makes Western Europe have flora like much more southern parts of North America for example) but satellite dishes work the same everywhere.
benatkin 6y ago
Ooh, this is great. I love GeoGuessr and have pointed several people to it. I have thought that a trained AI could do better than me on average. I've been posting my runs publicly to Strava for years. I recently started living in a small town and decided to stop posting my location, and using an open source tool called RunnerUp to track my runs. I have still been posting running photos that don't reveal my location. I know someday people will be able to figure out my location despite avoiding taking pictures with signs and stuff. If anyone wants to try to figure out one of my recent locations, my instagram link is in my bio.
- lastofthemojito 6y ago
  It's wild how we're anonymous on the internet, right up until we're not! I was watching an auction on bringatrailer.com (an auto auction website where people sell cars and discuss them in a comments section) the other day. One commenter realized he could distantly see his house in a stranger's auction photos! The person posting the photos probably never considered that the photo location would be precisely identifiable (and for this purpose, probably didn't care). But if you get enough eyes on something, someone is going to recognize it.

109 points by healeycodes 6y ago

I started working on a similar project - my approach was to train a model based on all of the geoguessr contest sets. Geoguessr gives you both the street view _and_ the correct answer when you guess wrong. They're all hand-curated so there are only so many points that you need to identify (the largest sets are 100k+ points) so the idea would be to run through it thousands of times with wrongs answers but use the correct answer to train the model It's only a matter of time before the game has cheat bots and that problem, much in the same way online chess has that issue

luguenth 6y ago
What would be the advantage to train a model on the whole dataset, you would be running it against later? Wouldn't that overfit the network? And for the cheating part; geoGuessr actually exposes the right answer in their API. You you could just use that to pinpoint the exact location automated.
- ajmurmann 6y ago
  > What would be the advantage to train a model on the whole dataset, you would be running it against later? Wouldn't that overfit the network? The opposite, it would almost be cheating since the model would just have memorized all the correct solutions. You'd usually want to train on a subset of the data and evaluate against the remainder to protect against overfitting. So to that point I agree with your evaluation. But after that you'd use it with real data in the wild for your actual use case. With geogussr you'd know already what all the "real-world" situations are and overfitting wouldn't matter as long as you retrain wherever they add new sets.

totetsu 6y ago

This could be useful for finding new places to go that are look similar to your holiday snaps. Also if I could do this with a picture of food I want to eat.. and it guess the probable location of a restaurant serving it, that would be great.

jibolso 6y ago

Nicely done. It will be interesting to see the results when you combine multiple modalities.

tobr 6y ago

GeoGuessr is a lot of fun. I played it a lot a few years ago, but only recently discovered that there’s a community of very serious players, and got interested in it again. For example check this video from yesterday where Tom Davies aka GeoWizard[1] gets a pretty good score on An Urban World: https://youtube.com/watch?v=TVt1GKBZMzc https://youtube.com/watch?v=TVt1GKBZMzc 1: https://en.wikipedia.org/wiki/GeoWizard https://en.wikipedia.org/wiki/GeoWizard

herbstein 6y ago
Tom Davis is very good at the game. I also recommend checking out his "straight line mission" videos. He tries crossing Wales (and more recently Norway) in a single straight line. Deviating by only a few feet across the trips. He manages to make it quite interesting to "just" watch a man walking in fields.
- herbstein 6y ago
  I thought a link was in order here. First video of his first "mission". Not everything he does is strictly legal, but it's never malicious. https://youtu.be/M7w986ni7_g https://youtu.be/M7w986ni7_g
sllabres 6y ago
The linked article https://somerandomstuff1.wordpress.com/2019/02/08/geoguessr-the-top-tips-tricks-and-techniques/ https://somerandomstuff1.wordpress.com/2019/02/08/geoguessr-... is interesting -- at least for me, a very casual player ;).
- lastofthemojito 6y ago
  I'm a fairly casual player too, and I guess I have my favorite aspects of the game - I've worked with foreign languages before so language identification is fun for me, and I'm a car enthusiast so I like informing guesses based on the cars I see. One thing this article raised that I'd never considered was the direction that satellite dishes face. Definitely going to keep my eye out for that!

chanind 6y ago

I wondered if this could be useful for open source journalism stuff like bellingcat does. Essentially combing through tons of photos online to figure out where / when they were taken and Put together an understanding of an incident

bvm 6y ago

You know how you get off a plane and everything about a place 3000 miles away feels different? Not just smells and heat, but the light seems different. The surfaces and materials are different. The uh...(actual) atmosphere seems different. Is for example, the atmosphere seeming different a real physical phenomenon that could be detected by ML?

qayxc 6y ago
Sure, why not? Chemical composition, temperature, and pressure of the atmosphere are just physical parameters (e.g. elements of a feature set) like any other (e.g. colour and intensity of pixels in an image). Everything that can be measured can be used as an input for ML algorithms. The things that "feel" different are the result of the "post-processing" your brain applies to the inputs it receives (olfactory, temperature, light intensity and -spectral composition, etc.). The sensation is based on physical phenomena, though it might be interesting how much of it remains if we were to take the knowledge about the change of location away, e.g. would you feel the same way if you didn't know you were 3000 miles away? In other words, does knowing about the change in location sharpen your senses such that you subconsciously look for changes in the environment?
TheAdamAndChe 6y ago
I've never traveled like this, merely drove cross-country and didn't experience this. Does it really feel so different?

hyper_reality 6y ago

Interesting post, but I disagree with the conclusion that with a bit of work AI could take down the best Geoguessrs. The best players already take into account all the metadata like camera quality and distinctive features of the Google car. Furthermore, they would surely have an edge in urban locations where proximate locations are visible in street-signs. I could be proven wrong though, and it would be super interesting to reverse engineer AI guesses that turn out to be surprisingly correct based off little info. To see just how good the top players are at instantly recognising a country using these clues, then check out https://www.youtube.com/watch?v=zEmoAYpTJuA https://www.youtube.com/watch?v=zEmoAYpTJuA . Or maybe don't if you don't want the game spoiled!

tobr 6y ago
> I disagree with the conclusion that with a bit of work AI could take down the best Geoguessrs. The best players already take into account all the metadata like camera quality and distinctive features of the Google car. Isn’t this exactly what an AI might do even better, though? There might be statistical variations in image quality between cameras and processing that are imperceptible to a human but easy for machine learning to pick up?
- hyper_reality 6y ago
  True, but less than half of world countries are covered by street view and it's possible for a human to learn the distinctive metadata features of every single one, together with all the common roadside features like poles, road markings, and vegetation. So I don't know if the level of granularity where statistical variations are noticed is required to succeed at least in determining the country. But in terms of picking the closest point within a country, perhaps with a large enough dataset the AI would be able to distinguish based on local weather conditions when the Google car was driving through certain regions. And this would trump the player's ability to read place names and signage.

lastofthemojito 6y ago

I'm a GeoGuessr fan, and like the author, experienced a bit of a GeoGuessr renaissance when I had a baby. I guess it's the sort of slow, quiet game that lends itself to being played between baby care tasks. In any case, the criteria discussed here is the sort of thing that all too often makes me skeptical of deep learning results. There are the things that I consider "real" discriminators when playing the game - architectural styles, building materials, road quality, driving direction, alphabets/languages on signage, flags, flora, topography, makes/types of vehicles, skin color and clothing style of humans, etc. Then there are the "artificial" things - glimpses at the street view vehicle, copyright notices, image quality, knowledge of which countries do or don't have Google Street View coverage, etc. Obviously by taking what I'm calling the "artificial" criteria into account a deep learning approach would score better today than if only the "real" criteria were considered. But I feel like if tomorrow, GeoGuessr swapped Apple Maps Look Around or Bing Maps StreetSide in place of Google Street View (or if Google released vastly updated imagery), the deep learning approach would fall apart, but humans who have built knowledge of the "real" features to look for would continue to do just about as well.

alkonaut 6y ago
Coming from 64 degrees north I always react when I see satellite dishes tilted up and realized a satellite dish is an exact indication - to the degree! - of latitude. Some climates can be deceiving depending on local conditions (Gulf Stream makes Western Europe have flora like much more southern parts of North America for example) but satellite dishes work the same everywhere.

benatkin 6y ago

Ooh, this is great. I love GeoGuessr and have pointed several people to it. I have thought that a trained AI could do better than me on average. I've been posting my runs publicly to Strava for years. I recently started living in a small town and decided to stop posting my location, and using an open source tool called RunnerUp to track my runs. I have still been posting running photos that don't reveal my location. I know someday people will be able to figure out my location despite avoiding taking pictures with signs and stuff. If anyone wants to try to figure out one of my recent locations, my instagram link is in my bio.

lastofthemojito 6y ago
It's wild how we're anonymous on the internet, right up until we're not! I was watching an auction on bringatrailer.com (an auto auction website where people sell cars and discuss them in a comments section) the other day. One commenter realized he could distantly see his house in a stranger's auction photos! The person posting the photos probably never considered that the photo location would be precisely identifiable (and for this purpose, probably didn't care). But if you get enough eyes on something, someone is going to recognize it.