SIFT is not enough for this task. There are many computer vision researchers working on this application (automatically inferring location of an image), and acc

SIFT is not enough for this task. There are many computer vision researchers working on this application (automatically inferring location of an image), and accuracies are pretty low right now, except for popular landmarks. The problem is that single SIFT features are not very distinctive, and so you need many of them in common between two images to get reliable matches. So you essentially need images taken from very similar locations. You can ameliorate this requirement slightly by using fancy tricks in how you aggregate SIFT features together, but the fundamental constraint remains. This precondition is satisfied at popular landmarks, since there is a good chance that someone's taken a photo from the same location you're standing at, but in general, this condition is not that easy to meet, and hence the poor accuracy of current approaches. Finally, SIFT is great only for roughly-2d, distinctively textured areas. However, a lot (maybe most?) of the world does NOT fall into this category -- many things are either not textured, not distinctive, or are 3d. Something other than local descriptors (of which SIFT is the best known example) will be needed to understand these kind of scenes. If you're curious about this line of research, a good place to start is the IM2GPS paper, which was the first major work (that I'm aware of) to look at this problem: http://graphics.cs.cmu.edu/projects/im2gps/ http://graphics.cs.cmu.edu/projects/im2gps/