4 ms·

I’m intrigued to see what kind of problems it’s going to be good/bad at. I think it’s going to be tricky to evaluate though because it has probably memorized al

I’m intrigued to see what kind of problems it’s going to be good/bad at. I think it’s going to be tricky to evaluate though because it has probably memorized all the easy images to eval it with. Eg anything pulled from Google Images (like that Pulp Fiction frame or city skyline photo) is not a good test. It recognizes common shots but if you pull a screenshot from Google Maps or a random screen cap from the movie it doesn’t do as well. I tried having it play Geoguessr via screenshots & it wasn’t good at it.

loupol 3y ago
I wonder how many images from Street View it has been trained on. I've seen top Geoguessr players be able to pretty consistently determine a location worldwide after seeing a photo for just one second. So I would assume training an LLM to do the same would definitely be doable.
- npinsker 3y ago
  It's been done recently! It's a bit better than (but competitive with) top players. https://www.youtube.com/watch?v=ts5lPDV--cU https://www.youtube.com/watch?v=ts5lPDV--cU
- inductive_magic 3y ago
  > I would assume training an LLM to do the same would definitely be doable. I wouldn't be so sure. The reasoning process of Geoguessr pros is symbolic, not statistical inference. /edit: as other commenters pointed out, something similar was done. While this wasn't an LLM, it was a deep learning model, so not symbolic -> https://www.theregister.com/2023/07/15/pigeon_model_geolocation/ https://www.theregister.com/2023/07/15/pigeon_model_geolocat...
- fellerts 3y ago
  Yep, some CS/AI grads from Stanford trained an AI on loads of Street View images and built a bot that is able to beat some of the best Geoguessr players: https://www.youtube.com/watch?v=ts5lPDV--cU https://www.youtube.com/watch?v=ts5lPDV--cU
  - bayesianbot 3y ago
    IIRC it wasn't that impressive in the end as instead of recognizing the places the AI apparently learnt to recognize subtle differences in street view cameras used in different locations? I might be wrong / thinking of the wrong model l and I'm on mobile without my browsing history so hard to check, but I think it was putting a lot of weight on some pixels that are noisy
    - zx_q12 3y ago
      Top geoguessr players use this technique as well. IIRC rainbolt mentioned that there is a section of a country where the street view camera has a small blemish from a raindrop on the camera so you can instantly tell where you are if you notice that.
    - thewataccount 3y ago
      From my understanding many of the best players immediately look down to tell what "generation streetview car" they're using, and seem to know what continents/times they're from.
skazazes 3y ago
It seems it will still be limited by its linguistic understanding of the surrounding context, at least in the first chicken sandwich picture. Although its interpretation could make some sense but is also mostly wrong if talking about physical size of a modern GPU's main processor compared to the size of the associated VRAM chips. It has missed the joke entirely as far as I am aware. I think the joke is actual about Nvidia's handling of product segmentation, selling massive processors with less memory than is reasonable to pair them with on their consumer gaming offerings, while loading up the nearly identical chips with more memory for scientific and compute applications...
- Melatonic 3y ago
  Ironically the exact processors need to run GPT-4V in the first place.....

loupol 3y ago

I wonder how many images from Street View it has been trained on. I've seen top Geoguessr players be able to pretty consistently determine a location worldwide after seeing a photo for just one second. So I would assume training an LLM to do the same would definitely be doable.

npinsker 3y ago
It's been done recently! It's a bit better than (but competitive with) top players. https://www.youtube.com/watch?v=ts5lPDV--cU https://www.youtube.com/watch?v=ts5lPDV--cU
inductive_magic 3y ago
> I would assume training an LLM to do the same would definitely be doable. I wouldn't be so sure. The reasoning process of Geoguessr pros is symbolic, not statistical inference. /edit: as other commenters pointed out, something similar was done. While this wasn't an LLM, it was a deep learning model, so not symbolic -> https://www.theregister.com/2023/07/15/pigeon_model_geolocation/ https://www.theregister.com/2023/07/15/pigeon_model_geolocat...
fellerts 3y ago
Yep, some CS/AI grads from Stanford trained an AI on loads of Street View images and built a bot that is able to beat some of the best Geoguessr players: https://www.youtube.com/watch?v=ts5lPDV--cU https://www.youtube.com/watch?v=ts5lPDV--cU
- bayesianbot 3y ago
  IIRC it wasn't that impressive in the end as instead of recognizing the places the AI apparently learnt to recognize subtle differences in street view cameras used in different locations? I might be wrong / thinking of the wrong model l and I'm on mobile without my browsing history so hard to check, but I think it was putting a lot of weight on some pixels that are noisy
  - zx_q12 3y ago
    Top geoguessr players use this technique as well. IIRC rainbolt mentioned that there is a section of a country where the street view camera has a small blemish from a raindrop on the camera so you can instantly tell where you are if you notice that.
  - thewataccount 3y ago
    From my understanding many of the best players immediately look down to tell what "generation streetview car" they're using, and seem to know what continents/times they're from.

skazazes 3y ago

It seems it will still be limited by its linguistic understanding of the surrounding context, at least in the first chicken sandwich picture. Although its interpretation could make some sense but is also mostly wrong if talking about physical size of a modern GPU's main processor compared to the size of the associated VRAM chips. It has missed the joke entirely as far as I am aware. I think the joke is actual about Nvidia's handling of product segmentation, selling massive processors with less memory than is reasonable to pair them with on their consumer gaming offerings, while loading up the nearly identical chips with more memory for scientific and compute applications...

Melatonic 3y ago
Ironically the exact processors need to run GPT-4V in the first place.....