3 ms·

Citations for that last claim?

taneq 3y ago
I think the Othello paper is the go-to for proving that transformer networks do actually create internal models of the things they're trained on. https://thegradient.pub/othello/ https://thegradient.pub/othello/
viraptor 3y ago
This post we're commenting on. Also https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell... Also, there's a paper I can't find now which shows you can find the name of the operation in the middle layers if you provide example of what to do. That is, if you prompt: "9 2 5 -> 5 2 9, 0 7 3 ->", you'll find the internals generate "reverse" even though it doesn't appear in the output.
- dartos 3y ago
  So you can programmatically approximate the scene intrinsics in this paper with traditional programs as well given a single image. All these scene data is already suggested by the data, what the AI model is doing is approximating the data it needs to generate a scene. It’s not just gluing data together, it’s discovering interrelationships in a very high dimensional feature space. It’s not creating anything it hasn’t seen either, which is why many image models (mostly smaller ones) have so much trouble with making fine details make sense, but are good with universal patterns like physics, lighting, the human shape, and composition. (not sure if you were trying to imply that, but it felt like it)
- DinaCoder99 3y ago
  If you try it with a longer sequence of numbers it seems to stumble. E.g. here's what gemini spits out when I input "9 2 5 8 4 3 7 1 0 2 9 4 -> 4 9 2 0 1 7 3 4 8 5 2 9, 0 7 3 8 6 2 9 4 1 7 5 2 0 3 4 8 5 1 ->": > You're correct! The pattern you identified applies to both sets of numbers. Here's the breakdown: > First set: > Original: 9 2 5 8 4 3 7 1 0 2 9 4 > Rearranged: 4 9 2 0 1 7 3 4 8 5 2 9 > Second set: > Original: 0 7 3 8 6 2 9 4 1 7 5 2 0 3 4 8 5 1 > Rearranged: 1 5 8 0 3 4 2 9 7 5 0 7 3 8 6 2 9 4 > In both cases, the rearrangement follows the same steps: > Move the first digit of each group of three to the last position. > Keep the middle digit in the same position. > Repeat steps 1 and 2 for all groups of three. > Therefore, the pattern holds true for both sets of numbers you provided. So—it's not clear there's recognition of a general process going on so much as recognizing a very specific process (one simple enough in form it's almost certainly in the training text somewhere).

I think the Othello paper is the go-to for proving that transformer networks do actually create internal models of the things they're trained on. https://thegradient.pub/othello/ https://thegradient.pub/othello/

viraptor 3y ago

This post we're commenting on. Also https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell... Also, there's a paper I can't find now which shows you can find the name of the operation in the middle layers if you provide example of what to do. That is, if you prompt: "9 2 5 -> 5 2 9, 0 7 3 ->", you'll find the internals generate "reverse" even though it doesn't appear in the output.

dartos 3y ago
So you can programmatically approximate the scene intrinsics in this paper with traditional programs as well given a single image. All these scene data is already suggested by the data, what the AI model is doing is approximating the data it needs to generate a scene. It’s not just gluing data together, it’s discovering interrelationships in a very high dimensional feature space. It’s not creating anything it hasn’t seen either, which is why many image models (mostly smaller ones) have so much trouble with making fine details make sense, but are good with universal patterns like physics, lighting, the human shape, and composition. (not sure if you were trying to imply that, but it felt like it)
DinaCoder99 3y ago
If you try it with a longer sequence of numbers it seems to stumble. E.g. here's what gemini spits out when I input "9 2 5 8 4 3 7 1 0 2 9 4 -> 4 9 2 0 1 7 3 4 8 5 2 9, 0 7 3 8 6 2 9 4 1 7 5 2 0 3 4 8 5 1 ->": > You're correct! The pattern you identified applies to both sets of numbers. Here's the breakdown: > First set: > Original: 9 2 5 8 4 3 7 1 0 2 9 4 > Rearranged: 4 9 2 0 1 7 3 4 8 5 2 9 > Second set: > Original: 0 7 3 8 6 2 9 4 1 7 5 2 0 3 4 8 5 1 > Rearranged: 1 5 8 0 3 4 2 9 7 5 0 7 3 8 6 2 9 4 > In both cases, the rearrangement follows the same steps: > Move the first digit of each group of three to the last position. > Keep the middle digit in the same position. > Repeat steps 1 and 2 for all groups of three. > Therefore, the pattern holds true for both sets of numbers you provided. So—it's not clear there's recognition of a general process going on so much as recognizing a very specific process (one simple enough in form it's almost certainly in the training text somewhere).