3 ms·

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

ImaCake 2y ago
Yeah, this entire thread seems utterly detached from my lived experience. LLMs are immensely useful for me at work but they certainly don't come close to the hype spouted by many commenters here. It would be great if it could handle more of our quite modest codebase but it's not able to yet
- m_ke 2y ago
  ARC is a silly benchmark, the other results in math and coding are much more impressive. o3 is just o1 scaled up, the main takeaway from this line of work that people should walk away with is that we now have a proven way to RL our way to super human performance on tasks where it’s cheap to sample and easy to verify the final output. Programming falls in that category, they focused on known benchmarks but the same process can be done for normal programs, using parsers, compilers, existing functions and unit tests as verifiers. Pre o1 we only really had next token prediction, which required high quality human produced data, with o1 you optimize for success instead of MLE of next token. Explained in simpler terms, it means it can get reward for any implementation of a function that reproduces the expected result, instead of the exact implementation in the training set. Put another way, it’s just like RLHF but instead of optimizing against learned human preferences, the model is trained to satisfy a verifier. This should work just as well in VLA models for robotics, self driving and computer agents.
causal 2y ago
I think that's part of what feels odd about this- in some ways it feels like the wrong type of test for an LLM, but in many ways it makes this achievement that much more remarkable
Jensson 2y ago
> If humans were given the json as input rather than the images, they’d have a hard time, too. We shine light in text patterns at humans rather than inject the text directly into the brain as well, that is extremely unfair! Imagine how much better humans would be at text processing if we injected and extracted information from their brains using the neurons instead of eyes and hands.
torginus 2y ago
Not sure how much that matters - I'm not an AI expert, but I did some intro courses where we had to train a classifier to recognize digits. How it worked basically was that we fed each pixel of the 2d grid of the image into an input of the network, essentially flattening it in a similar fashion. It worked just fine, and that was a tiny network.
- thegeomaster 2y ago
  The classifier was likely a convolutional network, so the assumption of the image being a 2D grid was baked into the architecture itself - it didn't have to be represented via the shape of the input for the network to use it.
  - torginus 2y ago
    I don't think so - convolutional neural networks also operate over 1D flat vectors - the spatial relationship of pixels is only learned from the training data.
    - thegeomaster 2y ago
      This is not true. CNNs perform 2D convolution, conceptually "sliding" a 2 dimensional kernel with learnable weights over the input image across two dimensions. Perhaps it wasn't a convolutional network after all, but a simple fully-connected feed-forward network taking all pixels as input? Could be viable for a toy example (MNIST).
deneas 2y ago
The JSON files still contain images, just not in a regular image format. You have a 2D array of numbers where each number maps to a color. If you really want a regular picture format, you can easily convert the arrays.

ImaCake 2y ago

Yeah, this entire thread seems utterly detached from my lived experience. LLMs are immensely useful for me at work but they certainly don't come close to the hype spouted by many commenters here. It would be great if it could handle more of our quite modest codebase but it's not able to yet

m_ke 2y ago
ARC is a silly benchmark, the other results in math and coding are much more impressive. o3 is just o1 scaled up, the main takeaway from this line of work that people should walk away with is that we now have a proven way to RL our way to super human performance on tasks where it’s cheap to sample and easy to verify the final output. Programming falls in that category, they focused on known benchmarks but the same process can be done for normal programs, using parsers, compilers, existing functions and unit tests as verifiers. Pre o1 we only really had next token prediction, which required high quality human produced data, with o1 you optimize for success instead of MLE of next token. Explained in simpler terms, it means it can get reward for any implementation of a function that reproduces the expected result, instead of the exact implementation in the training set. Put another way, it’s just like RLHF but instead of optimizing against learned human preferences, the model is trained to satisfy a verifier. This should work just as well in VLA models for robotics, self driving and computer agents.

causal 2y ago

I think that's part of what feels odd about this- in some ways it feels like the wrong type of test for an LLM, but in many ways it makes this achievement that much more remarkable

Jensson 2y ago

> If humans were given the json as input rather than the images, they’d have a hard time, too. We shine light in text patterns at humans rather than inject the text directly into the brain as well, that is extremely unfair! Imagine how much better humans would be at text processing if we injected and extracted information from their brains using the neurons instead of eyes and hands.

torginus 2y ago

Not sure how much that matters - I'm not an AI expert, but I did some intro courses where we had to train a classifier to recognize digits. How it worked basically was that we fed each pixel of the 2d grid of the image into an input of the network, essentially flattening it in a similar fashion. It worked just fine, and that was a tiny network.

thegeomaster 2y ago
The classifier was likely a convolutional network, so the assumption of the image being a 2D grid was baked into the architecture itself - it didn't have to be represented via the shape of the input for the network to use it.
- torginus 2y ago
  I don't think so - convolutional neural networks also operate over 1D flat vectors - the spatial relationship of pixels is only learned from the training data.
  - thegeomaster 2y ago
    This is not true. CNNs perform 2D convolution, conceptually "sliding" a 2 dimensional kernel with learnable weights over the input image across two dimensions. Perhaps it wasn't a convolutional network after all, but a simple fully-connected feed-forward network taking all pixels as input? Could be viable for a toy example (MNIST).

deneas 2y ago

The JSON files still contain images, just not in a regular image format. You have a 2D array of numbers where each number maps to a color. If you really want a regular picture format, you can easily convert the arrays.