5 ms·

Sequential modeling enables scalable learning for large vision models

132 points by famouswaffles 3y ago

samstevens 3y ago
Very odd that they don't do linear probing on the model features and measure ImageNet validation accuracy. In general, this paper is missing comparisons to existing vision models on existing benchmarks. The visual prompting is a neat trick and it's great to see scale continue to work, but without comparisons to other models or releasing the code/weights, I don't think this strategy is going to be competitive.
famouswaffles 3y ago
I'd simply been thinking about Large Vision Models in an annotation sense. QandA, Captioning...That sort of thing. Even though it makes so much sense, I never thought about it like this. Inpainting, Object Detection, Rotation, Lighting, Segmentation, Edge Detection, Pose Estimation, Surface Normal, Colorization and much more achieved by a single model. I believe this and Codi-2(https://codi-2.github.io/ https://codi-2.github.io/) offer a glimpse of the future of Large Multimodal Models.
- kaibee 3y ago
  So this is artificial general intelligence, right?
  - ben_w 3y ago
    The problem with the phrase "artificial general intelligence" is that everyone is arguing about the definition of all three words, and has a different threshold for the boolean pass/fail boundary.
  - famouswaffles 3y ago
    General Intelligence is a gradient, not a hard on/off. Obviously, these are machines so artificial. They're certainly not narrow in scope or abilities so general. They perform tasks we consider intelligent. So...sure. Like ENIAC, i imagine we'll build or have built agi well before everyone can agree it is so.
- BoiledCabbage 3y ago
  One of the things sci-fi really seemed to get right is that we will have AGI long before we'll have agreement that it is actually AGI. People will keep finding some small case or reason why not to call it AGI. And then finally once that last case is knocked down and we have agreement on a definition, we'll realize we crossed that threshold a "long" while back. And I'm not saying we have AGI now, just that it's now clear to me how this process will play out. (Where "long" in AI development timeliness probably doesn't mean the same thing "long" meant even in the 2010s.)
  - MorbidCuriosity 3y ago
    I think the last hurdles will be sentience and a theory of the mind. Theory of the mind is probably something that a LM could predict given enough data, but sentience? I don't think we understand enough about our own sentience for us to create it in a machine.
    - famouswaffles 3y ago
      LLMs can already predict theory of mind in text and keep good track of character motivations and knowledge in a story.
    - kenmorechalfant 3y ago
      It goes back to the old question of whether matter is fundamental and consciousness emerges from it or the other way around. My inclination is the latter, by way of Descartes. "I think therefore I am." If you can think "I am me" then you know with certainty that your first person identity exists but all other information that comes to you through your senses will always be an incomplete picture of the material universe. I think this is the strongest argument why consciousness precedes matter (or pervades through all matter, depending how you look at it). If sentience is a fundamental property of the universe, the way I see it, all matter shares the same soul. That makes it perfectly logical for a machine that mimics the pathways of the brain to behave just like a brain, to me.
    - naasking 3y ago
      > I don't think we understand enough about our own sentience for us to create it in a machine. Invention preceding understanding is the norm, not the exception. We created fire before understanding chemistry, and we constantly use pharmaceuticals without really understanding how they work. Invention first, then theory comes along to explain and generalize. For all we know, sentience is a necessary side-effect of semantic processing of any kind, in which case LLMs already have a form of sentience. So yes, you're right that we don't understand our own sentience. In fact, we understand so little that it could literally be staring us in the face right now and we don't realize it.
- deleted 3y ago
  [deleted]
toxik 3y ago
Would have been neat to see some animations since it is video frames in many cases.
- mpeg 3y ago
  The "add a frame to a video" use-cases are probably the least exciting here, the image annotation capabilities seem to me the bigger deal.
mickdarling 3y ago
Has anyone tried using transformers on weather forecasts yet?
- ben_w 3y ago
  Yes, several. • https://arxiv.org/pdf/2106.14742.pdf https://arxiv.org/pdf/2106.14742.pdf • https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/met.2098 https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/met.2... • https://ieeexplore.ieee.org/document/9671442 https://ieeexplore.ieee.org/document/9671442 Also non-transformer models, because of the scaling complexity on input: https://arxiv.org/pdf/2212.12794.pdf https://arxiv.org/pdf/2212.12794.pdf
cs702 3y ago
Upon reading this, my immediate thought is: It's only a matter of time before we have robots powered by large models pretrained to "predict the next token" across a bunch of different sensory modalities -- sight, sound, smell, touch, taste, etc. in a variety of artificial and natural settings, including social-interaction settings. Learning to read, learning to talk, learning to interact with the physical world, and so on -- all of it could very well be built upon the simple idea of learning to "predict the next token." We live in interesting times.
- catchnear4321 3y ago
  across a bunch of different gradients. senses will be the next step, humans grok that good and easy. once multiple gradients are considered, non-sensory gradients are going to be next. this is all a bunch of gobbledygook until it isn’t.
- raidicy 3y ago
  These sentiments are pretty close to my own. I read a paper that claimed that llms are General Pattern Machines and could be used to complete small games in gym environments. It seems to me that if these things really are General Pattern Machines all we have to do is figure out a way to represent any data as a pattern and try and predict the next step in the pattern right? The multi token[1] project which allows you to take any type of data and turn it into a token it's pretty interesting and seems like it's going in this direction. I would really like to see a framework where you can take any modality of any type turn it into a series of tokens and just cram it into a language model and effectively turning into a multimodal model with almost no effort. [0]https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ [1] https://github.com/sshh12/multi_token https://github.com/sshh12/multi_token
  - sshh12 3y ago
    Yeah this is exactly what I was thinking when building multi_token! Just bottlenecked so far by my own GPU, datasets, and freetime
    - 3y ago
agarsev 3y ago
I'm reading my thesis next tuesday, and the advances in AI the last couple of years have already made most of it obsolete :/ Anyway I'm excited and looking forward to the code and models to be released, hopefully I can use them for my research! I think it's easy to overlook how revolutionary the transformer "way" of doing things has been, and the fact that so many different tasks can be reformulated in a "language" way I believe hints at something deeper about how the universe, our minds and language work.
bambax 3y ago
Do captchas have a future? It seems inevitable that AI will beat humans on captcha real soon (if not already). What's next?
- jksk61 3y ago
  I don't know about you but I use chat gpt to solve captcha because I can't.
- jazzyjackson 3y ago
  I should patent this idea but here it goes anyway: in the future Captcha's will consist of requesting you make an antisemitic or misogynist remark to prove you are human, since the bots will be held to higher moral standards than man.
  - cooper_ganglia 3y ago
    This is a completely silly comment, but I will admit that it made me chuckle just because of how incredibly random and shocking it was, especially here on HN, lol
    - MarcoZavala 3y ago
      [dead]
  - PeterisP 3y ago
    Obviously, not all bots will be held to such standards - these are the standards which apply to western corporations who offer free bots for PR and marketing purposes, since 'politically incorrect' behavior stunts that goal, however, anyone keeping their own bot (e.g. a spammer wanting to solve millions of captchas) has no problems running an 'uncensored' bot - right now you can get reasonably large open source models without the RLHF pretraining and thus also no attempt at 'moral standards' other than those you choose to put in when finetuning the bot for your purposes. It's relatively trivial to flip the sign on that training data and have a bot that will instead refuse to make antifascist or feminist arguments; if someone wants a Hitlerbot to ghostwrite "My struggle" for them, there's nothing that could prevent them from finetuning such a model from one of the publicly available models; there is no one that can enforce any 'moral standards' on the bots other than their creators.
deleted 3y ago
[deleted]
mola 3y ago
So this can solve visual analogies from iq tests?
jerpint 3y ago
I would have loved to see videos on the blog post of completions
dwaltrip 3y ago
Hypothesis: Intelligence is prediction?
- romesc 3y ago
  Or prediction is intelligence. I think we need a more rigorous definition to go beyond tautology
- keenmaster 3y ago
  That intuitively makes sense to me for everything except higher-level goal selection. If someone works at an NGO instead of FAANG because they want to do good for the world, how can that be explained in terms of prediction? If someone minimizes consumption to reduce global warming, how did prediction result in them caring about global warming? I picked those examples on purpose, because they run against some core instincts. Maybe they’re predicting what maximizes peer approval, which optimizes other outcomes too? (Jk :D )
  - dwaltrip 3y ago
    You are right, prediction doesn’t provide one with base motivations or drives. But those seem fundamentally different than raw intelligence. Your examples aren’t an issue I don’t think. To take one: efforts to minimize one’s consumption to combat global warming could fall in the “help tribe survival” bucket, which makes evolutionary sense as a base drive. And it seems like a reasonably “intelligent” response to that base drive — the predicted results seems positive.
- trhway 3y ago
  which is very close as to say that intelligence is learning [from mis-prediction], a subject of cybernetics - common machinery across the scales and instances - using a model (brain in case of animals, model/embedding in case of computer, DNA in case of evolution through natural selection) a prediction (which for example in case of evolution is the next generation) is made, and based on the error of that prediction the model is updated for the generation of the next prediction (in case of evolution the DNA of the best fit is the model to be used in the next iteration).
fancyfredbot 3y ago
Next step, accept user input from VR glasses and we've basically got a holodeck.
iandanforth 3y ago
If this paper is coming out of BAIR at a max 3B parameter model I suspect we'll quickly see much larger models from the industrial players. Hopefully Mistral takes an interest and releases an OSI licensed model.

132 points by famouswaffles 3y ago

Very odd that they don't do linear probing on the model features and measure ImageNet validation accuracy. In general, this paper is missing comparisons to existing vision models on existing benchmarks. The visual prompting is a neat trick and it's great to see scale continue to work, but without comparisons to other models or releasing the code/weights, I don't think this strategy is going to be competitive.

famouswaffles 3y ago

I'd simply been thinking about Large Vision Models in an annotation sense. QandA, Captioning...That sort of thing. Even though it makes so much sense, I never thought about it like this. Inpainting, Object Detection, Rotation, Lighting, Segmentation, Edge Detection, Pose Estimation, Surface Normal, Colorization and much more achieved by a single model. I believe this and Codi-2(https://codi-2.github.io/ https://codi-2.github.io/) offer a glimpse of the future of Large Multimodal Models.

kaibee 3y ago
So this is artificial general intelligence, right?
- ben_w 3y ago
  The problem with the phrase "artificial general intelligence" is that everyone is arguing about the definition of all three words, and has a different threshold for the boolean pass/fail boundary.
- famouswaffles 3y ago
  General Intelligence is a gradient, not a hard on/off. Obviously, these are machines so artificial. They're certainly not narrow in scope or abilities so general. They perform tasks we consider intelligent. So...sure. Like ENIAC, i imagine we'll build or have built agi well before everyone can agree it is so.
BoiledCabbage 3y ago
One of the things sci-fi really seemed to get right is that we will have AGI long before we'll have agreement that it is actually AGI. People will keep finding some small case or reason why not to call it AGI. And then finally once that last case is knocked down and we have agreement on a definition, we'll realize we crossed that threshold a "long" while back. And I'm not saying we have AGI now, just that it's now clear to me how this process will play out. (Where "long" in AI development timeliness probably doesn't mean the same thing "long" meant even in the 2010s.)
- MorbidCuriosity 3y ago
  I think the last hurdles will be sentience and a theory of the mind. Theory of the mind is probably something that a LM could predict given enough data, but sentience? I don't think we understand enough about our own sentience for us to create it in a machine.
  - famouswaffles 3y ago
    LLMs can already predict theory of mind in text and keep good track of character motivations and knowledge in a story.
  - kenmorechalfant 3y ago
    It goes back to the old question of whether matter is fundamental and consciousness emerges from it or the other way around. My inclination is the latter, by way of Descartes. "I think therefore I am." If you can think "I am me" then you know with certainty that your first person identity exists but all other information that comes to you through your senses will always be an incomplete picture of the material universe. I think this is the strongest argument why consciousness precedes matter (or pervades through all matter, depending how you look at it). If sentience is a fundamental property of the universe, the way I see it, all matter shares the same soul. That makes it perfectly logical for a machine that mimics the pathways of the brain to behave just like a brain, to me.
  - naasking 3y ago
    > I don't think we understand enough about our own sentience for us to create it in a machine. Invention preceding understanding is the norm, not the exception. We created fire before understanding chemistry, and we constantly use pharmaceuticals without really understanding how they work. Invention first, then theory comes along to explain and generalize. For all we know, sentience is a necessary side-effect of semantic processing of any kind, in which case LLMs already have a form of sentience. So yes, you're right that we don't understand our own sentience. In fact, we understand so little that it could literally be staring us in the face right now and we don't realize it.
deleted 3y ago
[deleted]

toxik 3y ago

Would have been neat to see some animations since it is video frames in many cases.

mpeg 3y ago
The "add a frame to a video" use-cases are probably the least exciting here, the image annotation capabilities seem to me the bigger deal.

mickdarling 3y ago

Has anyone tried using transformers on weather forecasts yet?

ben_w 3y ago
Yes, several. • https://arxiv.org/pdf/2106.14742.pdf https://arxiv.org/pdf/2106.14742.pdf • https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/met.2098 https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/met.2... • https://ieeexplore.ieee.org/document/9671442 https://ieeexplore.ieee.org/document/9671442 Also non-transformer models, because of the scaling complexity on input: https://arxiv.org/pdf/2212.12794.pdf https://arxiv.org/pdf/2212.12794.pdf

cs702 3y ago

Upon reading this, my immediate thought is: It's only a matter of time before we have robots powered by large models pretrained to "predict the next token" across a bunch of different sensory modalities -- sight, sound, smell, touch, taste, etc. in a variety of artificial and natural settings, including social-interaction settings. Learning to read, learning to talk, learning to interact with the physical world, and so on -- all of it could very well be built upon the simple idea of learning to "predict the next token." We live in interesting times.

catchnear4321 3y ago
across a bunch of different gradients. senses will be the next step, humans grok that good and easy. once multiple gradients are considered, non-sensory gradients are going to be next. this is all a bunch of gobbledygook until it isn’t.
raidicy 3y ago
These sentiments are pretty close to my own. I read a paper that claimed that llms are General Pattern Machines and could be used to complete small games in gym environments. It seems to me that if these things really are General Pattern Machines all we have to do is figure out a way to represent any data as a pattern and try and predict the next step in the pattern right? The multi token[1] project which allows you to take any type of data and turn it into a token it's pretty interesting and seems like it's going in this direction. I would really like to see a framework where you can take any modality of any type turn it into a series of tokens and just cram it into a language model and effectively turning into a multimodal model with almost no effort. [0]https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ [1] https://github.com/sshh12/multi_token https://github.com/sshh12/multi_token
- sshh12 3y ago
  Yeah this is exactly what I was thinking when building multi_token! Just bottlenecked so far by my own GPU, datasets, and freetime
  - 3y ago

agarsev 3y ago

I'm reading my thesis next tuesday, and the advances in AI the last couple of years have already made most of it obsolete :/ Anyway I'm excited and looking forward to the code and models to be released, hopefully I can use them for my research! I think it's easy to overlook how revolutionary the transformer "way" of doing things has been, and the fact that so many different tasks can be reformulated in a "language" way I believe hints at something deeper about how the universe, our minds and language work.

bambax 3y ago

Do captchas have a future? It seems inevitable that AI will beat humans on captcha real soon (if not already). What's next?

jksk61 3y ago
I don't know about you but I use chat gpt to solve captcha because I can't.
jazzyjackson 3y ago
I should patent this idea but here it goes anyway: in the future Captcha's will consist of requesting you make an antisemitic or misogynist remark to prove you are human, since the bots will be held to higher moral standards than man.
- cooper_ganglia 3y ago
  This is a completely silly comment, but I will admit that it made me chuckle just because of how incredibly random and shocking it was, especially here on HN, lol
  - MarcoZavala 3y ago
    [dead]
- PeterisP 3y ago
  Obviously, not all bots will be held to such standards - these are the standards which apply to western corporations who offer free bots for PR and marketing purposes, since 'politically incorrect' behavior stunts that goal, however, anyone keeping their own bot (e.g. a spammer wanting to solve millions of captchas) has no problems running an 'uncensored' bot - right now you can get reasonably large open source models without the RLHF pretraining and thus also no attempt at 'moral standards' other than those you choose to put in when finetuning the bot for your purposes. It's relatively trivial to flip the sign on that training data and have a bot that will instead refuse to make antifascist or feminist arguments; if someone wants a Hitlerbot to ghostwrite "My struggle" for them, there's nothing that could prevent them from finetuning such a model from one of the publicly available models; there is no one that can enforce any 'moral standards' on the bots other than their creators.

deleted 3y ago

[deleted]

mola 3y ago

So this can solve visual analogies from iq tests?

jerpint 3y ago

I would have loved to see videos on the blog post of completions

dwaltrip 3y ago

Hypothesis: Intelligence is prediction?

romesc 3y ago
Or prediction is intelligence. I think we need a more rigorous definition to go beyond tautology
keenmaster 3y ago
That intuitively makes sense to me for everything except higher-level goal selection. If someone works at an NGO instead of FAANG because they want to do good for the world, how can that be explained in terms of prediction? If someone minimizes consumption to reduce global warming, how did prediction result in them caring about global warming? I picked those examples on purpose, because they run against some core instincts. Maybe they’re predicting what maximizes peer approval, which optimizes other outcomes too? (Jk :D )
- dwaltrip 3y ago
  You are right, prediction doesn’t provide one with base motivations or drives. But those seem fundamentally different than raw intelligence. Your examples aren’t an issue I don’t think. To take one: efforts to minimize one’s consumption to combat global warming could fall in the “help tribe survival” bucket, which makes evolutionary sense as a base drive. And it seems like a reasonably “intelligent” response to that base drive — the predicted results seems positive.
trhway 3y ago
which is very close as to say that intelligence is learning [from mis-prediction], a subject of cybernetics - common machinery across the scales and instances - using a model (brain in case of animals, model/embedding in case of computer, DNA in case of evolution through natural selection) a prediction (which for example in case of evolution is the next generation) is made, and based on the error of that prediction the model is updated for the generation of the next prediction (in case of evolution the DNA of the best fit is the model to be used in the next iteration).

fancyfredbot 3y ago

Next step, accept user input from VR glasses and we've basically got a holodeck.

iandanforth 3y ago

If this paper is coming out of BAIR at a max 3B parameter model I suspect we'll quickly see much larger models from the industrial players. Hopefully Mistral takes an interest and releases an OSI licensed model.