19 ms·

Transformers, originally designed to handle language, are taking on vision

190 points by theafh 5y ago

melling 5y ago
On Kaggle they are getting more usage.
andrewmatte 5y ago
I have not given transformers enough attention... but my impression is that this is still storing entities in the weights of the neural network instead of in a database where the can be operated on with CRUD. What are the knowledge discovery researchers doing with respect to transformers? And the SAT solver researchers? Here is an article on KDNuggets that explains transformers but doesn't answer my questions: https://www.kdnuggets.com/2021/06/essential-guide-transformers-key-modern-sota-ai.html https://www.kdnuggets.com/2021/06/essential-guide-transforme...
- ctoth 5y ago
  > The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information[0]. [0]: https://jalammar.github.io/illustrated-retrieval-transformer/ https://jalammar.github.io/illustrated-retrieval-transformer...
- stingraycharles 5y ago
  I think it’s relatively straightforward to serialize such a model into different representations, I completely understand that they keep the actual data inside pytorch state by default. Out of curiosity, what tools are researchers generally using to explore neural networks? I’m just an armchair ML enthusiast myself, but NN always appear very much like black boxes. What are the goals and methods for exploring neural network state nowadays?
- cleancoder0 5y ago
  First transformer models still dealt only with the training set. Eventually it was extended to work with an external data source that it queries. This is not a new thing, for example, image style transfer and some other image tasks that were attempted before the domination of NNs did the same thing (linear models would query the db for help and guided feature extraction). The greatest effect in transformers is the attention mechanism combined with self-supervised learning. Investigations in self-supervised learning tasks (article illustrates one word gap, but there are others) can result in superior models that are sometimes even easier to train. As for SAT, optimization, graph neural networks might end up being more effective (due to high structure of the inputs). I'm definitely awaiting for traveling salesman solver or similar, guided by NN, solving things faster and reaching optimality more frequently that optimized heuristic algos.
  - graycat 5y ago
    > I'm definitely awaiting for traveling salesman solver or similar, guided by NN, solving things faster and reaching optimality more frequently that optimized heuristic algos. Just in case we are not being clear, let's be clear. Bluntly in nearly every practical sense, the traveling salesman problem (TSP) is NOT very difficult. Instead we have had good approaches for decades. I got into the TSP writing software to schedule the fleet for FedEx. A famous, highly accomplished mathematician asked me what I was doing at FedEx, and as soon as I mentioned scheduling the fleet he waved his hand and concluded I was only wasting time, that the TSP was too hard. He was wrong, badly wrong. Once I was talking with some people in a startup to design the backbone of the Internet. They were convinced that the TSP was really difficult. In one word, WRONG. Big mistake. Expensive mistake. Hype over reality. I mentioned that my most recent encounter with combinatorial optimization was solving a problem with 600,000 0-1 variables and 40,000 constraints. They immediately, about 15 of them, concluded I was lying. I was telling the full, exact truth. So, what is difficult about the TSP? Okay, we would like an algorithm for some software that would solve TSP problems (1) to exact optimality, (2) in worst cases, (3) in time that grows no faster than some polynomial in the size of the input data to the problem. So, for (1) being provably within 0.025% of exact optimality is not enough. And for (2) exact optimality in polynomial time for 99 44/100% of real problems is not enough. In the problem I attacked with 600,000 0-1 variables and 40,000 constraints, a real world case of allocation of marketing resources, I came within the 0.025% of optimality. I know I was this close due to some bounding from some nonlinear duality -- easy math. So, in your > reaching optimality more frequently that optimized heuristic algos. heuristics may not be, in nearly all of reality probably are not, reaching "optimality" in the sense of (2). The hype around the TSP has been to claim that the TSP is really difficult. Soooo, given some project that is to cost $100 million, an optimal solution might save $15 million, and some software based on what has long been known (e.g., from G. Nemhauser) can save all but $1500 is not of interest. Bummer. Wasted nearly all of $15 million. For this, see the cartoon early in Garey and Johnson where they confess they can't solve the problem (optimal network design at Bell Labs) but neither can a long line of other people. WRONG. SCAM. The stockholders of AT&T didn't care about the last $1500 and would be thoroughly pleased by the $15 million without the $1500. Still that book wanted to say the network design problem could not yet be solved -- that statement was true only in the sense of exact optimality in polynomial time on worst case problems, a goal of essentially no interest to the stockholders of AT&T. For neural networks (NN), I don't expect (A) much progress in any sense over what has been known (e.g., Nemhauser et al.) for decades. And, (B) the progress NNs might make promise to be in performance aspects other than getting to exact optimality. Yes, there are some reasons for taking the TSP and the issue of P versus NP seriously, but optimality on real world optimization problems is not one of the main reasons. Here my goal is to get us back to reality and set aside some of the hype about how difficult the real world TSP is.
- axg11 5y ago
  I wrote a short post on retrieval transformers that you might find interesting [0]. It’s a twist on transformers that allows scaling “world knowledge” independently in a database-like manner. [0] - https://arsham.substack.com/p/retrieval-transformers-for-medicine?s=r https://arsham.substack.com/p/retrieval-transformers-for-med...
- adamsmith143 5y ago
  Isn't the benefit of NNs on some level that you can store finer grained and more abstract data than a standard DB?
  - macrolocal 5y ago
    Maybe. Transformers model associative memory in a way made precise by their connection to Hopfield networks. Individually, they're like look-up tables, but the queries can be ambiguous, even based on subtle higher-order patterns (which the network identifies on its own), and the returned values can be a mixture of stored information, weighted by statistically meaningful confidences.
- lolspace 5y ago
  > I have not given transformers enough attention... ( ͡° ͜ʖ ͡°)
  - fakethenews2022 5y ago
    Attention is all you need
CabSauce 5y ago
I thought they already had.
fithisux 5y ago
There are a lot of hype driven articles, less providing hand waving explanations and none about mathematics
dekhn 5y ago
I assume transformers will be replaced by something, just as transformers replaced other sequential models. That said, transformers have already earned a place in the annals of ML, if for no other reason than they were the critical to the first technology to solve protein structure prediction.
- PartiallyTyped 5y ago
  Transformers are really just fancy graph neural networks, the next step is diffusion networks.
mark_l_watson 5y ago
I think they will temporarily take over deep learning, but all of AI? No.
g42gregory 5y ago
I think Transformers are a (good) tool in AI toolbox and they are already being used a conjunction with many other recent tools such as Normalizing Flows, Diffusion Models, Energy-Based Models, etc…
graycat 5y ago
After a long "training set", I have learned that when I see a headline that ends with a question mark, don't bother to read the article; the answer to the question is nearly always "No". E.g., "Will Expert Systems AI Replace Nearly All Knowledge Workers?" No, they didn't. And I am fully confident they won't. So, due to the question mark, just stop reading. Then I apply this training to "Will Transformers Take over Artificial Intelligence?"
- version_five 5y ago
  https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headli...
zitterbewegung 5y ago
If you want to play with Transformers you can go here https://transformer.huggingface.co/ https://transformer.huggingface.co/ They have a really easy to use library in Python called Transformers. Below is an example of how to use it. >>> from transformers import pipeline # Allocate a pipeline for sentiment-analysis >>> classifier = pipeline('sentiment-analysis') >>> classifier('We are very happy to introduce pipeline to the transformers repository.') [{'label': 'POSITIVE', 'score': 0.9996980428695679}]
- PestoDiRucola 5y ago
  Huggingface is great! The only issue is the documentation which is rather lacking if you want to get more serious about writing custom models and solving more complex issues than what normally documented in the examples there.
omarhaneef 5y ago
For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML: https://twitter.com/karpathy/status/1468370605229547522 https://twitter.com/karpathy/status/1468370605229547522 And of course one of the early classic papers in the field, as a bonus: https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd05... (The paper is mentioned in the article)
- lucidrains 5y ago
  if one prefers video, Yannic Kilcher does an excellent explanation of the seminal paper https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro
  - amrrs 5y ago
    Glad to see you here lucidrains. Truly appreciate your recent open-source contributions and works like big sleep, deep daze. Everyone else check out this https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories
    - lucidrains 5y ago
      Thanks for the kind words, and credit goes to Ryan Murdock for Big Sleep and Deep Daze. I simply packaged it up to spread the usage
    - tomweingarten 5y ago
      +1, lucidrains is practically a one-person ML start-up
- algo_trader 5y ago
  Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs. [1]https://www.gwern.net/notes/Attention https://www.gwern.net/notes/Attention
tourist_on_road 5y ago
Transformers gained popularity due to the scalable nature of the architecture and how well it can be parallelized on existing GPU/XLA hardware. Modeling is always conditioned on the hardware available at hand. Transfomers lack inductive bias which make it generic building blocks unlike CNN/RNN like models and by injecting inductive bias like positional encoding, it can be well translated to various domains.
- Der_Einzige 5y ago
  All of this is just another externalization of the bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- saeranv 5y ago
  Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available? I'm not that familiar with "injecting inductive bias" via positional encodings, but it sounds really interesting. My crude understanding is that the positional encodings were used in the original Transformer architecture to encode the ordering of words for NLP. Are they more flexible then that? For example, can they be used to replicate the image-related inductive bias of CNNs and match CNN performance on small datasets (1000 - 10,000)? If not, then to me, it seems like only industries where it's possible to get access to a large amount of representative data (i.e. greater than a million?) benefit from transformers. In industries where there are bottlenecks to data generation, there's a clear benefit in leveraging the inductive bias in other architectures, such as the various ways CNNs have biases towards image recognition. I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?
  - tourist_on_road 5y ago
    Good point. The fact there is no inductive bias inherent to transformers makes it difficult to train a decent model on small datasets from scratch. However, there are recent research directions that try to address this problem [1]. Also baking in some sort of domain specific inductive bias into model architecture itself can address this problem as well [2]. [1]: Escaping the Big Data Paradigm with Compact Transformers: https://arxiv.org/abs/2104.05704 https://arxiv.org/abs/2104.05704 [2]: CvT: Introducing Convolutions to Vision Transformers: https://arxiv.org/abs/2103.15808 https://arxiv.org/abs/2103.15808
iamwil 5y ago
For those that want a high level overview of Transformers, we recently covered it in our podcast: https://www.youtube.com/watch?v=Kb0II5DuDE0 https://www.youtube.com/watch?v=Kb0II5DuDE0
goodmachine 5y ago
I really like Quanta: but this article was not so great.
deleted 5y ago
[deleted]
godelski 5y ago
There's pretty popular vision transformer tutorial on Medium. The authors were focused on making it work even with limited compute. https://medium.com/pytorch/training-compact-transformers-from-scratch-in-30-minutes-with-pytorch-ff5c21668ed5 https://medium.com/pytorch/training-compact-transformers-fro...
JPKab 5y ago
Has anyone seen successful use of transformers with tabular data, particularly high-cardinality categorical data sets? Curious.
- stevofolife 5y ago
  I am also very interested in this. I’m only aware of TabNet but I am not sure if it is SotA.
vpbusvltsmtk 5y ago
droffel 5y ago
Transformers: More than meets the eye
deleted 5y ago
[deleted]
dilawar 5y ago
Any comments on TinyML and transformers? Or they requires heavy duty CPU and GPUs?

190 points by theafh 5y ago

On Kaggle they are getting more usage.

I have not given transformers enough attention... but my impression is that this is still storing entities in the weights of the neural network instead of in a database where the can be operated on with CRUD. What are the knowledge discovery researchers doing with respect to transformers? And the SAT solver researchers? Here is an article on KDNuggets that explains transformers but doesn't answer my questions: https://www.kdnuggets.com/2021/06/essential-guide-transformers-key-modern-sota-ai.html https://www.kdnuggets.com/2021/06/essential-guide-transforme...

ctoth 5y ago
> The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information[0]. [0]: https://jalammar.github.io/illustrated-retrieval-transformer/ https://jalammar.github.io/illustrated-retrieval-transformer...
stingraycharles 5y ago
I think it’s relatively straightforward to serialize such a model into different representations, I completely understand that they keep the actual data inside pytorch state by default. Out of curiosity, what tools are researchers generally using to explore neural networks? I’m just an armchair ML enthusiast myself, but NN always appear very much like black boxes. What are the goals and methods for exploring neural network state nowadays?
cleancoder0 5y ago
First transformer models still dealt only with the training set. Eventually it was extended to work with an external data source that it queries. This is not a new thing, for example, image style transfer and some other image tasks that were attempted before the domination of NNs did the same thing (linear models would query the db for help and guided feature extraction). The greatest effect in transformers is the attention mechanism combined with self-supervised learning. Investigations in self-supervised learning tasks (article illustrates one word gap, but there are others) can result in superior models that are sometimes even easier to train. As for SAT, optimization, graph neural networks might end up being more effective (due to high structure of the inputs). I'm definitely awaiting for traveling salesman solver or similar, guided by NN, solving things faster and reaching optimality more frequently that optimized heuristic algos.
- graycat 5y ago
  > I'm definitely awaiting for traveling salesman solver or similar, guided by NN, solving things faster and reaching optimality more frequently that optimized heuristic algos. Just in case we are not being clear, let's be clear. Bluntly in nearly every practical sense, the traveling salesman problem (TSP) is NOT very difficult. Instead we have had good approaches for decades. I got into the TSP writing software to schedule the fleet for FedEx. A famous, highly accomplished mathematician asked me what I was doing at FedEx, and as soon as I mentioned scheduling the fleet he waved his hand and concluded I was only wasting time, that the TSP was too hard. He was wrong, badly wrong. Once I was talking with some people in a startup to design the backbone of the Internet. They were convinced that the TSP was really difficult. In one word, WRONG. Big mistake. Expensive mistake. Hype over reality. I mentioned that my most recent encounter with combinatorial optimization was solving a problem with 600,000 0-1 variables and 40,000 constraints. They immediately, about 15 of them, concluded I was lying. I was telling the full, exact truth. So, what is difficult about the TSP? Okay, we would like an algorithm for some software that would solve TSP problems (1) to exact optimality, (2) in worst cases, (3) in time that grows no faster than some polynomial in the size of the input data to the problem. So, for (1) being provably within 0.025% of exact optimality is not enough. And for (2) exact optimality in polynomial time for 99 44/100% of real problems is not enough. In the problem I attacked with 600,000 0-1 variables and 40,000 constraints, a real world case of allocation of marketing resources, I came within the 0.025% of optimality. I know I was this close due to some bounding from some nonlinear duality -- easy math. So, in your > reaching optimality more frequently that optimized heuristic algos. heuristics may not be, in nearly all of reality probably are not, reaching "optimality" in the sense of (2). The hype around the TSP has been to claim that the TSP is really difficult. Soooo, given some project that is to cost $100 million, an optimal solution might save $15 million, and some software based on what has long been known (e.g., from G. Nemhauser) can save all but $1500 is not of interest. Bummer. Wasted nearly all of $15 million. For this, see the cartoon early in Garey and Johnson where they confess they can't solve the problem (optimal network design at Bell Labs) but neither can a long line of other people. WRONG. SCAM. The stockholders of AT&T didn't care about the last $1500 and would be thoroughly pleased by the $15 million without the $1500. Still that book wanted to say the network design problem could not yet be solved -- that statement was true only in the sense of exact optimality in polynomial time on worst case problems, a goal of essentially no interest to the stockholders of AT&T. For neural networks (NN), I don't expect (A) much progress in any sense over what has been known (e.g., Nemhauser et al.) for decades. And, (B) the progress NNs might make promise to be in performance aspects other than getting to exact optimality. Yes, there are some reasons for taking the TSP and the issue of P versus NP seriously, but optimality on real world optimization problems is not one of the main reasons. Here my goal is to get us back to reality and set aside some of the hype about how difficult the real world TSP is.
axg11 5y ago
I wrote a short post on retrieval transformers that you might find interesting [0]. It’s a twist on transformers that allows scaling “world knowledge” independently in a database-like manner. [0] - https://arsham.substack.com/p/retrieval-transformers-for-medicine?s=r https://arsham.substack.com/p/retrieval-transformers-for-med...
adamsmith143 5y ago
Isn't the benefit of NNs on some level that you can store finer grained and more abstract data than a standard DB?
- macrolocal 5y ago
  Maybe. Transformers model associative memory in a way made precise by their connection to Hopfield networks. Individually, they're like look-up tables, but the queries can be ambiguous, even based on subtle higher-order patterns (which the network identifies on its own), and the returned values can be a mixture of stored information, weighted by statistically meaningful confidences.
lolspace 5y ago
> I have not given transformers enough attention... ( ͡° ͜ʖ ͡°)
- fakethenews2022 5y ago
  Attention is all you need

CabSauce 5y ago

I thought they already had.

fithisux 5y ago

There are a lot of hype driven articles, less providing hand waving explanations and none about mathematics

dekhn 5y ago

I assume transformers will be replaced by something, just as transformers replaced other sequential models. That said, transformers have already earned a place in the annals of ML, if for no other reason than they were the critical to the first technology to solve protein structure prediction.

PartiallyTyped 5y ago
Transformers are really just fancy graph neural networks, the next step is diffusion networks.

mark_l_watson 5y ago

I think they will temporarily take over deep learning, but all of AI? No.

g42gregory 5y ago

I think Transformers are a (good) tool in AI toolbox and they are already being used a conjunction with many other recent tools such as Normalizing Flows, Diffusion Models, Energy-Based Models, etc…

graycat 5y ago

After a long "training set", I have learned that when I see a headline that ends with a question mark, don't bother to read the article; the answer to the question is nearly always "No". E.g., "Will Expert Systems AI Replace Nearly All Knowledge Workers?" No, they didn't. And I am fully confident they won't. So, due to the question mark, just stop reading. Then I apply this training to "Will Transformers Take over Artificial Intelligence?"

version_five 5y ago
https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headli...

zitterbewegung 5y ago

If you want to play with Transformers you can go here https://transformer.huggingface.co/ https://transformer.huggingface.co/ They have a really easy to use library in Python called Transformers. Below is an example of how to use it. >>> from transformers import pipeline # Allocate a pipeline for sentiment-analysis >>> classifier = pipeline('sentiment-analysis') >>> classifier('We are very happy to introduce pipeline to the transformers repository.') [{'label': 'POSITIVE', 'score': 0.9996980428695679}]

PestoDiRucola 5y ago
Huggingface is great! The only issue is the documentation which is rather lacking if you want to get more serious about writing custom models and solving more complex issues than what normally documented in the examples there.

omarhaneef 5y ago

For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML: https://twitter.com/karpathy/status/1468370605229547522 https://twitter.com/karpathy/status/1468370605229547522 And of course one of the early classic papers in the field, as a bonus: https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd05... (The paper is mentioned in the article)

lucidrains 5y ago
if one prefers video, Yannic Kilcher does an excellent explanation of the seminal paper https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro
- amrrs 5y ago
  Glad to see you here lucidrains. Truly appreciate your recent open-source contributions and works like big sleep, deep daze. Everyone else check out this https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories
  - lucidrains 5y ago
    Thanks for the kind words, and credit goes to Ryan Murdock for Big Sleep and Deep Daze. I simply packaged it up to spread the usage
  - tomweingarten 5y ago
    +1, lucidrains is practically a one-person ML start-up
algo_trader 5y ago
Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs. [1]https://www.gwern.net/notes/Attention https://www.gwern.net/notes/Attention

tourist_on_road 5y ago

Transformers gained popularity due to the scalable nature of the architecture and how well it can be parallelized on existing GPU/XLA hardware. Modeling is always conditioned on the hardware available at hand. Transfomers lack inductive bias which make it generic building blocks unlike CNN/RNN like models and by injecting inductive bias like positional encoding, it can be well translated to various domains.

Der_Einzige 5y ago
All of this is just another externalization of the bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
saeranv 5y ago
Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available? I'm not that familiar with "injecting inductive bias" via positional encodings, but it sounds really interesting. My crude understanding is that the positional encodings were used in the original Transformer architecture to encode the ordering of words for NLP. Are they more flexible then that? For example, can they be used to replicate the image-related inductive bias of CNNs and match CNN performance on small datasets (1000 - 10,000)? If not, then to me, it seems like only industries where it's possible to get access to a large amount of representative data (i.e. greater than a million?) benefit from transformers. In industries where there are bottlenecks to data generation, there's a clear benefit in leveraging the inductive bias in other architectures, such as the various ways CNNs have biases towards image recognition. I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?
- tourist_on_road 5y ago
  Good point. The fact there is no inductive bias inherent to transformers makes it difficult to train a decent model on small datasets from scratch. However, there are recent research directions that try to address this problem [1]. Also baking in some sort of domain specific inductive bias into model architecture itself can address this problem as well [2]. [1]: Escaping the Big Data Paradigm with Compact Transformers: https://arxiv.org/abs/2104.05704 https://arxiv.org/abs/2104.05704 [2]: CvT: Introducing Convolutions to Vision Transformers: https://arxiv.org/abs/2103.15808 https://arxiv.org/abs/2103.15808

iamwil 5y ago

For those that want a high level overview of Transformers, we recently covered it in our podcast: https://www.youtube.com/watch?v=Kb0II5DuDE0 https://www.youtube.com/watch?v=Kb0II5DuDE0

goodmachine 5y ago

I really like Quanta: but this article was not so great.

deleted 5y ago

[deleted]

godelski 5y ago

There's pretty popular vision transformer tutorial on Medium. The authors were focused on making it work even with limited compute. https://medium.com/pytorch/training-compact-transformers-from-scratch-in-30-minutes-with-pytorch-ff5c21668ed5 https://medium.com/pytorch/training-compact-transformers-fro...

JPKab 5y ago

Has anyone seen successful use of transformers with tabular data, particularly high-cardinality categorical data sets? Curious.

stevofolife 5y ago
I am also very interested in this. I’m only aware of TabNet but I am not sure if it is SotA.

vpbusvltsmtk 5y ago

droffel 5y ago

Transformers: More than meets the eye

deleted 5y ago

[deleted]

dilawar 5y ago

Any comments on TinyML and transformers? Or they requires heavy duty CPU and GPUs?