4 ms·

Brief abstract for the lay person (like me): 1. Hopfield Networks are also known as "associative memory networks", a neural network model developed decades ago

Brief abstract for the lay person (like me): 1. Hopfield Networks are also known as "associative memory networks", a neural network model developed decades ago by a guy named Hopfield. 2. It's useful to plug these in somehow as layers in Deep Neural Networks today (particularly, in PyTorch). I hate non-informative titles!

virgil_disgr4ce 6y ago
Not to mention grammatically incorrect ones :/
- skrebbel 6y ago
  I think it's correct, in the same way that you can say "Rolling Stones is a great band". It's about the tech called "Hopfield Networks", not about any particular number of networks that are all you need.
  - caddemon 6y ago
    Not doubting what is officially grammatically correct, but that still sounds really weird to me. Like with sports teams I would only ever say "The Patriots are a good team" or "New England is a good team". Not "The Patriots is a good team". In any event, the authors definitely chose that title as a callback to the well known paper "Attention is all you need", which introduced Transformers. So that probably influenced their decision to use "is" instead of "are".
    - robotresearcher 6y ago
      Consider ‘My team is a good team’ vs. ‘My team are a good team’. I bet ‘is’ sounds better to you in this context, though ‘my team’ and ‘The Patriots’ are similar noun phrases that could refer to exactly the same thing. The difference is that Patriots is a plural. Replace it with Manchester United and ‘is’ sounds good again.
      - caddemon 6y ago
        Yeah it's definitely caused by the team name being plural, or at least sounding plural - I've never heard anyone say "The Red Sox is good" either. Regardless of what is technically grammatically correct I think real life usage has pretty much settled on that convention, at least in the US.
        drdeca 6y ago
        Something odd : While "The Red Sox are John's favorite team." seems more natural then "The Red Sox is John's favorite team.", phrasing it in the opposite order, "John's favorite team is the Red Sox." seems more natural then "John's favorite team are the Red Sox." . This seems like a strange discrepancy. Why is this the case? Maybe it is because "favorite team" is clearly singular, and is closer in the sentence to the "is"/"are" then the plural indicating sound in "Red Sox". Or maybe it is just whichever comes first which determines how the "to be" is conjugated? Hm, but what if instead of connecting a noun phrase (determiner phrase?) like "The Red Sox" to another noun phrase (determiner phrase) "John's favorite team", we instead connect it to an adjective? "The Red Sox are singular.", "The Red Sox is singular.", "Singular is The Red Sox." "Singular are The Red Sox." . Well, the "[Adjective] is [noun]" is kind of an unusual thing to say unless one is trying to sound like one is quoting poetry or yoda or something, but to the degree that either of them sound ok, I think "Singular are The Red Sox." sounds better than "Singular is The Red Sox." . Though, in this case, there doesn't seem to be anything grammatically suggested by the adjective that the thing be in the singular case (maybe I shouldn't have used "singular" as the adjective..) . Hm, what if instead of "John's favorite team [is/are] the Red Sox." , we instead look at "John's favorite [is/are] the Red Sox." ? In this case, it seems, less clear which is more natural? They seem about the same to me (but that might just be me, idk.) . Anyway : Weird!
        DavidSJ 6y ago
        This is language dependent. In English: It's only five minutes to the bus stop. In German: Es sind nur fünf Minuten zur Bushaltestelle. (It are only five minutes to the bus stop.) I think it's a question of whether the verb is supposed to agree with the subject or the complement.
isoprophlex 6y ago
Also... While cute, I found the examples of storing and retrieving images of The Simpsons characters not very informative about what goes on in that weight matrix that stores patterns. Edit: the linked pytorch implementation looks interesting, these layer types promise pretty incredible things https://github.com/ml-jku/hopfield-layers https://github.com/ml-jku/hopfield-layers
SneakyTornado29 6y ago
The title is a reference to the famous machine learning paper "Attention Is All You Need" which introduced the concept of transformers. Transformers have revolutionized how we process sequential data (i.e. natural language processing).
- bonoboTP 6y ago
  Which itself is a reference to the 1967 Beatles song All You Need is Love (which also includes the line "Love is all you need").
- bjourne 6y ago
  And recently, a paper titled Attention Is Not All You Need has made the rounds arguing that some of the claims made in the AIAYN paper may have been overstated. https://arxiv.org/abs/2103.03404 https://arxiv.org/abs/2103.03404
  - visarga 6y ago
    If you read the title, it only refers to the multi-head-attention part of BERT, excluding the feed forward and skip connections, hence calling it "Pure Attention". > Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.
- einpoklum 6y ago
  > famous machine learning paper "Attention Is All You Need" 1. It's a paper from 2017. Unless you follow academic ML research, you will not have heard of it. 2. That paper's title is also inscrutable unless you've gone and read at least the abstract.

virgil_disgr4ce 6y ago

Not to mention grammatically incorrect ones :/

skrebbel 6y ago
I think it's correct, in the same way that you can say "Rolling Stones is a great band". It's about the tech called "Hopfield Networks", not about any particular number of networks that are all you need.
- caddemon 6y ago
  Not doubting what is officially grammatically correct, but that still sounds really weird to me. Like with sports teams I would only ever say "The Patriots are a good team" or "New England is a good team". Not "The Patriots is a good team". In any event, the authors definitely chose that title as a callback to the well known paper "Attention is all you need", which introduced Transformers. So that probably influenced their decision to use "is" instead of "are".
  - robotresearcher 6y ago
    Consider ‘My team is a good team’ vs. ‘My team are a good team’. I bet ‘is’ sounds better to you in this context, though ‘my team’ and ‘The Patriots’ are similar noun phrases that could refer to exactly the same thing. The difference is that Patriots is a plural. Replace it with Manchester United and ‘is’ sounds good again.
    - caddemon 6y ago
      Yeah it's definitely caused by the team name being plural, or at least sounding plural - I've never heard anyone say "The Red Sox is good" either. Regardless of what is technically grammatically correct I think real life usage has pretty much settled on that convention, at least in the US.
      - drdeca 6y ago
        Something odd : While "The Red Sox are John's favorite team." seems more natural then "The Red Sox is John's favorite team.", phrasing it in the opposite order, "John's favorite team is the Red Sox." seems more natural then "John's favorite team are the Red Sox." . This seems like a strange discrepancy. Why is this the case? Maybe it is because "favorite team" is clearly singular, and is closer in the sentence to the "is"/"are" then the plural indicating sound in "Red Sox". Or maybe it is just whichever comes first which determines how the "to be" is conjugated? Hm, but what if instead of connecting a noun phrase (determiner phrase?) like "The Red Sox" to another noun phrase (determiner phrase) "John's favorite team", we instead connect it to an adjective? "The Red Sox are singular.", "The Red Sox is singular.", "Singular is The Red Sox." "Singular are The Red Sox." . Well, the "[Adjective] is [noun]" is kind of an unusual thing to say unless one is trying to sound like one is quoting poetry or yoda or something, but to the degree that either of them sound ok, I think "Singular are The Red Sox." sounds better than "Singular is The Red Sox." . Though, in this case, there doesn't seem to be anything grammatically suggested by the adjective that the thing be in the singular case (maybe I shouldn't have used "singular" as the adjective..) . Hm, what if instead of "John's favorite team [is/are] the Red Sox." , we instead look at "John's favorite [is/are] the Red Sox." ? In this case, it seems, less clear which is more natural? They seem about the same to me (but that might just be me, idk.) . Anyway : Weird!
        DavidSJ 6y ago
        This is language dependent. In English: It's only five minutes to the bus stop. In German: Es sind nur fünf Minuten zur Bushaltestelle. (It are only five minutes to the bus stop.) I think it's a question of whether the verb is supposed to agree with the subject or the complement.

isoprophlex 6y ago

Also... While cute, I found the examples of storing and retrieving images of The Simpsons characters not very informative about what goes on in that weight matrix that stores patterns. Edit: the linked pytorch implementation looks interesting, these layer types promise pretty incredible things https://github.com/ml-jku/hopfield-layers https://github.com/ml-jku/hopfield-layers

SneakyTornado29 6y ago

The title is a reference to the famous machine learning paper "Attention Is All You Need" which introduced the concept of transformers. Transformers have revolutionized how we process sequential data (i.e. natural language processing).

bonoboTP 6y ago
Which itself is a reference to the 1967 Beatles song All You Need is Love (which also includes the line "Love is all you need").
bjourne 6y ago
And recently, a paper titled Attention Is Not All You Need has made the rounds arguing that some of the claims made in the AIAYN paper may have been overstated. https://arxiv.org/abs/2103.03404 https://arxiv.org/abs/2103.03404
- visarga 6y ago
  If you read the title, it only refers to the multi-head-attention part of BERT, excluding the feed forward and skip connections, hence calling it "Pure Attention". > Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.
einpoklum 6y ago
> famous machine learning paper "Attention Is All You Need" 1. It's a paper from 2017. Unless you follow academic ML research, you will not have heard of it. 2. That paper's title is also inscrutable unless you've gone and read at least the abstract.