2 ms·
Specifically about attention: >> The original transformer experiment tested the hypothesis: "if we organize layers of attention mechanisms in a certain way and
by YeGoblynQueenne 6y ago
Specifically about attention:
>> The original transformer experiment tested the hypothesis: "if we organize
layers of attention mechanisms in a certain way and feed them tons of text
they will be able to process that text effectively enough to build very rich
and consistent language model".
I read the paper when it came out and I checked it again now to confirm:
there's no such hypothesis in there. In fact "Attention is all you need" is a
typical example of a post-hoc paper that describes what a research team did
that worked and how well it worked. "We tweaked these knobs and out came
STUFF!". Typically for deep learning papers it lacks a theory section, the
space of which is instead taken by an "Architecture" section wich well,
describes the architecture. There are no theorems, or proofs. There is nothing
that connects some kind of theoretical claim to the experiments. The main
claim of the paper is "we build this system and it has better performance than
previous systems". Like I say in my earlier comment, that's not an
interesting scientific claim.
I'm sorry but personally I find that kind of work irritating. "We tried some
stuff and got some results". Woo-hoo. But, why did you try that stuff and why
did you get those results? Did you try to get the same results without that
stuff? Did you try to get some other results with the same stuff? Can you
explain what is going on in your system and why it does that when I twist
this knob? If I twist that knob, can you tell me what it will do without
having to run it first to find out? Typically, 99% of the time, the answer to
all this is "no" (i.e. no ablation experiments, no theoretical explanations,
etc. etc, no nothing). It's like I say above, just throwing stuff at the wall
to see what sticks. And then writing a paper to describe it.
Oh, and calling stuff suggestive names like "attention". If I call it
"boredom", am I more or less justified than the authors?