The use of the word “head” in machine learning does not seem consistent, in case anyone else is confused by that. There’s multihead attention and multiple outp

The use of the word “head” in machine learning does not seem consistent, in case anyone else is confused by that. There’s multihead attention and multiple output heads as a concept in the paper. Multihead attention is about focusing on different areas of the input in transformer architectures, and the biological analogy here is head as a central processing unit. An output head refers to the final layer of a neural network, of which you could have more than one producing different outputs based on the same previous layers. This is also a loose biological analogy, but instead of head as cpu, think more along the lines of head being on one end of the body. In neither case is there any analogy to a tape head that reads data.