3 ms·

I don't understand the claims well enough to evaluate them. But the way this document is written, and where it choose to put its attention (no pun intended) str

by abeppu 4y ago

I don't understand the claims well enough to evaluate them. But the way this document is written, and where it choose to put its attention (no pun intended) strike me as odd. The sections on the transformer and its components seem like they should be backed with a lot more specifics tying back to the (Combinatorial) Hopf algebra described in the preceding sections. The Hopf algebra section defines a bunch of notation and structure, and introduces the label "Hopf Coherence" for some invariant, which is expressed in terms of that pile of notation. The Combinatorial Hopf Algebra section introduces a "shuffle product". But we never actually get a clear explanation of how this stuff lines up with parts of the transformer / attention mechanism. There's an _assertion_ of Hopf coherence between two paths, and in 6.2 there's a description of the QK and OV circuits as having a co-product product relationship, which isn't especially clear but is at least a direction. What is the antipode in the transformer context? If Hopf coherence is m(id x S)\Delta = \epsilon * u, and OV is (or is like?) the product m, and QK is the coproduct \Delta ... what are the unit u, counit \epsilon, and antipode S? It looks like the author never spells it out, and certainly never proves the assertion. Instead, there are these hard-to-comprehend claims in prose, interspersed with math defining the most trivial things. In 6.2.1, it reminds us of what it means for a row to sum to 1. Then it quotes a definition of co-injection. In 6.2.2 they remind us what a symmetric matrix is. The shuffle product defined in 5.1 doesn't get used, except to say that it will be part of future work. I'm not saying there isn't any insight here, but I think because the author shows in sections 3-5 that they expect many readers will not have this background, they should do a clearer job drawing the relevant connections explicitly, and backing up claims.

adamnemecek 4y ago
You are right, the paper can and will be improved. I think that I assume too much that people have read the Pang2014 paper. Hopf coherence is not new, it's a name for an old concept that did not have a name. Unit and counit together make the residual stream. The antipode is a part of the OV circuit (see section 6.2.2). You are right re: basic definitions, I wanted the definitions in there and briefly considered a separate section that provides an overview (as is customary) but there were not enough of them to warrant that but they do seem somewhat out of place sometimes.

by abeppu 4y ago

adamnemecek 4y ago

You are right, the paper can and will be improved. I think that I assume too much that people have read the Pang2014 paper. Hopf coherence is not new, it's a name for an old concept that did not have a name. Unit and counit together make the residual stream. The antipode is a part of the OV circuit (see section 6.2.2). You are right re: basic definitions, I wanted the definitions in there and briefly considered a separate section that provides an overview (as is customary) but there were not enough of them to warrant that but they do seem somewhat out of place sometimes.