So a transformer is just a "graph" network. You have a bunch of nodes in the graph, and depending on the kind of transformer, the nodes form different topologie

So a transformer is just a "graph" network. You have a bunch of nodes in the graph, and depending on the kind of transformer, the nodes form different topologies. Now, here is the catch, with graph neural networks, we may reorder the nodes, and as long as the edges are kept okay, we are fine. This is because graph networks are permutation invariant, but vision tasks are not. Nodes do not necessarily know their ordering with respect to the input. We use positional embeddings to inform the nodes where they are with respect to the input, because it matters. To see how, or why, take any image, flip it upside down, cut it into patches, and then flip the patches. The patches know what their content is, but not where. The edges in the graph merely point to the neighbours for further computation. Positional embeddings are just vectors that you index or compute at runtime. These vectors then are concatenated or added onto the vectors of the patches or tokens. We use positional embeddings for all sorts of tasks, such as text (position of token), vision tasks (position of patch), or even sound tasks (e.g. point in time).