It's a nice demo. Unfortunately the article mixes up things in the explanation for causal masking, because it seems the author conflates aspects from training a

It's a nice demo. Unfortunately the article mixes up things in the explanation for causal masking, because it seems the author conflates aspects from training and inference. First, causal masking exists to prevent the model from "peeking" at future tokens during training, and second (at least for GPT-like architectures) for enforcing the autoregressive aspect during inference. During inference we only use the last token anyways, so it will attend the entire input sequence. So this token is definitely not decided only from the last token's embedding.