Nice article related to the last point (nearly orthogonal vectors): https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub

Nice article related to the last point (nearly orthogonal vectors): https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html