Very nice. Well-written. Feels "natural." Besides helping with interpretability, my immediate thought is that maybe we could pretrain models faster by adding r

Very nice. Well-written. Feels "natural." Besides helping with interpretability, my immediate thought is that maybe we could pretrain models faster by adding regularization terms in the objective function that induce representations of distinct categories to be in subspaces that are orthogonal to each other, and representations of subcategories to be in orthogonal subspaces that can form polytopes. The data necessary for doing so is readily available: Wordnet synsets. Induce representations of synsets to be orthogonal to each other and representations of hierarchically related synsets to be arranged in polytopes. There's already some evidence that we can leverage Wordnet synsets to pretrain some models faster. Take a look at https://news.ycombinator.com/item?id=40160728 https://news.ycombinator.com/item?id=40160728 for example. Thank you for sharing this on HN.