This tool is really lovely, great work! I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to imp

This tool is really lovely, great work! I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to improve interpretability. PS: I share your curiosity with respect to things like deep dream. My brief summary of this paper is that you can use GPT4 to summarize what's similar about a set of highlighted words in context which is clever but doesn't fundamentally inform much that we didn't already know about how these models work. I wonder if there's some diffusion based approach that could be used to diffuse from noise in the residual stream towards a maximized activation at a particular point. [1] https://transformer-circuits.pub/2022/solu/index.html https://transformer-circuits.pub/2022/solu/index.html