3 ms·
> The team is pretty deep in investigating the idea of superposition, where individual neurons encode for multiple concepts. Wonder if it's a matter of perspec
by TeMPOraL 3y ago
> The team is pretty deep in investigating the idea of superposition, where individual neurons encode for multiple concepts.
Wonder if it's a matter of perspective - that is, of transform. Consider an image. Most real-world images have pixels with high locality - distant pixels are less correlated than immediate neighbours.
Now take an FFT of that. You get an equivalent 2D image containing the same information, but suddenly each pixel contains information about every pixel of the original image! You can do some interesting things there, like erasing the centre of the picture (higher frequencies), which will give you blurred original image when you run FFT on the frequency-image to get proper pixels again.
- tel 3y agoI think that’s basically correct, the FFT representation is a better feature representation.