3 ms·
Some ART concepts 1) Normalize input (batch norm, 2015) 2) Competitive dynamics / lateral inhibition (softmax in attention layers, 2017) 3) Cluster best matc
by intalentive 3y ago
Some ART concepts
1) Normalize input (batch norm, 2015)
2) Competitive dynamics / lateral inhibition (softmax in attention layers, 2017)
3) Cluster best matching activation vectors (top-k keys, 2023)