I don’t mean to say that is _all_ that is separating us from the transformer architecture. I do think we’re more efficient than gradient descent, and sgd+attent

I don’t mean to say that is _all_ that is separating us from the transformer architecture. I do think we’re more efficient than gradient descent, and sgd+attention is a function of how modern parallel processing works more than it is a function of any deep insight into the brain’s computation or learning methods. I just think we’re really making a silly comparison when we compare a stream of text to the multi sensory experiences humans have directly, reducing what we are doing to just reading words off a screen or page.