The negativity here reminds me that there is a spectrum among engineers: between high-level and low-level thinking, between goals and means, between abstraction

The negativity here reminds me that there is a spectrum among engineers: between high-level and low-level thinking, between goals and means, between abstractions and operations, between simplification and optimization. Most sit somewhere in the middle, but if you put two people on opposite sides into a discussion, you’re going to see fireworks. “My implementation runs 10x as fast” verses “My implementation is maintainable.” Both types of thinking are necessary, but you need to know where the other person is coming from. This essay is a thought-piece, exploring possibilities. It isn’t a specification for a new form of computation. It’s an imagination of new possibilities. And as such, it has a lot of merit. Take a look at “Weight Agnostic Neural Networks” by Adam Gaier and David Ha at Google Brain: https://weightagnostic.github.io https://weightagnostic.github.io. Their research shows that very small RL neural networks can be evolved with shared weights to solve control tasks. My immediate thought after reading the paper was that an FPGA would be well suited for this kind of model — certainly at inference, but possibly at training time considering that the model is trained without backprop. The limit on training becomes the latency in updating the FPGA during the training loop. Moreover, you should be able to run many versions of a model on a single FPGA by dividing it up into batches. If there’s one thing we know about DL, it is that computational power always wins (The Bitter Lesson - http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html). You can tackle that in two ways: increasing compute or increasing compute efficiency. The Transformer started out as the latter: replacing sequential RNNs with parallel Attention. It seems likely that the next big breakthrough will also find a way to get 10x computation out of current hardware. Maybe the idea of non-differentiable highly-customized networks will be that breakthrough. Maybe the essay is right that we’ll find the breakthrough with a new paradigm in hardware itself, Maybe not. It’s worth thinking about, if only because it is so fascinating.