I don't feel like the bitter lesson becomes applicable until you've demonstrated some initial degree of success with your technique on a single machine/GPU/CPU/

I don't feel like the bitter lesson becomes applicable until you've demonstrated some initial degree of success with your technique on a single machine/GPU/CPU/thread. If you cannot make it work in practical environments, it's going to be an uphill battle the entire way. This is why I've moved toward CPU-only techniques in my experimentation. Being able to execute arbitrary UTMs with high performance provides a significantly richer computational landscape to work with than matrix multiplication. I am perfectly happy with something taking longer as if it's on a linear basis. I.e., adding another CPU provides ~2x search speed. I am NOT happy with ideas like taking a 100x hit on token generation rate because half my parameters are paged out to disk at any given moment due to not having a step-size amount of vram. The rigidity of the GPU solution stack makes exploration of clever techniques largely a slap on the wrist experience. Anything with rapid control flow changes is verboten for a fancy pants cluster. The latency domain of L1 cache in the CPU is impossible to compete with if you need to serialize all of your events. I strongly believe that control flow is where the magic happens. This is where you can cut through 100 million parameters of linear math bullshit and solve the problem with a lookup table and 2 interpreter cycles. You get about half a billion of these cycles to work with per second per thread, so there is a lot of room to play with ideas.