> I’m sure I’m not alone, in that each time you pull the curtain off a piece of ‘magic’, you have the same thought: Oooooh yeah. I mean, well duh.. how else wou

> I’m sure I’m not alone, in that each time you pull the curtain off a piece of ‘magic’, you have the same thought: Oooooh yeah. I mean, well duh.. how else would you do that? I can't believe I couldn't see it. This is the great paradox of good ideas. The best ideas are obvious, but only in retrospect. You would be very unlikely to encounter the idea by randomly fumbling around, yet it feels so simple and obvious (and often even easy to implement!) after someone else points it out and you mull it over. Usually it requires looking in a different dimension than the one you were focused on rather than looking further along familiar dimensions, which is our default behavior. In this way there’s a parallel to modern machine learning, where backprop on gigantic models allows us to find very short paths in very high dimensions, rather than finding very long circuitous paths in low dimensions. It turns out this also solves the overfitting problem, in the same way that “retrospectively obvious” is a good filter for ideas.