I think there's also a challenging line to draw about where a defining a search space stops and where encoding knowledge begins. When you define attention modu

I think there's also a challenging line to draw about where a defining a search space stops and where encoding knowledge begins. When you define attention modules for an LLM, encode a search heuristic into A*, or define a feature space for a random forest, you are encoding domain knowledge, i.e. adding bias in exchange for faster learning relative to an even more general model. At any given time, the best performing computation heavy techniques have embedded more structural knowledge than zero, while much less than some experts believed necessary.