3 ms·

It is important to note that the search space is just the training data. That space does not span enough dimensions to actually solve the problems in question.

It is important to note that the search space is just the training data. That space does not span enough dimensions to actually solve the problems in question. For example, we act in the world so as to set the state of possible causal variables -- and thereby can distinguis causes. Above, graph algorithms operate on a discrete graph space and hence on the graph. GPT here isn't doing that. Seeing NNs as distance-based searches in a compressed 'training space' demystifies what solutions can be found. Here the text space defined by the training set does not permit a graph traversal, so eg., the NN 'uses' edges that don't exist We are not searches across such spaces, which are radically impoverished and not parametrised by time or action

YorkshireSeason 4y ago
It is not the case that "the search space is just the training data", the search space in NNs are some variants of real multivariate functions. The training data can be all manner of things, in LLMs, they are text in natural language. Graphs or natural language text are mapped from a discrete space to a continuous metric space. It is true that LLMs cannot currently solve complex mathematics problems reliably. Whether that's an intrinsic shortcoming of transformers, or not is an interesting open question, that is being investigated as we speak. A lot of well-known DL research teams bet on transformers beating humans in mathematics in the next few years, see e.g. https://imo-grand-challenge.github.io/