Thanks, this is great. Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the

Thanks, this is great. Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the form "for function class X there exists an architecture A of complexity O(N) such that blah". And then this is compared favorably with some other function class whose dimension is O(N). But there's something tricky about this: you leave the architecture unspecified. You are comparing a single space of functions with an enormous number of spaces, one for each architecture with the specified complexity, and then saying "well if I pick the right architecture I win". Doesn't seem like a fair comparison.