You're right about the classical theory part. But the infinite dimensional is just one representation that is mathematically convenient - for example, you could

You're right about the classical theory part. But the infinite dimensional is just one representation that is mathematically convenient - for example, you could consider a 2D image of 10x10 as a vector from a 100 dim space, each scalar being a pixel, or as one from a 2^100 space, with a vector of all zeros except at it's "index" in the list of all 10x10 images (this repr also allows you to express superposition within it by turning on multiple indices). Or as an element of a much lower dimensional subspace embedded in those spaces, like ML uses. In general infinite dimensional should just be taken to mean that using the inductive bias endowed by a specific representation isn't necessary for whatever we want to use the theory for. It's also how you (imo, crudely) represent functions as first-class objects in math.