4 ms·

While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical appr

by nine_k 2y ago

While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. I would consider a different approach, when the training phase watches games (or video recordings) and refines the formulas that describe its physics, the geometry of the area, the optics, etc. The result would be a "map" that is "playable" without much if any inference involved, and with no time limitation dictated by the size of the context to keep. Very certainly, video game map generation by AI is a thing, and creating models of motion by watching and then fitting reasonably simple functions (fewer than millions of parameters) is also known. I cannot be the first person to think about such possibilities, so I wonder what does the current SOTA look like there.

furyofantares 2y ago
> Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. I think this is precisely why they're doing it. Video games are where the data is, because the analytical solutions can generate it. They aren't trying to make a video game. They're trying to make an android.
halflings 2y ago
> I cannot be the first person to think about such possibilities Differentiable Rendering [1] is the closest thing to what you are describing. And yes, people have been working on this for the same reason you outline, it is more data/compute efficient and hence should generalize better. [1] https://blog.qarnot.com/article/an-overview-of-differentiable-rendering https://blog.qarnot.com/article/an-overview-of-differentiabl... But also: > While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. A bit of the same debate as people calling LLMs a "blurry JPEG of the web" and hence useless. Yes this is a statistical approximation to an analytical problem... but that's a very reductive framing to what is going on. To find the symbolic/analytical solution here would require to constrain the problem greatly: not all things on the screen have a differentiable representation, for example complex simulations might involve some kind of custom internal loop/simulation. You waste compute to get a solution that can just be trained on billions of unlabeled (synthetic) examples, and then generalize to previously unseen prompts/environments.

by nine_k 2y ago

furyofantares 2y ago

> Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. I think this is precisely why they're doing it. Video games are where the data is, because the analytical solutions can generate it. They aren't trying to make a video game. They're trying to make an android.

halflings 2y ago

> I cannot be the first person to think about such possibilities Differentiable Rendering [1] is the closest thing to what you are describing. And yes, people have been working on this for the same reason you outline, it is more data/compute efficient and hence should generalize better. [1] https://blog.qarnot.com/article/an-overview-of-differentiable-rendering https://blog.qarnot.com/article/an-overview-of-differentiabl... But also: > While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. A bit of the same debate as people calling LLMs a "blurry JPEG of the web" and hence useless. Yes this is a statistical approximation to an analytical problem... but that's a very reductive framing to what is going on. To find the symbolic/analytical solution here would require to constrain the problem greatly: not all things on the screen have a differentiable representation, for example complex simulations might involve some kind of custom internal loop/simulation. You waste compute to get a solution that can just be trained on billions of unlabeled (synthetic) examples, and then generalize to previously unseen prompts/environments.