4 ms·

I'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this.

I'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this. What is the best way to ship training code in a game? Do I embed Python and PyTorch or something? Do I code my own NN training algorithm? Do I use a library such as Tinygrad?

alar44 5y ago
Do you know how to program? I have no idea what you're asking.
- Buttons840 5y ago
  Yes, I know how to program and know a good amount about machine learning in Python. The problem is these machine learning frameworks are rather heavy to be shipping around, although I guess you could do it. I was wondering if there's any lighter weight machine learning libraries, which might not have all the bells and whistles, but would be better to ship and train models on consumers computers. Most of the time people ship pretrained models, but don't actually update that model on the end users computer.
  - upbeat_general 5y ago
    I think the real issue here is with the training speed. Presumably a GPU is required to run the game, but is it restricted to just NVIDIA GPUs/a subset of those? I don’t know of any such libraries offhand and my guess would be that’s because the size of the library generally matters less if there’s a requirement to have a powerful GPU.
  - CornCobs 5y ago
    Client side training with the usual python stack sounds like a configuration nightmare. Would it try to use the users GPU if it exists? Would it fallback to CPU? Does your user need a specific GPU card to play?
phillypham 5y ago
Typically, these libraries are unusable for this purpose because you don't want to ship a Python interpreter with your video game. I usually prefer to to rewrite my training step as a pure function, so the model weights are just inputs and the gradient updates are outputs. You need to serialize your computation graph in some way, so it can be run in C++ or some other low-level language. TensorFlow is known for doing this well since it was original design goal of the project. Some of the other frameworks that originally targeted researchers make this harder. Most mature frameworks have some way of doing this now, though, and projects like https://onnx.ai/ https://onnx.ai/ may solve this in general. It gets more complicated if your model has dynamic control flow, but you get the idea.
pizza 5y ago
You probably want a lightweight reinforcement learning library in C/C++ e.g. something from here https://github.com/search?l=C%2B%2B&q=reinforcement+learning&type=Repositories https://github.com/search?l=C%2B%2B&q=reinforcement+learning... You could also compile a neural net into a less python-tied format e.g. ONNX or torchscript. In general a siloed pytorch env would be massive, I'm assuming at least a gig or two.
girvo 5y ago
In theory, once could use Arraymancer to build the tools to do this, and not have to ship a Python interpreter. That said, I've not used it in anger. https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer
generalizations 5y ago
I suppose you could collect training data during gameplay and process it after the fact. Then you can use the heavy python framework, but it doesn't get in the way. Have the user run the training program on the accumulated data whenever they want to increase the level of customization.
andai 5y ago
If your game has enemies, you could give each unit its own brain. Behind the scenes you could have a "gene pool". When a unit dies it's removed from the pool. When it performs well (eg. damages player or wins the game) the unit can reproduce. Then you add the usual stuff (mutate parameters, cross-breeding fittest units). For training you could just make two AI teams play against each other (off the cuff, 10-100k games should do the trick). Once you release the game, you could sample the best AI from each player's machine. Free distributed neuroevolution cluster ;) I've wanted to do this for a while but I haven't made any proper games yet. Someday!
astrange 5y ago
I think video games already do this; video game AI existed before deep learning, it just used things like decision trees and MCMC.
sdenton4 5y ago
It's likely worth looking into federated learning. You could have a core model, which then trains locally on player inputs. The federated learning research space is focused on this sort of small model update on phones, so should have plenty of people who have thought hard about the problem of keeping binary sizes small and training compute manageable. https://arxiv.org/abs/1909.11875 https://arxiv.org/abs/1909.11875
accountofme 5y ago
Doesn't forza the racing game do this? Edit: spelling
NtGuy25 5y ago
There's actually alot of research in this topic. You should look into "Genetic Programming" on scholar.google.com and you'll find some good presentations. For example https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=genetic+programming+video+games&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=gen... and that references a paper where they do this with MUGEN (Ai v Ai fighting game) http://irep.ntu.ac.uk/id/eprint/30021/1/PubSub7423_8186_Martinez-Arellano.pdf http://irep.ntu.ac.uk/id/eprint/30021/1/PubSub7423_8186_Mart... Genetic programming sounds like a complicated term, but it's basically an easy way to take successful characteristics and breed them into something else.

alar44 5y ago

Do you know how to program? I have no idea what you're asking.

Buttons840 5y ago
Yes, I know how to program and know a good amount about machine learning in Python. The problem is these machine learning frameworks are rather heavy to be shipping around, although I guess you could do it. I was wondering if there's any lighter weight machine learning libraries, which might not have all the bells and whistles, but would be better to ship and train models on consumers computers. Most of the time people ship pretrained models, but don't actually update that model on the end users computer.
- upbeat_general 5y ago
  I think the real issue here is with the training speed. Presumably a GPU is required to run the game, but is it restricted to just NVIDIA GPUs/a subset of those? I don’t know of any such libraries offhand and my guess would be that’s because the size of the library generally matters less if there’s a requirement to have a powerful GPU.
- CornCobs 5y ago
  Client side training with the usual python stack sounds like a configuration nightmare. Would it try to use the users GPU if it exists? Would it fallback to CPU? Does your user need a specific GPU card to play?

phillypham 5y ago

Typically, these libraries are unusable for this purpose because you don't want to ship a Python interpreter with your video game. I usually prefer to to rewrite my training step as a pure function, so the model weights are just inputs and the gradient updates are outputs. You need to serialize your computation graph in some way, so it can be run in C++ or some other low-level language. TensorFlow is known for doing this well since it was original design goal of the project. Some of the other frameworks that originally targeted researchers make this harder. Most mature frameworks have some way of doing this now, though, and projects like https://onnx.ai/ https://onnx.ai/ may solve this in general. It gets more complicated if your model has dynamic control flow, but you get the idea.

pizza 5y ago

You probably want a lightweight reinforcement learning library in C/C++ e.g. something from here https://github.com/search?l=C%2B%2B&q=reinforcement+learning&type=Repositories https://github.com/search?l=C%2B%2B&q=reinforcement+learning... You could also compile a neural net into a less python-tied format e.g. ONNX or torchscript. In general a siloed pytorch env would be massive, I'm assuming at least a gig or two.

girvo 5y ago

In theory, once could use Arraymancer to build the tools to do this, and not have to ship a Python interpreter. That said, I've not used it in anger. https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer

generalizations 5y ago

I suppose you could collect training data during gameplay and process it after the fact. Then you can use the heavy python framework, but it doesn't get in the way. Have the user run the training program on the accumulated data whenever they want to increase the level of customization.

andai 5y ago

If your game has enemies, you could give each unit its own brain. Behind the scenes you could have a "gene pool". When a unit dies it's removed from the pool. When it performs well (eg. damages player or wins the game) the unit can reproduce. Then you add the usual stuff (mutate parameters, cross-breeding fittest units). For training you could just make two AI teams play against each other (off the cuff, 10-100k games should do the trick). Once you release the game, you could sample the best AI from each player's machine. Free distributed neuroevolution cluster ;) I've wanted to do this for a while but I haven't made any proper games yet. Someday!

astrange 5y ago

I think video games already do this; video game AI existed before deep learning, it just used things like decision trees and MCMC.

sdenton4 5y ago

It's likely worth looking into federated learning. You could have a core model, which then trains locally on player inputs. The federated learning research space is focused on this sort of small model update on phones, so should have plenty of people who have thought hard about the problem of keeping binary sizes small and training compute manageable. https://arxiv.org/abs/1909.11875 https://arxiv.org/abs/1909.11875

accountofme 5y ago

Doesn't forza the racing game do this? Edit: spelling

NtGuy25 5y ago

There's actually alot of research in this topic. You should look into "Genetic Programming" on scholar.google.com and you'll find some good presentations. For example https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=genetic+programming+video+games&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=gen... and that references a paper where they do this with MUGEN (Ai v Ai fighting game) http://irep.ntu.ac.uk/id/eprint/30021/1/PubSub7423_8186_Martinez-Arellano.pdf http://irep.ntu.ac.uk/id/eprint/30021/1/PubSub7423_8186_Mart... Genetic programming sounds like a complicated term, but it's basically an easy way to take successful characteristics and breed them into something else.