4 ms·

I'm guessing the compute needed to create this particular demo is prohibitive for any real-world use case, but it's an impressive proof of concept. Someday it w

by causal 2y ago

I'm guessing the compute needed to create this particular demo is prohibitive for any real-world use case, but it's an impressive proof of concept. Someday it will run on a phone tho.

echelon 2y ago
Not at all. If you look at the publications in the world model space, this can be achieved by any pre-seed company. You just need lots of data, but that's easy to gather in an automated fashion with scripting against game engines. These are basically learning game playthroughs. There are some folks on Discord training a playable Super Smash Bros Melee world model. There's a large corpus of annotated Smash Bros gameplay video complete with button inputs. Same for many FPS games. This is once again a "wow" demo from something that doesn't have much exposure, but is actually quite easy to develop and train. There are multiple university papers in this domain that do the exact same thing, they just don't have a fancy demo page. World models will be commonplace in short order, just like image and video models.
- 93po 2y ago
  i upvoted you depsite some downvotes because i would like to hear your argument on why this isn't expensive. if it takes a high end consumer video card that costs $1-2k something like 10 seconds to generate a single low res image, presumably effectively real-time video combined with user inputs would be extremely, extremely demanding and therefore expensive. i mean basic napkin math for 25FPS is 250 GPUs, which is half a million dollars in hardware, and around $20/hour in electricity alone
  - echelon 2y ago
    You can run and play several publicly available world models on lower end graphics cards today, and that's with "academic" code: crude, unoptimized, written just to accomplish the science. Here's one: https://diamond-wm.github.io/ https://diamond-wm.github.io/ There are lots of others. From their paper: "Our model was trained in 12 days on an RTX 4090, and can be played at ~10 FPS on an RTX 3090." This is incredibly accessible, and "this is the worst it will ever look". There are so many low hanging fruit optimizations, and that's without even considering a different model architecture. Here's someone's world model side project from over two years ago that you can play in your browser: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/demo/ This stuff will be everywhere soon, and you'll be running it on consumer hardware.

by causal 2y ago

I'm guessing the compute needed to create this particular demo is prohibitive for any real-world use case, but it's an impressive proof of concept. Someday it will run on a phone tho.

echelon 2y ago

Not at all. If you look at the publications in the world model space, this can be achieved by any pre-seed company. You just need lots of data, but that's easy to gather in an automated fashion with scripting against game engines. These are basically learning game playthroughs. There are some folks on Discord training a playable Super Smash Bros Melee world model. There's a large corpus of annotated Smash Bros gameplay video complete with button inputs. Same for many FPS games. This is once again a "wow" demo from something that doesn't have much exposure, but is actually quite easy to develop and train. There are multiple university papers in this domain that do the exact same thing, they just don't have a fancy demo page. World models will be commonplace in short order, just like image and video models.

93po 2y ago
i upvoted you depsite some downvotes because i would like to hear your argument on why this isn't expensive. if it takes a high end consumer video card that costs $1-2k something like 10 seconds to generate a single low res image, presumably effectively real-time video combined with user inputs would be extremely, extremely demanding and therefore expensive. i mean basic napkin math for 25FPS is 250 GPUs, which is half a million dollars in hardware, and around $20/hour in electricity alone
- echelon 2y ago
  You can run and play several publicly available world models on lower end graphics cards today, and that's with "academic" code: crude, unoptimized, written just to accomplish the science. Here's one: https://diamond-wm.github.io/ https://diamond-wm.github.io/ There are lots of others. From their paper: "Our model was trained in 12 days on an RTX 4090, and can be played at ~10 FPS on an RTX 3090." This is incredibly accessible, and "this is the worst it will ever look". There are so many low hanging fruit optimizations, and that's without even considering a different model architecture. Here's someone's world model side project from over two years ago that you can play in your browser: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/demo/ This stuff will be everywhere soon, and you'll be running it on consumer hardware.