13 ms·

Diffusion for World Modeling

• deleted 2y ago
  [deleted]
• yakorevivan 2y ago
  [dead]
• madaxe_again 2y ago
  I earnestly think this is where all gaming will go in the next five years - it’s going to be so compelling that stuff already under development will likely see a shift to using diffusion models. As this is demonstrating, a sufficiently honed model can produce realtime graphics - and some of the demos floating around where people are running GTA San Andreas through non-realtime models hint as to where this will go. I give it the same five years before there are games entirely indistinguishable from reality, and I don’t just mean graphical fidelity - there’s no reason that the same or another model couldn’t provide limitless physics - bust a hole through that wall, set fire to this refrigerator, whatever.
  • advael 2y ago
    I'm not sure that's a warranted assumption based on this result, exciting as it is, we are still seeing replication of an extant testable world model, rather than extrapolation that can produce novel mechanics without them being in the training data. I'm not saying this isn't a stepping stone to that, I just think your prediction's a little optimistic based on the scope of that problem
  • casenmgreen 2y ago
    Not a chance. There are fundamental limitations with what are in the end all essentially neural nets; there is no understanding, only prediction. Prediction alone is not enough to emulate reality, which is why for example genuinely self-driving cars have not, and will not, emerge. A fundamental advance in AI technology will be required for that, something which leads to genuine intelligence, and we are no closer to that than ever we were.
    • jiggawatts 2y ago
      For simulations like games, it's a trivial matter to feed the neural game engine pixel-perfect metadata. Instead of rendering the final shaded and textured pixels, the engine would output just the material IDs, motion vectors, and similar "meta" data that would normally be the inputs into a real-time shader. The AI can use this as inputs to render a photorealistic output. It can be trained using offline-rendered "ground-truth" raytraced scenes. Potentially, video labelled in a similar way could be used to give it a flair of realism. This is already what NVIDIA DLSS and similar AI upscaling tech uses. The obvious next step is not just to upscale rendered scenes, but to do the rendering itself.
    • fancyfredbot 2y ago
      Looking at the examples of 2600 games in the paper I'm not sure you can tell that they are just predictions. Have you considered how you'd tell the difference between a prediction and understanding in practice?
    • francoisfleuret 2y ago
      "there is no understanding, only prediction" I have no idea what this means.
      • tourmalinetaco 2y ago
        The MLM has no idea what it’s making, where you are in the map, what you left behind, and what you picked up. It can accurately predict what comes next, but if you pick up an item and do a 360° turn the item will be back and you can repeat the process.
      • GaggiX 2y ago
        When a human does it, it's understanding, when an AI does it, it's prediction, I thinks it's very clear /s
        • therouwboat 2y ago
          Does what? In normal game world things tend to stay where they are without player having to do anything.
  • TinkersW 2y ago
    It requires a monster GPU to run 10 fps at what looks like sub 720p... I think it may be abit more than 5 years..
  • viraptor 2y ago
    It's not that great yet. Given a model which can generate the game view in ~real time and a model which can generate the models and textures, why would you ever use the first option, apart from a cool tech demo? I'm sure there's space for new dreamy games where invisible space behind you transforms when you turn around, but for other genres... why? Destructible environment has been possible for quite a while, but once you allow that everywhere, you can get games into unplayable state. They need to be designed around that mechanic to work well: Noita, Worms, Teardown, etc. I don't believe the "limitless physics" would matter after a few minutes.
  • qayxc 2y ago
    I think you're missing the most important point: these models need to be trained on something and that something is a fully developed, working game. You're basically saying that game development would need to do the work twice: step 1: develop a fully functional game, step 2: spend ridiculous effort (in terms of time and compute) on training a model to emulate the game in a half-baked fashion. It's a solution looking for a problem.
    • manmal 2y ago
      The world model can still be rendered in very low res, and then the diffusion skin/remaster is applied. And this would also be an exciting route to go at remastering old games. I‘d pay a lot to play NFS Porsche again, with photorealism. Or imagine Command & Conquer Red Alert, „rendered“ with such a model.
      • qayxc 2y ago
        NVIDIA's RTX Remix [1] suite of tools already does that. It doesn't require any model training or dozens of hours of pre-recorded gameplay either. You can drop in low-res textures and have AI tools upscale them. Models can be replaced, as well as lighting and the best part: it's all under your control. You're not at the merci of obscure training material that might or might not result in a consistent look-and-feel. More knobs, more control, less compute required. [1] https://www.nvidia.com/en-us/geforce/rtx-remix/ https://www.nvidia.com/en-us/geforce/rtx-remix/
        • manmal 2y ago
          TIL, thanks for posting. The workflow I was sketching out is simpler though: Render a legacy game or low fidelity modern game as-is, and run it through a diffusion model in real time.
    • FeepingCreature 2y ago
      You can crosstrain on reality.
  • Arch485 2y ago
    It seems extremely unlikely to me that ML models will ever run entire games. Nobody wants a game that's "entirely indistinguishable from reality" anyways. If they did, they would go outside. I think it's possible specific engine components could be ML-driven in the future, like graphics or NPC interactions. This is already happening to a certain degree. Now, I don't think it's impossible for an ML model to run an entire game. I just don't think making + running your game in a predictive ML model will ever be more effective than making a game the normal way.
    • jsheard 2y ago
      Yep, the fuzziness and opaqueness of ML models makes developing an entire game state inside one a non-starter in my opinion. You need precise rules, and you need to be able to iterate on those rules quickly, neither of which are feasible with our current understanding of ML models. Nobody wants a version of CS:GO where fundamental constants like weapon damage run on dream logic. If ML has any place in games it's for specific subsystems which don't need absolute precision, NPC behaviour, character animation, refining the output of a renderer, that kind of thing.
• mungoman2 2y ago
  This is getting ridiculous! Curious, since this is a strong loop old frame + input -> new frame, What happens if a non-CS image is used to start it off? Or a map the model has never seen. Will the model play ball, or will it drift back to known CS maps?
  • Arch-TK 2y ago
    Looks like it only knows Dust 2 since every single "dream" (I'm going to call them that since looking at this stuff feels like dreaming about Dust 2) is of that map only.
• advael 2y ago
  Dang this is the first paper I've seen in a while that makes me think I need new GPUs
• 6510 2y ago
  Can it use a seed that makes the same map every time?
• croo 2y ago
  For anyone who actually tried it : Does it respects/builds some kind of game map in the process or is it just a bizarre psychedelic dream walk experience where you cannot go back the same place twice and space dimensions are just funny? Is a game map finite?
  • delusional 2y ago
    Just tried it out, and no. It doesn't have any sort of "map" awareness. It's very much in the "recall/replay" category of "AI" where it seems to accurately recall stuff that is part of the training dataset, but as soon as you do something not in there (like walk into a wall), it completely freaks out and spits out gibberish. Plausible gibberish, but gibberish none the less.
    • neongreen 2y ago
      Can you upload a screen recording? I don’t think I can run the model locally but it’d be super interesting to see what happens if you run into a wall
    • kqr 2y ago
      This should mainly be a matter of giving it more training though, right? It sounds like to amount of training it's gotten is relatively sparse.
      • treyd 2y ago
        It doesn't have any ability to reason about what you did more than a couple of seconds ago. Its memory is what's currently on the screen and what the user's last few inputs were.
      • delusional 2y ago
        Theoretically. In practice, that's not clear. As you add more training data you have to ask yourself what the point is. we already have a pretty good simulation of Counter Strike.
  • aidos 2y ago
• cousin_it 2y ago
  I continue to be puzzled by people who don't notice the "noise of hell" in NN pictures and videos. To me it's always recognizable and terrifying, has been from the start.
  • HKH2 2y ago
    Eyes have a lot of noise too.
  • npteljes 2y ago
    What do you mean by noise of hell in particular? I do notice that the images are almost always uncanny in a way, but maybe we're not meaning the same thing. Could you elaborate on what you experience?
  • taneq 2y ago
    Like a subtle but unsettling babble/hubbub/cacophony? If so then I think I kind of know what you mean.
    • TechDebtDevin 2y ago
      There's definately a bit of an uncanny valley in the land of top tier diffusion models. A generative video of someone smiling is way more likely to illicit this response for me than a generative image or single frame. It definately has something to do with the movement.
    • cousin_it 2y ago
      Yes, that's exactly it.
• fancyfredbot 2y ago
  Strangely the paper doesn't seem to give much detail on the cs-go example. Actually the paper explicitly mentions it's limited to discrete control environments. Unless I'm missing something the mouse input for counterstrike isn't discrete and wouldn't work. I'm not sure why the title says it was trained on 2x4090 either as I can't see this on either the linked page or the paper. The paper mentions a GPU year of 4090 compute was used to train the Atari model.
  • c1b 2y ago
    CSGO model is only 1.5 gb & training took 12 days on a 4090 https://github.com/eloialonso/diamond/tree/csgo?tab=readme-ov-file https://github.com/eloialonso/diamond/tree/csgo?tab=readme-o...
    • fancyfredbot 2y ago
      Thanks, that's the detail I was looking for on the training. It's amazing results like this can be achieved at such a low costs! I thought this kind of work was out of reach for the GPU poor. The part about the continuous control still seems weird to me though. If anyone understands that then very interested to hear more.
• TealMyEal 2y ago
  What's the end goal here? personalised games for everyone. ultra-graphics. i dont really see how this is going to be better than our engine based systems. I love being a horse in the 1900s that automobilewill never take off /s
  • qayxc 2y ago
    It's a research paper. Not everything that comes out of research has an immediate real-world application in mind. Games are just an accessible and easy to replicate context to work in, they're not an end goal or target application. The research is about AI agents interacting with and creating world models. Such world models could just as well be alien environments - i.e. the kind of stuff an interstellar and even interplanetary probe would need to be able to do, as two-way communication over large distances is impractical.
  • visarga 2y ago
    The goal is to train agents that can imagine consequences before acting. But it could also become a cheap way to create experiences and user interfaces on the fly, imagine if you can have any app UI dreamed up like that, not just games. Generative visual interfaces could be a big leap over text mode.
• francoisfleuret 2y ago
  This is 300M parameters model (1/1300th of the big llama-3) trained with 5M frames with 12 days of a GTX4090. This is what a big tech company was doing in 2015. The same stuff at industrial scale à la large LLMs would be absolutely mind blowing.
  • GaggiX 2y ago
    If 12 days with an RTX4090 is all you need, some random people on the Internet will soon start training their own.
    • rizky05 2y ago
      [dead]
  • Sardtok 2y ago
    Two 4090s, but yeah.
    • Sardtok 2y ago
      Never mind, the repo on Github says 12 days on a 4090, so I'm unsure why the title here says two.
  • gjulianm 2y ago
    What exactly would be the benefit of that? We already have Counter Strike working far more smooth than this, without wasting tons of compute.
    • eproxus 2y ago
      But please, think of the shareholders!
    • ben_w 2y ago
      As with diffusion models in general, the point isn't the specific example but that it's generalisable. 5 million frames of video data with corresponding accelerometer data, and you get this for genuine photorealism.
• w-m 2y ago
  If you're not bored with it yet, here's a Deep Dive (NotebookLM, generated podcast). I fed it the project page, the arXiv paper, the GitHub page, and the two twitter threads by the authors. https://notebooklm.google.com/notebook/a240cb12-8ca1-41b4-ab1f-1d029ae3142e/audio https://notebooklm.google.com/notebook/a240cb12-8ca1-41b4-ab... (7m59s) As always, it's not actually much of a technical deep dive, but gives a quite decent overview of the pieces involved, and its applications.
  • thierrydamiba 2y ago
    How did you get the output to be so long? My podcasts are 3 mins max…
    • w-m 2y ago
      Oh wow, really? Even if you feed it whole research papers? The ones I tried until now were more in the 8-10 minute range. I haven’t looked in to how to control the output yet. Hopefully that’ll get a little more transparent and controllable soon.
• ThouYS 2y ago
  I don't really understand the intuition on why this helps RL. The original game has a lot more detail, why can't it be used directly?
  • FeepingCreature 2y ago
    In the real world, you can't just boot up a copy of reality to play out strategies. You need an internal model.
    • tourmalinetaco 2y ago
      So, effectively, these video game models are proof-of-concepts to say “we can make models with extremely accurate predictions using minimal resources”?
      • vbezhenar 2y ago
        May be I'm wrong but my understanding is that you can film some area using, say, dashcams and then generate this kind of neuro model. Then you can train robot to walk in this area with this neuro-model. It can perform billions of training sessions without touching physical world. Alternatively you can somehow perform 3D scan of area, recreate its 3D model and use, say, game engine to simulate, but that probably requires more effort and not necessarily better.
        • usrusr 2y ago
          And the leg motions we sometimes see in sleeping dogs suggest that this is very much a way how having dreams is useful!
      • usrusr 2y ago
        Not sure where you see the "minimal resources" here? But I'd just counter all questions about "why" with the blanket response of "for understanding natural intelligence": the way biology innovates is that it throws everything against the wall and not pick the one thing that sticks as the winner and focus on that mechanism, it keeps the sticky bits and also everything else as long as their cost isn't prohibitive. Symbolic modeling ("this is an object that can fall down"), prediction chains based on visual similarity patterns (this), hardwired reflexes (we tend to not trust anything that looks and moves like a spider or snake) and who knows what else, it's all there, it all runs in parallel, invited or not, and they all influence each other in subtle and less subtle ways. The interaction is not engineered, it's more like crosstalk that's allowed to happen and has more upside than downside, or else evolution would have preferred variations of the setup that have less of the kind of crosstalk in question. But in our quest to understand us, it's super exciting to see candidates for processes that perhaps play some role in our minds, in isolation, no matter if that role is big or small.
• snickerer 2y ago
  I see where this is going. The next step to create training data is a real human with a bodycam. There is only the need to connect the real body movement (step forward, turning left, etc) to typical keyboard and mouse game control events, to feed them into the model, too. I think that is what the devs here are dreaming about.
  • devttyeu 2y ago
    The "We live in a simulation" argument just started looking a lot more conceivable.
    • iwontberude 2y ago
      Not really because is it simulators all the way down? Simulation theory explains nothing and only adds more unexplainable phenomenon.
    • tiborsaas 2y ago
      I'm already very suspicious, we just got the same room number in the third hotel in a row. Someone got lazy with the details :)
  • devttyeu 2y ago
    Could probably make a decent dataset from VR headset tracking cameras + motion sensors + passthrough output + decoded hand movements
  • CaptainFever 2y ago
    Or a cockpit cam for the world's most realistic flight simulator. /lighthearted
• delusional 2y ago
  I just checked it out right quick. It works perfectly well on an AMD card with ROCM pytorch. It seems decent in short bursts. As it goes on it quite quickly loses detail and the weapon has a tendency to devolve into colorful garbage. I would also like to point out that none of the videos show what happens when you walk into a wall. It doesn't handle it very gracefully.
• thenthenthen 2y ago
  When my game starts to look like this, I know it is time to quit hahha, maybe a helpful tool in gaming addiction therapy? The morphing of the gun/skins and the environment (the sandbags) wow. Would like to play this and see what happens when you walk backwards, turn around quick, use ‘noclip’ :D
• DrSiemer 2y ago
  Where it gets really interesting is if we can train a model on the latest GTA, plus maybe related real life footage, and then use it to live upgrade the visuals of an old game like Vice City. The lack of temporal consistency will still make it feel pretty dreamlike, but it won't matter that much, because the base is consistent and it will look amazing.
  • davedx 2y ago
    A game like GTA has way too much functionality and complex branching for this to work I think (beyond eg doing aimless drives around the city — which would be very cool though)
    • DrSiemer 2y ago
      Gta 5 has everything Vice City has and more. In the Doom AI dream it's possible to shoot people. Maybe in this CS model as well? I think the model does not have to know anything about the functionality. It can just dream up what is most probable to happen based on the training data.
  • InsideOutSanta 2y ago
    Just redrawing images drawn by an existing game engine works, and generates amazing results, although like you point out, temporal consistency is not great. It might interpret the low-res green pixels on a far-away mountain as fruit trees in one frame, and as pines in the next. Here's a demo from 2021 doing something like that: https://www.youtube.com/watch?v=3rYosbwXm1w https://www.youtube.com/watch?v=3rYosbwXm1w
  • sorenjan 2y ago
    In addition to the sibling comment's older example there's new work done with GTA too. https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_with_a_photorealistic_filter_with_runway/ https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_wit...
    • DrSiemer 2y ago
• mixtureoftakes 2y ago
  this is crazy when trying to run on a mac it only plays in a very small window, how could this be configured?
• mk_stjames 2y ago
  This was Schmidhuber's group is 2018: https://worldmodels.github.io/ https://worldmodels.github.io/ Just want to point that out.
  • afh1 2y ago
    Ahead of its time for sure. Dream is an accurate term here, that driving scene does resemble driving in dreams.
  • hervature 2y ago
    I assume you are pointing this out because it is the first reference in the paper and getting the recognition it deserves and you are simply providing this link for convenience to those who do not go to the references.
    • mk_stjames 2y ago
      Yes, it was very nice to see it was the first citation in the paper (and cited several times throughout). The World Models paper is still one of the most amazing papers I've ever read. And I just really keep wanting to show that, in case people really don't see that, many in-the-know... knew.
  • Grimblewald 2y ago
    that was fun to play, gets incredibly unstable if you survive too long.
• jmchambers 2y ago
  I _think_ I understand the basic premise behind stable diffusion, i.e., reverse the denoising process to generate realistic images but, as far as I know, this is always done at the pixel level. Is there any research attempting to do this at the 3D asset level, i.e., subbing in game engine assets (with position and orientation) until a plausible scene is recreated? If it were possible to do it that way, couldn't it "dream" up real maps, with real physics, and so avoid the somewhat noisy output these types of demo generate?
  • desdenova 2y ago
    I think the closest we have right now is 3D gaussian splatting. So far it's only been used to train a scene from photographs from multiple angles and rebuild it volumetrically by adjusting densities in a point-cloud. But it might be possible to train a model on multiple different scenes, and perform diffusion on a random point cloud to generate new scenes. Rendering a point cloud in real time is also very efficient, so it could be used to create insanely realistic game worlds instead of polygonal geometry. It seems someone already thought of that: https://ar5iv.labs.arxiv.org/html/2311.11221 https://ar5iv.labs.arxiv.org/html/2311.11221
    • jmchambers 2y ago
      Interesting, I guess that takes things even further and removes the need for hand-crafted 3D assets altogether, which is probably how things will end up going in gaming, long-term. I was suggesting a more modest approach, I guess, one where the reverse-denoising process involves picking and placing existing 3D assets, e.g., those in GTA 5, so that the process is actually building a plausible map, using those 3D assets, but on the fly... Turn your car right and a plausible street decorated with buildings, trees and people is dreamt up by the algorithm. All the lighting and physics would still be done in-engine, with stable diffusion acting as a dynamic map creator, with an inherent knowledge of how to decorate a street with a plausible mix of assets. I suppose it could form the basis of a procedurally generated game world where, given the same random seed, it could generate whole cities or landscapes that would be the same on each player's machine. Just an idea...
• suggeststrongid 2y ago
  [dead]
• marcyb5st 2y ago
  So, this is pretty exciting. I can how this can already be used to generate realistic physics approximations in a game engine. You create a bunch of snippets of gameplay using a much heavier and realistic physics engine (perhaps even CGI). The model learn to approximate the physics and boom, now you have a lightweight physics engine. Perhaps you can even have several that are specialized (e.g. one for smoke dynamics, one for explosions, ...). Even if it allucinates, wouldn't be worse than the physics bugs that are so common in games.
  • hobs 2y ago
    A physics bug would be a consistent problem you can fix. There's no such guarantee about an ML model. This would likely only be ok in the context of a game specifically made to be janky.
    • kqr 2y ago
      This obsession people have with determinism! I'd much rather take a low rate of weird bugs than common consistent ones. I don't believe reproducibility of bugs makes for better gameplay generally.
      • paulryanrogers 2y ago
        Reproducibility does make bugs more likely to be fixed, or at least fixable. Also, games introduce randomness in a controlled way so users don't get frustrated by it appearing in unexpected places. I don't want characters to randomly appear and disappear. It's fine if bullet trajectory varies more randomly as they get further away.
        • skydhash 2y ago
          Also most engines have been worked on for years. So more often than not, core elements like audio, physics, input,... are very stable and the remaining bugs are either "can't fix" or "won't fix".
      • NotMichaelBay 2y ago
• smusamashah 2y ago
  This video https://x.com/Sentdex/status/1845146540555243615 https://x.com/Sentdex/status/1845146540555243615 looks way too much like my dreams. This is almost exactly that happens when I sometimes try to jump high, it transforms me to a different place just like that. Things keep changing just like that. It's amazing to see how close it is to a real dream experience.
  • kleene_op 2y ago
    I noticed that all text looked garbled up when I had some lucid dreams. When diffusion models started to gain attention, I made the connection that text generated in generated images also looked garbled up. Maybe all of those are clues that parts of the human subconscious mind operate pretty close to the principles behind diffusion models.
    • sci_prog 2y ago
      Also the AI generated images that can't get the fingers right. Have you ever tried to look at your hands while lucid dreaming and try counting fingers? There are some really interesting parallels between the dreams and diffusion models.
      • dartos 2y ago
        Of course, due to the very nature of dreams, your awareness of diffusion models and their output flavors how you perceive even past dreams. Our brains love retroactively altering fuzzy memories.
        • hombre_fatal 2y ago
          On the other hand, psychedelics give you perceptions similar to even early deepdream genai images. On LSD, I was swimming in my friend’s pool (for six hours…) amazed at all the patterns on his pool tiles underwater. I couldn’t get enough. Every tile had a different sophisticated pattern. The next day I went back to his place (sober) and commented on how cool his pool tiles were. He had nfi what I was talking about. I walk out to the pool and sure enough it’s just a grid of small featureless white tiles. Upon closer inspection they have a slight grain to them. I guess my brain was connecting the dots on the grain and creating patterns. It was quite a trip to be so wrong about reality. Not really related to your claim I guess but I haven’t thought of this story in 10 years and don’t want to delete it.
• iwontberude 2y ago
  This is crazy looking, I know it’s basically useless but it’s cool anyways.
• shahzaibmushtaq 2y ago
  As I used to play CS 1.6 and CS: GO in my free time before the pandemic, this playable CS diffusion world map has been trained by a noob player for research purposes. After reading the comments I can assume that if you play outside of the scope it was trained on, the game loses its functionality. Nevertheless, R&D for a good cause is something we all admire.
  • crossroadsguy 2y ago
    How is the last version CS 2.0 (I think)? It’s been free to play like GO I guess. Is it like GO where physics felt too dramatised (could just be my opinion)? Or realistic in a snappy way like 1.6?
    • shahzaibmushtaq 2y ago
      Honestly, I heard about CS 2.0 from you. And you are right what you just said about GO.
      • crossroadsguy 2y ago
        Yeah I also didn't know about it until two days back when I was discussing CS with a college friend when we checked it out we realised there is something new after CSGO as well. And of course "proudly Windows only".
        • shahzaibmushtaq 2y ago
          For Windows and Linux, and still grossed around $2 billion according to various reports and sources
          • crossroadsguy 2y ago
            Yeah. I wish 1.6 worked on Apple Silicon and I wish there was an active scene of that still.
• Zealotux 2y ago
  Could we imagine parts of game elements to become "targets" for models? For example hair and fur physics have been notoriously difficult to nail, but it should be easier to use AI to simulate some fake physics on top of the rendered frame, right? Is anyone working on that?
• gadders 2y ago
  Cool achievement, but I want AI to give me smarter NPCs, not simulate the map.
  • thelastparadise 2y ago
    The NPCs need a model of the world in their brain in order to act normal.
• styfle 2y ago
  But does it work on macOS? (The latest CS removed support for macOS)
• ilaksh 2y ago
  I wonder if there is some way to combine this with a language model, or somehow have the language model in the same latent space or something. Is that was vision-language models already do? Somehow all of the language should be grounded in the world model. For models like Gemini that can answer questions about video, it must have some level of this grounding already. I don't understand how this stuff works, but compressing everything to one dimension as in a language model for processing seems inefficient. The reason our language is serial is because we can only make one sound at a time. But suppose the "game" trained on was a structural engineering tool. The user asks about some scenario for a structure and somehow that language is converted to an input visualization of the "game state". Maybe some constraints to be solved for are encoded also somehow as part of that initial state. Then when it's solved (by an agent trained through reinforcement learning that uses each dreamed game state as input?), the result "game state" is converted somehow back into language and combined with the original user query to provide an answer. But if I understand properly, the biggest utility of this is that there is a network that understands how the world works, and that part of the network can be utilized for predicting useful actions or maybe answering questions etc. ?
  • LarsDu88 2y ago
    To combine with a language model simply replace the action vector with a language model latent. Alternative as of last year there are now purely diffusion based text decoder models
  • empath75 2y ago
    Not everything needs to be a single giant neural network. You could have a bunch of weakly coupled specialized networks sending data back and forth over a normal api.
• LarsDu88 2y ago
  Iterative denoising diffusion is such a hurdle for getting this sort of thing running at reasonable fps
• siev 2y ago
  [dead]
• akomtu 2y ago
  The current batch of ML models looks a lot like filling in holes in the wall of text, drawings or movies: you erase a part of the wall and tell it to fix it. And it fills in the hole using colors from the nearby walls in the kitchen and similar walls and we watch this in awe thinking it must've figured out the design rules of the kitchen. However what it's really done is it interpolated the gaps with some sort of basic functions, trigonometric polynomials for example, and it used thousands of those. This solution wouldn't occur to us because our limited memory isn't enough for thousands of polynomials: we have to find a compact set of rules or give up entirely. So when these ML models predict the motion of planets, they approximate the Newton's law with a long series of basic functions.