8 ms·

Steps towards prompt-based creation of virtual worlds

69 points by Schiphol 4y ago

zyang 4y ago
Prompt based world building is going to be the killer app for VR. Zuck was too early.
- technotony 4y ago
  I agree with you that this is the killer app to unlock the metaverse. When everyone can create their own spaces usage will explode like Facebook or YouTube. What makes you think he's too early though? As long as he can keep funding until that point he'll win when it happens surely?
  - zyang 4y ago
    State of art AI is progressing in that direction but we are still years away from creating a fleshed out world from prompt. Zuck funneled billions intro VR chart/Unity clone with little to show. It was too early.
- dahdum 4y ago
  > Zuck was too early. Who will be best positioned when these killer apps start coming out? Meta has like 90% headset market share, and while Apple may release something "cool" with AR it's unlikely to be a direct threat. Microsoft is cranking along with WMR and Hololens, but they aren't leading innovation or really trying to. Valve has good hardware but can't compete with Meta due to their scale and capital. Nobody is putting money into VR like Meta. Zuck is betting the company, and I'd probably do the same in his place. FB/IG print money but growth is substantially dead and long term prospects are dim.
  - andybak 4y ago
    > Who will be best positioned when these killer apps start coming out? Hopefully everyone. Use OpenXR as a base and release on every headset.
    - dahdum 4y ago
      You can release on every headset but 90% of your market is Oculus and they'll collect 15-30% commission on every sale.
  - bee_rider 4y ago
    90% of a nascent market is not a huge lead — not sent from my blackberry.
    - TedDoesntTalk 4y ago
      > and while Apple may release something "cool" with AR it's unlikely to be a direct threat 2nd (or 5th) mover advantage?
      - bee_rider 4y ago
        Advantage of moving when the path has been revealed I think.
    - moffkalast 4y ago
      I'm sure Nokia will take over the market again, any minute now... any minute now...
  - glass3 4y ago
    >Valve has good hardware but can't compete with Meta due to their scale and capital. Valve is no startup but should capital ever be the limiting factor, venture capital should be ready to finance the expansion. However, Valve has the Instagram problem. If people don't pay for their worlds but accept them filled with branded artifacts, then Meta has that infrastructure and Valve does not. Apple's entrance into the advertising market could be seen as a preparation for their VR products, like their gaming platform (similar to Netflix). As Meta, I wouldn't bet that Apple just creates an AR device. Microsoft on the other hand needs a VR headset for their Xbox because Sony has one. They are already cooperating with Meta on the business headset. To me, this suggests that MS will also cooperate with Meta on the gaming headset. If Apple manages to sell as many headsets as they sell phones then gaming consoles are dwarfed. When all is said and done, I would love to see a documentary about Sony's strategy meetings. How can they align themselves to stay relevant?
  - wslh 4y ago
    I don't think current market share is an indicator of future market share here, here innovation and non-predictable stuff could change things completely. VR is not yet popular so the potential market was not tested yet. I own Meta devices and all I can say is they have great technologies from the Hardware side. Can also say that Unity is working towards a great SDK. In B2B: how many companies use VR headsets for meetings?
- teddyh 4y ago
  If worlds can be built from a prompt, worlds will be worthless.
  - ben_w 4y ago
    Conversations with other real humans can only be built from prompts, they're not worthless. I assume it will end up with a power-law distribution just like poetry and literature, most seen as low-value, a few as generation-defining.
    - teddyh 4y ago
      Then the value is not in the created world, but in the curation.
- 6502nerdface 4y ago
  It's going to be like those scenes in Star Trek: TNG when they give instructions to the holodeck.
- est 4y ago
  > Prompt based world building bonus: - prompt based story for RPGs - prompt based dungeons challenges - synthetic voice acting for NPCs - prompt based outfits/face for story characters
  - nivenkos 4y ago
    We're so much closer to these things now, the progress in the last few years has been incredible.
    - BizarroLand 4y ago
      Retirement homes in 40 years are just going to be warehouses full of pods with people in them enjoying fulldive through infinite universes. When you run out of money it fills with carbon dioxide, waits an hour, then dumps your body and is ready for the next one.
- ben_w 4y ago
  The problem with Meta's vision for Metaverse isn't being early, it's being bland and technically uninspired. VRChat, and before it Second Life, are both better placed on the same journey. Heck, even Second Life running in a browser in 2010 has a better value proposition than Meta's vision, even if the frame rate was bad enough that I recon it would cause almost universal nausea in VR: https://www.youtube.com/watch?v=cOi11vq4bxU https://www.youtube.com/watch?v=cOi11vq4bxU
CatWChainsaw 4y ago
Great comfort for when the real one is WALL-E world! :)
- deleted 4y ago
  [deleted]
nickpeterson 4y ago
It would be neat to describe features of a Minecraft world and have it generated on the fly, feels doable but I could be way off.
nsxwolf 4y ago
This was the most impressive part of the Star Trek holodeck - not the simulated matter and holographic walls, but the computer you could have a conversation with and create a custom “program”. We seem to be heading quickly to that being a reality.
- ben_w 4y ago
  Between neural radiance fields, CLIP embedding, VR headsets, and voice interfaces, if that hasn't already been done, a tech demo that glues those things together is almost certainly going to happen in the next few months as a result of you suggesting it here.
net_ 4y ago
I've thought a bit about how I would add text-based procedural generation to my virtual world engine (I would love to have an "edit a book to change the world" experience, like Myst's linking books). The hard part is the composition of smaller things into larger concepts, since you have limited data. For example, developers often devise a system of generic water tiles that they can fit together to form any sort of river. But how do you algorithmically go from the text "I want a river that starts large, gets narrow, then turns East" to a valid map layout? The water tiles and the ways they fit together are specific to the world, so there's nowhere near enough data to train a model. You could hardcode procedural logic based on key phrases ("river" + "start large" + "get narrow" + "turn east"), but that's a lot of work for a very limited implementation. The most promising idea I've come up with is a node-based intermediate representation. You could use general data on rivers to determine how they're shaped, then use that to translate the user's prompt into a series of nodes on a map. Then, you just need to hand-implement a system to iterate the nodes and place tiles based on whether the next node is straight ahead, turning, etc. This approach could generalize to other concepts like mountains. At that point, it's just about making the text-based stage sufficiently predictive of what the user wants instead of just being a Logo-like.
- vintermann 4y ago
  The incremental-procedural generation that impresses me most is long distance connections. We want worlds that are too big to be completely stored, so there's conceptually a tree, where high-level features (continents? cities?) exist on the top, and low-level features (the colour of the fur of a specific citizen's pet?) towards the bottom. To get the same tree, no matter the order you expand it, is the key desirable feature of incremental procedural generation. Because then you get the feeling that it's in a sense "all there", and not just made up as you go along. But making a tree like that is easy. The hard part is the long-distance connections. If you step down the tree to a continent, to a country, to a city, to a person... and you want that guy to have an aunt. Possibly on another continent. And you want to make it so that if you restarted with the same seed, and stepped down to the same aunt, she would still have that same nephew. Most incremental procedural games have very little in the way of long distance connections, and it makes them lose their lustre quickly. What does it matter if there are 10000 procedurally generated cities in your world, if they don't have anything to do with each other? A few games get by this by not being incremental, but simulating all that hard stuff up front. That's how they have aunts in Dwarf Fortress. But while it's impressive, it doesn't scale up. I think the way forward must be to figure out how to embellish (that is, expanding nodes in a seed-generated tree) just what you need, when you need it. Language models are literally good at embellishing, but then you feel that lack of consistency and coherence, because there's no seed-generated tree at the root of it.
fergie 4y ago
Bit of a metaquestion but what am I reading here exactly? Is this academic research? I cant see any reference to where the work was carried out, or in what context (Cornell? Or has the paper just been submitted to them?).
- IanCal 4y ago
  It's a research paper on something called a preprint platform. It's a place almost anything fitting can be put for free - to share work before it goes through the arduous process of getting it published (side rant about publishing not meaning publishing goes here). Cornell run/host arxiv, but aren't doing heavy moderation of content (journals should check much more, arxiv will be checking more for spam/non researchy stuff). It's a bit of work between three authors, all affiliated with Microsoft and one who is also affiliated with the University of California San Diego.
utopiah 4y ago
My own stuff in JavaScript relying on SantaCoder https://twitter.com/utopiah/status/1621760472461594624 https://twitter.com/utopiah/status/1621760472461594624 just eval() all things.

69 points by Schiphol 4y ago

zyang 4y ago

Prompt based world building is going to be the killer app for VR. Zuck was too early.

technotony 4y ago
I agree with you that this is the killer app to unlock the metaverse. When everyone can create their own spaces usage will explode like Facebook or YouTube. What makes you think he's too early though? As long as he can keep funding until that point he'll win when it happens surely?
- zyang 4y ago
  State of art AI is progressing in that direction but we are still years away from creating a fleshed out world from prompt. Zuck funneled billions intro VR chart/Unity clone with little to show. It was too early.
dahdum 4y ago
> Zuck was too early. Who will be best positioned when these killer apps start coming out? Meta has like 90% headset market share, and while Apple may release something "cool" with AR it's unlikely to be a direct threat. Microsoft is cranking along with WMR and Hololens, but they aren't leading innovation or really trying to. Valve has good hardware but can't compete with Meta due to their scale and capital. Nobody is putting money into VR like Meta. Zuck is betting the company, and I'd probably do the same in his place. FB/IG print money but growth is substantially dead and long term prospects are dim.
- andybak 4y ago
  > Who will be best positioned when these killer apps start coming out? Hopefully everyone. Use OpenXR as a base and release on every headset.
  - dahdum 4y ago
    You can release on every headset but 90% of your market is Oculus and they'll collect 15-30% commission on every sale.
- bee_rider 4y ago
  90% of a nascent market is not a huge lead — not sent from my blackberry.
  - TedDoesntTalk 4y ago
    > and while Apple may release something "cool" with AR it's unlikely to be a direct threat 2nd (or 5th) mover advantage?
    - bee_rider 4y ago
      Advantage of moving when the path has been revealed I think.
  - moffkalast 4y ago
    I'm sure Nokia will take over the market again, any minute now... any minute now...
- glass3 4y ago
  >Valve has good hardware but can't compete with Meta due to their scale and capital. Valve is no startup but should capital ever be the limiting factor, venture capital should be ready to finance the expansion. However, Valve has the Instagram problem. If people don't pay for their worlds but accept them filled with branded artifacts, then Meta has that infrastructure and Valve does not. Apple's entrance into the advertising market could be seen as a preparation for their VR products, like their gaming platform (similar to Netflix). As Meta, I wouldn't bet that Apple just creates an AR device. Microsoft on the other hand needs a VR headset for their Xbox because Sony has one. They are already cooperating with Meta on the business headset. To me, this suggests that MS will also cooperate with Meta on the gaming headset. If Apple manages to sell as many headsets as they sell phones then gaming consoles are dwarfed. When all is said and done, I would love to see a documentary about Sony's strategy meetings. How can they align themselves to stay relevant?
- wslh 4y ago
  I don't think current market share is an indicator of future market share here, here innovation and non-predictable stuff could change things completely. VR is not yet popular so the potential market was not tested yet. I own Meta devices and all I can say is they have great technologies from the Hardware side. Can also say that Unity is working towards a great SDK. In B2B: how many companies use VR headsets for meetings?
teddyh 4y ago
If worlds can be built from a prompt, worlds will be worthless.
- ben_w 4y ago
  Conversations with other real humans can only be built from prompts, they're not worthless. I assume it will end up with a power-law distribution just like poetry and literature, most seen as low-value, a few as generation-defining.
  - teddyh 4y ago
    Then the value is not in the created world, but in the curation.
6502nerdface 4y ago
It's going to be like those scenes in Star Trek: TNG when they give instructions to the holodeck.
est 4y ago
> Prompt based world building bonus: - prompt based story for RPGs - prompt based dungeons challenges - synthetic voice acting for NPCs - prompt based outfits/face for story characters
- nivenkos 4y ago
  We're so much closer to these things now, the progress in the last few years has been incredible.
  - BizarroLand 4y ago
    Retirement homes in 40 years are just going to be warehouses full of pods with people in them enjoying fulldive through infinite universes. When you run out of money it fills with carbon dioxide, waits an hour, then dumps your body and is ready for the next one.
ben_w 4y ago
The problem with Meta's vision for Metaverse isn't being early, it's being bland and technically uninspired. VRChat, and before it Second Life, are both better placed on the same journey. Heck, even Second Life running in a browser in 2010 has a better value proposition than Meta's vision, even if the frame rate was bad enough that I recon it would cause almost universal nausea in VR: https://www.youtube.com/watch?v=cOi11vq4bxU https://www.youtube.com/watch?v=cOi11vq4bxU

CatWChainsaw 4y ago

Great comfort for when the real one is WALL-E world! :)

deleted 4y ago
[deleted]

nickpeterson 4y ago

It would be neat to describe features of a Minecraft world and have it generated on the fly, feels doable but I could be way off.

nsxwolf 4y ago

This was the most impressive part of the Star Trek holodeck - not the simulated matter and holographic walls, but the computer you could have a conversation with and create a custom “program”. We seem to be heading quickly to that being a reality.

ben_w 4y ago
Between neural radiance fields, CLIP embedding, VR headsets, and voice interfaces, if that hasn't already been done, a tech demo that glues those things together is almost certainly going to happen in the next few months as a result of you suggesting it here.

net_ 4y ago

I've thought a bit about how I would add text-based procedural generation to my virtual world engine (I would love to have an "edit a book to change the world" experience, like Myst's linking books). The hard part is the composition of smaller things into larger concepts, since you have limited data. For example, developers often devise a system of generic water tiles that they can fit together to form any sort of river. But how do you algorithmically go from the text "I want a river that starts large, gets narrow, then turns East" to a valid map layout? The water tiles and the ways they fit together are specific to the world, so there's nowhere near enough data to train a model. You could hardcode procedural logic based on key phrases ("river" + "start large" + "get narrow" + "turn east"), but that's a lot of work for a very limited implementation. The most promising idea I've come up with is a node-based intermediate representation. You could use general data on rivers to determine how they're shaped, then use that to translate the user's prompt into a series of nodes on a map. Then, you just need to hand-implement a system to iterate the nodes and place tiles based on whether the next node is straight ahead, turning, etc. This approach could generalize to other concepts like mountains. At that point, it's just about making the text-based stage sufficiently predictive of what the user wants instead of just being a Logo-like.

vintermann 4y ago
The incremental-procedural generation that impresses me most is long distance connections. We want worlds that are too big to be completely stored, so there's conceptually a tree, where high-level features (continents? cities?) exist on the top, and low-level features (the colour of the fur of a specific citizen's pet?) towards the bottom. To get the same tree, no matter the order you expand it, is the key desirable feature of incremental procedural generation. Because then you get the feeling that it's in a sense "all there", and not just made up as you go along. But making a tree like that is easy. The hard part is the long-distance connections. If you step down the tree to a continent, to a country, to a city, to a person... and you want that guy to have an aunt. Possibly on another continent. And you want to make it so that if you restarted with the same seed, and stepped down to the same aunt, she would still have that same nephew. Most incremental procedural games have very little in the way of long distance connections, and it makes them lose their lustre quickly. What does it matter if there are 10000 procedurally generated cities in your world, if they don't have anything to do with each other? A few games get by this by not being incremental, but simulating all that hard stuff up front. That's how they have aunts in Dwarf Fortress. But while it's impressive, it doesn't scale up. I think the way forward must be to figure out how to embellish (that is, expanding nodes in a seed-generated tree) just what you need, when you need it. Language models are literally good at embellishing, but then you feel that lack of consistency and coherence, because there's no seed-generated tree at the root of it.

fergie 4y ago

Bit of a metaquestion but what am I reading here exactly? Is this academic research? I cant see any reference to where the work was carried out, or in what context (Cornell? Or has the paper just been submitted to them?).

IanCal 4y ago
It's a research paper on something called a preprint platform. It's a place almost anything fitting can be put for free - to share work before it goes through the arduous process of getting it published (side rant about publishing not meaning publishing goes here). Cornell run/host arxiv, but aren't doing heavy moderation of content (journals should check much more, arxiv will be checking more for spam/non researchy stuff). It's a bit of work between three authors, all affiliated with Microsoft and one who is also affiliated with the University of California San Diego.

utopiah 4y ago

My own stuff in JavaScript relying on SantaCoder https://twitter.com/utopiah/status/1621760472461594624 https://twitter.com/utopiah/status/1621760472461594624 just eval() all things.