3 ms·

Holodeck: Language Guided Generation of 3D Embodied AI Environments

55 points by geox 3y ago

imzadi 3y ago
As a VR enthusiast, I'm curious how this might be implemented for on-the-fly integration into a game.
- sitzkrieg 3y ago
  this would be cool for dungeon crawlers period
- ilaksh 3y ago
  I think it would make sense in VR as a standalone system with multiplayer which people could use for any type of game or work. I actually think this sort of thing will become typical within 5-10 years as the systems become faster and we get comfortable VR and MR glasses.
jimbokun 3y ago
I recently rewatched the Star Trek TNG episode where Geordi LaForge accidentally creates a self aware Moriarty as an opponent for Data role playing Sherlock Holmes. It struck me how similar the Holodeck prompts resemble prompts to get ChatGPT to tell a story with certain parameters.
- yjftsjthsd-h 3y ago
  Honestly LLMs in general give me that vibe. Especially character description; "Computer, create an assistant for me. Make him enthusiastic and knowledgeable, and always provide well-written, clear, factual responses." Star Trek or Ollama?
  - justinlloyd 3y ago
    I hooked up a small label printer to a custom application an LLM running on a Jetson the other week that now lives in my workshop permanently. "Hey jiji, I need a label for this piece of 3/4 birch. It's the left panel of the upper cabinet for the 3D printer. 14 inches by 23 and a half inches." And the LLM spits out the parameters that invoke a local function to print the label which I can stick on the material to identify it. It's only a small thing, but very much reminded me of the holodeck in TNG when I first started using it.
justinlloyd 3y ago
I am currently working on a AAA FPS MP RPG - we're a well funded team with decades of experience in big studios, but not yet announced. The very large world is being generated by a variety of DL & ML algorithms. Thousands of buildings in a city with the interiors of each fully realized. This research work in the paper is almost precisely what we're doing and I suspect the AI team on the game are even now cribbing notes from the paper.

55 points by geox 3y ago

imzadi 3y ago

As a VR enthusiast, I'm curious how this might be implemented for on-the-fly integration into a game.

sitzkrieg 3y ago
this would be cool for dungeon crawlers period
ilaksh 3y ago
I think it would make sense in VR as a standalone system with multiplayer which people could use for any type of game or work. I actually think this sort of thing will become typical within 5-10 years as the systems become faster and we get comfortable VR and MR glasses.

jimbokun 3y ago

I recently rewatched the Star Trek TNG episode where Geordi LaForge accidentally creates a self aware Moriarty as an opponent for Data role playing Sherlock Holmes. It struck me how similar the Holodeck prompts resemble prompts to get ChatGPT to tell a story with certain parameters.

yjftsjthsd-h 3y ago
Honestly LLMs in general give me that vibe. Especially character description; "Computer, create an assistant for me. Make him enthusiastic and knowledgeable, and always provide well-written, clear, factual responses." Star Trek or Ollama?
- justinlloyd 3y ago
  I hooked up a small label printer to a custom application an LLM running on a Jetson the other week that now lives in my workshop permanently. "Hey jiji, I need a label for this piece of 3/4 birch. It's the left panel of the upper cabinet for the 3D printer. 14 inches by 23 and a half inches." And the LLM spits out the parameters that invoke a local function to print the label which I can stick on the material to identify it. It's only a small thing, but very much reminded me of the holodeck in TNG when I first started using it.

justinlloyd 3y ago

I am currently working on a AAA FPS MP RPG - we're a well funded team with decades of experience in big studios, but not yet announced. The very large world is being generated by a variety of DL & ML algorithms. Thousands of buildings in a city with the interiors of each fully realized. This research work in the paper is almost precisely what we're doing and I suspect the AI team on the game are even now cribbing notes from the paper.