5 ms·
Fable 5.1 World Modeling
- surreal_ 3mo agofable 5.1 generated an interactive 3D union square, and the agent filmed its own tour guide vid inside it. you can walk Powell to Stockton, read the actual storefronts, cross a working intersection, watch a cable car go by. went inside Apple and the Nintendo store, lower level included check source code + more worlds soon
- giancarlostoro 3mo agoHad to vouch for your comment, for some reason it was not visible. > Geometry is derived from OpenStreetMap (ODbL) and USGS 3DEP (public domain). This is probably how I would have tackled this, saves on time, especially if Fable just writes code to convert OSM data to a reasonable ThreeJS model.
- andy99 3mo agoDo you have data on the tokens or $ cost of this?
- surreal_ 3mo agothis is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/PROMPT.md https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop. ~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
- jaybuff 3mo agoThis isn't what I expected when I heard "world model"!
- surreal_ 3mo agoik, "world model" has too many definitions at this point. imo the bar is just: can it simulate the world.
- benatkin 3mo agoNot really. Whether you agree with LeCun's assertion that "LLMs Are a Dead End" or not, world modeling is the same domain. "World Modeling" in this post is confusing because it seems to reference the deep, specific machine learning that goes into these frontier world models and self-driving cars, where it's really just vibe coding turned at the problem of modeling the sort of 3D worlds you'd find in a video game.
- hliyan 3mo agoAgreed. Author should not be using this term. A world model, as I understand, is what an LLM is missing that a human brain has: in addition to a linguistically described world, we also posses a sensory model of the world, a physics model so to speak, against which the random token/thought chains generated by our brains can be validated.
- deleted 3mo ago[deleted]
- Reebz 2mo agobeing reductive, causality is the key term for a world model. the rollout is the key action to demonstrate causality. lots of nice LLMs can generate a 3d representation of a "visual world". These days the focus is on sensory data (image, video, etc.), however world models can be more than just 3d spatial representations - they could be tabular. Again being reductive, its how do you predict the next state from the current state (the rollout).
- hadlock 3mo agoThis is the kind of demo that would really benefit from a youtube demo of 2-3 min Fake edit: there is a longer video here 1 min long: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/union-square-walkthrough.mp4 https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo.
- Willish42 3mo ago> I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo. Here ya go: - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/src/life/Pedestrians.ts https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/src/life/Traffic.ts https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
- avdelazeri 3mo agoI wish you would provide more information. About time, costs, reliability (how many duds did you have? did you need to do any babysitting or etc) and so on But it is a very impressive looking demo, for sure.
- surreal_ 3mo agothis is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union- https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop. ~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
- threecheese 3mo agoDid you need to iterate on the prompt, or did you have a model help you author it? I frequently have problems with orchestration instructions in-prompt, and your is huge. Maybe this is just better with Fable? I honestly haven’t used it much.
- Willish42 3mo agothis is really neat. I wonder if people will start building open world games or "AR" games in the vein of Pokemon Go and Ingress based on similar tech. I've had a similar idea for a long time but don't think it was feasible before now due to AI
- surreal_ 3mo agoyes we're working on it! trying to push a few open world rpg games with real economy and game design. it's also super interesting to benchmark the current models' capability in this direction, since this is a naturally hard and multimodal coding task
- somenameforme 3mo agoI'm not sure about expanding domains, but I expect that very close to 100% of games are now using LLMs to a very significant degree during development, even if most won't admit it. The "AI disclaimers" for games need to go away, simply because it's just rewarding and incentivizing lying and punishing truth.
- ramesh31 3mo agoCool. But the question remains, what is the point of this? I don't mean to pick on this project specifically, but it's indicative of AI usage in general. You end up with something that's like "wow an AI made this" but it's not really anything that interesting or useful in and of itself. I've yet to see a single agent generated piece of software above that bar.
- dogma1138 3mo agoBecause it’s a very complex task to do especially with any level of quality and consistency. Classical algorithmic approaches for data to CAD are quite unreliable heck commercial CAD software still struggles turning scanned 2D plans into sketches with any degree of reliability. Something of this scale would normally require 1000’s and 1000’s of man hours in manual modeling, placement and quality control. Some of the latest prompt to CAD agents and models I’ve seen are the most impressive use of AI I’ve seen in a long time. At least for me this is rather exiting since it gives me the ability to turn many more ideas into reality as a hobbyist rather than spending the entire weekend in Fusion 360.
- trees101 3mo agoI'm interested in this, what is the best current method/tool for this task?
- jryle70 3mo agoWhy don't you try to do it yourself with Fable and compare it with this one? Even with AI there are still a lot of work, knowledge, and gotchas involved. What's the point of such comment?
- ChickeNES 3mo ago> I've yet to see a single agent generated piece of software above that bar. Begs the question, an earnest one: what software would be above that bar?
- surreal_ 3mo agothe topology/texturing critique is fair for mesh generation, but code-generated worlds mostly sidestep it. when the model writes three.js or blender scripts, geometry comes from primitives, csg, and parametric construction, so topology is clean by construction rather than something you clean up after. texturing is still a gap, agreed, though procedural materials cover more than people expect.
- vopi 3mo agoCool demo! That's funny. I literally sketched out something similar 30 minutes ago on paper. I wonder if there was some recent, semi-related article/link that subconsciously prompted the relevance.
- surreal_ 3mo agowe're working on a survey paper about world modeling via code, with folks from frontier labs (qwen omni, oai, etc) and academic institutions (e.g., oxford, stanford, etc) reach out to us about collab: team@philolabs.ai
- T-A 3mo agoThis, maybe? https://news.ycombinator.com/item?id=49525160 https://news.ycombinator.com/item?id=49525160
- kodefreeze 3mo agoI have experimented with this type of modeling for a RTS game I'm building: https://playwarpact.com https://playwarpact.com Few things: 1. Opus 5 is just as good for this tbh and cheaper. 2. They don't generate optimized 3d models. They have high poly count for simple geometries. 3. A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes of the 3d model and then bake textures that include a lot more details like windows, doors etc with tools like Meshy. Can post a tutorial if there's interest. The models need some more RL to be able to do this autonomously.
- vismit2000 3mo agoRegarding your game shared here: https://news.ycombinator.com/item?id=48943351 https://news.ycombinator.com/item?id=48943351
- andai 3mo agoWait so Opus modeled your cool scifi buildings?
- deleted 3mo ago[deleted]
- lukan 3mo ago"Can post a tutorial if there's interest." There is.
- kodefreeze 3mo agoWill write it up!
- ElFitz 3mo agoThanks!
- kodefreeze 2mo ago
- julien_dev 3mo agoThese things always look impressive but I fail to see how they are usable beyond demos or simple games. In my experience the topology is often a mess and texturing is difficult as well
- eagleinparadise 3mo agoi discovered doing this in claude design and i can tell you... it's useful in a professional domain. Although I am not going to share what I'm using it for, to keep an edge in my domain ;)
- ryan_n 3mo agoDon’t worry I’m sure it’ll be in the training data soon enough…
- nomel 3mo agoWell, nothing now starts in a completed state. See pre-production footage for movies and video games, for how scenes progress. Uncharted 4: https://youtu.be/NedDxIGQVs0&t=147 https://youtu.be/NedDxIGQVs0&t=147 Coco: https://youtu.be/nl_JkjgHfFU&t=10 https://youtu.be/nl_JkjgHfFU&t=10 Up: https://www.youtube.com/watch?v=JHfLGgOs6gY https://www.youtube.com/watch?v=JHfLGgOs6gY And, eventually you can just run it through Nvidia DLSS 7 for better hair physics and textures. ;)
- breadsniffer01 3mo agoWoah amazing clips thanks for sharing! The GH repo is definitely in early stages and prob laying the ground work for using the real world as part of the pre production process for a game/movie, etc.
- pelagicAustral 3mo agoBe nice to turn some of these into Counter Strike maps... mhmm...
- lanyard-textile 3mo agoMaking callouts by SF street intersections, lol
- deleted 3mo ago[deleted]
- bb88 3mo agoThis reminds me of the experience VRML wanted to be 3 decades ago. The only difference is that links went to other worlds living on another server. But that seems trivial to implement with this.
- bastawhiz 3mo agoVRML was incredible. It's a "hypertext" markup language but instead of documents it produces 3D environments. It's a shame it never really caught on.
- kevthecoder 3mo agoThere's some work going on at the Metaverse Standards Forum to progress these ideas with Web of Worlds (https://webofworlds.github.io/ https://webofworlds.github.io/). It's scope is a lot broader than VRML, it covers assets, identity, avatars, payments, etc. There's a link to the presentation at SIGGRAPH - https://youtu.be/N2_lb77gKQ8 https://youtu.be/N2_lb77gKQ8.
- ProofHouse 3mo agoI know a guy in LA who basically spent every waking minute of his life past year or two building a spec for this. Announced soon apparently. I’ve spent hours reading some of it. Pretty damn compelling.
- bb88 2mo agoA little late I know, but yes it was. The problem was the hardware didn't really support what VRML could do at the time. I had access to an SGI irix workstation and VRML (and 3d scenes) was still challenging for it back in the early to mid 1990s.
- gimlids 3mo ago“Fable 5.1 World Modeling” is an incredibly misleading title! This has very little to do with Fable and using an unexpected sense of “world model”
- minimaltom 3mo agoThis is beautiful! I would love if we could model out a whole city, ideally using a much cheaper model.
- mrinterweb 3mo agoI'm really curious how GML-5.3-flash would do. Very affordable, and it seems to do pretty well with 3D modeling.
- surreal_ 3mo agoyes experimenting with it actually, will update here! in fact we've generated most of the tourist spots in sf, should be reflected in the repo soon too
- minimaltom 3mo agoHell yeah, whats your thoughts on doing it for like a whole city?
- throwaway613746 3mo ago[dead]
- ameddserM 3mo agoThis looks great.
- lauralin 3mo agoVery good
- hartator 3mo agoStarted the Matrix
- heyl987 3mo agosimulate the universes and then observe the parallel universes
- luciana1u 3mo ago[flagged]
- WordNotSee 3mo agoFundamentally it’s an image model predicting out frames of a sequence. It’s not enough to call it just an image model though. Not quite video either because it’s so specific to rendering images of the world. Thus, “world model”? The problem I have with “world” is it should imply so much more depth. To me this is actually a “POV image model” or “First-person perspective model” (maybe we call this an FPM). There could also be TPMs (third-person models). This maps better to modern game and 3D dev phrasing, where “world model” might imply 3D object collections spatially organized in a cohesive file format - complete with interactivity, audio, and all the baseline constituents of what one might call a “world”. FPMs and TPMs, think about it. Reserve POVMs for photographic realism.
- RugnirViking 3mo agoI think perhaps you've misunderstood what the linked repo actually is? It seems to indeed be a 3d model of an area. It's not an image model, and nothing to do with frames.
- WordNotSee 3mo agoYou are correct - this is threejs and not video/generative rendering which I’ve seen a lot of. I guess “world” works if interactivity and sound etc. are considered other modalities. Video kinda has the same issue (for me) where the audio part of the video is not part of the model. When I think “world” I think more than just the visuals. Like a “film” model to me would imply sound and more than just the video. We typically call 3D art “models” too, so I guess it can’t be called a model model, idk the word “world” doesn’t seem right to me
- OldDrunkGuy 3mo agoI shitpost on this website constantly and trash Anslopic on a daily basis. I am your biggest hater. But god dammit if I’m not also your biggest respecter. reaches out for a handshake You guys are onto something with this one. Keep going. You crazy bastards! You crazy fuckin bastards you hear me?! Hahahahah WOO! WOO!
- deleted 3mo ago[deleted]
- chemael 3mo ago[flagged]
- surreal_ 3mo agoadded the hand-painted style kyoto: https://github.com/PhiloLabs/fable51-worlds/tree/main/kyoto-higashiyama https://github.com/PhiloLabs/fable51-worlds/tree/main/kyoto-...
- deadbabe 3mo agoI don’t really care about the real world, I want someone to make a thing where you take scenes from movies and shows and can then walk around in those worlds constructed by AI, to very high fidelity, and maybe in VR. Someone go do this, I’ll be on the lookout for the hackernews article.
- 1saadcodes 3mo agoThis is really cool, but $33 for a single generated world makes it hard to see this being useful for games just yet. Not to mention how this would go in a much larger project
- olmo23 3mo ago$33 is expensive? Compared to what?
- iammjm 3mo agoThere are free open source tools that convert osm data into 3D models, for example https://osm2world.org/ https://osm2world.org/. Also you could use freely available national 3d point cloud data, at least for some areas. both are $0 options. btw I am not saying $33 dollars here is expensive; just giving you some comparisons
- 3ds 3mo agoY U NO github pages? Or where can I explore the demo app? It seems like it's just a static page! All you need to do is create the gh-pages branch! And then you have it at https://philolabs.github.io/fable51-worlds/kyoto-higashiyama https://philolabs.github.io/fable51-worlds/kyoto-higashiyama
- tintor 3mo agothis is not a world model
- RugnirViking 3mo agowhat's going on with these overlays? the readme is presenting those as though they are evidence of the accuracy of the project, but its completely unable to line up the shots https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/08-compare-vp10-saks.jpg https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/09-compare-vp09-macys.jpg https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
- MyFirstSass 3mo agoAI generated everything. Dude probably didn't even look at the examples.
- half_fish 3mo agoGood work! I'm curious about how much token consumed to have this... Would it be a business secret?
- larodi 3mo agoThis demo is based on the wonderful (and actually hand-crafted) Three.js. It is a beautiful demo. The model expounded on the prompt and produced a very convincing web product based on Three.js. Wonder why wouldn't author try to produce this in plain WebGL... my guess - it is much harder and costly to do so. So, I wonder when is Anthropic (or OpenAI (or... NAME_IT)) going to release one single stable super-working library that does... just anything, that we can then reuse the way these guys demonstrate somebody model's caps, while actually standing on the shoulder of giants. Because they do stand on the shoulder of gigantic work done by Three.js team. Same goes for demos based on D3, imgui, etc.
- tomaytotomato 3mo agoI am making a spiritual successor the Rainbow Six Rogue Spear Opus has been great at building the game engine but it does struggle with world building for me. Any tips welcomed!
- gshklovski 3mo agoNice! I had a similar positive experience treating OpenStreetMaps as the highest tier data source and then using screenshots to tune the aesthetics. I wonder how high fidelity we can get these views using just ThreeJS and Fable5.1 iteration cycles. Here is my Palisades Tahoe world that I made in a similar way: https://ski-red-dog-face.vercel.app/ https://ski-red-dog-face.vercel.app/
- santiago-pl 3mo agoDid anyone compare it with gpt-6?
- surreal_ 2mo agoyes we added the side by side comparison, check out the readme!