6 ms·
Fable 5.1 World Modeling
- surreal_ 1mo agofable 5.1 generated an interactive 3D union square, and the agent filmed its own tour guide vid inside it. you can walk Powell to Stockton, read the actual storefronts, cross a working intersection, watch a cable car go by. went inside Apple and the Nintendo store, lower level included check source code + more worlds soon
- giancarlostoro 1mo agoHad to vouch for your comment, for some reason it was not visible. > Geometry is derived from OpenStreetMap (ODbL) and USGS 3DEP (public domain). This is probably how I would have tackled this, saves on time, especially if Fable just writes code to convert OSM data to a reasonable ThreeJS model.
- andy99 1mo agoDo you have data on the tokens or $ cost of this?
- surreal_ 1mo agothis is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/PROMPT.md https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop. ~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
- jaybuff 1mo agoThis isn't what I expected when I heard "world model"!
- surreal_ 1mo agoik, "world model" has too many definitions at this point. imo the bar is just: can it simulate the world.
- benatkin 1mo agoNot really. Whether you agree with LeCun's assertion that "LLMs Are a Dead End" or not, world modeling is the same domain. "World Modeling" in this post is confusing because it seems to reference the deep, specific machine learning that goes into these frontier world models and self-driving cars, where it's really just vibe coding turned at the problem of modeling the sort of 3D worlds you'd find in a video game.
- hliyan 1mo agoAgreed. Author should not be using this term. A world model, as I understand, is what an LLM is missing that a human brain has: in addition to a linguistically described world, we also posses a sensory model of the world, a physics model so to speak, against which the random token/thought chains generated by our brains can be validated.
- deleted 1mo ago[deleted]
- Reebz 28d agobeing reductive, causality is the key term for a world model. the rollout is the key action to demonstrate causality. lots of nice LLMs can generate a 3d representation of a "visual world". These days the focus is on sensory data (image, video, etc.), however world models can be more than just 3d spatial representations - they could be tabular. Again being reductive, its how do you predict the next state from the current state (the rollout).
- hadlock 1mo agoThis is the kind of demo that would really benefit from a youtube demo of 2-3 min Fake edit: there is a longer video here 1 min long: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/union-square-walkthrough.mp4 https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo.
- Willish42 1mo ago> I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo. Here ya go: - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/src/life/Pedestrians.ts https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/src/life/Traffic.ts https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
- avdelazeri 1mo agoI wish you would provide more information. About time, costs, reliability (how many duds did you have? did you need to do any babysitting or etc) and so on But it is a very impressive looking demo, for sure.
- surreal_ 1mo agothis is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union- https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop. ~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
- threecheese 1mo agoDid you need to iterate on the prompt, or did you have a model help you author it? I frequently have problems with orchestration instructions in-prompt, and your is huge. Maybe this is just better with Fable? I honestly haven’t used it much.
- Willish42 1mo agothis is really neat. I wonder if people will start building open world games or "AR" games in the vein of Pokemon Go and Ingress based on similar tech. I've had a similar idea for a long time but don't think it was feasible before now due to AI
- surreal_ 1mo agoyes we're working on it! trying to push a few open world rpg games with real economy and game design. it's also super interesting to benchmark the current models' capability in this direction, since this is a naturally hard and multimodal coding task
- somenameforme 1mo agoI'm not sure about expanding domains, but I expect that very close to 100% of games are now using LLMs to a very significant degree during development, even if most won't admit it. The "AI disclaimers" for games need to go away, simply because it's just rewarding and incentivizing lying and punishing truth.
- ramesh31 1mo agoCool. But the question remains, what is the point of this? I don't mean to pick on this project specifically, but it's indicative of AI usage in general. You end up with something that's like "wow an AI made this" but it's not really anything that interesting or useful in and of itself. I've yet to see a single agent generated piece of software above that bar.
- dogma1138 1mo agoBecause it’s a very complex task to do especially with any level of quality and consistency. Classical algorithmic approaches for data to CAD are quite unreliable heck commercial CAD software still struggles turning scanned 2D plans into sketches with any degree of reliability. Something of this scale would normally require 1000’s and 1000’s of man hours in manual modeling, placement and quality control. Some of the latest prompt to CAD agents and models I’ve seen are the most impressive use of AI I’ve seen in a long time. At least for me this is rather exiting since it gives me the ability to turn many more ideas into reality as a hobbyist rather than spending the entire weekend in Fusion 360.
- trees101 1mo agoI'm interested in this, what is the best current method/tool for this task?
- jryle70 1mo agoWhy don't you try to do it yourself with Fable and compare it with this one? Even with AI there are still a lot of work, knowledge, and gotchas involved. What's the point of such comment?
- ChickeNES 1mo ago> I've yet to see a single agent generated piece of software above that bar. Begs the question, an earnest one: what software would be above that bar?
- surreal_ 1mo agothe topology/texturing critique is fair for mesh generation, but code-generated worlds mostly sidestep it. when the model writes three.js or blender scripts, geometry comes from primitives, csg, and parametric construction, so topology is clean by construction rather than something you clean up after. texturing is still a gap, agreed, though procedural materials cover more than people expect.
- vopi 1mo agoCool demo! That's funny. I literally sketched out something similar 30 minutes ago on paper. I wonder if there was some recent, semi-related article/link that subconsciously prompted the relevance.
- surreal_ 1mo agowe're working on a survey paper about world modeling via code, with folks from frontier labs (qwen omni, oai, etc) and academic institutions (e.g., oxford, stanford, etc) reach out to us about collab: team@philolabs.ai
- T-A 1mo agoThis, maybe? https://news.ycombinator.com/item?id=49525160 https://news.ycombinator.com/item?id=49525160
- kodefreeze 1mo agoI have experimented with this type of modeling for a RTS game I'm building: https://playwarpact.com https://playwarpact.com Few things: 1. Opus 5 is just as good for this tbh and cheaper. 2. They don't generate optimized 3d models. They have high poly count for simple geometries. 3. A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes of the 3d model and then bake textures that include a lot more details like windows, doors etc with tools like Meshy. Can post a tutorial if there's interest. The models need some more RL to be able to do this autonomously.
- vismit2000 1mo agoRegarding your game shared here: https://news.ycombinator.com/item?id=48943351 https://news.ycombinator.com/item?id=48943351
- andai 1mo agoWait so Opus modeled your cool scifi buildings?
- deleted 1mo ago[deleted]
- lukan 1mo ago"Can post a tutorial if there's interest." There is.
- kodefreeze 1mo agoWill write it up!
- ElFitz 1mo agoThanks!
- kodefreeze 25d ago
- julien_dev 1mo agoThese things always look impressive but I fail to see how they are usable beyond demos or simple games. In my experience the topology is often a mess and texturing is difficult as well
- eagleinparadise 1mo agoi discovered doing this in claude design and i can tell you... it's useful in a professional domain. Although I am not going to share what I'm using it for, to keep an edge in my domain ;)
- ryan_n 1mo agoDon’t worry I’m sure it’ll be in the training data soon enough…
- nomel 1mo agoWell, nothing now starts in a completed state. See pre-production footage for movies and video games, for how scenes progress. Uncharted 4: https://youtu.be/NedDxIGQVs0&t=147 https://youtu.be/NedDxIGQVs0&t=147 Coco: https://youtu.be/nl_JkjgHfFU&t=10 https://youtu.be/nl_JkjgHfFU&t=10 Up: https://www.youtube.com/watch?v=JHfLGgOs6gY https://www.youtube.com/watch?v=JHfLGgOs6gY And, eventually you can just run it through Nvidia DLSS 7 for better hair physics and textures. ;)
- breadsniffer01 1mo agoWoah amazing clips thanks for sharing! The GH repo is definitely in early stages and prob laying the ground work for using the real world as part of the pre production process for a game/movie, etc.
- pelagicAustral 1mo agoBe nice to turn some of these into Counter Strike maps... mhmm...
- lanyard-textile 1mo agoMaking callouts by SF street intersections, lol
- deleted 1mo ago[deleted]
- bb88 1mo agoThis reminds me of the experience VRML wanted to be 3 decades ago. The only difference is that links went to other worlds living on another server. But that seems trivial to implement with this.
- bastawhiz 1mo agoVRML was incredible. It's a "hypertext" markup language but instead of documents it produces 3D environments. It's a shame it never really caught on.
- kevthecoder 1mo agoThere's some work going on at the Metaverse Standards Forum to progress these ideas with Web of Worlds (https://webofworlds.github.io/ https://webofworlds.github.io/). It's scope is a lot broader than VRML, it covers assets, identity, avatars, payments, etc. There's a link to the presentation at SIGGRAPH - https://youtu.be/N2_lb77gKQ8 https://youtu.be/N2_lb77gKQ8.
- ProofHouse 1mo agoI know a guy in LA who basically spent every waking minute of his life past year or two building a spec for this. Announced soon apparently. I’ve spent hours reading some of it. Pretty damn compelling.
- bb88 21d agoA little late I know, but yes it was. The problem was the hardware didn't really support what VRML could do at the time. I had access to an SGI irix workstation and VRML (and 3d scenes) was still challenging for it back in the early to mid 1990s.
- gimlids 1mo ago“Fable 5.1 World Modeling” is an incredibly misleading title! This has very little to do with Fable and using an unexpected sense of “world model”
- minimaltom 1mo agoThis is beautiful! I would love if we could model out a whole city, ideally using a much cheaper model.
- mrinterweb 1mo agoI'm really curious how GML-5.3-flash would do. Very affordable, and it seems to do pretty well with 3D modeling.
- surreal_ 1mo agoyes experimenting with it actually, will update here! in fact we've generated most of the tourist spots in sf, should be reflected in the repo soon too
- minimaltom 1mo agoHell yeah, whats your thoughts on doing it for like a whole city?
- throwaway613746 1mo ago[dead]
- ameddserM 1mo agoThis looks great.
- lauralin 1mo agoVery good
- hartator 1mo agoStarted the Matrix
- heyl987 1mo agosimulate the universes and then observe the parallel universes
- luciana1u 1mo ago[flagged]
- WordNotSee 1mo agoFundamentally it’s an image model predicting out frames of a sequence. It’s not enough to call it just an image model though. Not quite video either because it’s so specific to rendering images of the world. Thus, “world model”? The problem I have with “world” is it should imply so much more depth. To me this is actually a “POV image model” or “First-person perspective model” (maybe we call this an FPM). There could also be TPMs (third-person models). This maps better to modern game and 3D dev phrasing, where “world model” might imply 3D object collections spatially organized in a cohesive file format - complete with interactivity, audio, and all the baseline constituents of what one might call a “world”. FPMs and TPMs, think about it. Reserve POVMs for photographic realism.
- RugnirViking 1mo agoI think perhaps you've misunderstood what the linked repo actually is? It seems to indeed be a 3d model of an area. It's not an image model, and nothing to do with frames.
- WordNotSee 1mo agoYou are correct - this is threejs and not video/generative rendering which I’ve seen a lot of. I guess “world” works if interactivity and sound etc. are considered other modalities. Video kinda has the same issue (for me) where the audio part of the video is not part of the model. When I think “world” I think more than just the visuals. Like a “film” model to me would imply sound and more than just the video. We typically call 3D art “models” too, so I guess it can’t be called a model model, idk the word “world” doesn’t seem right to me
- OldDrunkGuy 1mo agoI shitpost on this website constantly and trash Anslopic on a daily basis. I am your biggest hater. But god dammit if I’m not also your biggest respecter. reaches out for a handshake You guys are onto something with this one. Keep going. You crazy bastards! You crazy fuckin bastards you hear me?! Hahahahah WOO! WOO!
- deleted 1mo ago[deleted]
- chemael 1mo ago[flagged]
- surreal_ 1mo agoadded the hand-painted style kyoto: https://github.com/PhiloLabs/fable51-worlds/tree/main/kyoto-higashiyama https://github.com/PhiloLabs/fable51-worlds/tree/main/kyoto-...
- deadbabe 1mo agoI don’t really care about the real world, I want someone to make a thing where you take scenes from movies and shows and can then walk around in those worlds constructed by AI, to very high fidelity, and maybe in VR. Someone go do this, I’ll be on the lookout for the hackernews article.
- 1saadcodes 1mo agoThis is really cool, but $33 for a single generated world makes it hard to see this being useful for games just yet. Not to mention how this would go in a much larger project
- olmo23 1mo ago$33 is expensive? Compared to what?
- iammjm 1mo agoThere are free open source tools that convert osm data into 3D models, for example https://osm2world.org/ https://osm2world.org/. Also you could use freely available national 3d point cloud data, at least for some areas. both are $0 options. btw I am not saying $33 dollars here is expensive; just giving you some comparisons
- 3ds 1mo agoY U NO github pages? Or where can I explore the demo app? It seems like it's just a static page! All you need to do is create the gh-pages branch! And then you have it at https://philolabs.github.io/fable51-worlds/kyoto-higashiyama https://philolabs.github.io/fable51-worlds/kyoto-higashiyama
- tintor 1mo agothis is not a world model
- RugnirViking 1mo agowhat's going on with these overlays? the readme is presenting those as though they are evidence of the accuracy of the project, but its completely unable to line up the shots https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/08-compare-vp10-saks.jpg https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... https://github.com/PhiloLabs/fable51-worlds/blob/main/union-square-sf/media/09-compare-vp09-macys.jpg https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
- MyFirstSass 1mo agoAI generated everything. Dude probably didn't even look at the examples.
- half_fish 1mo agoGood work! I'm curious about how much token consumed to have this... Would it be a business secret?
- larodi 1mo agoThis demo is based on the wonderful (and actually hand-crafted) Three.js. It is a beautiful demo. The model expounded on the prompt and produced a very convincing web product based on Three.js. Wonder why wouldn't author try to produce this in plain WebGL... my guess - it is much harder and costly to do so. So, I wonder when is Anthropic (or OpenAI (or... NAME_IT)) going to release one single stable super-working library that does... just anything, that we can then reuse the way these guys demonstrate somebody model's caps, while actually standing on the shoulder of giants. Because they do stand on the shoulder of gigantic work done by Three.js team. Same goes for demos based on D3, imgui, etc.
- tomaytotomato 1mo agoI am making a spiritual successor the Rainbow Six Rogue Spear Opus has been great at building the game engine but it does struggle with world building for me. Any tips welcomed!
- gshklovski 1mo agoNice! I had a similar positive experience treating OpenStreetMaps as the highest tier data source and then using screenshots to tune the aesthetics. I wonder how high fidelity we can get these views using just ThreeJS and Fable5.1 iteration cycles. Here is my Palisades Tahoe world that I made in a similar way: https://ski-red-dog-face.vercel.app/ https://ski-red-dog-face.vercel.app/
- santiago-pl 1mo agoDid anyone compare it with gpt-6?
- surreal_ 27d agoyes we added the side by side comparison, check out the readme!