8 ms·
Show HN: Stable Diffusion powered level editor for a 2D game
Hey folks, I’ve been working on using control-net to take in a video game level (input as a depth image) and output a beautiful illustration of that level. Play with it here: dimensionhopper.com or read the blog post about what it took to get it to work. Been a super fun project.
- Ireallyapart 3y agoThis is good for procedural generated 2D worlds. Think Hollow Knight, but expansive across infinite environments. Just randomly generate the control image and have the LLM generate the theme. Combine that with LLM generated lore and the possibilities are unlimited. We have the technology to do this right now.
- arijo 3y agoAny books, articles you could suggest?
- Ireallyapart 3y agoThere's no books man. This stuff is too new. But all the components are in place and exist. This guy just demonstrated what's required to generate a theme, and it's not far off from extrapolating further from that in using a LLM to generate Lore and just some random maze generator to create the base control image.
- baq 3y agoYeah. Show this to the Dwarf Fortress team. I hope they're already working on it anyway.
- Ireallyapart 3y agoDoubtful. The developer is really old school. Additionally he's already spent years on integrating his huge lore generator into the fabric of the game. An LLM will throw a wrench into the whole process.
- 6510 3y agothink random events that change the entire world
- robobenjie 3y agoYeah, in the map editor there is, in fact a random button that generates. I havn't gotten around to making sure that the random level is playable (and about 1 in 4 have unreachable areas) but that wouldn't be that hard to add. (I've been focused on the creative aspect of creating your own levels because right now that part is more fun).
- zokier 3y agoI have far more simpler (I imagine?) case already in mind, from the recent Cities discussion thread: >> I would've expected at least a not grid-based zoning so that buildings on curves look more natural. All these empty pieces of land in between buildings look really bad and kind of force us to make grid cities. And that is not even an innovation, it was already present in the SimCity series. But some procedurally generated buildings for smooth corners and connecting buildings would be nice. > Its hard to make assets that would work with every curve. When you see screenshots of nice cities like this, people are using mods to hand place assets with them clipping into each other to make a unified wall of buildings along the curve or corner. https://news.ycombinator.com/item?id=36294742 https://news.ycombinator.com/item?id=36294742 ML generated building configurations for city builder games. Readily adaptable to any shape, and as a bonus can break up excessive repetition a bit. If you want to be ambitious, train a model on real-world aerial photos.
- mywittyname 3y agoWould this require ML? I would think this could be accomplished with some rudimentary procedural generation. Divide the large space up into building sized plots, then generate a building to fit the space.
- erezsh 3y agoVery cool! When building it, did you encounter any unexpected obstacles, or found any cool tips that you'd like to share?
- robobenjie 3y agoThe main one is that making the control-net depth input look like something helps a ton. You can creating levels that have more 'structure' (large flat platforms, platforms that line up with others) and levels that are more random and see that the structure works way better. I played around a lot with turning the control-net on and off at the beginning and end of generation, which seemed to help when I was playing in the webui but then I didn't immediately find the API in diffusers and the results I was getting were great so I didn't keep looking.
- q_andrew 3y agoDepth is a useful parameter for controlnet, especially when you want really specific forms. I've found that it can hamper outputs because blank sections of solid color are interpreted as flat walls, when really I'm trying to make those parts ambiguous!
- mock-possum 3y agoSeems like a good source of cheap visual design!
- robobenjie 3y agoSure, but I'm more interested in things that were just impossible before. You can hire a artist to illustrate a level, or have AI do it cheaper, but you can't have an artist illustrate a level the player made while they wait. I think there are whole play patterns that are possible because the cost and especially speed of creating the art are many orders of magnitude different.
- zacharycohn 3y agoI didn't think of that while reading your article. You should plaster that comment all over this project to help people see the possibilities.
- robobenjie 3y agoGreat feedback. I added it to the start of the article. Thanks!
- fnordpiglet 3y agoI think even more interesting is generating entire styles and story lines that evolve infinitely and coherently based off of seeds. Players could even inject a concept - “steampunk” or “discworld” and an LLM could construct the story, with characters, and visual themes.
- robobenjie 3y agoSeveral of the themes (including "alien jungle", which is my favorite) we're created by chatGpt. I totally want to try evolving the game in that direction.
- kraftman 3y agoI think this could be great for letting the players design custom weapons/Armor/spells very roughly and then using AI to convert it to something that looks good in the game
- nico 3y agoThis is such a cool project. Very inspiring, thank you for posting it Great example of using AI as a tool to make something exceptional
- robobenjie 3y agoThanks! It has been a ton of fun too.
- ionwake 3y agoamazing advances this year. Remember the guy who created the 2d platformer thats based on time, what was it called again? He spent around $100k+ just for the art, which I am pretty sure was a huge expenditure for him, with this software he could have done it virtually for free without much artistic talent at all.
- DeRock 3y agoThe game was called “Braid”
- drorco 3y agoAt the quality of the current output, I think players still easily differentiate between AI generated art and hand-created art. Maybe in future versions this will be less noticeable. As a game dev, I think at this stage AI can be a helpful utility, but it does not replace a designer's touch for professionally looking games.
- AlecSchueler 3y agoThe AI stuff is a style though. I'm seeing it happening now in the art world, where the quirks of the model become part of the appeal of the work. Won't be long until a game with good enough mechanics comes along and blows up I think.
- __loam 3y agoFor me the model "quirks", eg. all the mistakes they make, are a huge turnoff.
- NBJack 3y agoFor better or worse, I agree with your sentiment, but that will probably change. Consider how many kinds of foods and clothing are mass produced; we often consider something made by hand to be precious, and even a higher value, but we have become accustomed to the tradeoffs for cheaper solutions. It may not be our generation, but it's conceivable future generations will be less inclined to differentiate as we do (if only based on the exposure to what this kind of art generation offers at an early age).
- robobenjie 3y agoI'm also curious if anyone has made a level that worked particularly well/poorly or has a great custom theme (that maybe I should add to the dropdown) :)
- spywaregorilla 3y agoI wish this was around when I was a kid :(
- fnordpiglet 3y agoProtip: you don’t have to grow up
- gadflyinyoureye 3y agoWhy not dimension jump? https://www.reddit.com/r/DimensionalJumping/ https://www.reddit.com/r/DimensionalJumping/
- anhner 3y agowhat the fuck did i just read
- gadflyinyoureye 3y agoThe internet is a weird and wonderful place.
- spywaregorilla 3y agoIt appears to be a group of people who are more willing to believe in transdimensional travel being responsible for changes in their life than things happening as a result of their outlook, luck, or their own agency.
- deleted 3y ago[deleted]
- tltimeline2 3y ago[flagged]
- tweetle_beetle 3y agoI feel like Jump 'n Bump probably has a special place in the hearts of people who had access to the internet at a particular time. The internet was available, but multiplayer online gaming was still out of reach for many with it - there was an amazing niche of fairly polished indie local multiplayer games. Imagine being told while playing it back then what would be possible a few decades later. (see also https://github.com/midzer/jumpnbump https://github.com/midzer/jumpnbump) Now need someone to do other games of that time, place and genre: Tremor 3, C-Dogs, etc.
- xyzal 3y agoWow, I nearly forgot C-Dogs! Played the hell out of it on my 386. Later, when I got my first Pentium, the same with Tapan Kaikki.
- bsenftner 3y agoLove to see a write up on your Hugging Face Diffusers experience, setting that up, what your dev cycle & stack look like, if you're hosting that server on a GPU cloud instance or what. Those kind of details are very interesting.
- robobenjie 3y agoI'm on an EC2 instance. Most of the effort was on cuda/pytorch/pip nonsense. Once stable diffusion webui was working, diffusers worked basically out of the box, which was really nice. (Trickiest bit was figuring out that I needed to use their tool to convert my safetensor file and that the version of python I was using wasn't working with it for some reason). Stack is flask + gunicorn which was what chatgpt recommended (lol). I had a websockets version of the progress bar working on flask-socketio on my local machine but could never get the server version through nginx to work correctly. So eventually I just gave up and switched it to polling so I could launch.
- thih9 3y agoDoes the 2d data like platforms and hit boxes still match the input entered by the human? If yes, I wouldn't say this is using AI for level editing, this seems using AI for level artwork generation. Impressive nonetheless, just different. HN's submission title ("Show HN: Stable Diffusion powered level editor for a 2D game") made me think of the former. Article title ("2D Platformer using Stable Diffusion for live level art creation") was more accurate to me.
- AlecSchueler 3y agoYour quote says represent the levels not generate the levels right?
- porcc 3y agoTo get even less "structured" backgrounds you should try replacing the support backgrounds and the far backgrounds with a light bit of depth noise.
- robobenjie 3y agoWhile I have folks attention, I want to try training a model to generate monster/creature walk animations. Anyone know of a dataset of walk cycle sprite sheets that I could massage and label to see if I can make that work?
- psychphysic 3y agoDoes https://opengameart.org/ https://opengameart.org/ work?
- __loam 3y agoMight be nice to actually ask for permission here before scraping everything and feeding it into a training set.
- bluefirebrand 3y agoThat seems counter to the current AI mentality though. Clearly if it's online and available it's part of your AI playground so go nuts. Edit: In case my sarcasm isn't clear, I hate this mentality and I am just bitterly griping about AI into the void. You should definitely ask permission to use data before training AI on it, but that will put you behind other AI people who aren't asking permission
- TillE 3y agoEverything on OGA has a license attached. Just respect the appropriate licenses and you're fine.
- __loam 3y agoI think the problem is that the people doing the scraping often disrespect the license as we saw with co-pilot
- nightski 3y agoIt's not a full dataset per say, but you might be interested in Sebastian Starke's DeepPhase work [1]. [1] https://youtu.be/wAbLsRymXe4 https://youtu.be/wAbLsRymXe4 [2] https://github.com/sebastianstarke/AI4Animation https://github.com/sebastianstarke/AI4Animation
- jamesgreenleaf 3y agoI wonder how far off we are from something like the battle school game from the book Ender's Game. That is, an immersive video game that uses player actions, choices, exploration etc. in order to generate not only new content, but entirely new game rules on the fly. It feels like we're getting closer and closer to Ender's holographic terminal with VR interfaces + AI content.
- bemmu 3y agoWhile not the dreamy part you asked for, you can already play the Zero G battle part in “Echo VR”.
- duringwork12 3y agowe are building something like that but the images are sadly static per story page and it has no psychological analysis. I've been interested in adding pupil dilation as a measure of cognitive load as a signal on one of the things you read but i think customers would find it too creepy.
- TalEs 3y agoWith Apples new Vision Pro, the ability for it to do iris scans and also view the content in VR, your idea might actually be very interesting, with user consent of course.
- MattRix 3y agoIt’s not that hard to get these LLMs to generate new game rules, I’ve made some prototypes that use them to do that. The hard part is getting them to generate rules that are actually fun to play and somewhat balanced. The fact is that it’s hard to know if something is fun until you actually play it.
- jakearmitage 3y agoI've recently tried using InvokeAI to apply a specific style as a texture mod for the original Max Payne, along with RTX Remix. Instead of making the textures "modern", I was attempting to mix them into a noir rendition, similar to Sin City, but less cartoonish. Unfortuantely, it was really hard to get InvokeAI to restrict to the UV boundaries, with data always leaking and not looking really good when rendered.
- nagonago 3y agoI was going to comment that that the contrast between the beautiful illustrations and red blood that violently explodes when you kill the opponent is pretty funny. But then I looked up the original Jump 'n Bump and it's just as gory, if not more! Good ol' 90s games.
- Agentlien 3y agoI played a lot of Jump 'n Bump as a kid. It came with one of André LaMothe's game programming books.
- creativenolo 3y agoWould love to see a video of the game
- MagicMoonlight 3y agoWhen they start making games like GTA with generative AI we will get to the real fun
- ilaksh 3y agoYou could do something similar with a 3d game template and a way to describe scene additions or changes via API calls. I've seen a demo like that.