13 ms·
Building a Distributed Turn-Based Game System in Elixir
- AlchemistCamp 5y agoThis post is essentially an ad on Fly's blog, but maybe some others with experience can weigh in on its tradeoffs vs the other popular providers with first class Elixir support. I haven't used it yet and am curious about new offerings. That said, the guide linked from this post requires more config than the competing options. For example, Render also handles clustered deployment with libcluster setup, and it's very, very simple to set up. http://render.com http://render.com
- mrkurt 5y agoThe thing we do that's (maybe) interesting is run processes all over the world. It's helpful for liveview because it minimizes latency to end users. Here's an example Liveview cluster running in 17 regions, if you scroll down you'll see the round trip time to the server you're connected to: https://liveview-counter.fly.dev/ https://liveview-counter.fly.dev/ Liveview is crazy powerful if you can keep the latency down.
- AlchemistCamp 5y agoLiveView should already have less latency than an app doing traditional AJAX requests, since an open web socket is a lot better than making a call to axios! The less latency the better, though, of course. I'm seeing roughly 80ms, which is very good.
- watermelon0 5y agoHTTP with keep-alive supports keeping around idle connections, which can be used to make subsequent AJAX calls. HTTP/2 has this by default, and allows multiplexing multiple requests on a single connection, so even making multiple parallel calls won't require establishing new connections. Basically, performance between AJAX and web sockets should be comparable in most cases.
- msmithstubbs 5y agoNice demo! How is the server chosen? I'm seeing ~200ms latency. I'm connected to LAX which I don't think would be shortest round trip.
- mrkurt 5y agoWhere are you connecting from, out of curiosity? 200ms is poor! Fly apps get anycast IPs, so it's basically bgp getting you to our "nearest" edge proxy, then we connect you to the closest vm from there. https://debug.fly.dev https://debug.fly.dev will show you which region you hit first (the fly-region request header). --edit-- I snooped on your HN profile and found Australia. We don't have an instance of this app up in AU, but we could. You're likely connecting to us in Sydney, and LAX is most likely the fastest over our backhaul from there.
- avinassh 5y agoI am connecting from India and my latency is about ~180ms
- mrkurt 5y agoIn a couple of months we'll have instances in Chennai!
- Moru 5y agoI'm connecting from Sweden and got about 40 ms. But now and then it peaks to 4000-8000 ms while my ping within the city stays at 2 ms.
- mrkurt 5y agoThose spikes are hilarious, they're probably from a liveview DoS. People are fond of using the debug console to "click" hundreds of times per second and there's no rate limiting in the app.
- MawKKe 5y agoFrom Finland: Through LTE I get served via 'fra' with ~60ms (as one might expect). However if I switch to Wifi I get 'hkg' with ~260ms?? Not exactly next-door :)
- iudqnolq 5y agoFyi at least on mobile the buttons stay focused after tap, making it look (incorrectly) like the ack is lagging.
- matthewcford 5y agoI like that you can mount a volume to a region, render can only do a single instance.
- brightball 5y agoThat is extremely interesting. I've never seen a cloud provider that prominently advertised Anycast IP addresses.
- valzam 5y agoI always think it's amazing how much simpler your architecture can get if you don't move any and all state to some other service in the name of keeping your app stateless. Of course there are good reasons why the industry has moved towards stateless business logic services but with all the BEAM goodness (resiliency, hot code reload) it might be feasible to write apps that actually hold state again.
- mrkurt 5y agoApps with state are wonderful to work with. I'm pretty excited to build apps that cache in process and don't rely on 47 external services to do CRUD.
- robotmay 5y agoAgreed. Elixir/BEAM gives you some really interesting new ways of solving problems. A couple of years ago I built a subscription system that interacted with Stripe that only kept the state whilst a user was interacting with the system (used a GenServer as a read-through/write-through Stripe API cache that exited after they stopped using it). It was by far the least irritating subscription system I've ever worked on.
- cpursley 5y agoInteresting. Do you mind (and if you can) elaborate on how you model that subscription system?
- robotmay 5y agoSure thing! Basically I set out to use Stripe's built-in tools as much as possible, rather than duplicating the state on our side - storing partial data in your own DB is one of the biggest pain points of subscription systems in my experience. So for this app it didn't have a DB at all - Stripe was the source of truth. The way it worked was fairly straightforward: a user would click a link from their account area and it would include their Stripe customer ID in the request (all our users had already had accounts created due to our main booking system). The app then span up a new GenServer to represent that customer, which would pull down all the data it needed from Stripe's API as soon as it initialised. There was also some generic data stored in a "global" GenServer to cover stuff like plans etc (I can't remember how I had that refreshing, probably via webhooks). Then as the user went through the subscription process or management process, any changes they made would be made through calls to their own personal GenServer, which on write would first write to the Stripe API, then update its own cache with the new state returned from Stripe to ensure consistency. These GenServers were kept alive by a timer set on interactions with it, and would automatically shut down 30 minutes after the user stopped interacting with it, discarding all the data it held. Then when they next return, it fetches the state from Stripe again. It also listened for webhooks and would update running GenServer instances with data it received to ensure they were consistent, and just ignored any for users that weren't currently running. Overall I was really happy with it - it was really performant due to the data caching but also didn't suffer from staleness issues :)
- laviroman 5y agoElixir and especially Phoenix is such a blessing. I recommend everybody to try them.
- joelbluminator 5y agoThere are no Elixir jobs where I live, waste of time.
- laviroman 5y agoThen start looking online or start your own business (using Elixir because it is the easiest and least money-expensive for a single developer) and stop blaming the whole world. Internet is everywhere. So, if you can't move to a bigger city to find Elixir jobs, look for the remote position, working from home is pretty normal for at least a year now. Or, as I said, start your own online projects using Elixir. Again, using Elixir and its ecosystem is the easiest for a single developer. To be honest, people who operate in the Elixir ecosystem are often much better programmers (and thinkers; a much welcomed side product of functional programming) than JavaScript people where the quality of code is usually much worse. But it's partially the problem of JavaScript where it's easier to write bad code. And Elixir is much better especially for big projects than JavaScript. Maintaining a big project in JS vs Ex? Elixir is much much better at that. Programming things in JavaScript is torture. In Elixir it's pleasure. It's like the improved version of Ruby, very nice language.
- deleted 5y ago[deleted]
- cpursley 5y agoYeah, I don't get it. I've worked remotely with Elixir for several years from an non-tech oriented Eastern European... And these days it seems like more than half dev jobs are remote first.
- deleted 5y ago[deleted]
- 5y ago
- edrobap 5y agoI thought they have used TURN [1] to create a truly distributed p2p app. But it turns out that they have build a cluster of servers to possibly improve the scalability by distributing load. It not clear how they have partitioned the data to distribute load. If all state are eventually going to get transferred to each node, it may not achieve much by having cluster of nodes. [1] https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_...
- jayunit 5y agoI'm curious about how race conditions would be handled when multiple users, on different regional LiveView servers, take conflicting actions. In the "Let's walk it through" section, it seems like the Player-to-LiveView connection will process user input (e.g. a Tic-Tac-Toe move) and update the UI to acknowledge this, at which point the user can be assured that the LiveView server accepted their input. But it seems like this happens before the GameServer has also accepted the input. What if Player 2 made a conflicting play and their change was accepted by the GameServer before Player 1's change reached the GameServer? Given, in Tic-Tac-Toe, the game is simple enough that this is neatly avoided: each regional LiveView server has enough information to only allow the current player to make a play. But in more complex applications, how might you (anyone; curious for discussion) handle this? One answer is something like: The LiveView server is effectively producing optimistic updates, and the GameServer would need to produce an authoritative ordering of events and tell the various LiveServers which of the optimistic updates lost a race and should be backed out.
- laviroman 5y agoIt's the same as with your mobile phone when you lost your wi-fi signal. Everything pauses and everybody has to wait. Have you played games like HOMAM 1 or 2? You can't do anything when the CPU is playing its players. You can watch where he goes and what he does but that's it. When he is finished then you go. When there is a network error - some message or please wait... or loading spinner message should be shown in the meantime. For turn based RPG games or Chess etc. this is a non issue. Of course, real time action games etc. is not a good idea for this technology.
- 1_player 5y ago> What if Player 2 made a conflicting play and their change was accepted by the GameServer before Player 1's change reached the GameServer? Not sure I understand the question but, I don't see how this would happen. On the BEAM it's processes all the way down. There's a process for that instance of the game, which is basically a big state machine, and 2 processes representing the client state, one for each player. When the game (process) starts, it expects a message from player 1 (process), then one from player 2, and so on. If there's a client timeout or network disconnection, the player process affected crashes, and if the app has been architected well, the other player process and game process are in a supervision tree, so they crash as well, perhaps notifying the other player that the game has ended because of a disconnection from the other peer. But none of this will accept a move from the player 2 when it's player 1's turn.
- POiNTx 5y agoHow would you handle animations and more interactive page elements? Is it possible to reach the same kind of UI as something built with Svelte/Vue/React or are there certain limitations?
- mrkurt 5y agoIf you're within ~50ms of the liveview server you can get really close to client side interaction responsiveness. Animations are still better off handled client side, though you can go far with web components and liveview updated HTML: https://github.com/hmans/three-elements https://github.com/hmans/three-elements
- 1_player 5y agoDon't need to be so close. AFAIK when Live View was announced, Jose Valim and Chris McCord tested a 60 fps server side rendered animation from across the pond and it was perfectly smooth. Definitely not it's intended use case, but it works perfectly on regular Internet latencies in the 100-300ms ranges.
- sa1 5y agoServer rendered animation sounds less susceptible to latency than interactive elements.
- 1_player 5y agoHow so? 60fps means it requires constant updates every 16.6ms or you'd get a very noticeable frame delay or skip. Again, not really the best use of Live View, because it'd be very hard to guarantee that stability for all users in different Internet conditions. Whereas interactive elements just require an update in 100ms or 200ms after the user pressed a button, which is not that hard to achieve.
- sa1 5y agoWe’re not measuring latency due to processing here. We’re measuring latency due to round trips. For server rendered animations, all the frames might be delayed due to geographical distance, but the time gap between the arrival of individual frames is not affected by geographical distance. The server is not waiting for the client to request the next frame, it’s always pushing. For the user, it’s similar to watching a streaming video after letting it buffer a bit. For interactive elements, every interaction has to go through the round trip. So the gap between interactions is affected by distance. Server rendered games would suffer much more than server rendered animation.
- Darmody 5y agoThis is cool. I was thinking of using Elixir/Phoenix for a browser game. A different concept, though. My first choice was node but the more I use Javascript the more I hate it.
- Rodeoclash 5y agoI build the majority of my speculative projects in Elixir these days. I did want to try something new but after a few hours of fighting NextJS / Typescript / Vercel / Serverside rendering I just gave up and went back to Elixir again. It's fantastically productive.
- Darmody 5y agoI'll dive a bit this weekend into it. I was hesitant but apparently everybody is super productive and super happy with it so I have no choice.
- tomgp 5y agoI’ve been thinking of trying out elixir for exactly this kind of thing. Having done similar projects in Nodejs (experience was fine tbh) and used Xstate or Redux for game state management on the server side. What would people recommend for that role in Elixir (or is it a case of roll yr own)
- andyjones11 5y agoElixir makes state management very easy by default. Elixir implicitly encourages you to write code in-terms of state, events and transitions - so managing game state can be done using the default tools/abstractions. When I write LiveView components I usually end up building a state-machine.
- gallexme 5y agoThere's also gen_statem which provides state machine like behavior ontop of genservers
- iudqnolq 5y agoDon't do this unless you've got some experience with alternatives. It's definitely not the standard in the Elixir community. Also keep in mind OTP (the Erlang stdlib) is sort of one company's kitchen sink helper lib. It's got some gems, but do read other people's opinions about which are worth using. I wish it's docs included more about when not no use it. Edit: Definitely look up "contexts". They're the Phoenix word for module that handle storing/retrieving a piece of data. If you're writing an app with low perf requirements consider using the db directly for this with no in memory persistence. I did this with a low usage internal dashboard and it's was dead simple. Edit2: Thanks to reply, previously said contexts were an Elixir concept
- gallexme 5y agoContexts are Phoenix specific. They are not the standard of the elixir community. Gen_statem is erlang standard, for state machines https://hexdocs.pm/gen_state_machine/GenStateMachine.html https://hexdocs.pm/gen_state_machine/GenStateMachine.html is a a fine wrapper for that If u have finite states I would go for a state machine
- gallexme 5y agoI'm currently building a distributed mmo emulator using elixir, and it's going well (performance/bug/effort wise). Distributed via horde/libcluster libraries, ranch as acceptor pool, using livebooks for debugging and performance testing. I couldn't really ask for anything better, as solo dev in less than 100h basic distribution solved(one process per map, one per player. Maps are spread around the erlang cluster using horde, if a map crashes players seeemlesly reconnect in less than a second, and the map may actually be on a whole different system, especially when the node becomes non viable) Basic monster ai, fighting logic. Respawn/monster drops, extensive ddbuggability using observer (gui tool included in erlang, allows me to send messages to processes/see their status/ what they do, maybe kill them in the whole erlang cluster ) Can just use livebooks to do more extensive programming (just connect to cluster, execute a mark down files code blocks, see results live ingame and in the livebook) It's a very great and easy language for this. I may consider moving couple of things to rust using rustler tho for the fun of learning about nifs
- dsgrillo 5y agoThis sounds quite interesting! Any chance you mind sharing that code on GitHub?
- gallexme 5y agoIt's on gitlab. But a lot stuff isn't on it yet/changes a lot, and I take my time :) https://gitlab.com/zen_core/zen_core https://gitlab.com/zen_core/zen_core And u need assets from the original game to use it I made a small example clip to show how I use livebook to simulate player load(they use exactly the same genserver as real players, just that the packets are already deserialzed ) https://cdn.discordapp.com/attachments/819516082154242048/833612125044867072/2021-04-19_09-54-22.mp4 https://cdn.discordapp.com/attachments/819516082154242048/83...
- klohto 5y agoDamn, HN would be the last place where I would expect to see Metin. Is the community still there? It was huge in Central Europe ~15 years ago.
- robotmay 5y agoEver since I heard about Fly I was really keen to try out something using Elixir. I've been severely lacking in ideas that I can complete within two weeks though (my maximum attention span)! This was well-timed for me - nice to have a reference working config to play around with, especially for clustering. How do you find the BEAM handles a distributed cluster across multiple regions? Are there any oddities that crop up due to inter-site latency?
- atonse 5y agoAfter clinging to my favorite stack for five years (elixir+ember js) my next project will ditch ember. LiveView is the nail in that coffin. It doesn’t help that Ember, though I STILL love the technology and framework, is deader than dead community-wise except for a few silos. It’s very tough to find talent. And the benefits just aren’t there anymore compared to using LiveView. I still use ember on about 8 projects but no more. Have others with this stack reached similar conclusions?
- henrysdev 5y agoGreat article. I'm currently building a turn-based multiplayer improv piano game for the web using Elixir/Phoenix. In a nutshell, you compete with other players on creating freestyle piano solos over backing tracks using a MIDI keyboard. Players and audience members then listen through the solos and vote for their favorite. The Elixir/Phoenix stack has been an absolute superpower for building my game backend. Some examples: - Phoenix channels as a wonderful abstraction over managing WS connections. Implementing basic chat took ~3 minutes on the backend. - The ability to model game logic with a finite state machine in a GenServer. The lifecycle of a game round is progressed forward by receiving incoming client events. - The ability to have many game servers running simultaneously as a dynamic pool under a DynamicSupervisor. Games can end and new games can be created, all isolated and under watchdog-like supervision. - ETS as an out-of-the-box memory cache for session data. Persisting user data between pages and sessions without needing to deal with an actual persistence layer. For the curious, the project is called Midi Matches and is currently in public alpha: Game (desktop chromium only): http://midimatches.com/ http://midimatches.com/ Gameplay Video: https://www.youtube.com/watch?v=UVD2wOCB_jE&t=41s https://www.youtube.com/watch?v=UVD2wOCB_jE&t=41s Repo: https://github.com/henrysdev/midimatches https://github.com/henrysdev/midimatches
- deleted 5y ago[deleted]
- voppe 5y agoI can't overstate how good Elixir + LiveView is, especially for prototyping. I have been experimenting with it ever since it first came out, mostly games, and I have to say that building even simple multiplayer games is a lot of fun. An example: https://dev.voppe.it/chess https://dev.voppe.it/chess (FYI there is no actual game here, just pick emojis, place them in the arena and watch them fight to the death via a poorly designed combat system. Yes, those placing emojis other than you are other players.) This was a prototype that I built to see if real-time multiplayer games were feasible. Apparently LiveView manages high tickrates decently enough to be a valid solution for (quasi) real-time games. This game for example runs at a whopping 8 ticks/s! It can handle more, as I've tried developing with faster tickrates, I think as high as 24 t/s, but I want to avoid server strain. With enough HTML/CSS/SVG wizardry you can get away with quite a lot. But the most amazing thing was the fact that there was no need to fiddle around with state syncing. Everything just works! I have nothing but praise for it. Try LiveView if you have the chance!
- isuckatcoding 5y agoI understood all the components but given I don’t do a lot of complex distributed programming , I kind of got lost about what Horde and libcluster do. My world view is very simple: backend app server and front end code. Can anyone ELI5?
- kinjba11 5y agoSee also for more discussion on LiveView: Phoenix LiveDashboard (April 16, 2020) https://news.ycombinator.com/item?id=22892401 https://news.ycombinator.com/item?id=22892401