9 ms·

League of Legends Chat Service Architecture

139 points by bandris 11y ago

bandris 11y ago
An Erlang success story.
k__ 11y ago
I thought LoL didn't have a chat because it "increases toxicity by 250%"
- eterm 11y ago
  Officially this is why they don't support voice communication.
  - Omniusaspirer 11y ago
    Which is also a complete cop-out meant to save $$$ that lacks any basis in fact or reality. Dota 2 is churning along quite comfortably with integrated voice chat despite having more inter-region mixing. Any time this subject comes up the Dota 2 community nearly unanimously agrees that voice chat in no way encourages "toxicity" and more likely actually prevents it by assigning a human voice to the player as opposed to just disembodied words.
    - xfalcox 11y ago
      As an avid Dota 2 player I can attest this. If people start flaming you, replying on voice chat and saying something positive, like "Sorry, my wife was calling me. But don't worry we can win this", will change the tides immediately.
      - jgrowl 11y ago
        Yup. The most toxic players in my experience tend to be keyboard warriors anyway.
    - iN7h33nD 11y ago
      Not to mention the fact that most toxic people can only spam the chat wheel commands and cannot actually talk as they are chat banned for X hours. EDIT: Are there any articles like this on the architecture of Dota 2 or Steam?
      - Omniusaspirer 11y ago
        Look into SteamRE for a good primer on interfacing with Valve chat services, although if you want gory details of server architecture I don't think they've ever written about that in detail.
      - Namidairo 11y ago
        And if players start turning to auto-matching voice services (Curse Voice and Razer Connect did this at one stage), they'll go and threaten bans for all.
    - Mimu 11y ago
      It does not encourage it. It also sometimes lead to funny stuff like someone broadcasting sad music if we are losing or epic music if we are winning ahah. Although the mute system is still a must have.
    - Mimu 11y ago
      It does not encourage it. It also sometimes lead to funny stuff like someone broadcasting sad music if we are losing or epic music if we are winning ahah. Although the mute system is still a must have.
    - res0nat0r 11y ago
      Or they're just doing what WoW did, WoW if I remember correctly added in-game voice support for raids years ago, but dropped it because of lack of user uptake. No one was using it because it wasn't as good or easy to use as already existing third party programs, so they dropped it. Riot is probably thinking the same, and can spend their resources elsewhere on something that isn't already done better by someone else.
      - Omniusaspirer 11y ago
        WoW already had persisting social structures that lead to those groups setting up voice servers that provided them a lot more intricate controls. People who chose to speak in game had already assembled the infrastructure to do so, why use a worse system? I played Eve Online and it was the same scenario, every single group ran a voice server and often individuals did as well. Contrast this to LoL where you're dropped into a game randomly with others random players for <60 minutes who you don't know but are forced to communicate with to maximize strategy. Communication is utterly critical yet the best option for doing so is slowly typing a message in a micro intensive game or a nondescript ping. The resulting player behavior is exactly as one would expect.
        res0nat0r 11y ago
        Maybe also they're taking their cues from Xbox Live and all of the raging that comes from a similar aged user-base dropped in with random groups and don't feel like enabling young kids the ability to tell random strangers to go kill themselves...
        Omniusaspirer 11y ago
        You'll always have outliers, but nothing stops those same kids from doing so right now with text chat (and probably at a higher rate). Mute functionality is there for a reason.
        pikzen 11y ago
        And we've solved this problem ever since the invention of the "Mute this player" button.
    - tehwebguy 11y ago
      Maybe, but they just bought a stake in Curse Voice
    - bigtunacan 11y ago
      I'm an avid LoL player and not having voice chat support built in is the largest downside to LoL. Plenty of players are "side-loading" their voice chat through a variety of applications. This is a huge PITA as I need Skype, TeamSpeak, whatever the rest of my team is using. Players aren't consistently using voice chat since it's not built in. If you are in a match up where one team is on voice chat and the other is not; the team with voice has a huge advantage as text chat is less visible and has more delay.
    - jcoby 11y ago
      I have ~1k games in Dota 2 and have run across less than a dozen ragers in voice. A quick report+mute and they're taken care of. Usually getting a chat ban in the process. If anything it makes the game better when people start using voice. I've had games that I've given up on and people started using voice to ask for things like shared unit control for dominated creeps to stack ancients and actually communicate where they're going instead of trying to guess what's going on in four silent teammates' minds. OTOH, I had one game on EU W where two Russians were talking (nicely) to each other and some British guy comes in and just says "I can't stand Russians. Good bye." and leaves. And I've had my fair share of LAN cafe players with an open mic blaring music and their friends conversations. Mute and move on. I really do wish we could mute the mic and not chat and vice-versa.
      - mreiland 11y ago
        I can't speak for Dota, but I quit playing league because I got tired of how badly Riot has implemented their ranked system. There's so much talk about toxicity in league, and very little about what CAUSES the toxicity. Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth. For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss. I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts. If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game. Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where. https://www.youtube.com/watch?v=5UfO7qOHx5M https://www.youtube.com/watch?v=5UfO7qOHx5M The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling. I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me. I haven't played a game since, although I still watch LCS.
  - discreditable 11y ago
    It's also why they no longer display user's pings on the loading screen.
    - Karunamon 11y ago
      I find this trend of social engineering via hiding info to be really annoying/disturbing. LoL doesn't even have a way for a player to find out what their ranking is - the ELO on the back end is hidden for some inane reason, and now I can't even know that I have to play with a certain guy a bit more carefully because he's half a second behind everyone else. Knowing that I can't trust another player to launch a skill shot attack on target is valuable info - it changes the dynamic of how you'd play with that person.
      - simoncion 11y ago
        > I find this trend of social engineering via hiding info to be really annoying/disturbing. I totally understand where you're coming from. However, there exist people who play MOBAS who are easily frustrated and -emotionally- twelve years old. Hiding the rank and ping of all players gives such people two fewer things to get angry or upset about. If they're not angry or upset, they might just either play well and win the game, or -barring that- be a pleasant person to play with. :) > Knowing that I can't trust another player to launch a skill shot attack on target [because of high ping] is valuable info... Back in the day, I used to play Descent and Descent 2 [0] over both a 28.8 and -later- a 56k analog modem. My ping times varied from 300ms to 700ms. There were a few weapons in that game that were -effectively- skill shots and tricky to land, but -if you were aware of your ping, and your opponent's likely motion-, landing the shots was totally doable. In short, just because your teammate is a HPB doesn't mean that he can't cope with that ping. He might always play with that ping, and -thus- be completely accustomed to it. [0] Both games are 3D, 6DoF shooters.
        Karunamon 11y ago
        Here's the problem though - Descent was optimized for those comparatively awful connections. League doesn't specify any minimums, but players report games becoming difficult around 100ms and nigh unplayable >=300ms. (From a casual google) With LoL being as team-based as it is, knowing if/when my Blitzcrank is going to be able to land his rocket grab lets me know when to launch my ultimate and the rest of the team to join the fight. Having a high ping guy initiate a teamfight is.. unwise. But quite honestly, I don't believe that hypothesis. If someone is going to get bothered to the point of toxicity by ping, something that is almost completely out of the player's control, they're going to get just as bothered being killed by Malzahar's global ultimate, someone picking the wrong team composition, not getting the lane they want, someone speaking a language other than perfect english, and so on. It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child. As someone who is not an emotional child (at least I think not): that's disheartening. I also don't believe Riot is serious about dealing with the problem - it takes a long history of being a serious jerk before the bans start getting trotted out (talk of "correction cards" aside). And it makes sense why - banned people don't buy champions and skins.
  - Natsu 11y ago
    There's a solution to that (considering that text is acceptable and voice is not for whatever reason) - use voice to text software to convert the user's voice to text and send it over the existing text chat.
burrox 11y ago
Ah this is the reason why I hate that my workplace blocks gaming websites.
- lackbeard 11y ago
  Here's a snapshot that might work for you: https://archive.is/CFboh https://archive.is/CFboh
- driverdan 11y ago
  Look for a new job. Engineers are in high demand, you can find a place that doesn't treat you like a child.
- lectrick 11y ago
  I used to work for those sorts of companies. I fired them because I'm a mature professional who can figure out how best to utilize my time, TYVM. Find a company that people actually enjoy coming to work and collaborating in. I mean, if you're an engineer, you're already doing the #1 job in the country, why poison the well at a company with a mistrustful IT policy? Sorry but I have no end of disrespect for that kind of bullshit. If you're at a job and you're jerking around, people will eventually figure it out regardless of stupid content filtering. And in this case, this is a lost learning experience for you, as the article is very interesting.
traduz 11y ago
It's nice to see how the backend of my favorite game works. Specially this article, right now I'm developing a chat service for a company and god I hate openfire.
- kaylarose 11y ago
  My team also recently built a chat service into a mobile app for a cruise line, and had to have high resiliency to network drops and offline users - while having a mission-critical "guaranteed delivery" requirement. We also chose OpenFire since it was an off-the-shelf XMPP server, and (presumably) was easily extensibile via plugins. After working with (hacking around) OpenFire for two years, I 100% agree with your statement. The clustering plugin routinely fails (to the point where we have actually investigated not even clustering it anymore), the admin interface routinely displays "wrong" data, and other fun bugs we found along the way. Does it work for us (with a lot of client XEPs and additional custom plugins)? Yes, it does an acceptable job. Would I choose the same product again, if given the choice? No.
tuukkah 11y ago
For comparison, is there info on how Facebook is using MQTT for chat? EDIT: here's something, but not on backend architecture: https://www.facebook.com/notes/facebook-engineering/building-facebook-messenger/10150259350998920 https://www.facebook.com/notes/facebook-engineering/building...
togilvie 11y ago
All this fancy architecture doesn't change the fact that....I CALLED TOP FIRST!
- nanny 11y ago
  >I CALLED TOP FIRST! We'll never know for sure. There have been many times where I'll type in chat: "top", my message will appear, and then somebody else will say "top" immediately after. But, a asking a third person reveals that the second player's message actually arrived first, even though my client shows my message first. The chat architecture is terrible.
  - forrestthewoods 11y ago
    Generally speaking users expect instantaneous feedback. It would feel super shitty if every time you hit enter there was lag until a server roundtrip posted the message.
    - simoncion 11y ago
      > It would feel super shitty if every time you hit enter... I and many people I know don't agree with this. Instant feedback is nice, but not required. Moreover, if we need to choose between (instant feedback) and (knowledge of server receipt of a message and knowledge of message ordering), we'll choose the latter every time. (I'm glaring at you, Google Hangouts.) Having the chat server lie to you about the state of the world always feels far worse than a delay between the time you submit your message and the time it appears.
      - forrestthewoods 11y ago
        Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter. Even if the server corrected the order messages were received it'd feel shitty to not get your lane because someone has a faster roundtrip to the chat server.
runholm 11y ago
Message ordering is still messed up causing a lot of confusing situations ruining games. "Rito plz."
mentos 11y ago
As an indie game dev I would like to deal with as little of a stack as possible. So what high level services could you use to achieve a similar chat feature without having to roll and maintain all of your own infrastructure? To me something like Firebase could work for this? It seems like when game dev companies grow they love to blow out all of their own in house solutions so they can have complete ownership and control. But is the cost of ownership of your stack really worth the maintenance costs? Why not outsource as much of the stack to another company that can take advantage of economies of scale?
- TimothyFitz 11y ago
  Chat at "indie scale" is fairly easy. 10-20k concurrent users fit into one or two machines with off the shelf software (ejabberd) with minimal expertise. 1k-2k concurrent users could easily be done in almost any stack. If you're doing more than 20k concurrent users, odds are you can afford to hire someone to scale up or build out a team to do it, or find a 3rd party service and pay them to host your chat.
  - Thaxll 11y ago
    Indie don't have 20k CCU. Openfire is a known server for XMPP: http://www.igniterealtime.org/projects/openfire/ http://www.igniterealtime.org/projects/openfire/
    - scott_karana 11y ago
      > Openfire is a known server for XMPP Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc) They were talking about indie game studies, not indie chat protocols.
- weego 11y ago
  You could find someone who would do managed ejabberd. ejabberd is an XMPP server written in erlang. As soon as I saw they talked about erlang I was pretty confident that ejabberd was the next name they would drop. Very surprised they suggest they rolled their own.
anthonyrrubin 11y ago
Here is a related talk from Strange Loop last year: "Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek https://www.youtube.com/watch?v=_jsMpmWaq7I https://www.youtube.com/watch?v=_jsMpmWaq7I
scrollaway 11y ago
I didn't know Riot kept an engineering blog. I'm so happy to find this out! I really wish more game studios would do this (Blizzard especially).
- AntiRush 11y ago
  They actually just started it a few months ago: http://www.riotgames.com/articles/20150817/1697/our-engineering-tech-blog-blasts http://www.riotgames.com/articles/20150817/1697/our-engineer...
- bargl 11y ago
  I agree, I really like seeing the EVE development blog. There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up. They also mention that their users warned them about this battle before hand which was really cool.
  - simoncion 11y ago
    > There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up. They also mention that their users warned them about this battle before hand which was really cool. So, I know that I'm expecting too much, but remember that the only way CCP can handle large fights is to simulate them at 1/10th realtime, and get advance notice of said fights so that they can move the simulation for the target system to a single, beefy server ahead of time. It's a goddamn shame that they don't have the resources available to figure out the -admittedly difficult- problem that is * Reworking their simulation to be parallelizable across nodes * Dynamically shifting slices of a battle within a system across physical nodes
sng 11y ago
don't really have to deal with scaling when so many players are chat restricted.
BuckRogers 11y ago
My languages of choice are Python(2, of course) and Erlang. Most of my fellows seem obsessed with multicore programming, which IMO is only useful for latency bound situations such as an OS or game engine. For everything else, multinode wins handily. I'm pleased to see yet-another one of my favorite applications building on the fantastic BEAM VM.
- derefr 11y ago
  I love Erlang and the concept of multinode programming, but sometimes I wish it could lean back into the multicore/SIMD world. A lot of the time, actors are the right choice; but for things like nodes in a physics engine, you obviously want to take a big pile of them and stream them through a matrix multiplication or something of the sort, if you can. If there was a language that let me define something in terms of an individual actor-modelled components, but then allowed you to optimize some of them into batched GPGPU passes (and apply strict simulation ticking and so forth in the process), I feel like it would "win" all sorts of domains. Erlang could be that language with a little work.
- sanderjd 11y ago
  > Python(2, of course) Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long been mystified by the almost proud unwillingness of the community to embrace Python 3. It seems like a better language to me. What am I missing?
  - strong_ai 11y ago
    afaik the two main reasons are 1) the expense of porting from 2.* to 3.* and 2) some libraries do not yet support 3 http://python3wos.appspot.com/ http://python3wos.appspot.com/
  - BuckRogers 11y ago
    Few reasons and there's more but I won't go on forever. 1)The "latest version" of a programming language is not the same as having the latest version of Firefox. Having the latest version can actually be a detriment if you lose libraries as with Python3. 2)It may be unpopular, but I tend to follow the market to a certain degree. Python is my language of choice for home projects and employment. But almost all employers use Python2.x. So I follow them just to keep 1 less hassle out of my life. 3)Losing the long tail of libraries that 2.x has. Many things support Python3, and I'm not against it. I'm just waiting for the day when Python3 is more convenient than Python2 to move. I wasn't one asking for the Py3 changes, and they certainly weren't done for me. So I don't feel like being inconvenienced or doing any porting/heavy lifting in the meantime. 4)My petty reason. I think everyone should just do what's in their best interests. I advertised that "2 of course" because it is a given, but rarely stated. I've met a number of pushy Python3 folks and find them distasteful, which hasn't helped budge me in the slightest. They've probably made me a little edgy on that topic. 5)It's a programming language, so what helps you build something the fastest should be what we use. Today that's still Python2. Some people feel very passionately about Python3 and I hope they spend many hours porting and building the ecosystem for my arrival one day. What seems to be going on instead is they're hoping to convert more devotees that will go out there and do a bunch of free ports. 6)I'm more concerned with moving too early than I am too late when the Python3 ecosystem one day, maybe, eclipses Python2. They're just not that different, I've used both and it's no different from building software with 2. But they have vastly different breadth of libraries in 2's favor. They also screwed up on a few things with 3, it isn't really a flat "upgrade" either. 7)Python3 is just adding more bloat. I generally prefer features are REMOVED from a language until there's nothing left that could reasonably be removed from the core language. Ideally I'd like to see a small version of Python with the basics never touched and features added through addons or a macro system. Latest version, upgrade, and support-ends-2020 are all political statements in regards to Python3 that mean nothing in reality. Latest version- what can't be backported? Upgrade- more like sidegrade once all things are considered. Support EOL- total smoke and mirrors, code will not stop running post 2020. They're tossing every feature against the wall that they can to fix that. I'm one that holds feature-soup in disdain though so this is unattractive to me, and I prefer a well thought out "one way to do it" (Zen of Python) approach. Python is well beyond the basics at this point and not a small language, but at least 2.7 is frozen and I can choose to use the backports people provide if I want. I like a smaller core language if possible because I have to read other people's codebases.
debacle 11y ago
Why not just use IRC?
- baldfat 11y ago
  I seriously don't understand how IRC has withstood the test of time and gets relegated to the back room. 3d.tv (Twitch before there was twitch) Used IRC in a way I could just go to the IRC channel and connect to it and the people could write IRC programs. Twitch.tv IRC is a little more hidden. http://help.twitch.tv/customer/portal/articles/1302780-twitch-irc http://help.twitch.tv/customer/portal/articles/1302780-twitc...
  - debacle 11y ago
    It only really confuses me because there's not really a lot of value in writing your own "not quite IRC" type chat system. Just think about the development and administration time that went into setting something up that's probably not as good as something you can apt-get install.
    - mentos 11y ago
      I can't agree more. Are there any great examples of IRC being used for the backend of a games chat system? How would you envision using IRC in this case? Would have been great to see RiotGames write their own custom framework to interface between their game and an IRC backend, open source it for other companies to contribute to and prevent further reinventing of the wheel.
      - 6581 11y ago
        Worms Armageddon and some other Worms games use IRC for lobby chat. http://worms2d.info/WormNET_(Worms_Armageddon) http://worms2d.info/WormNET_(Worms_Armageddon)
      - debacle 11y ago
        Pretty sure Eve did/does. Twitch does.
mattlong 11y ago
What happens if a server get partitioned for an extended period of time? In their example, it sounds like Alice and Bob would be isolated from everyone else until connectivity to the cluster is re-established. Is there a mechanism that makes clients connected to an isolated server reconnect to the load balancer?
- lectrick 11y ago
  Assuming state isn't stored in the processes (which it typically isn't in Erlang) and is instead in Riak, I believe it would be Riak's responsibility to resolve the split. Which it is likely capable of doing.
lectrick 11y ago
TL;DR Erlang and distributed Riak using CRDT's (but the article is interesting!)
Senlaar 11y ago
Thanks for posting this! I help to produce the blog at Riot and I really hope the tech community finds it useful. In case anyone's interested, we just posted new content this week - an overview of our RFC process, and the continuation of a Docker&Jenkins tutorial: http://engineering.riotgames.com http://engineering.riotgames.com

139 points by bandris 11y ago

bandris 11y ago

An Erlang success story.

k__ 11y ago

I thought LoL didn't have a chat because it "increases toxicity by 250%"

eterm 11y ago
Officially this is why they don't support voice communication.
- Omniusaspirer 11y ago
  Which is also a complete cop-out meant to save $$$ that lacks any basis in fact or reality. Dota 2 is churning along quite comfortably with integrated voice chat despite having more inter-region mixing. Any time this subject comes up the Dota 2 community nearly unanimously agrees that voice chat in no way encourages "toxicity" and more likely actually prevents it by assigning a human voice to the player as opposed to just disembodied words.
  - xfalcox 11y ago
    As an avid Dota 2 player I can attest this. If people start flaming you, replying on voice chat and saying something positive, like "Sorry, my wife was calling me. But don't worry we can win this", will change the tides immediately.
    - jgrowl 11y ago
      Yup. The most toxic players in my experience tend to be keyboard warriors anyway.
  - iN7h33nD 11y ago
    Not to mention the fact that most toxic people can only spam the chat wheel commands and cannot actually talk as they are chat banned for X hours. EDIT: Are there any articles like this on the architecture of Dota 2 or Steam?
    - Omniusaspirer 11y ago
      Look into SteamRE for a good primer on interfacing with Valve chat services, although if you want gory details of server architecture I don't think they've ever written about that in detail.
    - Namidairo 11y ago
      And if players start turning to auto-matching voice services (Curse Voice and Razer Connect did this at one stage), they'll go and threaten bans for all.
  - Mimu 11y ago
    It does not encourage it. It also sometimes lead to funny stuff like someone broadcasting sad music if we are losing or epic music if we are winning ahah. Although the mute system is still a must have.
  - Mimu 11y ago
    It does not encourage it. It also sometimes lead to funny stuff like someone broadcasting sad music if we are losing or epic music if we are winning ahah. Although the mute system is still a must have.
  - res0nat0r 11y ago
    Or they're just doing what WoW did, WoW if I remember correctly added in-game voice support for raids years ago, but dropped it because of lack of user uptake. No one was using it because it wasn't as good or easy to use as already existing third party programs, so they dropped it. Riot is probably thinking the same, and can spend their resources elsewhere on something that isn't already done better by someone else.
    - Omniusaspirer 11y ago
      WoW already had persisting social structures that lead to those groups setting up voice servers that provided them a lot more intricate controls. People who chose to speak in game had already assembled the infrastructure to do so, why use a worse system? I played Eve Online and it was the same scenario, every single group ran a voice server and often individuals did as well. Contrast this to LoL where you're dropped into a game randomly with others random players for <60 minutes who you don't know but are forced to communicate with to maximize strategy. Communication is utterly critical yet the best option for doing so is slowly typing a message in a micro intensive game or a nondescript ping. The resulting player behavior is exactly as one would expect.
      - res0nat0r 11y ago
        Maybe also they're taking their cues from Xbox Live and all of the raging that comes from a similar aged user-base dropped in with random groups and don't feel like enabling young kids the ability to tell random strangers to go kill themselves...
        Omniusaspirer 11y ago
        You'll always have outliers, but nothing stops those same kids from doing so right now with text chat (and probably at a higher rate). Mute functionality is there for a reason.
        pikzen 11y ago
        And we've solved this problem ever since the invention of the "Mute this player" button.
  - tehwebguy 11y ago
    Maybe, but they just bought a stake in Curse Voice
  - bigtunacan 11y ago
    I'm an avid LoL player and not having voice chat support built in is the largest downside to LoL. Plenty of players are "side-loading" their voice chat through a variety of applications. This is a huge PITA as I need Skype, TeamSpeak, whatever the rest of my team is using. Players aren't consistently using voice chat since it's not built in. If you are in a match up where one team is on voice chat and the other is not; the team with voice has a huge advantage as text chat is less visible and has more delay.
  - jcoby 11y ago
    I have ~1k games in Dota 2 and have run across less than a dozen ragers in voice. A quick report+mute and they're taken care of. Usually getting a chat ban in the process. If anything it makes the game better when people start using voice. I've had games that I've given up on and people started using voice to ask for things like shared unit control for dominated creeps to stack ancients and actually communicate where they're going instead of trying to guess what's going on in four silent teammates' minds. OTOH, I had one game on EU W where two Russians were talking (nicely) to each other and some British guy comes in and just says "I can't stand Russians. Good bye." and leaves. And I've had my fair share of LAN cafe players with an open mic blaring music and their friends conversations. Mute and move on. I really do wish we could mute the mic and not chat and vice-versa.
    - mreiland 11y ago
      I can't speak for Dota, but I quit playing league because I got tired of how badly Riot has implemented their ranked system. There's so much talk about toxicity in league, and very little about what CAUSES the toxicity. Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth. For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss. I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts. If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game. Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where. https://www.youtube.com/watch?v=5UfO7qOHx5M https://www.youtube.com/watch?v=5UfO7qOHx5M The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling. I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me. I haven't played a game since, although I still watch LCS.
- discreditable 11y ago
  It's also why they no longer display user's pings on the loading screen.
  - Karunamon 11y ago
    I find this trend of social engineering via hiding info to be really annoying/disturbing. LoL doesn't even have a way for a player to find out what their ranking is - the ELO on the back end is hidden for some inane reason, and now I can't even know that I have to play with a certain guy a bit more carefully because he's half a second behind everyone else. Knowing that I can't trust another player to launch a skill shot attack on target is valuable info - it changes the dynamic of how you'd play with that person.
    - simoncion 11y ago
      > I find this trend of social engineering via hiding info to be really annoying/disturbing. I totally understand where you're coming from. However, there exist people who play MOBAS who are easily frustrated and -emotionally- twelve years old. Hiding the rank and ping of all players gives such people two fewer things to get angry or upset about. If they're not angry or upset, they might just either play well and win the game, or -barring that- be a pleasant person to play with. :) > Knowing that I can't trust another player to launch a skill shot attack on target [because of high ping] is valuable info... Back in the day, I used to play Descent and Descent 2 [0] over both a 28.8 and -later- a 56k analog modem. My ping times varied from 300ms to 700ms. There were a few weapons in that game that were -effectively- skill shots and tricky to land, but -if you were aware of your ping, and your opponent's likely motion-, landing the shots was totally doable. In short, just because your teammate is a HPB doesn't mean that he can't cope with that ping. He might always play with that ping, and -thus- be completely accustomed to it. [0] Both games are 3D, 6DoF shooters.
      - Karunamon 11y ago
        Here's the problem though - Descent was optimized for those comparatively awful connections. League doesn't specify any minimums, but players report games becoming difficult around 100ms and nigh unplayable >=300ms. (From a casual google) With LoL being as team-based as it is, knowing if/when my Blitzcrank is going to be able to land his rocket grab lets me know when to launch my ultimate and the rest of the team to join the fight. Having a high ping guy initiate a teamfight is.. unwise. But quite honestly, I don't believe that hypothesis. If someone is going to get bothered to the point of toxicity by ping, something that is almost completely out of the player's control, they're going to get just as bothered being killed by Malzahar's global ultimate, someone picking the wrong team composition, not getting the lane they want, someone speaking a language other than perfect english, and so on. It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child. As someone who is not an emotional child (at least I think not): that's disheartening. I also don't believe Riot is serious about dealing with the problem - it takes a long history of being a serious jerk before the bans start getting trotted out (talk of "correction cards" aside). And it makes sense why - banned people don't buy champions and skins.
- Natsu 11y ago
  There's a solution to that (considering that text is acceptable and voice is not for whatever reason) - use voice to text software to convert the user's voice to text and send it over the existing text chat.

burrox 11y ago

Ah this is the reason why I hate that my workplace blocks gaming websites.

lackbeard 11y ago
Here's a snapshot that might work for you: https://archive.is/CFboh https://archive.is/CFboh
driverdan 11y ago
Look for a new job. Engineers are in high demand, you can find a place that doesn't treat you like a child.
lectrick 11y ago
I used to work for those sorts of companies. I fired them because I'm a mature professional who can figure out how best to utilize my time, TYVM. Find a company that people actually enjoy coming to work and collaborating in. I mean, if you're an engineer, you're already doing the #1 job in the country, why poison the well at a company with a mistrustful IT policy? Sorry but I have no end of disrespect for that kind of bullshit. If you're at a job and you're jerking around, people will eventually figure it out regardless of stupid content filtering. And in this case, this is a lost learning experience for you, as the article is very interesting.

traduz 11y ago

It's nice to see how the backend of my favorite game works. Specially this article, right now I'm developing a chat service for a company and god I hate openfire.

kaylarose 11y ago
My team also recently built a chat service into a mobile app for a cruise line, and had to have high resiliency to network drops and offline users - while having a mission-critical "guaranteed delivery" requirement. We also chose OpenFire since it was an off-the-shelf XMPP server, and (presumably) was easily extensibile via plugins. After working with (hacking around) OpenFire for two years, I 100% agree with your statement. The clustering plugin routinely fails (to the point where we have actually investigated not even clustering it anymore), the admin interface routinely displays "wrong" data, and other fun bugs we found along the way. Does it work for us (with a lot of client XEPs and additional custom plugins)? Yes, it does an acceptable job. Would I choose the same product again, if given the choice? No.

tuukkah 11y ago

For comparison, is there info on how Facebook is using MQTT for chat? EDIT: here's something, but not on backend architecture: https://www.facebook.com/notes/facebook-engineering/building-facebook-messenger/10150259350998920 https://www.facebook.com/notes/facebook-engineering/building...

togilvie 11y ago

All this fancy architecture doesn't change the fact that....I CALLED TOP FIRST!

nanny 11y ago
>I CALLED TOP FIRST! We'll never know for sure. There have been many times where I'll type in chat: "top", my message will appear, and then somebody else will say "top" immediately after. But, a asking a third person reveals that the second player's message actually arrived first, even though my client shows my message first. The chat architecture is terrible.
- forrestthewoods 11y ago
  Generally speaking users expect instantaneous feedback. It would feel super shitty if every time you hit enter there was lag until a server roundtrip posted the message.
  - simoncion 11y ago
    > It would feel super shitty if every time you hit enter... I and many people I know don't agree with this. Instant feedback is nice, but not required. Moreover, if we need to choose between (instant feedback) and (knowledge of server receipt of a message and knowledge of message ordering), we'll choose the latter every time. (I'm glaring at you, Google Hangouts.) Having the chat server lie to you about the state of the world always feels far worse than a delay between the time you submit your message and the time it appears.
    - forrestthewoods 11y ago
      Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter. Even if the server corrected the order messages were received it'd feel shitty to not get your lane because someone has a faster roundtrip to the chat server.

runholm 11y ago

Message ordering is still messed up causing a lot of confusing situations ruining games. "Rito plz."

mentos 11y ago

As an indie game dev I would like to deal with as little of a stack as possible. So what high level services could you use to achieve a similar chat feature without having to roll and maintain all of your own infrastructure? To me something like Firebase could work for this? It seems like when game dev companies grow they love to blow out all of their own in house solutions so they can have complete ownership and control. But is the cost of ownership of your stack really worth the maintenance costs? Why not outsource as much of the stack to another company that can take advantage of economies of scale?

TimothyFitz 11y ago
Chat at "indie scale" is fairly easy. 10-20k concurrent users fit into one or two machines with off the shelf software (ejabberd) with minimal expertise. 1k-2k concurrent users could easily be done in almost any stack. If you're doing more than 20k concurrent users, odds are you can afford to hire someone to scale up or build out a team to do it, or find a 3rd party service and pay them to host your chat.
- Thaxll 11y ago
  Indie don't have 20k CCU. Openfire is a known server for XMPP: http://www.igniterealtime.org/projects/openfire/ http://www.igniterealtime.org/projects/openfire/
  - scott_karana 11y ago
    > Openfire is a known server for XMPP Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc) They were talking about indie game studies, not indie chat protocols.
weego 11y ago
You could find someone who would do managed ejabberd. ejabberd is an XMPP server written in erlang. As soon as I saw they talked about erlang I was pretty confident that ejabberd was the next name they would drop. Very surprised they suggest they rolled their own.

anthonyrrubin 11y ago

Here is a related talk from Strange Loop last year: "Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek https://www.youtube.com/watch?v=_jsMpmWaq7I https://www.youtube.com/watch?v=_jsMpmWaq7I

scrollaway 11y ago

I didn't know Riot kept an engineering blog. I'm so happy to find this out! I really wish more game studios would do this (Blizzard especially).

AntiRush 11y ago
They actually just started it a few months ago: http://www.riotgames.com/articles/20150817/1697/our-engineering-tech-blog-blasts http://www.riotgames.com/articles/20150817/1697/our-engineer...
bargl 11y ago
I agree, I really like seeing the EVE development blog. There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up. They also mention that their users warned them about this battle before hand which was really cool.
- simoncion 11y ago
  > There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up. They also mention that their users warned them about this battle before hand which was really cool. So, I know that I'm expecting too much, but remember that the only way CCP can handle large fights is to simulate them at 1/10th realtime, and get advance notice of said fights so that they can move the simulation for the target system to a single, beefy server ahead of time. It's a goddamn shame that they don't have the resources available to figure out the -admittedly difficult- problem that is * Reworking their simulation to be parallelizable across nodes * Dynamically shifting slices of a battle within a system across physical nodes

sng 11y ago

don't really have to deal with scaling when so many players are chat restricted.

BuckRogers 11y ago

My languages of choice are Python(2, of course) and Erlang. Most of my fellows seem obsessed with multicore programming, which IMO is only useful for latency bound situations such as an OS or game engine. For everything else, multinode wins handily. I'm pleased to see yet-another one of my favorite applications building on the fantastic BEAM VM.

derefr 11y ago
I love Erlang and the concept of multinode programming, but sometimes I wish it could lean back into the multicore/SIMD world. A lot of the time, actors are the right choice; but for things like nodes in a physics engine, you obviously want to take a big pile of them and stream them through a matrix multiplication or something of the sort, if you can. If there was a language that let me define something in terms of an individual actor-modelled components, but then allowed you to optimize some of them into batched GPGPU passes (and apply strict simulation ticking and so forth in the process), I feel like it would "win" all sorts of domains. Erlang could be that language with a little work.
sanderjd 11y ago
> Python(2, of course) Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long been mystified by the almost proud unwillingness of the community to embrace Python 3. It seems like a better language to me. What am I missing?
- strong_ai 11y ago
  afaik the two main reasons are 1) the expense of porting from 2.* to 3.* and 2) some libraries do not yet support 3 http://python3wos.appspot.com/ http://python3wos.appspot.com/
- BuckRogers 11y ago
  Few reasons and there's more but I won't go on forever. 1)The "latest version" of a programming language is not the same as having the latest version of Firefox. Having the latest version can actually be a detriment if you lose libraries as with Python3. 2)It may be unpopular, but I tend to follow the market to a certain degree. Python is my language of choice for home projects and employment. But almost all employers use Python2.x. So I follow them just to keep 1 less hassle out of my life. 3)Losing the long tail of libraries that 2.x has. Many things support Python3, and I'm not against it. I'm just waiting for the day when Python3 is more convenient than Python2 to move. I wasn't one asking for the Py3 changes, and they certainly weren't done for me. So I don't feel like being inconvenienced or doing any porting/heavy lifting in the meantime. 4)My petty reason. I think everyone should just do what's in their best interests. I advertised that "2 of course" because it is a given, but rarely stated. I've met a number of pushy Python3 folks and find them distasteful, which hasn't helped budge me in the slightest. They've probably made me a little edgy on that topic. 5)It's a programming language, so what helps you build something the fastest should be what we use. Today that's still Python2. Some people feel very passionately about Python3 and I hope they spend many hours porting and building the ecosystem for my arrival one day. What seems to be going on instead is they're hoping to convert more devotees that will go out there and do a bunch of free ports. 6)I'm more concerned with moving too early than I am too late when the Python3 ecosystem one day, maybe, eclipses Python2. They're just not that different, I've used both and it's no different from building software with 2. But they have vastly different breadth of libraries in 2's favor. They also screwed up on a few things with 3, it isn't really a flat "upgrade" either. 7)Python3 is just adding more bloat. I generally prefer features are REMOVED from a language until there's nothing left that could reasonably be removed from the core language. Ideally I'd like to see a small version of Python with the basics never touched and features added through addons or a macro system. Latest version, upgrade, and support-ends-2020 are all political statements in regards to Python3 that mean nothing in reality. Latest version- what can't be backported? Upgrade- more like sidegrade once all things are considered. Support EOL- total smoke and mirrors, code will not stop running post 2020. They're tossing every feature against the wall that they can to fix that. I'm one that holds feature-soup in disdain though so this is unattractive to me, and I prefer a well thought out "one way to do it" (Zen of Python) approach. Python is well beyond the basics at this point and not a small language, but at least 2.7 is frozen and I can choose to use the backports people provide if I want. I like a smaller core language if possible because I have to read other people's codebases.

debacle 11y ago

Why not just use IRC?

baldfat 11y ago
I seriously don't understand how IRC has withstood the test of time and gets relegated to the back room. 3d.tv (Twitch before there was twitch) Used IRC in a way I could just go to the IRC channel and connect to it and the people could write IRC programs. Twitch.tv IRC is a little more hidden. http://help.twitch.tv/customer/portal/articles/1302780-twitch-irc http://help.twitch.tv/customer/portal/articles/1302780-twitc...
- debacle 11y ago
  It only really confuses me because there's not really a lot of value in writing your own "not quite IRC" type chat system. Just think about the development and administration time that went into setting something up that's probably not as good as something you can apt-get install.
  - mentos 11y ago
    I can't agree more. Are there any great examples of IRC being used for the backend of a games chat system? How would you envision using IRC in this case? Would have been great to see RiotGames write their own custom framework to interface between their game and an IRC backend, open source it for other companies to contribute to and prevent further reinventing of the wheel.
    - 6581 11y ago
      Worms Armageddon and some other Worms games use IRC for lobby chat. http://worms2d.info/WormNET_(Worms_Armageddon) http://worms2d.info/WormNET_(Worms_Armageddon)
    - debacle 11y ago
      Pretty sure Eve did/does. Twitch does.

mattlong 11y ago

What happens if a server get partitioned for an extended period of time? In their example, it sounds like Alice and Bob would be isolated from everyone else until connectivity to the cluster is re-established. Is there a mechanism that makes clients connected to an isolated server reconnect to the load balancer?

lectrick 11y ago
Assuming state isn't stored in the processes (which it typically isn't in Erlang) and is instead in Riak, I believe it would be Riak's responsibility to resolve the split. Which it is likely capable of doing.

lectrick 11y ago

TL;DR Erlang and distributed Riak using CRDT's (but the article is interesting!)

Senlaar 11y ago

Thanks for posting this! I help to produce the blog at Riot and I really hope the tech community finds it useful. In case anyone's interested, we just posted new content this week - an overview of our RFC process, and the continuation of a Docker&Jenkins tutorial: http://engineering.riotgames.com http://engineering.riotgames.com