7 ms·

Our recent server issues

iliekcomputers 5y ago
I'm not sure I completely understand. They say that the only thing that was affected was the tournament because its events needed to be processed synchronously, but I remember the entire site being unavailable for people. Was that unrelated? On a side note, huge props to Lichess, the fact that they can compete with chess.com which has so many resources behind it is very impressive. Everyone who plays chess should consider becoming a patron.
- Santosh83 5y ago
  During the first crash immediately after the initial start of the tournament, the entire site did indeed go offline for a few minutes. Even address resolution failed. Then things went smoothly for about an hour after which the 2nd crash came. This one just seemed to affect the particular tournament (participants couldn't get fresh pairings) while the rest of the site was still working, as the article mentions.
  - iliekcomputers 5y ago
    Ah I see. That makes sense. Would be interesting to know what caused the entire site to go down in the beginning. Wonder if it was just too much traffic.
bryan0 5y ago
Hikaru was live streaming the event so you can see the series of failures and how they affected the tournament here: https://youtu.be/YKfvNl8UoxA https://youtu.be/YKfvNl8UoxA
- shawnz 5y ago
  You can also see agadmator's stream here: https://www.youtube.com/watch?v=w0efZdnPnHQ https://www.youtube.com/watch?v=w0efZdnPnHQ And a game review with some commentary about the issue: https://www.youtube.com/watch?v=lh_NeBZhqPM https://www.youtube.com/watch?v=lh_NeBZhqPM
assbuttbuttass 5y ago
Sounds like they need backpressure. Isn't that the usual solution to a queue growing without bound?
- c0balt 5y ago
  Load sheddding might also be a good solution
  - progbits 5y ago
    How would that work here? "Load" is players finishing games and requiring new match pairings / rating updates for the tournament to continue. You can tell them to wait, sure. But as long as the rate of games finishing exceeds the rate at which they can process them I don't see how that would improve the situation, the tournament would be stuck anyway. One option is to eg. limit the total number of players in the tournament up front but they explicitly said they didn't want to do that.
    - bcrosby95 5y ago
      Depends upon of every event is strictly necessary for the proper functioning of the system, or if some are just nice-to-have.
      - jeremyjh 5y ago
        This is where it helps to have domain knowledge before pontificating on someone else's architecture. It is an arena tournament. If you want people to join the tournament and get paired with games after finishing their current game you'll need those events. There are no "nice to haves".
        than3 5y ago
        > There are no 'nice to haves' That's an assumption, there's a lot of assumptions flying around right now. The simple answer is we don't know. Often there are a lot of design tradeoffs, bcrosby was likely questioning whether all events going into that specific queue were strictly necessary for matchmaking. We don't even know that this was actually the problem. So far, no professional post-mortem, or root cause analysis has been released. There are certainly ways one can setup these type of services so they dynamically scale correctly, but since the architecture they are using isn't fully disclosed its difficult to say anything with certainty and this type of guesswork is entirely unproductive.
bo1024 5y ago
It would be very interesting to hear about the technical details!
- EarthIsHome 5y ago
  Further down in the article, there's a more technical explanation under the heading "Can you elaborate on the technical issue?"
jph 5y ago
> Eventually there was no way of keeping up with the queue A chess congestion pile up... sounds like an event stream rook-ie mistake. :-) Seriously congrats to Lichess for growing. It's an amazing site. Donate if you can.
nijave 5y ago
Anyone take a look into the code and see why it can't be parallelized? The bottom of the FAQ mentions that but I'd think at least certain aspects should be parallelizable or at least be prioritizable (like maybe forgoing leaderboard updates to focus on more important events?)
- jeremyjh 5y ago
  If you think about it, you can't really generate pairings in parallel because each thread would need write access to the entire pool to ensure no one is paired twice and that you have a consistent view of all the results to that point before creating a new pair. You could maybe create a lock for each participant but that might actually be slower, and would definitely be more difficult to reason about and can lead towards bugs just as serious under load.
  - saghm 5y ago
    I'm not sure exactly what the algorithm is for choosing pairs, but naively it seems like it should be possible to bucket the players in some way, at which point the pairings within each bucket can be generated in parallel, with buckets getting merged if they get too small. The tricky part is coming up with a way to bucket them that's both efficient and will lead to enough pairings, and I'm not sure what the criteria is. If it's completely random, then ideally any bucketing should work fine. If it's based on ratings or existing records, it seems like starting by grouping players within certain ranges of that stat should work (assuming the boundaries can be picked based on the distribution of the players). I don't know enough to know for sure that bucket grouping would work here, but it doesn't seem like knowing the entire set of results beforehand should be necessary (unless there's some standard algorithm for choosing pairings that's very prescriptive).
    - dmurray 5y ago
      The pairing is based on ratings and existing records, and happily isn't dependent on a standard prescriptive algorithm, unlike traditional Swiss tournaments. It would be fine to accept some reordering of game-finishing events. I'm certain the process can be parallelized - it just hadn't been considered necessary before.
than3 5y ago
I expect that what they've pointed to as the cause is only part of the problem. We'll never know the full picture unless they share it. Being upfront, my experience of the people in charge of the organization there doesn't have much goodwill left. Nothing against Thibault personally, I think he's done some great things but seems to be busy with whatever he's interested in and management isn't it, and he has some unprofessional people with access that work for the project/charity. I had volunteered my services years ago as a System Administrator (no charge) with suggestions, but with limited modes of communication, multiple issues going stale, no response with an auto issue closed. They have issues that don't get addressed, and their process doesn't appare to be aimed to cultivate qualified volunteers or improve the bus factor of the project. To make things worse, when I expressed disagreement with constructive feedback regarding one of their process decisions, one of the other dev's with access apparently took offense and the next day I found my lichess account had been edited by an admin without notice or notification. I could not log in (wrong password), the password and email for password recovery were changed, and trying to access the profile URI directly showed the account as banned. It seemed this was done out of spite, and definitely without any kind of due process. Appeals by email went unanswered within the 90 day cutoff I gave them. As a result I submitted a complaint to the french charities regulatory body and moved on since the group wasn't worth wasting any more of my time. I haven't heard back so who knows if anything came of what I reported. In my opinion, they've got more internal problems than they let on, and to me this is just spillover. Its unfortunate because any failure like this impacts so many people, but I don't find it surprising given my limited experience of the people there.
- Shadonototra 5y ago
  - than3 5y ago
    Yes, that is the only place you can communicate with them. If they haven't tampered with the history it should still be there for anyone that wants to dig. I was simply describing my experience and yet someone always tries to minimize or discredit assuming some other motive; whatever, the world isn't all rainbows and sunshine. You seem to have missed the point, didn't read, both, or have some other agenda. To turn it around, what's your motive for being 'unusually' defensive? You know there are three lichess right, the organization, the service, and the OSS code. What are we talking about here? The former, not the latter two. At this point I could care less how they run their ship. I've learned that what they sell to the public, and the process by which they operate are different things, and through their actions that they don't value processes that are generally accepted and work. As a result, I won't have anything to do with them, but its annoying how many people blindly have to assert the assumption that Lichess (the organization) can do no wrong, and the marketed narrative is the whole picture, and implying that anyone saying differently is attacking or has some subversive plan to hurt the community. Honestly, that kind of propaganda strategy sounds just like something one might hear in the Nuremberg trials (Henry Goering interview with Goebbels). To be clear, I only posted today because this spoiled my newsfeed. If they want to be transparent they should publish a professional post-mortem.
powera 5y ago
The "easy" solution is to put a cap on tournament size. Apart from when Agadmator wants his fans to be in the same tournament as Magnus Carlsen etc., there is basically no need to hold chess tournaments with over 1000 players.
- tephra 5y ago
  The lichess marathon tournaments usually are a lot more than 1000 players.
  - mwt 5y ago
    For example, a recent 24-hour marathon had 24,000 players. Though not all playing at once, surely there were periods of time in which several thousand were playing/pairing at once. https://lichess.org/tournament/autumn21 https://lichess.org/tournament/autumn21
Thorentis 5y ago
For some things there is simply no way to parallelise events when things need to be processed in order, and the limitation is simply the processing capacity. In this instance I'm surprised Lichess didn't have surge capacity to automatically scale up for a big tournament like this. Sure, it might be expensive, but that's when optimisation should come in (to save costs). Having to cancel a major event like this due to poor optimisation is a deployment failure moreso than a coding failure.
- nijave 5y ago
  You're basically talking mainframes looking at vertically scaling for high throughput on something serial
- weird-eye-issue 5y ago
  It's not going to go faster if you add more workers to a queue that has to be processed one by one
  - Thorentis 5y ago
    It's not just a case of adding more workers, but adding more processing power and removing bottlenecks so that each event can leave the queue faster.
    - weird-eye-issue 5y ago
      I doubt it's as simple as adding processing power. That only helps in a linear fashion and it sounds like it was a bigger problem. Probably was IO bound. Also removing bottlenecks is not related to surge capacity but rather code level optimizations that they didn't realize were needed until this happened so it's easy to say that after the fact...
lom 5y ago
You could maybe paralyze the matchmaking process by making buckets in which players of all the elos are ranked. Nobody in the top 100 will be paired with someone who is in the last 2000 anyways. So why not split it up on every 1000th player and treat them separately. The leaderboard then only needs to modified (read can be done without race conditions) by the top 100 or whatever.
f3rnando 5y ago

I'm not sure I completely understand. They say that the only thing that was affected was the tournament because its events needed to be processed synchronously, but I remember the entire site being unavailable for people. Was that unrelated? On a side note, huge props to Lichess, the fact that they can compete with chess.com which has so many resources behind it is very impressive. Everyone who plays chess should consider becoming a patron.

Santosh83 5y ago
During the first crash immediately after the initial start of the tournament, the entire site did indeed go offline for a few minutes. Even address resolution failed. Then things went smoothly for about an hour after which the 2nd crash came. This one just seemed to affect the particular tournament (participants couldn't get fresh pairings) while the rest of the site was still working, as the article mentions.
- iliekcomputers 5y ago
  Ah I see. That makes sense. Would be interesting to know what caused the entire site to go down in the beginning. Wonder if it was just too much traffic.

bryan0 5y ago

Hikaru was live streaming the event so you can see the series of failures and how they affected the tournament here: https://youtu.be/YKfvNl8UoxA https://youtu.be/YKfvNl8UoxA

shawnz 5y ago
You can also see agadmator's stream here: https://www.youtube.com/watch?v=w0efZdnPnHQ https://www.youtube.com/watch?v=w0efZdnPnHQ And a game review with some commentary about the issue: https://www.youtube.com/watch?v=lh_NeBZhqPM https://www.youtube.com/watch?v=lh_NeBZhqPM

assbuttbuttass 5y ago

Sounds like they need backpressure. Isn't that the usual solution to a queue growing without bound?

c0balt 5y ago
Load sheddding might also be a good solution
- progbits 5y ago
  How would that work here? "Load" is players finishing games and requiring new match pairings / rating updates for the tournament to continue. You can tell them to wait, sure. But as long as the rate of games finishing exceeds the rate at which they can process them I don't see how that would improve the situation, the tournament would be stuck anyway. One option is to eg. limit the total number of players in the tournament up front but they explicitly said they didn't want to do that.
  - bcrosby95 5y ago
    Depends upon of every event is strictly necessary for the proper functioning of the system, or if some are just nice-to-have.
    - jeremyjh 5y ago
      This is where it helps to have domain knowledge before pontificating on someone else's architecture. It is an arena tournament. If you want people to join the tournament and get paired with games after finishing their current game you'll need those events. There are no "nice to haves".
      - than3 5y ago
        > There are no 'nice to haves' That's an assumption, there's a lot of assumptions flying around right now. The simple answer is we don't know. Often there are a lot of design tradeoffs, bcrosby was likely questioning whether all events going into that specific queue were strictly necessary for matchmaking. We don't even know that this was actually the problem. So far, no professional post-mortem, or root cause analysis has been released. There are certainly ways one can setup these type of services so they dynamically scale correctly, but since the architecture they are using isn't fully disclosed its difficult to say anything with certainty and this type of guesswork is entirely unproductive.

bo1024 5y ago

It would be very interesting to hear about the technical details!

EarthIsHome 5y ago
Further down in the article, there's a more technical explanation under the heading "Can you elaborate on the technical issue?"

jph 5y ago

> Eventually there was no way of keeping up with the queue A chess congestion pile up... sounds like an event stream rook-ie mistake. :-) Seriously congrats to Lichess for growing. It's an amazing site. Donate if you can.

nijave 5y ago

Anyone take a look into the code and see why it can't be parallelized? The bottom of the FAQ mentions that but I'd think at least certain aspects should be parallelizable or at least be prioritizable (like maybe forgoing leaderboard updates to focus on more important events?)

jeremyjh 5y ago
If you think about it, you can't really generate pairings in parallel because each thread would need write access to the entire pool to ensure no one is paired twice and that you have a consistent view of all the results to that point before creating a new pair. You could maybe create a lock for each participant but that might actually be slower, and would definitely be more difficult to reason about and can lead towards bugs just as serious under load.
- saghm 5y ago
  I'm not sure exactly what the algorithm is for choosing pairs, but naively it seems like it should be possible to bucket the players in some way, at which point the pairings within each bucket can be generated in parallel, with buckets getting merged if they get too small. The tricky part is coming up with a way to bucket them that's both efficient and will lead to enough pairings, and I'm not sure what the criteria is. If it's completely random, then ideally any bucketing should work fine. If it's based on ratings or existing records, it seems like starting by grouping players within certain ranges of that stat should work (assuming the boundaries can be picked based on the distribution of the players). I don't know enough to know for sure that bucket grouping would work here, but it doesn't seem like knowing the entire set of results beforehand should be necessary (unless there's some standard algorithm for choosing pairings that's very prescriptive).
  - dmurray 5y ago
    The pairing is based on ratings and existing records, and happily isn't dependent on a standard prescriptive algorithm, unlike traditional Swiss tournaments. It would be fine to accept some reordering of game-finishing events. I'm certain the process can be parallelized - it just hadn't been considered necessary before.

than3 5y ago

I expect that what they've pointed to as the cause is only part of the problem. We'll never know the full picture unless they share it. Being upfront, my experience of the people in charge of the organization there doesn't have much goodwill left. Nothing against Thibault personally, I think he's done some great things but seems to be busy with whatever he's interested in and management isn't it, and he has some unprofessional people with access that work for the project/charity. I had volunteered my services years ago as a System Administrator (no charge) with suggestions, but with limited modes of communication, multiple issues going stale, no response with an auto issue closed. They have issues that don't get addressed, and their process doesn't appare to be aimed to cultivate qualified volunteers or improve the bus factor of the project. To make things worse, when I expressed disagreement with constructive feedback regarding one of their process decisions, one of the other dev's with access apparently took offense and the next day I found my lichess account had been edited by an admin without notice or notification. I could not log in (wrong password), the password and email for password recovery were changed, and trying to access the profile URI directly showed the account as banned. It seemed this was done out of spite, and definitely without any kind of due process. Appeals by email went unanswered within the 90 day cutoff I gave them. As a result I submitted a complaint to the french charities regulatory body and moved on since the group wasn't worth wasting any more of my time. I haven't heard back so who knows if anything came of what I reported. In my opinion, they've got more internal problems than they let on, and to me this is just spillover. Its unfortunate because any failure like this impacts so many people, but I don't find it surprising given my limited experience of the people there.

Shadonototra 5y ago
- than3 5y ago
  Yes, that is the only place you can communicate with them. If they haven't tampered with the history it should still be there for anyone that wants to dig. I was simply describing my experience and yet someone always tries to minimize or discredit assuming some other motive; whatever, the world isn't all rainbows and sunshine. You seem to have missed the point, didn't read, both, or have some other agenda. To turn it around, what's your motive for being 'unusually' defensive? You know there are three lichess right, the organization, the service, and the OSS code. What are we talking about here? The former, not the latter two. At this point I could care less how they run their ship. I've learned that what they sell to the public, and the process by which they operate are different things, and through their actions that they don't value processes that are generally accepted and work. As a result, I won't have anything to do with them, but its annoying how many people blindly have to assert the assumption that Lichess (the organization) can do no wrong, and the marketed narrative is the whole picture, and implying that anyone saying differently is attacking or has some subversive plan to hurt the community. Honestly, that kind of propaganda strategy sounds just like something one might hear in the Nuremberg trials (Henry Goering interview with Goebbels). To be clear, I only posted today because this spoiled my newsfeed. If they want to be transparent they should publish a professional post-mortem.

powera 5y ago

The "easy" solution is to put a cap on tournament size. Apart from when Agadmator wants his fans to be in the same tournament as Magnus Carlsen etc., there is basically no need to hold chess tournaments with over 1000 players.

tephra 5y ago
The lichess marathon tournaments usually are a lot more than 1000 players.
- mwt 5y ago
  For example, a recent 24-hour marathon had 24,000 players. Though not all playing at once, surely there were periods of time in which several thousand were playing/pairing at once. https://lichess.org/tournament/autumn21 https://lichess.org/tournament/autumn21

Thorentis 5y ago

For some things there is simply no way to parallelise events when things need to be processed in order, and the limitation is simply the processing capacity. In this instance I'm surprised Lichess didn't have surge capacity to automatically scale up for a big tournament like this. Sure, it might be expensive, but that's when optimisation should come in (to save costs). Having to cancel a major event like this due to poor optimisation is a deployment failure moreso than a coding failure.

nijave 5y ago
You're basically talking mainframes looking at vertically scaling for high throughput on something serial
weird-eye-issue 5y ago
It's not going to go faster if you add more workers to a queue that has to be processed one by one
- Thorentis 5y ago
  It's not just a case of adding more workers, but adding more processing power and removing bottlenecks so that each event can leave the queue faster.
  - weird-eye-issue 5y ago
    I doubt it's as simple as adding processing power. That only helps in a linear fashion and it sounds like it was a bigger problem. Probably was IO bound. Also removing bottlenecks is not related to surge capacity but rather code level optimizations that they didn't realize were needed until this happened so it's easy to say that after the fact...

lom 5y ago

You could maybe paralyze the matchmaking process by making buckets in which players of all the elos are ranked. Nobody in the top 100 will be paired with someone who is in the last 2000 anyways. So why not split it up on every 1000th player and treat them separately. The leaderboard then only needs to modified (read can be done without race conditions) by the top 100 or whatever.

f3rnando 5y ago