3 ms·

Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and

Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics. At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it. After a bit of searching I found the responsible code. It was a client side bug which crashed the server. The line even had a comment mentioning that it wasn't entirely correct, but that the probability of a crash was about one in a million. Well, we had millions of players.

tasuki 5y ago
A client side bug was crashing the server? I always thought the client depended on the server, not vice versa.
- mianas 5y ago
  I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.
  - Agentlien 5y ago
    This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?
    - mianas 5y ago
      When I read "wasn't entirely correct", and "probability of a crash was about one in a million", I wondered how you'd work that value out. Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users. I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.
- Agentlien 5y ago
  It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
  - tasuki 5y ago
    > The server lacked validation in this case and crashed when processing the invalid data. As a backend dev, I'd point to this as the root cause, not the client sending invalid data.
    - Agentlien 5y ago
      I see where you're coming from. I would say it's the cause of the crash but not the root cause of the issue as a whole. Both obviously need to be fixed. Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future. Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.

tasuki 5y ago

A client side bug was crashing the server? I always thought the client depended on the server, not vice versa.

mianas 5y ago
I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.
- Agentlien 5y ago
  This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?
  - mianas 5y ago
    When I read "wasn't entirely correct", and "probability of a crash was about one in a million", I wondered how you'd work that value out. Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users. I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.
Agentlien 5y ago
It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
- tasuki 5y ago
  > The server lacked validation in this case and crashed when processing the invalid data. As a backend dev, I'd point to this as the root cause, not the client sending invalid data.
  - Agentlien 5y ago
    I see where you're coming from. I would say it's the cause of the crash but not the root cause of the issue as a whole. Both obviously need to be fixed. Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future. Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.