3 ms·

That was an interesting read. I'd love to watch the actual presentation if it's available somewhere. One thing that struck me as odd from a software design poi

by Udo 15y ago

That was an interesting read. I'd love to watch the actual presentation if it's available somewhere. One thing that struck me as odd from a software design point of view is the "tile table". The name and the absurdly high number of I/Os per second suggest that each tile was stored in a separate database record? The game looks like a Farmville clone, so wouldn't it have been more economical to store the entire farm of a player in one blob? Is there someone from Wooga here to shed some light on the architectural decisions that went into the game?

bermanoid 15y ago
I've got to agree - 1M daily users and 50k db hits per second indicates something is being done in a ridiculously inefficient manner somewhere, and I'm really surprised that the solution was to fiddle around on the server-side. Suppose your players are active for an average of, say, 20 minutes - that's a damn generous upper bound for a Facebook game. That would mean that 1M / (24*3) = 13,888 users are active (on average) at any time. Which means that they must each be generating db events at a rate of about 3.6 events / sec / user, which is ludicrous (at the very least, these should probably be somehow combined so that each client only hits the db once in a while, preferably in response to a user action). And in reality, the average play session is probably quite a bit shorter than 20 minutes, which means the event rate is even higher per active user. What the hell are they doing that is causing so much database traffic? Is the game actually doing something a lot more sophisticated than I would assume it is based on the looks of it (to be fair, I've never played it), or is the client-side design just really that messed up? I mean, it's cool that they can handle that much traffic, and all, and the server guys should be proud, but IMO they should really be able to do better on the client-side to prevent this level of scaling from being necessary...
- joelhaasnoot 15y ago
  Have you played these games? Everything you do on your farm/plot/area is a click on a tile. You're essentially trying to click on as many tiles in as little time as possible for your own sanity: there's no "select all".
  - jules 15y ago
    You can preload/cache all or nearly all data on the client side, and batch updates.
    - jrirei 15y ago
      We do preload all data on client side as the client must never wait on server responses for better user experience. But we cannot batch on client side as we cannot foresee when the user will kill the client. And actually we have more than 40K parallel user sessions as players come back more than once each day.
jrirei 15y ago
The presentation was recorded by InfoQ. I hope they will make it available during in the next 1-2 months. There's a lot I said but did not put on the slides. You are correct in your assumption that each Tile had been a record in a MySQL table (now it is a value i a Redis hash). Actually we considered using a "blob" approach. But in the client we cannot batch requests as we cannot foresee when the user will simply kill the Flash client to go to some other site. So when a user request (i.e. a game event) arrives in the server there is no way to know if another request will follow. So we have to persist that change right away. This is using a stateless server. In a later game called Magic Land we are going for a stateful Erlang server. There we keep the whole user state in RAM while the user plays and persist state changes every minute or so. Here we can do without any database and just use S3 for persistence. Works just great. On the upcoming Erlang User Conference we will give an update on that project and slides will be available at Slideshare next week, too. In the meantime please have a look at this old slide set to explain the concept in detail: http://www.slideshare.net/wooga/erlang-the-big-switch-in-social-games http://www.slideshare.net/wooga/erlang-the-big-switch-in-soc...
- Udo 15y ago
  Thanks for clearing this up. > Actually we considered using a "blob" approach. But in the client we cannot batch requests as we cannot foresee when the user will simply kill the Flash client to go to some other site. So when a user request (i.e. a game event) arrives in the server there is no way to know if another request will follow. So we have to persist that change right away. < That's an understandable dilemma. I love thinking about stuff like this and see how other people are dealing with these challenges, so please forgive my Sunday morning quarterbacking ;-) Wouldn't the issue have been solvable by creating an relatively simple persistent software layer between the app code manipulating the tiles and the backend storage? I understand that you moved to this model with your Erlang game, but I'd like to know if a persistence/caching layer was considered for the farming game? More generally, somewhere in here is an idea for a great Node.js server project that takes coarse grained datasets from a contentious database and serves as an interface for finer grained portions of that data.
  - jrirei 15y ago

by Udo 15y ago

bermanoid 15y ago

I've got to agree - 1M daily users and 50k db hits per second indicates something is being done in a ridiculously inefficient manner somewhere, and I'm really surprised that the solution was to fiddle around on the server-side. Suppose your players are active for an average of, say, 20 minutes - that's a damn generous upper bound for a Facebook game. That would mean that 1M / (24*3) = 13,888 users are active (on average) at any time. Which means that they must each be generating db events at a rate of about 3.6 events / sec / user, which is ludicrous (at the very least, these should probably be somehow combined so that each client only hits the db once in a while, preferably in response to a user action). And in reality, the average play session is probably quite a bit shorter than 20 minutes, which means the event rate is even higher per active user. What the hell are they doing that is causing so much database traffic? Is the game actually doing something a lot more sophisticated than I would assume it is based on the looks of it (to be fair, I've never played it), or is the client-side design just really that messed up? I mean, it's cool that they can handle that much traffic, and all, and the server guys should be proud, but IMO they should really be able to do better on the client-side to prevent this level of scaling from being necessary...

joelhaasnoot 15y ago
Have you played these games? Everything you do on your farm/plot/area is a click on a tile. You're essentially trying to click on as many tiles in as little time as possible for your own sanity: there's no "select all".
- jules 15y ago
  You can preload/cache all or nearly all data on the client side, and batch updates.
  - jrirei 15y ago
    We do preload all data on client side as the client must never wait on server responses for better user experience. But we cannot batch on client side as we cannot foresee when the user will kill the client. And actually we have more than 40K parallel user sessions as players come back more than once each day.

jrirei 15y ago

The presentation was recorded by InfoQ. I hope they will make it available during in the next 1-2 months. There's a lot I said but did not put on the slides. You are correct in your assumption that each Tile had been a record in a MySQL table (now it is a value i a Redis hash). Actually we considered using a "blob" approach. But in the client we cannot batch requests as we cannot foresee when the user will simply kill the Flash client to go to some other site. So when a user request (i.e. a game event) arrives in the server there is no way to know if another request will follow. So we have to persist that change right away. This is using a stateless server. In a later game called Magic Land we are going for a stateful Erlang server. There we keep the whole user state in RAM while the user plays and persist state changes every minute or so. Here we can do without any database and just use S3 for persistence. Works just great. On the upcoming Erlang User Conference we will give an update on that project and slides will be available at Slideshare next week, too. In the meantime please have a look at this old slide set to explain the concept in detail: http://www.slideshare.net/wooga/erlang-the-big-switch-in-social-games http://www.slideshare.net/wooga/erlang-the-big-switch-in-soc...

Udo 15y ago
Thanks for clearing this up. > Actually we considered using a "blob" approach. But in the client we cannot batch requests as we cannot foresee when the user will simply kill the Flash client to go to some other site. So when a user request (i.e. a game event) arrives in the server there is no way to know if another request will follow. So we have to persist that change right away. < That's an understandable dilemma. I love thinking about stuff like this and see how other people are dealing with these challenges, so please forgive my Sunday morning quarterbacking ;-) Wouldn't the issue have been solvable by creating an relatively simple persistent software layer between the app code manipulating the tiles and the backend storage? I understand that you moved to this model with your Erlang game, but I'd like to know if a persistence/caching layer was considered for the farming game? More generally, somewhere in here is an idea for a great Node.js server project that takes coarse grained datasets from a contentious database and serves as an interface for finer grained portions of that data.
- jrirei 15y ago