8 ms·

It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers

It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? Oh, I guess it's 100% utilization, not 100% load. As in, they don't spin down servers to save power during off-hours I guess. The time dilation is a neat solution to the server load problem, but it's sooo annoying as a player. In beta, it was interesting to watch the entire game desync and grind to a halt, but we could still chat and look around. Interesting, but frustrating.

incision 14y ago
>It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? I wasn't sure how to interpret that either. I've always been a bit surprised that the the backend systems for these MMOs aren't a bit more flexible. Though, EvE did launch nearly 10 years ago and it has never had a huge number of subscribers. I have no idea whether more recent games have solved these problems.
- namidark 14y ago
  I can't think of any games that deal with PVP on this scale either.
- mmanfrin 14y ago
  I wouldn't call over 400,000 players '[not] a huge number'. It's the 2nd or 3rd most popular paid-subscription MMO.
  - incision 14y ago
    The specific, relative number of subscribers is not the point. That said, I wouldn't call something huge when there are comparables (even if it's just one or two) which dwarf it and others which (at some point) exceeded it.
  - flexd 14y ago
    As you can see here http://eve-offline.net/?server=tranquility http://eve-offline.net/?server=tranquility , there is about 44k people online at this point, probably a bit less since most people have more than one account. That seems like a fairly substantial amount of users in a single world. Is there any other game world with this many active users in a single world?
- Goronmon 14y ago
  I have no idea whether more recent games have solved these problems. Most games solve this problem with completely separate servers and population caps. I know in Guild Wars 2 they actually stop displaying players past a certain number to improve performance. This works fine in calm areas like cities, but in PvP it causes issues with being killed by "invisible" groups that the game fails to load in time. I may be remembering incorrectly, but last I remember reading CCP was basically in uncharted territory on the tech front as far as EVE Online is concerned. No other game developer has even tried to tackle this problem at the level they have, and I can't think of many applications in the world that could comparable in scale and complexity to what the EVE servers have to deal with. Edit: To summarize, even being 10 years old, no game has come close to matching it in this context.
  - xenophanes 14y ago
    Guild Wars 2 is stupid about this. At least on release, they stopped displaying ENEMIES FIRST. So if you're in a big enough group, a couple enemies can walk up and none of you can see them b/c you hit the limit from displaying allies, but they can still see some of you guys and kill you while invisible. If they prioritized displaying just party members (parties only go up to 5 people) and maybe people on your friends list, and then enemies, it would have worked better.
    - saraid216 14y ago
      They supposedly just fixed this in the patch yesterday. As a player who gets owned by unrendered thieves, I'm not convinced. -_- (OTOH, I'm also pretty bad at PvP anyways, so whatever.)
- outworlder 14y ago
  400k+ active players IS a huge number. Specially considering they are in the same logical server. Other MMO's handle this by having copies of the world and splitting the population - which is the same approach used by Ultima Online back in 1997. So actually, EVE is way ahead of the rest, technology-wise.
  - incision 14y ago
    >So actually, EVE is way ahead of the rest, technology-wise. That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. In any case, I'm not making a dig at EvE, rather acknowledging that the game has had a long life and has never been a multi-million subscriber behemoth. Therefore, they might make concessions, or allowed the persistence of previous limitations / decisions in design/infrastructure for lack of resources [1][2]. 1: http://news.ycombinator.com/item?id=5135873 http://news.ycombinator.com/item?id=5135873 2: http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onlines-server-model/ http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onli...
    - Goronmon 14y ago
      That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. The "typical" MMO doesn't even try to handle large populations. The use of separate servers, population caps and login queues are how just about every one else deals with congestion problems. Even from your second link there is the quote "Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry." Sure that is from 2008, but newer MMOs are built using the same overall server architecture you saw back in the days of Ultima Online.
    - vacri 14y ago
      400k subscribing users is a great big pile of cash to work with.
iharris 14y ago
If memory serves, I believe that they mean that the servers run at 100% capacity, not 100% load ie. CCP does not throttle back system resources during periods of light use.
cdash 14y ago
The real problem for them that forces them to use time dilation is the fact that a solar system can only run on one core. They have a few machines with a 4.4 Ghz or greater processor I believe for system that are high load but currently they are extremely limited in what they can do hardware wise.
alexkus 14y ago
Two main solutions (given that the article mentions that they have huge machines already):- A) Allow a single solar system to span multiple machines. Very hard, especially if the server software isn't architected for this. Retrofitting this can be nigh on impossible. B) Have a few huge machines that can be used to host scenarios like this and, more importantly, have a way of migrating users over to the huge machine seemlessly. The latter can be done but it's tricky, especially if transferring game state between instances of the server is not simple (I'm not talking about transferring the VM itself with something like vMotion). It comes down to:- 1) Being able to make the bigger machine act as a temporary proxy pushing connections data back to the smaller machine. 2) Having a way of telling clients to make a new connection to the bigger machine and, once that connection is made (and the data is being proxied to the smaller machine) cut the connection to the smaller machine. Users see no loss of service or reconnects at all. 3) Once all clients are now being proxied by the bigger machine; pause and transfer the game state from the smaller machine to the big machine and then continue. Obviously it works best if a chunk of state can be transferred in the background and then the final transfer (and pause) is as short as possible in order to transfer over the bang up to the minute state. Option (A) is always the proverbial "In v2 of the server we'll do it a completely different way..."
- roc 14y ago
  Which never seems to come around, because the new hardware is so much faster that it can host what were previously problematic server loads without a problem, and you've got a million other things to write. Yet players have a tendency to figure out when places are too overcrowded to be fun. So your old problematic load is almost never representative of how many players wanted to be in that area, but merely how many players were willing to put up with that level of degraded performance. So upon release (or sufficiently close to it to start stress testing, which is conveniently when it's too late to really change architecture) the new limits are quickly hit.
- saraid216 14y ago
  They've been struggling with the issue of multi-thousand player fights for a while now, and have moved towards both of these solutions but are obviously not quite there yet. For instance, the article actually talks about having said huge machines. There's a way in EVE to inform the GMs about anticipated big fights, at which point they'll do the reinforcement preemptively. In this case, there wasn't such a convenient warning.

incision 14y ago

>It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? I wasn't sure how to interpret that either. I've always been a bit surprised that the the backend systems for these MMOs aren't a bit more flexible. Though, EvE did launch nearly 10 years ago and it has never had a huge number of subscribers. I have no idea whether more recent games have solved these problems.

namidark 14y ago
I can't think of any games that deal with PVP on this scale either.
mmanfrin 14y ago
I wouldn't call over 400,000 players '[not] a huge number'. It's the 2nd or 3rd most popular paid-subscription MMO.
- incision 14y ago
  The specific, relative number of subscribers is not the point. That said, I wouldn't call something huge when there are comparables (even if it's just one or two) which dwarf it and others which (at some point) exceeded it.
- flexd 14y ago
  As you can see here http://eve-offline.net/?server=tranquility http://eve-offline.net/?server=tranquility , there is about 44k people online at this point, probably a bit less since most people have more than one account. That seems like a fairly substantial amount of users in a single world. Is there any other game world with this many active users in a single world?
Goronmon 14y ago
I have no idea whether more recent games have solved these problems. Most games solve this problem with completely separate servers and population caps. I know in Guild Wars 2 they actually stop displaying players past a certain number to improve performance. This works fine in calm areas like cities, but in PvP it causes issues with being killed by "invisible" groups that the game fails to load in time. I may be remembering incorrectly, but last I remember reading CCP was basically in uncharted territory on the tech front as far as EVE Online is concerned. No other game developer has even tried to tackle this problem at the level they have, and I can't think of many applications in the world that could comparable in scale and complexity to what the EVE servers have to deal with. Edit: To summarize, even being 10 years old, no game has come close to matching it in this context.
- xenophanes 14y ago
  Guild Wars 2 is stupid about this. At least on release, they stopped displaying ENEMIES FIRST. So if you're in a big enough group, a couple enemies can walk up and none of you can see them b/c you hit the limit from displaying allies, but they can still see some of you guys and kill you while invisible. If they prioritized displaying just party members (parties only go up to 5 people) and maybe people on your friends list, and then enemies, it would have worked better.
  - saraid216 14y ago
    They supposedly just fixed this in the patch yesterday. As a player who gets owned by unrendered thieves, I'm not convinced. -_- (OTOH, I'm also pretty bad at PvP anyways, so whatever.)
outworlder 14y ago
400k+ active players IS a huge number. Specially considering they are in the same logical server. Other MMO's handle this by having copies of the world and splitting the population - which is the same approach used by Ultima Online back in 1997. So actually, EVE is way ahead of the rest, technology-wise.
- incision 14y ago
  >So actually, EVE is way ahead of the rest, technology-wise. That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. In any case, I'm not making a dig at EvE, rather acknowledging that the game has had a long life and has never been a multi-million subscriber behemoth. Therefore, they might make concessions, or allowed the persistence of previous limitations / decisions in design/infrastructure for lack of resources [1][2]. 1: http://news.ycombinator.com/item?id=5135873 http://news.ycombinator.com/item?id=5135873 2: http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onlines-server-model/ http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onli...
  - Goronmon 14y ago
    That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. The "typical" MMO doesn't even try to handle large populations. The use of separate servers, population caps and login queues are how just about every one else deals with congestion problems. Even from your second link there is the quote "Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry." Sure that is from 2008, but newer MMOs are built using the same overall server architecture you saw back in the days of Ultima Online.
  - vacri 14y ago
    400k subscribing users is a great big pile of cash to work with.

iharris 14y ago

If memory serves, I believe that they mean that the servers run at 100% capacity, not 100% load ie. CCP does not throttle back system resources during periods of light use.

cdash 14y ago

The real problem for them that forces them to use time dilation is the fact that a solar system can only run on one core. They have a few machines with a 4.4 Ghz or greater processor I believe for system that are high load but currently they are extremely limited in what they can do hardware wise.

alexkus 14y ago

Two main solutions (given that the article mentions that they have huge machines already):- A) Allow a single solar system to span multiple machines. Very hard, especially if the server software isn't architected for this. Retrofitting this can be nigh on impossible. B) Have a few huge machines that can be used to host scenarios like this and, more importantly, have a way of migrating users over to the huge machine seemlessly. The latter can be done but it's tricky, especially if transferring game state between instances of the server is not simple (I'm not talking about transferring the VM itself with something like vMotion). It comes down to:- 1) Being able to make the bigger machine act as a temporary proxy pushing connections data back to the smaller machine. 2) Having a way of telling clients to make a new connection to the bigger machine and, once that connection is made (and the data is being proxied to the smaller machine) cut the connection to the smaller machine. Users see no loss of service or reconnects at all. 3) Once all clients are now being proxied by the bigger machine; pause and transfer the game state from the smaller machine to the big machine and then continue. Obviously it works best if a chunk of state can be transferred in the background and then the final transfer (and pause) is as short as possible in order to transfer over the bang up to the minute state. Option (A) is always the proverbial "In v2 of the server we'll do it a completely different way..."

roc 14y ago
Which never seems to come around, because the new hardware is so much faster that it can host what were previously problematic server loads without a problem, and you've got a million other things to write. Yet players have a tendency to figure out when places are too overcrowded to be fun. So your old problematic load is almost never representative of how many players wanted to be in that area, but merely how many players were willing to put up with that level of degraded performance. So upon release (or sufficiently close to it to start stress testing, which is conveniently when it's too late to really change architecture) the new limits are quickly hit.
saraid216 14y ago
They've been struggling with the issue of multi-thousand player fights for a while now, and have moved towards both of these solutions but are obviously not quite there yet. For instance, the article actually talks about having said huge machines. There's a way in EVE to inform the GMs about anticipated big fights, at which point they'll do the reinforcement preemptively. In this case, there wasn't such a convenient warning.