3 ms·

I was surprised to find a game as big as WoW to have a data strucutre which clearly violates some basic forms [1] of database formalization. Isn't database norm

by dnate 8y ago

I was surprised to find a game as big as WoW to have a data strucutre which clearly violates some basic forms [1] of database formalization. Isn't database normalizaton something that any cs program teaches in the first few semesters? [1] https://en.wikipedia.org/wiki/Database_normalization https://en.wikipedia.org/wiki/Database_normalization

mkreis 8y ago
There are very good reasons to violate it. Like performance - you would need to make a sub-select or join, which is costly. Also if you are 100% sure that "no one will ever need more spell effects", it is a valid way to go.
jeremyjh 8y ago
You can't always run a normalized database though; often you need to denormalize for performance reasons. In very massive systems, a normalized database is rare.
- ckaygusu 8y ago
  I'm yet to have the privilege to work on a database which I would call truly "massive", so I'm oblivious to how things work in such environment in reality. I've seen the question and the answer you've given repeats many times, but aren't there exists a plethora of tools like table partitioning, materialized views etc. to allow one to have the database in a normalized form and run it fast? Don't they work in practice?
  - jeremyjh 8y ago
    Table partitioning does nothing to solve the performance problems of joins of very large datasets. Materialized views can help but they are not a panacea. You have to trigger the updates yourself, and because they are updated asynchronously, you will not have a consistent view of your data. E.g. you could write a row to the database and not see it updated in your view for minutes.
- masklinn 8y ago
  More importantly here normalising the database has conceptual and interactive overhead: in the original model all spell properties are in a single table which makes import/export and devtools easy, while the normalised schema splits the same thing over 3 different tables (at least, not sure the example is even complete). I see it as a case of YAGNI. Also this specific issue likely wasn't a problem of mass: the number of spells in the game is not that high, even today. The issue was throughput, IIRC they had high expectations and they got 10x as many signups as they'd expected.
actsasbuffoon 8y ago
Normalization is the ideal, but sometimes you have to de-normalize for performance. Big, complex joins can be a performance killer. Most apps won't reach the scale where they need to worry about this, but WoW definitely has reached it.
- Const-me 8y ago
  > Most apps won't reach the scale where they need to worry about this That’s true for servers running on modern hardware (lots of RAM, extremely fast SSDs), but when I’m working on desktop, mobile or embedded apps, some of them need embedded databases, and I still need to worry about it. On desktops not all users have SSDs, so when the data is larger than a couple of GB i.e. can’t be cached in RAM, every IO takes 4-5ms because HDD seek latency. Mobile/embedded devices typically have flash memory so the IO latency is lower, but that flash memory is not as fast as modern desktop or servers SSDs. E.g. some Android phones have 2000 random read IOPS, 220 random write IOPS https://www.phonearena.com/news/Android-storage-speed-comparison-which-phone-has-the-fastest-IO-performance_id65588 https://www.phonearena.com/news/Android-storage-speed-compar... P.S. I even need to worry about this when I’m designing RAM-only data structures. On modern systems, RAM is block device (the block size is 128 bit for dual-channel RAM), and there’s a pre-fetcher silicon in CPUs making sequential access much faster than random access.
scrollaway 8y ago
I RE'd WoW for ~10 years. You'd be horrified if you knew how many hacks and crazy-bad technical designs are in the game. The team sometimes infamously has (or had, as of a couple expansions ago) hundreds of thousands of open issues and sometimes squashes bugs which are several expansions old. Incidentally, they also quite often create new bugs in super-old content due to changes in newer content. I mean, the work they do is incredible. But it blows my mind how they manage to maintain this game given the amount of intertwined legacy within it. I'm not sure what exactly caused it to become as bad as it did. It's probably a mix of the game being far more popular than they originally anticipated, the team being incredibly large, and the engine itself not being very accomodating.
- vcfg 8y ago
  I think it’s a miracle old instances work. They rewrote some of them with Cataclysm, but others have been pristine for 10+ years. And they say Windows works hard for compatibility... :P

by dnate 8y ago

mkreis 8y ago

There are very good reasons to violate it. Like performance - you would need to make a sub-select or join, which is costly. Also if you are 100% sure that "no one will ever need more spell effects", it is a valid way to go.

jeremyjh 8y ago

You can't always run a normalized database though; often you need to denormalize for performance reasons. In very massive systems, a normalized database is rare.

ckaygusu 8y ago
I'm yet to have the privilege to work on a database which I would call truly "massive", so I'm oblivious to how things work in such environment in reality. I've seen the question and the answer you've given repeats many times, but aren't there exists a plethora of tools like table partitioning, materialized views etc. to allow one to have the database in a normalized form and run it fast? Don't they work in practice?
- jeremyjh 8y ago
  Table partitioning does nothing to solve the performance problems of joins of very large datasets. Materialized views can help but they are not a panacea. You have to trigger the updates yourself, and because they are updated asynchronously, you will not have a consistent view of your data. E.g. you could write a row to the database and not see it updated in your view for minutes.
masklinn 8y ago
More importantly here normalising the database has conceptual and interactive overhead: in the original model all spell properties are in a single table which makes import/export and devtools easy, while the normalised schema splits the same thing over 3 different tables (at least, not sure the example is even complete). I see it as a case of YAGNI. Also this specific issue likely wasn't a problem of mass: the number of spells in the game is not that high, even today. The issue was throughput, IIRC they had high expectations and they got 10x as many signups as they'd expected.

actsasbuffoon 8y ago

Normalization is the ideal, but sometimes you have to de-normalize for performance. Big, complex joins can be a performance killer. Most apps won't reach the scale where they need to worry about this, but WoW definitely has reached it.

Const-me 8y ago
> Most apps won't reach the scale where they need to worry about this That’s true for servers running on modern hardware (lots of RAM, extremely fast SSDs), but when I’m working on desktop, mobile or embedded apps, some of them need embedded databases, and I still need to worry about it. On desktops not all users have SSDs, so when the data is larger than a couple of GB i.e. can’t be cached in RAM, every IO takes 4-5ms because HDD seek latency. Mobile/embedded devices typically have flash memory so the IO latency is lower, but that flash memory is not as fast as modern desktop or servers SSDs. E.g. some Android phones have 2000 random read IOPS, 220 random write IOPS https://www.phonearena.com/news/Android-storage-speed-comparison-which-phone-has-the-fastest-IO-performance_id65588 https://www.phonearena.com/news/Android-storage-speed-compar... P.S. I even need to worry about this when I’m designing RAM-only data structures. On modern systems, RAM is block device (the block size is 128 bit for dual-channel RAM), and there’s a pre-fetcher silicon in CPUs making sequential access much faster than random access.

scrollaway 8y ago

I RE'd WoW for ~10 years. You'd be horrified if you knew how many hacks and crazy-bad technical designs are in the game. The team sometimes infamously has (or had, as of a couple expansions ago) hundreds of thousands of open issues and sometimes squashes bugs which are several expansions old. Incidentally, they also quite often create new bugs in super-old content due to changes in newer content. I mean, the work they do is incredible. But it blows my mind how they manage to maintain this game given the amount of intertwined legacy within it. I'm not sure what exactly caused it to become as bad as it did. It's probably a mix of the game being far more popular than they originally anticipated, the team being incredibly large, and the engine itself not being very accomodating.

vcfg 8y ago
I think it’s a miracle old instances work. They rewrote some of them with Cataclysm, but others have been pristine for 10+ years. And they say Windows works hard for compatibility... :P