3 ms·
At a guess, then, your dataset is larger because of duplicated data. One of the nice things about normalisation is that, by design, it removes such duplication
by jacques_chester 15y ago
At a guess, then, your dataset is larger because of duplicated data.
One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression.
In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on.
Compression would help a lot, if it was available (some brief Googling suggests it isn't).
(edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
- cmer 15y agoThe problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down. Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique. And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
- jacques_chester 15y ago> The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.