4 ms·
The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. No
by cmer 15y ago
The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down.
Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique.
And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
- jacques_chester 15y ago> The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.