4 ms·

It's mostly meta data and hashes of stuff. Uncompressed. Most of it in MongoDB, which is probably one of the worst technical decisions we made, for all the reas

by cmer 15y ago

It's mostly meta data and hashes of stuff. Uncompressed. Most of it in MongoDB, which is probably one of the worst technical decisions we made, for all the reasons that have been discussed at length.

jacques_chester 15y ago
At a guess, then, your dataset is larger because of duplicated data. One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression. In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on. Compression would help a lot, if it was available (some brief Googling suggests it isn't). (edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
- cmer 15y ago
  The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down. Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique. And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
  - jacques_chester 15y ago
    > The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.
mwerty 15y ago
Do you have a link on why mongodb is bad?
- cmer 15y ago
  This has been discussed so many times here, especially recently. Search HN you'll find tons of info. MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock. I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...
yourapostasy 15y ago
If you had a chance to do it over again, which store would you have picked instead of Mongo?
- cmer 15y ago
  I haven't researched this much, but my gut tells me Cassandra and/or Postgres. One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs. I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock. The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.

by cmer 15y ago

It's mostly meta data and hashes of stuff. Uncompressed. Most of it in MongoDB, which is probably one of the worst technical decisions we made, for all the reasons that have been discussed at length.

jacques_chester 15y ago

At a guess, then, your dataset is larger because of duplicated data. One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression. In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on. Compression would help a lot, if it was available (some brief Googling suggests it isn't). (edit) My guess is that you already know all of the above. I'm not really helping, am I? :/

cmer 15y ago
The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down. Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique. And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
- jacques_chester 15y ago
  > The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.

mwerty 15y ago

Do you have a link on why mongodb is bad?

cmer 15y ago
This has been discussed so many times here, especially recently. Search HN you'll find tons of info. MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock. I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...

yourapostasy 15y ago

If you had a chance to do it over again, which store would you have picked instead of Mongo?

cmer 15y ago
I haven't researched this much, but my gut tells me Cassandra and/or Postgres. One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs. I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock. The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.