4 ms·

I was surprised at how small their database is! We (Defensio) store about 300gb of data, and most if it is purged after 60 days. We're quite far from being 274

by cmer 15y ago

I was surprised at how small their database is! We (Defensio) store about 300gb of data, and most if it is purged after 60 days. We're quite far from being 274th largest website in the world as well, I assume. It's just very interesting to see how such a huge website can use so little storage.

jaequery 15y ago
I'm really surprised by this as well.
zitterbewegung 15y ago
What do they have in the database other than text fields and profile information? They don't store many pictures or anything else do they? Thats probably why they don't need lots of space.
jacques_chester 15y ago
How are you storing your data? And is it compressed? ... I suppose for spam, the problem is that as a side-effect of trying to fool Bayesian filters, there's a lot of incompressible gibberish.
- cmer 15y ago
  It's mostly meta data and hashes of stuff. Uncompressed. Most of it in MongoDB, which is probably one of the worst technical decisions we made, for all the reasons that have been discussed at length.
  - jacques_chester 15y ago
    At a guess, then, your dataset is larger because of duplicated data. One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression. In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on. Compression would help a lot, if it was available (some brief Googling suggests it isn't). (edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
    - cmer 15y ago
      The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down. Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique. And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
      - jacques_chester 15y ago
        > The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.
  - mwerty 15y ago
    Do you have a link on why mongodb is bad?
    - cmer 15y ago
      This has been discussed so many times here, especially recently. Search HN you'll find tons of info. MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock. I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...
  - yourapostasy 15y ago
    If you had a chance to do it over again, which store would you have picked instead of Mongo?
    - cmer 15y ago
      I haven't researched this much, but my gut tells me Cassandra and/or Postgres. One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs. I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock. The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.
Devilboy 15y ago
Yea 100GB is tiny! Our database is several times that big and we're just some random B2B company with a few dozen employees. And we don't use sharding or whatever, it's all just one machine with lots of RAM. We've not even moved it to SSDs yet.

by cmer 15y ago

jaequery 15y ago

I'm really surprised by this as well.

zitterbewegung 15y ago

What do they have in the database other than text fields and profile information? They don't store many pictures or anything else do they? Thats probably why they don't need lots of space.

jacques_chester 15y ago

How are you storing your data? And is it compressed? ... I suppose for spam, the problem is that as a side-effect of trying to fool Bayesian filters, there's a lot of incompressible gibberish.

cmer 15y ago
It's mostly meta data and hashes of stuff. Uncompressed. Most of it in MongoDB, which is probably one of the worst technical decisions we made, for all the reasons that have been discussed at length.
- jacques_chester 15y ago
  At a guess, then, your dataset is larger because of duplicated data. One of the nice things about normalisation is that, by design, it removes such duplications. The data is reduced to the smallest usable form reasonably possible without compression. In your case, and I'm just guessing here, there'll be many many cases of duplicate data. Whereas SO would store a user email once, you might be storing it hundreds or thousands of times. And so on. Compression would help a lot, if it was available (some brief Googling suggests it isn't). (edit) My guess is that you already know all of the above. I'm not really helping, am I? :/
  - cmer 15y ago
    The problem with de-duplication is that it actually takes resources to do that. At the rate we're serving requests, everything has to go as fast as possible. Normalizing can actually slow things down. Our app is not quite the typical web app. we can't cache much, we serve 8-9 digits api request every day and we try to do it as quickly as possible. Every request is different and unique. And don't worry, I'm not insulted :) Hopefully this conversation will give some insight to people about how we do things.
    - jacques_chester 15y ago
      > The problem with de-duplication is that it actually takes resources to do that. Normalisation doesn't deduplicate by performing comparisons; it works by ensuring that there is one and only one logical place for the data to go. On OLTP tasks highly normalised databases as a rule perform better. Where normalised data extracts its pound of flesh is in 1. join times and 2. the indices created to support reads and joins. Yours is a case where, given the pattern of traffic, a highly denormalised data store makes sense. I just thought I'd explain to listeners-in why relational data stores can often do something with less total disk usage.
- mwerty 15y ago
  Do you have a link on why mongodb is bad?
  - cmer 15y ago
    This has been discussed so many times here, especially recently. Search HN you'll find tons of info. MongoDB is great at small scale. Basically, if everything fits in memory, it's awesome. If not, it slowly becomes a major pain in the butt, especially because of the global write lock. I love all the concepts behind Mongo. Replication is so easy, and the data structure is extremely flexible. I just wish the implementation was as good. Hopefully one day...
- yourapostasy 15y ago
  If you had a chance to do it over again, which store would you have picked instead of Mongo?
  - cmer 15y ago
    I haven't researched this much, but my gut tells me Cassandra and/or Postgres. One of the main reasons why we escaped MySQL was that we were stuck with out schema. Adding an index or a column basically copies the table to a temporary table, apply the change and copies is back in place. That's my understanding, which might be a little bit wrong. All I know is that making one very simple change took hours and hours because our tables were so big. It was quite ridiculous. We couldn't afford the downtime, so we were stuck with what we had, which was no longer sufficient for our needs. I understand that Postgres doesn't have that limitation, but since our schema used to change so much and we has so many joined tables, the MongoDB data structure seemed like a great fit. Mongo was also amazingly fast in our tests, but those tests didn't properly take into consideration the global write lock. The global write lock problem is a very well known issue now but we started using MongoDB before 1.0; way back when nobody really knew anything about it. At least people are now more informed, although it doesn't seem to prevent many from making the bad decision of using it.

Devilboy 15y ago

Yea 100GB is tiny! Our database is several times that big and we're just some random B2B company with a few dozen employees. And we don't use sharding or whatever, it's all just one machine with lots of RAM. We've not even moved it to SSDs yet.