6 ms·

Silly question: they have 5.7TB in their database... How come? It's a dating app founded in 2012, I can understand that one can accumulate such much data in 11

Silly question: they have 5.7TB in their database... How come? It's a dating app founded in 2012, I can understand that one can accumulate such much data in 11 years, but sure you can periodically archive "unused" data and move it out of your primary database, right? I mean, are the 5.7TB of data actually needed in a daily basis by their app? (I assume data for analytical purposes is not stored in their primary DB, which is fair to assume I believe)

diziet 6y ago
Imagine there are 10m users. That's 600kb per user.
- mbyio 6y ago
  And you have to account for indexes, temporary tables used for data analysis, etc. And most of it is probably not compressed. So with that perspective it isn't that much data at all.
- stickfigure 6y ago
  That's an incredibly large amount per user? I have worked on a couple online dating sites, including one that was fairly popular (Let's Date - which stiffed me for my last invoice before they went belly up grrr). Unless you're storing images in the database, it's really hard to generate 600k for a dating workload - even with indexes. The only thing I can imagine generating 600k per user is putting something like "hit tracking" in the database. Which I've done - yes it adds up - but it's also relatively easy to move to some other kind of store.
  - curryst 6y ago
    If messages between users go into their main database, then that would be a pretty reasonable amount.
    - stickfigure 6y ago
      600k is a sizable book. The Adventures of Huckleberry Finn is 600k. And that's the average per-user; most users will never send or receive messages. The only thing I can imagine is that they do an incredible amount of activity tracking.
- mrweasel 6y ago
  That's really a good way of looking at it. I though it sounded like a lot of data, well, 600kb is a lot of textual data, but who knows what they have stuffed into the database. I worked for an e-commerce site, with a few million customers, even more orders, data-duplication all over the place, and still we where using a perhaps a 200GB of database storage.
jamesmishra 6y ago
5.7 TB is small by database standards. I work at a much smaller company and deal with "proportionally" much more data. I don't know if it would be worth the engineering effort to try and archive old data in a way that still makes it transparently accessible to users that go looking for it--especially when modern databases ought to be able to scale up and out without manual archiving.
- systems 6y ago
  its huge by database standard, i worked in large multinationals and dealt with some of the their largest databases 5.7 is enormous by database standard , there is no way you can get good query performance on a 5.7 tb database without solid physical partitioning and heavily optimized queries, and most normal companies even with 200-500 GB database use datamarts to have good performance without a super complex architectures and geniuses working fro you in db admin department the more i think about it, the more i think that 5.7 TB would be unusably huge, and if you have this much data, most wont even bother to partition, the db will be broken into several (hundreds) smaller databases
  - derekperkins 6y ago
    5TB is not that large and it's not that difficult to get good performance. We operated a 50TB single instance of MySQL for 5 years with a tiny team before migrating to Vitess, and it was basically zero maintenance. We did partition our largest table, which just requires a little extra SQL, but is otherwise transparent.
    - throwdbaaway 6y ago
      You guys are talking past each other, as your workloads appear to be different. With a traditional RDBMS, size usually wouldn't be a bottleneck, as long as you can add enough disks to a single server. Write operations per second, that's the metric I would care about. A 50TB instance with low amount of write operations can be zero maintenance, while a 500GB instance with high amount of write operations can be a real pain.
sharadov 6y ago
Not a lot, but there may be options to partition, but again, you can't comment unless you know the design.
philipwhiuk2 6y ago
Purely at a guess, people's images are stored in the app as blobs because it's "easier"
- tommyzli 6y ago
  Nope.
  - jbverschoor 6y ago
    Probably some excessive data hoarding of location information, and perhaps too much meta data for each request. Anyways, seems more like a liability than an achievement
- kevas 6y ago
  What was the strategy for moving things over? By age? Monitoring queries and determining what data isn’t being queried? Something else?
YuriNiyazov 6y ago
As someone who manages an order of magnitude greater database than that for an app founded in 2008 (and that's after multiple archivings of unused data, and no severe crimes like storing image blobs in the DB), I can tell you that, uhm, Coffee Meets Bagel is either not that successful, or is doing a very good job at managing their DB size.
hermanradtke 6y ago
Agreed. I ran a 1 billion dollar GMV e-commerce company and our primary OLTP database was around 60 GB. Everything else was moved to an OLAP database.
wejick 6y ago
I think it's because they use Django and naturally it's a monolithic architecture. We will hardly find database this big in microservice world, instead there will be several smaller databases. Now whether several smaller DBs is easier to manage compared to one big one, it would be debatable. However with that huge DB, I would prefer having several smaller one.

diziet 6y ago

Imagine there are 10m users. That's 600kb per user.

mbyio 6y ago
And you have to account for indexes, temporary tables used for data analysis, etc. And most of it is probably not compressed. So with that perspective it isn't that much data at all.
stickfigure 6y ago
That's an incredibly large amount per user? I have worked on a couple online dating sites, including one that was fairly popular (Let's Date - which stiffed me for my last invoice before they went belly up grrr). Unless you're storing images in the database, it's really hard to generate 600k for a dating workload - even with indexes. The only thing I can imagine generating 600k per user is putting something like "hit tracking" in the database. Which I've done - yes it adds up - but it's also relatively easy to move to some other kind of store.
- curryst 6y ago
  If messages between users go into their main database, then that would be a pretty reasonable amount.
  - stickfigure 6y ago
    600k is a sizable book. The Adventures of Huckleberry Finn is 600k. And that's the average per-user; most users will never send or receive messages. The only thing I can imagine is that they do an incredible amount of activity tracking.
mrweasel 6y ago
That's really a good way of looking at it. I though it sounded like a lot of data, well, 600kb is a lot of textual data, but who knows what they have stuffed into the database. I worked for an e-commerce site, with a few million customers, even more orders, data-duplication all over the place, and still we where using a perhaps a 200GB of database storage.

jamesmishra 6y ago

5.7 TB is small by database standards. I work at a much smaller company and deal with "proportionally" much more data. I don't know if it would be worth the engineering effort to try and archive old data in a way that still makes it transparently accessible to users that go looking for it--especially when modern databases ought to be able to scale up and out without manual archiving.

systems 6y ago
its huge by database standard, i worked in large multinationals and dealt with some of the their largest databases 5.7 is enormous by database standard , there is no way you can get good query performance on a 5.7 tb database without solid physical partitioning and heavily optimized queries, and most normal companies even with 200-500 GB database use datamarts to have good performance without a super complex architectures and geniuses working fro you in db admin department the more i think about it, the more i think that 5.7 TB would be unusably huge, and if you have this much data, most wont even bother to partition, the db will be broken into several (hundreds) smaller databases
- derekperkins 6y ago
  5TB is not that large and it's not that difficult to get good performance. We operated a 50TB single instance of MySQL for 5 years with a tiny team before migrating to Vitess, and it was basically zero maintenance. We did partition our largest table, which just requires a little extra SQL, but is otherwise transparent.
  - throwdbaaway 6y ago
    You guys are talking past each other, as your workloads appear to be different. With a traditional RDBMS, size usually wouldn't be a bottleneck, as long as you can add enough disks to a single server. Write operations per second, that's the metric I would care about. A 50TB instance with low amount of write operations can be zero maintenance, while a 500GB instance with high amount of write operations can be a real pain.

sharadov 6y ago

Not a lot, but there may be options to partition, but again, you can't comment unless you know the design.

philipwhiuk2 6y ago

Purely at a guess, people's images are stored in the app as blobs because it's "easier"

tommyzli 6y ago
Nope.
- jbverschoor 6y ago
  Probably some excessive data hoarding of location information, and perhaps too much meta data for each request. Anyways, seems more like a liability than an achievement
kevas 6y ago
What was the strategy for moving things over? By age? Monitoring queries and determining what data isn’t being queried? Something else?

YuriNiyazov 6y ago

As someone who manages an order of magnitude greater database than that for an app founded in 2008 (and that's after multiple archivings of unused data, and no severe crimes like storing image blobs in the DB), I can tell you that, uhm, Coffee Meets Bagel is either not that successful, or is doing a very good job at managing their DB size.

hermanradtke 6y ago

Agreed. I ran a 1 billion dollar GMV e-commerce company and our primary OLTP database was around 60 GB. Everything else was moved to an OLAP database.

wejick 6y ago

I think it's because they use Django and naturally it's a monolithic architecture. We will hardly find database this big in microservice world, instead there will be several smaller databases. Now whether several smaller DBs is easier to manage compared to one big one, it would be debatable. However with that huge DB, I would prefer having several smaller one.