4 ms·

I find these kinds of stories interesting, but without some feel for the size of the data, they're not very useful/practical. I've heard of Bump, and used it o

by tlianza 14y ago

I find these kinds of stories interesting, but without some feel for the size of the data, they're not very useful/practical. I've heard of Bump, and used it once or twice, but I don't actually know how big or popular it is. If we're talking about a database for a few million users, only a tiny percentage of which are actively "bumping" at any time, it's really hard for me to imagine this is an interesting scaling problem. Ex. If I just read an article about a "data migration" who's scale is something a traditional DBMS would yawn at, the newsworthiness would have to be re-evaluated.

polynomial 14y ago
I was actually going to make a joke about "if the number of people I know who actually use Bump is any indication, it's not clear they even need a large data store."
yahelc 14y ago
They celebrated 80 million installations as of 2 months ago (up from 50 million 8 months ago). http://blog.bu.mp/introducing-bump-pay-a-new-project-and-app-fr http://blog.bu.mp/introducing-bump-pay-a-new-project-and-app... That's a growth rate of 5 million installs a month; if they kept up that pace, they're at 90 million installs. To put that in perspective, Instagram "only" has 50 million users. http://www.quora.com/Instagram/How-many-users-does-Instagram-have http://www.quora.com/Instagram/How-many-users-does-Instagram... More bump data here: http://bu.mp/static/images/infographic_9-2011_6.pdf http://bu.mp/static/images/infographic_9-2011_6.pdf I'm not a user, but it seems like they have serious data.
- heretohelp 14y ago
  Even at 90 million users, with anything approaching a reasonable level of activity, we're not talking about serious data. 90 million rows of denormalized data isn't a big deal, and if I had to guess, their ops per second is probably no higher than what a dedicated single, or maybe a small master-slave postgres deployment could handle. Again, something a DBA would yawn at. And I say this as someone who scaled up an API for a service that plugged into multiple ad networks concurrently for a total of billions of impressions per month with a high level of reliability. Using NoSQL and an RDBMS combined. People who want to preach the NoSQL message should probably have some actual experience. Otherwise, it just makes very viable NoSQL solutions look really bad.
  - ricardobeat 14y ago
    90 million users certainly does not equate to 90 million rows of data.
    - heretohelp 14y ago
      I was being lackadaisical, deal with it.
timdoug 14y ago
I'm not sure what exactly qualifies as respectable scale, but the Mongo master was running out of space and IO capacity with 24 SSDs and 90 million user records, and was replaced by a sixteen-node Riak cluster. I'll happily share any other statistics you're interested in. Edit: the Riak cluster actually contains lots of other data (communications, object metadata, etc.); we didn't need sixteen boxes for the user records.
- tlianza 14y ago
  90 million users is a great datapoint, yes! In my book that's more than respectable. The only other stat that I'm curious about is the total size of the DB. Certainly databases with tens of millions of records can be held completely in RAM these days... but that also depends on how big each record is.
  - timdoug 14y ago
    All-told the users database when we started migration was about 600 GB on disk, so not the most easily stored database in RAM, but not impossible if you get enough large machines. In fact, we use Redis a lot at Bump, although almost exclusively for queueing and transient state, and not as a persistent database. For a period of time we did store long-lasting metadata in Redis, and as we became more popular instead of throwing engineering effort at the problem we threw more memory, culminating with a handful of boxes with 192 GB of RAM each. We've since moved that entire database to Riak. :)
- cies 14y ago
  i was sort of hoping you'd tell us why you wanted to move away from mongo.. but it was your mongo master deamon that was the bottle neck; right?
- jonshea 14y ago
  To be clear, you are comparing a sharded Riak cluster against an unsharded MongoDB installation?

by tlianza 14y ago

polynomial 14y ago

I was actually going to make a joke about "if the number of people I know who actually use Bump is any indication, it's not clear they even need a large data store."

yahelc 14y ago

They celebrated 80 million installations as of 2 months ago (up from 50 million 8 months ago). http://blog.bu.mp/introducing-bump-pay-a-new-project-and-app-fr http://blog.bu.mp/introducing-bump-pay-a-new-project-and-app... That's a growth rate of 5 million installs a month; if they kept up that pace, they're at 90 million installs. To put that in perspective, Instagram "only" has 50 million users. http://www.quora.com/Instagram/How-many-users-does-Instagram-have http://www.quora.com/Instagram/How-many-users-does-Instagram... More bump data here: http://bu.mp/static/images/infographic_9-2011_6.pdf http://bu.mp/static/images/infographic_9-2011_6.pdf I'm not a user, but it seems like they have serious data.

heretohelp 14y ago
Even at 90 million users, with anything approaching a reasonable level of activity, we're not talking about serious data. 90 million rows of denormalized data isn't a big deal, and if I had to guess, their ops per second is probably no higher than what a dedicated single, or maybe a small master-slave postgres deployment could handle. Again, something a DBA would yawn at. And I say this as someone who scaled up an API for a service that plugged into multiple ad networks concurrently for a total of billions of impressions per month with a high level of reliability. Using NoSQL and an RDBMS combined. People who want to preach the NoSQL message should probably have some actual experience. Otherwise, it just makes very viable NoSQL solutions look really bad.
- ricardobeat 14y ago
  90 million users certainly does not equate to 90 million rows of data.
  - heretohelp 14y ago
    I was being lackadaisical, deal with it.

timdoug 14y ago

I'm not sure what exactly qualifies as respectable scale, but the Mongo master was running out of space and IO capacity with 24 SSDs and 90 million user records, and was replaced by a sixteen-node Riak cluster. I'll happily share any other statistics you're interested in. Edit: the Riak cluster actually contains lots of other data (communications, object metadata, etc.); we didn't need sixteen boxes for the user records.

tlianza 14y ago
90 million users is a great datapoint, yes! In my book that's more than respectable. The only other stat that I'm curious about is the total size of the DB. Certainly databases with tens of millions of records can be held completely in RAM these days... but that also depends on how big each record is.
- timdoug 14y ago
  All-told the users database when we started migration was about 600 GB on disk, so not the most easily stored database in RAM, but not impossible if you get enough large machines. In fact, we use Redis a lot at Bump, although almost exclusively for queueing and transient state, and not as a persistent database. For a period of time we did store long-lasting metadata in Redis, and as we became more popular instead of throwing engineering effort at the problem we threw more memory, culminating with a handful of boxes with 192 GB of RAM each. We've since moved that entire database to Riak. :)
cies 14y ago
i was sort of hoping you'd tell us why you wanted to move away from mongo.. but it was your mongo master deamon that was the bottle neck; right?
jonshea 14y ago
To be clear, you are comparing a sharded Riak cluster against an unsharded MongoDB installation?