3 ms·

Dropbox does about 1 billion file updates pr. day across millions and millions of active users, petabytes of data and millions of devices. I think it's very ign

by amix 13y ago

Dropbox does about 1 billion file updates pr. day across millions and millions of active users, petabytes of data and millions of devices. I think it's very ignorant to think that their problem is just solved by using powerful machines and caching...

general_failure 13y ago
Ok. So can you give me a few pointers on topics to read up on how they might scale up? Everything I have read in this field is all about - more machines, split data into several machines aka sharing for nice map reduce queries etc. Genuine question not snarky. I have even read yahoo's fast website tricks - it's all really hacks and tricks, nothing mind blowing. What I mean is given some extra time most people can figure things out by some googling and trial and error. This is unlike wolf 3d. Unless you spend considerable time understaning math and graphics you cannot do it.
- smartwater 13y ago
  DropBox can't just easily Google an answer like you would with a Wordpress blog. Their bottleneck wouldn't be web based, so how they load their javascript or if they use image spriting is basically irrelevant. Scaling a website with exponential growth is very difficult. Fix one problem, two more pop up. It never ends. Oops just spent 50k on SSD's to find out it was a load balancer issue.
  - general_failure 13y ago
    Please Bbe specific. I have never seen any real life load balancing code. We just used elb, it was mostly configuration. Drop box uses was AFAIK.
- amix 13y ago
  Scaling DropBox is not only about sharding the data; it's a huge operation that requires management of thousands of machines, complex and fault tolerant algorithms and management of a lot of people (from database administrators, to sys-ops, to very smart developers). It's also not something that you can Google the answers to as very few companies have handled 100+ millions of users, billions of daily updates and petabytes of data. Comparing this to wolf3d is comparing apples to oranges as scaling DropBox and implementing wolf3d are entirely different problem domains . But I would say that wolf3d is a much easier problem that's solved by just one man (John Carmack), while DropBox requires tons of resources and hundreds of people with different skill-sets.

by amix 13y ago

general_failure 13y ago

Ok. So can you give me a few pointers on topics to read up on how they might scale up? Everything I have read in this field is all about - more machines, split data into several machines aka sharing for nice map reduce queries etc. Genuine question not snarky. I have even read yahoo's fast website tricks - it's all really hacks and tricks, nothing mind blowing. What I mean is given some extra time most people can figure things out by some googling and trial and error. This is unlike wolf 3d. Unless you spend considerable time understaning math and graphics you cannot do it.

smartwater 13y ago
DropBox can't just easily Google an answer like you would with a Wordpress blog. Their bottleneck wouldn't be web based, so how they load their javascript or if they use image spriting is basically irrelevant. Scaling a website with exponential growth is very difficult. Fix one problem, two more pop up. It never ends. Oops just spent 50k on SSD's to find out it was a load balancer issue.
- general_failure 13y ago
  Please Bbe specific. I have never seen any real life load balancing code. We just used elb, it was mostly configuration. Drop box uses was AFAIK.
amix 13y ago
Scaling DropBox is not only about sharding the data; it's a huge operation that requires management of thousands of machines, complex and fault tolerant algorithms and management of a lot of people (from database administrators, to sys-ops, to very smart developers). It's also not something that you can Google the answers to as very few companies have handled 100+ millions of users, billions of daily updates and petabytes of data. Comparing this to wolf3d is comparing apples to oranges as scaling DropBox and implementing wolf3d are entirely different problem domains . But I would say that wolf3d is a much easier problem that's solved by just one man (John Carmack), while DropBox requires tons of resources and hundreds of people with different skill-sets.