4 ms·

Mogile relies on a central mysql db for metadata (e.g. replica locations) so it doesn't scale well at all, especially not when your workload emphasizes many sma

by jbellis 17y ago

Mogile relies on a central mysql db for metadata (e.g. replica locations) so it doesn't scale well at all, especially not when your workload emphasizes many small files ( because then you exhaust mysql that much faster relative to your disk capacity).

recurser 17y ago
We're running about 90 terabytes of mogile storage containing 20-30 million files and haven't had any problems at all. If we do, the plan is to shard them along the same lines as our database sharding. I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. What kind of scaling problems have you had?
- jbellis 17y ago
  so your average file size is 4.5MB. that's a pretty good fit. > I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. I think you're right about that part. Better solutions don't come off the shelf. Yet. :) > What kind of scaling problems have you had? the system I wrote at Mozy needed to scale to PBs of space and billions of files, so it was obvious that the mysql part would be a problem. more generally I think mogile is in a pretty narrow niche: typically either you really need a lot of space and files, like flickr or facebook or ..., in which case mogilefs doesn't cut it, or you have a smaller data set but you want random access to your files in which case mogilefs also doesn't cut it.
  - recurser 17y ago
    Ha sorry I didn't realize who I was talking to, yeah if you built mozy then mogile would probably hit the wall fairly early :) I work for one of the larger Japanese web companies, and the CEO has basically declared war on amazon japan (literally - they have anti-amazon tshirts, slogans etc) so S3 was not an option. Otherwise, a lot of people in this space would probably find S3 a perfect fit.

4 ms·

Mogile relies on a central mysql db for metadata (e.g. replica locations) so it doesn't scale well at all, especially not when your workload emphasizes many sma

by jbellis 17y ago

recurser 17y ago
We're running about 90 terabytes of mogile storage containing 20-30 million files and haven't had any problems at all. If we do, the plan is to shard them along the same lines as our database sharding. I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. What kind of scaling problems have you had?
- jbellis 17y ago
  so your average file size is 4.5MB. that's a pretty good fit. > I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. I think you're right about that part. Better solutions don't come off the shelf. Yet. :) > What kind of scaling problems have you had? the system I wrote at Mozy needed to scale to PBs of space and billions of files, so it was obvious that the mysql part would be a problem. more generally I think mogile is in a pretty narrow niche: typically either you really need a lot of space and files, like flickr or facebook or ..., in which case mogilefs doesn't cut it, or you have a smaller data set but you want random access to your files in which case mogilefs also doesn't cut it.
  - recurser 17y ago
    Ha sorry I didn't realize who I was talking to, yeah if you built mozy then mogile would probably hit the wall fairly early :) I work for one of the larger Japanese web companies, and the CEO has basically declared war on amazon japan (literally - they have anti-amazon tshirts, slogans etc) so S3 was not an option. Otherwise, a lot of people in this space would probably find S3 a perfect fit.