Commodity CDN's will always have some amount of cache misses. For most services that's fine; there isn't too much additional overhead in serving a static Javasc

Commodity CDN's will always have some amount of cache misses. For most services that's fine; there isn't too much additional overhead in serving a static Javascript file a few times. If you're doing computational work for each request, you want to try to avoid unnecessary work as much as possible. It's helpful for us to hold onto master images after fetching -- these images can be large and may not be served to us from a particularly fast source. The speed of light does come into play, so anything we can do to shorten distances is beneficial. This cache layer respects the Cache-Control headers of the original objects, so the behavior there is under our user's control. There's a similar need to cache rendered output ("derivative images"), because while most operations are very fast they do still add a little bit of latency. This helps with our rendering capacity, but it's really about improving time-to-serve latency wherever we can. We evaluated a few open source object stores and decided on MogileFS. It largely came down to the simplicity of what Mogile does and how it does it. We figured that we could rip pieces out and adapt it to our needs with a lot less work than more elaborate systems (like ceph). With that said, we didn't actually do an implementation test with ceph, so it may in fact be great for our use case. Not sure off-hand about the number of servers deployed for use with Mogile, but it isn't a gigantic deployment. We aren't storing anything that can't be re-acquired, so we can adjust our storage to hit particular cache hit ratios rather than storing the enormous long-tail forever. The first generation of storage servers are 16 core / 32gb / 6x4T systems -- more like what I would use for computationally related storage like HDFS than a large object store. We could stick with this hardware spec, or we might move to SSDs if latency becomes an issue (I personally doubt it will). We might also move to more dense systems, with more like 20-30 drives. That's all based on our metrics and how things perform in practice.