If the data isn't "optional" (i.e. if there's a cache miss, you can't just go to a traditional database for a somewhat higher - but acceptable - cost), "memcach

If the data isn't "optional" (i.e. if there's a cache miss, you can't just go to a traditional database for a somewhat higher - but acceptable - cost), "memcached with persistence" approach isn't going to cut it. You now have a distributed system with state, which is a much more difficult problem. That's why there are so many distributed storage systems: there's no "one fits all" solution that can handle every theoretical corner case (even Google hasn't solved that). In my (biased) view, the eventually consistent stores (e.g. Dynamo-inspired ones or even "Friendfeed/Facebook model" of sharded and replicated MySQL databases storing Blobs) do seem most reasonable for web-type problems.