3 ms·
Speaking as a CDN responsible for caching artifacts from petabytes of storage on relatively limited pools of "edge servers", I feel most any discussion of conte
by Terretta 13y ago
Speaking as a CDN responsible for caching artifacts from petabytes of storage on relatively limited pools of "edge servers", I feel most any discussion of content-based addressing needs a review of consistent hashing as well.
http://en.wikipedia.org/wiki/Consistent_hashing http://en.wikipedia.org/wiki/Consistent_hashing
Also, we've been hashing content into content keys for a decade and haven't had a collision even using hashes one shouldn't use according to this discussion. In practice, “fast” has been more important to us than “secure”, as we are dealing with multi gigabyte files. Consider your trade offs carefully and keep an eye on published work.
- diminish 13y agoa technical question; how do you know you didn't have any collisions, do you get alerts when one occurs?
- Terretta 13y agoBecause there'd be a key already there for a new file.
- eru 13y agoAs a CDN, do you have any opinion on submodular load balancing (as mentioned in http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.147.2201 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.147....)? We encountered the same problem at work, but in a different domain.
- Terretta 13y agoWithout revealing our IP, yes, we believe it's a really good idea to use just "good enough" algorithms for traffic balancing, packing edge servers with content, content promotion and demotion, etc. At scale, we find blind strategies (round robin for balancing, LRU for content allocation) do okay (say, 70% of theoretic optimal) with almost no signal (e.g., server up/down), and many CDNs using off the shelf equipment are in that category. Using another 20% of hardware doesn't really affect their overhead. We're a wholesaler and rely on a competitive edge in the margins, so in our IP we've discovered and take advantage of classes of "good enough" strategies (say, 95% of optimal) with just a couple high performance signals (e.g. overall server load). To get above those to, say, 96% or 97%, requires gathering detailed metrics with deltas (e.g., HDD and memory stats), and lengthy computation. For us, those final incremental gains are not worth the complexity.
- eru 13y agoThanks!