I built Haystack with two other engineers and then founded the Everstore team which works on Haystack/F4/Cold Storage. I moved off of managing Everstore to focu

I built Haystack with two other engineers and then founded the Everstore team which works on Haystack/F4/Cold Storage. I moved off of managing Everstore to focus on Traffic in fall of 2011, so F4 and Cold Storage happened after my time, but I worked on Haystack and the storage management layer above it for five years. There were a few things that were interesting or novel about Haystack. CDNs were only giving us a 80% hit rate on photos, and even back in 2007 when we first starting working on Haystack, that was an immense number of misses to serve from the origin. That heavily informed the design goals of Haystack, which were to have a very I/O efficient userland filesystem that used direct I/O to squeeze as much iops from the underlying drives as possible. We also wanted zero indirect I/O, so we used an efficient in-memory index so that all seeks on disk were to service production reads versus indirect index blocks or other metadata. Lastly, Haystack is very very simple. We made it as simple as we could and eschewed anything that looked too clever. I actually think its simplicity is one of the things that made it Hard. It would have been much easier to design something more complex. If you have any questions about the Haystack paper after reading it, I'd be happy to answer what I can. Just send me an email. Regarding the datacenters, you're kind of simplifying it, but that's okay. There's a lot more orchestration, automation, and integration, and I wish I could give you a tour of one of our OCP datacenters so you can see it firsthand. There's plenty of things I wish I could talk about on the datacenter front that I feel are truly amazing and without precedent, but I can't until they are public. There's an excellent chance that they will become public and likely even part of OCP in the future though. I'm really proud of how much of Facebook's datacenter architecture has been shared with the world for free. It gives me warm fuzzies all the time. Anyways, thanks for the fun thread, and let me know if you ever want to talk storage sometime.