> I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta I looked just glanced at that video, it looked familiar. What

> I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all). Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options. My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available. > Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market. > I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale. Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company. > (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price. At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale. Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms. One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along). [1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.