Servers: C1: Have a look at the FatTwin Line for more Disks per U. More PCIe Slots too. Disks D3: Check the measurements, having it not fit is painful D4: M

Servers: C1: Have a look at the FatTwin Line for more Disks per U. More PCIe Slots too. Disks D3: Check the measurements, having it not fit is painful D4: More, smaller drives. Make sure you go PMR not SMR if you do go for 8TB Network N1: The "SDN" aspect of the supermicro one is not really any different than any other. Look at https://bm-switch.com/ https://bm-switch.com/ and get one that supports Cumulus Linux. Buy one with an x86 CPU. If you want to do "SDN" things or run custom monitoring, not dealing with PPC is great. N3: Probably not needed, but not terribly expensive if it provides benefit. N4: see N1, no. N5: Cheap 1G switch that supports cumulus, x86, probably broadcom \ helix4 Networking General: - I wouldn't advise using the 10GbE Copper --Go for 25GbE with DAC, it's basically the same price, Mellanox NICs are small/cheap - Transit is cheap, you can get 500Mbps on a 10G port for $325/mo from Cogent - If your bandwidth needs to scale up, data center locations matter more than you think 25GbE adapter -- minimal additional cost for 2.5x the perf, lower latency as well: http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2816&idcategory=6 http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2... a 32 port 100GbE switch is about $7000-12,000 -- you can break that down to 128x 25GbE, and use the 25GbE ports running at 10GbE mode for your carrier uplinks. Could even do 100GbE to your Ceph nodes if you wanted, but be aware of PCIe bottlenecks -- x8 is about 64Gbps, x16 will do 125Gbps. Dual port 40G on x8 or dual port 100G on x16 will not provide more than those numbers. Consider Supermicro NVMe servers (Ultra series) for DBs, and 2.5in NVMe SSDs instead of PCIe. Rack: Don't assume 45U 40-48 is common. Consider buying 2 racks. Power: 19kW seems high for a single rack, you will need a good datacenter to support that density. Density costs money more racks is cheaper generally. 208v * 30A = 5000W usable per feed, unless they are talking about 208v 3-phase, which gets you 8600W usable per feed, which again is only 17kW and you need 18-20kW. helpful reference: http://www.raritan.com/blog/detail/3-phase-208v-power-strips-rack-pdus-demystified-part-ii-understanding-capac http://www.raritan.com/blog/detail/3-phase-208v-power-strips... You can only use 80% of your power provided. You also need Rack PDUs, higher density PDUs cost more money consider buying port-switchable PDUs. Raritan makes good ones. Ask supermicro (or your reseller) for a "Power Sheet" it will tell you almost exactly how much power your server will use. I've had good luck with ThinkMate Hosting H1: Yes, too many to mention H2: Do it yourselves H4: yes Hosting general: Cross connects cost money, a number of facilities offer free xconnects, this can add up. Other notes: - You want a small toolbox in the data center - Buy more cables than you think you'll need - You'll always forget something - There are a number of companies that will lease you servers for pretty decent rates