3 ms·

As someone who spends millions of dollars on computers - yes, absolutely. You severely underestimate the cost of rolling out another cable type one per server.

by phil21 10y ago

As someone who spends millions of dollars on computers - yes, absolutely. You severely underestimate the cost of rolling out another cable type one per server. In fact there are entire billion-dollar segments of the industry that specifically exist to remove said cables. Anything more than power and a few ethernet connected to a rack full of 40+ servers gets real messy real fast, and starts to actually impact cooling and thus power usage in a very measurable way. Even things like screws in server rails vs. tool-less impact the bottom line in most large scale server ops. For the 3 total servers that actually need very precise timekeeping we effectively do this. For the tens of thousands of others, that's what IP and network protocols were made for.

walrus01 10y ago
I cannot even imagine trying to deploy coaxial cables or serial throughout a datacenter via overhead cabling trays/ladder racks to 120 individual 52U racks each full of 10kW of servers in hot/cold separation systems. It would be a complete nightmare. Cable management is already to the point where are specifically using 1.6mm uniboot diameter fiber patch cables because the traditional yellow fat stuff starts becoming unwieldy.
- PhantomGremlin 10y ago
  traditional yellow fat stuff starts becoming unwieldy I'm not sure what you're referring to here. The traditional 10BASE5 cable hasn't been used in literally decades. There's none of it deployed in modern datacenters. https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5 AKA 0.375 inch diameter "frozen yellow garden hose". So standard fiber optic patch cable is the new "traditional"?
  - walrus01 10y ago
    no, I'm talking about the different types of short distance 9/125 fiber patch cables used between servers and top of rack switches, or in large end of row aggregation switch environments and with large-scale patch panels (for 144/288 count cables): http://tiffanycommunication.altervista.org/wp-content/uploads/2016/09/uniboot-LC-fiber-patch-cable.jpg http://tiffanycommunication.altervista.org/wp-content/upload... and another example of duplex uniboot inside one cable jacket: http://www.fibertronics-store.com/images/1378852143657-2109354257.png http://www.fibertronics-store.com/images/1378852143657-21093... versus zipcord/duplex fiber which has a figure-8 shape, and the total jacket diameter is fatter in the range of 2.2 to 3mm: http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jpg http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jp... ignore the SC or LC connectors, the difference is the cable diameter itself. Figure-8 shaped duplex two strand fiber cables that are fatter and occupy more space than a single circular jacket that contains two strands and a single boot connector. This matters for cable management when you're dealing with a switch that might have 4 x 48-port 10Gbe SFP+ linecards in it, all heavily populated. All of the cables coming off the switch in a neatly organized way and going vertically up some fiber cable management to a patch panel.
someguydave 10y ago
> that's what IP and network protocols were made for. No, it's not and that's the problem. IP was meant for wide area internetworking, not local low-latency time distribution. With enough engineering one could potentially include the time and frequency reference signals directly onto existing cabling - for example they could be put onto a fiber at a different wavelength from the ethernet traffic. But it would take quite a bit of money to convince a NIC manufacturer to do this. Some NICs already come with 1pps reference: http://www.solarflare.com/ptp-adapters http://www.solarflare.com/ptp-adapters
- detaro 10y ago
  What scenarios do you see where PTP (which gets improvements from hardware-support in widely available network cards and switches, but doesn't require extra cabling etc) isn't good enough? On a geeky level syncing everything to a central clock is fun, but I don't see much practical reasons to actually build complex hardware for it given that we have working alternatives on top of our existing infrastructure.
  - someguydave 10y ago
    >What scenarios do you see One scenario which we certainly don't have today: access to a tick-accurate global clock across a large number of computers. It would be interesting to see how distributed scheduling and communications between computers would be coordinated if each computer had access to an internal clock timestamp that exactly matched all of its peers.

by phil21 10y ago

walrus01 10y ago

I cannot even imagine trying to deploy coaxial cables or serial throughout a datacenter via overhead cabling trays/ladder racks to 120 individual 52U racks each full of 10kW of servers in hot/cold separation systems. It would be a complete nightmare. Cable management is already to the point where are specifically using 1.6mm uniboot diameter fiber patch cables because the traditional yellow fat stuff starts becoming unwieldy.

PhantomGremlin 10y ago
traditional yellow fat stuff starts becoming unwieldy I'm not sure what you're referring to here. The traditional 10BASE5 cable hasn't been used in literally decades. There's none of it deployed in modern datacenters. https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5 AKA 0.375 inch diameter "frozen yellow garden hose". So standard fiber optic patch cable is the new "traditional"?
- walrus01 10y ago
  no, I'm talking about the different types of short distance 9/125 fiber patch cables used between servers and top of rack switches, or in large end of row aggregation switch environments and with large-scale patch panels (for 144/288 count cables): http://tiffanycommunication.altervista.org/wp-content/uploads/2016/09/uniboot-LC-fiber-patch-cable.jpg http://tiffanycommunication.altervista.org/wp-content/upload... and another example of duplex uniboot inside one cable jacket: http://www.fibertronics-store.com/images/1378852143657-2109354257.png http://www.fibertronics-store.com/images/1378852143657-21093... versus zipcord/duplex fiber which has a figure-8 shape, and the total jacket diameter is fatter in the range of 2.2 to 3mm: http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jpg http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jp... ignore the SC or LC connectors, the difference is the cable diameter itself. Figure-8 shaped duplex two strand fiber cables that are fatter and occupy more space than a single circular jacket that contains two strands and a single boot connector. This matters for cable management when you're dealing with a switch that might have 4 x 48-port 10Gbe SFP+ linecards in it, all heavily populated. All of the cables coming off the switch in a neatly organized way and going vertically up some fiber cable management to a patch panel.

someguydave 10y ago

> that's what IP and network protocols were made for. No, it's not and that's the problem. IP was meant for wide area internetworking, not local low-latency time distribution. With enough engineering one could potentially include the time and frequency reference signals directly onto existing cabling - for example they could be put onto a fiber at a different wavelength from the ethernet traffic. But it would take quite a bit of money to convince a NIC manufacturer to do this. Some NICs already come with 1pps reference: http://www.solarflare.com/ptp-adapters http://www.solarflare.com/ptp-adapters

detaro 10y ago
What scenarios do you see where PTP (which gets improvements from hardware-support in widely available network cards and switches, but doesn't require extra cabling etc) isn't good enough? On a geeky level syncing everything to a central clock is fun, but I don't see much practical reasons to actually build complex hardware for it given that we have working alternatives on top of our existing infrastructure.
- someguydave 10y ago
  >What scenarios do you see One scenario which we certainly don't have today: access to a tick-accurate global clock across a large number of computers. It would be interesting to see how distributed scheduling and communications between computers would be coordinated if each computer had access to an internal clock timestamp that exactly matched all of its peers.