3 ms·

So people would be willing to spend millions of dollars on computers and would also be unwilling to spend a few thousand more for cabling?

walrus01 10y ago
You underestimate the complication of cabling and managing that cabling when building a facility that is on the scale of eight thousand or more 1U size servers (or the proprietary open compute/blade type equivalent of the same in 1U). Then the RF and/or serial cabling from each server has to go somewhere, to some equipment (what, a new special 10 MHz reference 1U top of rack device that is a bespoke creation?), which both consumes rack space and electricity. And is managed over some sort of management network.
- someguydave 10y ago
  >what, a new special 10 MHz reference 1U top of rack device that is a bespoke creation? It's not a "bespoke creation", such things have been available for decades: https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/gps-instruments/xli#overview https://www.microsemi.com/products/timing-synchronization-sy...
  - walrus01 10y ago
    and each one of those 1U units costs how much, and can feed reference signals to how many individual end-point servers? It doesn't scale to a massive hosting/hypervisor operation. products like the one linked above are priced and intended for providing timing to crucial core network infrastructure. Such as if you're an ISP with a POP that has a 1+1 pair of Juniper MX960 at the main IX point in a major city. Where the ISP would then have its own set of NTP servers serving as internally-accessible-only stratum 2 to all of their other network equipment that is topologically nearby within the same AS.
    - someguydave 10y ago
      >and each one of those 1U units costs how much It depends on what capabilities you want. A good oscillator with GPS discipline capabilities would cost between $100-$10000 depending on specifications. In particular, if you want it to have high stability with time and temperature when GPS fails you must spend much more money. >and can feed reference signals to how many individual end-point servers? In theory one of these units could feed a whole datacenter (thousands of endpoints), if you also purchased enough cabling and distribution amplifiers.
- someguydave 10y ago
  >You underestimate the complication I'm not underestimating anything. If people want to have highly accurate time available at a hardware level, they can pay for it. I'm complaining that it isn't even an option from server vendors, who are quite capable of implementing it.
phil21 10y ago
As someone who spends millions of dollars on computers - yes, absolutely. You severely underestimate the cost of rolling out another cable type one per server. In fact there are entire billion-dollar segments of the industry that specifically exist to remove said cables. Anything more than power and a few ethernet connected to a rack full of 40+ servers gets real messy real fast, and starts to actually impact cooling and thus power usage in a very measurable way. Even things like screws in server rails vs. tool-less impact the bottom line in most large scale server ops. For the 3 total servers that actually need very precise timekeeping we effectively do this. For the tens of thousands of others, that's what IP and network protocols were made for.
- walrus01 10y ago
  I cannot even imagine trying to deploy coaxial cables or serial throughout a datacenter via overhead cabling trays/ladder racks to 120 individual 52U racks each full of 10kW of servers in hot/cold separation systems. It would be a complete nightmare. Cable management is already to the point where are specifically using 1.6mm uniboot diameter fiber patch cables because the traditional yellow fat stuff starts becoming unwieldy.
  - PhantomGremlin 10y ago
    traditional yellow fat stuff starts becoming unwieldy I'm not sure what you're referring to here. The traditional 10BASE5 cable hasn't been used in literally decades. There's none of it deployed in modern datacenters. https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5 AKA 0.375 inch diameter "frozen yellow garden hose". So standard fiber optic patch cable is the new "traditional"?
    - walrus01 10y ago
      no, I'm talking about the different types of short distance 9/125 fiber patch cables used between servers and top of rack switches, or in large end of row aggregation switch environments and with large-scale patch panels (for 144/288 count cables): http://tiffanycommunication.altervista.org/wp-content/uploads/2016/09/uniboot-LC-fiber-patch-cable.jpg http://tiffanycommunication.altervista.org/wp-content/upload... and another example of duplex uniboot inside one cable jacket: http://www.fibertronics-store.com/images/1378852143657-2109354257.png http://www.fibertronics-store.com/images/1378852143657-21093... versus zipcord/duplex fiber which has a figure-8 shape, and the total jacket diameter is fatter in the range of 2.2 to 3mm: http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jpg http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jp... ignore the SC or LC connectors, the difference is the cable diameter itself. Figure-8 shaped duplex two strand fiber cables that are fatter and occupy more space than a single circular jacket that contains two strands and a single boot connector. This matters for cable management when you're dealing with a switch that might have 4 x 48-port 10Gbe SFP+ linecards in it, all heavily populated. All of the cables coming off the switch in a neatly organized way and going vertically up some fiber cable management to a patch panel.

walrus01 10y ago

You underestimate the complication of cabling and managing that cabling when building a facility that is on the scale of eight thousand or more 1U size servers (or the proprietary open compute/blade type equivalent of the same in 1U). Then the RF and/or serial cabling from each server has to go somewhere, to some equipment (what, a new special 10 MHz reference 1U top of rack device that is a bespoke creation?), which both consumes rack space and electricity. And is managed over some sort of management network.

someguydave 10y ago
>what, a new special 10 MHz reference 1U top of rack device that is a bespoke creation? It's not a "bespoke creation", such things have been available for decades: https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/gps-instruments/xli#overview https://www.microsemi.com/products/timing-synchronization-sy...
- walrus01 10y ago
  and each one of those 1U units costs how much, and can feed reference signals to how many individual end-point servers? It doesn't scale to a massive hosting/hypervisor operation. products like the one linked above are priced and intended for providing timing to crucial core network infrastructure. Such as if you're an ISP with a POP that has a 1+1 pair of Juniper MX960 at the main IX point in a major city. Where the ISP would then have its own set of NTP servers serving as internally-accessible-only stratum 2 to all of their other network equipment that is topologically nearby within the same AS.
  - someguydave 10y ago
    >and each one of those 1U units costs how much It depends on what capabilities you want. A good oscillator with GPS discipline capabilities would cost between $100-$10000 depending on specifications. In particular, if you want it to have high stability with time and temperature when GPS fails you must spend much more money. >and can feed reference signals to how many individual end-point servers? In theory one of these units could feed a whole datacenter (thousands of endpoints), if you also purchased enough cabling and distribution amplifiers.
someguydave 10y ago
>You underestimate the complication I'm not underestimating anything. If people want to have highly accurate time available at a hardware level, they can pay for it. I'm complaining that it isn't even an option from server vendors, who are quite capable of implementing it.

phil21 10y ago

As someone who spends millions of dollars on computers - yes, absolutely. You severely underestimate the cost of rolling out another cable type one per server. In fact there are entire billion-dollar segments of the industry that specifically exist to remove said cables. Anything more than power and a few ethernet connected to a rack full of 40+ servers gets real messy real fast, and starts to actually impact cooling and thus power usage in a very measurable way. Even things like screws in server rails vs. tool-less impact the bottom line in most large scale server ops. For the 3 total servers that actually need very precise timekeeping we effectively do this. For the tens of thousands of others, that's what IP and network protocols were made for.

walrus01 10y ago
I cannot even imagine trying to deploy coaxial cables or serial throughout a datacenter via overhead cabling trays/ladder racks to 120 individual 52U racks each full of 10kW of servers in hot/cold separation systems. It would be a complete nightmare. Cable management is already to the point where are specifically using 1.6mm uniboot diameter fiber patch cables because the traditional yellow fat stuff starts becoming unwieldy.
- PhantomGremlin 10y ago
  traditional yellow fat stuff starts becoming unwieldy I'm not sure what you're referring to here. The traditional 10BASE5 cable hasn't been used in literally decades. There's none of it deployed in modern datacenters. https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5 AKA 0.375 inch diameter "frozen yellow garden hose". So standard fiber optic patch cable is the new "traditional"?
  - walrus01 10y ago
    no, I'm talking about the different types of short distance 9/125 fiber patch cables used between servers and top of rack switches, or in large end of row aggregation switch environments and with large-scale patch panels (for 144/288 count cables): http://tiffanycommunication.altervista.org/wp-content/uploads/2016/09/uniboot-LC-fiber-patch-cable.jpg http://tiffanycommunication.altervista.org/wp-content/upload... and another example of duplex uniboot inside one cable jacket: http://www.fibertronics-store.com/images/1378852143657-2109354257.png http://www.fibertronics-store.com/images/1378852143657-21093... versus zipcord/duplex fiber which has a figure-8 shape, and the total jacket diameter is fatter in the range of 2.2 to 3mm: http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jpg http://www.pctstore.com/v/vspfiles/photos/SCASCADP30S10-5.jp... ignore the SC or LC connectors, the difference is the cable diameter itself. Figure-8 shaped duplex two strand fiber cables that are fatter and occupy more space than a single circular jacket that contains two strands and a single boot connector. This matters for cable management when you're dealing with a switch that might have 4 x 48-port 10Gbe SFP+ linecards in it, all heavily populated. All of the cables coming off the switch in a neatly organized way and going vertically up some fiber cable management to a patch panel.