11 ms·

In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacement

by rtp4me 4y ago

In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacements (velcro, tie wraps, etc). There is a fine line between cleanliness and usability. For the folks that work with me, I explain the importance of air flow, proper cable length, and component serviceability. You definitely don't want "piano wire" stretched across the rack that prevents someone from replacing a PSU or NIC without affecting other gear in the rack.

vidarh 4y ago
Couldn't agree more. And the latter (tied down) quite often devolve to the former over time exactly because it's so much effort to maintain that suddenly an urgent repair happens and ties get cut and cables torn out and something new just thrown in, and the effort of re-doing the original setup means it gets deferred. I stopped tieing cables together years ago for that reason. The more the perceived effort in getting what needs to be done out of the way + fixing the cabling afterwards, the less chance it gets done properly. Instead I'd aim to set an example of stretching the cables out of the way, labelling and getting length roughly right that is simple enough to work with in a rush and trivial to "fix up" afterwards. In large environments where crises are more likely to involved routing traffic away from entire racks or data centres, you can afford being a stickler for these things. Most places are not that large, and being too strict creates more problems than it solves.
jonhohle 4y ago
Is there any good resources for learning more about this? I have a lot of legacy AV equipment mixed with modern digital components and networking and despite best efforts, I typically fall into either a rats nest or something that’s too constrained to repair/modify without undoing everything.
- Zircom 4y ago
  I mean honestly there's not too much expertise involved, it's mostly common sense. It's just that it takes so much longer to do it right at the beginning to ensure its maintainable and ordered rather than just doing it the easy way and hoping that it's someone else's problem by the time someone actually needs to find the other end or replace a cable. Making your own cables that are the exact length they need to be instead of using preterminated ones and just stuffing the slack into the side of the rack, routing them down the side and through the floor or the cabling trays on the ceiling, standardized naming schemes for labeling devices/ports on either end of cables, documented rack diagrams, etc.
  - chasd00 4y ago
    i once had to make up about a 100 ethernet cables of varying lengths at once. Man, untwisting, sorting, and separating the individual wires to insert into the connector before crimping made for some very sore fingers.
- rtp4me 4y ago
  Not sure of any specific resources, just years of running cables :-) Some of my lessons learned: * Whatever you do, be very mindful of future events (repairs, replacements, etc). Spending an extra 5mins upfront when you are in a hurry will save you lots of time later. * Label each end of the cable if possible. It is a PIA in the beginning but will be a real life-saver when you need to do maintenance * Replace old (fat) cables with new, thinner cables of the proper length when possible. I typically add 1ft extra to any cable for a little slack. * Think of airflow. Lots of cables bundled in the back of a rack can seriously cause airflow issues. Try to run all cables along the sides of the rack. * NEVER run cables across the back of gear - especially for multi-node chassis that require rear access for maintenance * Velcro is your friend. * Stick with a cable color standard if possible. For me, I use white for IPMI, blue for data, red for external access * Keep an updated diagram in the rack for the next guy/gal who needs to do work * Always remove old/un-needed cables after a maintenance job. * Keep the rack clean!
  - dontbesquare 4y ago
    Velcro is your friend. So much this!
  - systems_glitch 4y ago
    Abandoned-in-place wiring is the bane of every tech's and installer's existence. It's impossible to have worked on legacy equipment and not spent some time reterminating abandoned cable :/ Not to mention having to weed through it when tracing problems. We're in an ongoing battle with abandoned-in-place wiring in our current building. There's data wiring back to ARCnet (93 ohm coax), and power wiring going back to knob and tube days.
    - h2odragon 4y ago
      > 93 ohm coax ooo nifty. be on the lookout for vampire taps. they never truly die.
      - rtp4me 4y ago
        Wow - vampire taps. That brings back some memories :-)
- jcrawfordor 4y ago
  "Finger duct" is often the easiest way to get relatively clean results in a situation like AV where cabling is sort of poorly standardized (connections front and back, inline DC supplies, etc). It's fairly cheap and often run vertically on rack posts and horizontally above and below cable-dense areas like patch panels and switches, but in an AV situation you probably only need one vertical channel to get a big improvement in results. Finger duct can get pretty messy without planning but I think it's usually the best results to effort ratio.
  - systems_glitch 4y ago
    Often used inside industrial equipment as well. Routes pneumatic lines pretty good.
Zircom 4y ago
My first real IT job was doing all the grunt work (and whatever sysadmin tasks I could talk the greybeards into trusting the new kid with) in our two DCs, one at the corporate office and the other at our manufacturing plant. Our DC at the corporate office was OK aside from the patch panel which was a fucking mess, that and when they originally built the thing however long ago they installed the switches at the top of the racks BACKWARDS so the ports where at the FRONT of the rack instead of the back with the server ports and they just ran with it instead of taking 20 minutes to flip them around, so instead of neat orderly cables going up and down between the ports you had these giant bundles on either side of the servers going from front to back I had to unbind and untangle anytime one needed replacing >.> This[0] is the what I managed to get our patch panels there reduced too and slightly cleaned too after removing all of these[1] cables that were still plugged into dead ports that my predecessor just never bothered to take out after retiring or recabling servers that had been on the other end. I'm 99% sure he was just running new cables and configuring a new port anytime someone called to complain about an unreachable server instead of trying to find the other end of the relevant cable. [0] - https://i.imgur.com/PIPhAal.jpeg https://i.imgur.com/PIPhAal.jpeg [1] - https://i.imgur.com/PIPhAal.jpeg https://i.imgur.com/PIPhAal.jpeg Wish I still had some pictures of the plant DC though because holy cow it was even worse, no patch panels anywhere, but they did have a drop floor so I could pull the tiles up and run cables down there if I was so inclined. My predecessor, not so much. He seemed to prefer running them between the tops of the racks, with random cables crisscrossing above your head, always drooping down and getting snagged on the step ladder as I carried it between racks, or getting caught in the doors constantly. And they would never let us have any downtime to clean up the cabling, nor could I talk them into just switching the network configs to use a different damn port on the actual servers so I could just run new fucking cables ;_; I had to settle for obsessively tracing and labeling each end of the cable with the port# and device that was on the other end of the cable, because otherwise every call about checking on a server turned into an hour of either tracing cables either under the floor, pulling tiles one by one to follow it to the other end, or a jungle safari sorting through the tangled mess of cables on top of the racks following the one in question as it arced over multiple aisles to it's final destination. Which of course I had to do anyways to get them all labeled, but at least this way I only had to do it once and it was done forever. Good times.
walrus01 4y ago
one of the key things to do in a 'serious' datacenter is separate the fiber from the power. this is why serious facilities have a dedicated channel of yellow plastic overhead fiber tray which contains ONLY 1.6mm or 2.0mm jacketed 9/125 singlemode fiber, with nice smooth corners and spigots on it coming down above racks. and then the AC power is in conduit and long rectangular metal enclosures. and the -48VDC cabling on steel ladder rack, often with waxed string lacing if it was done by people who are being serious about it. on a per rack cabinet level it's also really important to pick one vertical side for your power cables, and data cables vertically up the OTHER side, and stick to it consistently.
- systems_glitch 4y ago
  We often end up with AC power down both sides for A and B busses. Prefer the "zero U" type solutions for that.
- deleted 4y ago
  [deleted]
- gtfoutttt 4y ago
  Why is this? Does power interfere with the fiber?
  - walrus01 4y ago
    No, not at all, it's for proper organization, management, labeling, flexibility to add and remove things, etc.
  - deleted 4y ago
    [deleted]
- ilyt 4y ago
  Then you get a bunch of boxes where "connectivity" and "power" is on opposite side than the other boxes ;/
- kazen44 4y ago
  what is also very common is having power ran across different power rails under the floor of the DC (different feeds obviously), which is then terminated upwards to a PDU into the customers rack. Fiber comes in from the top, with a MPO or patchpanel at the top of rack for customer termination.
gertrunde 4y ago
A particularly frustrating variant of the 'piano wire' issue: A redundant pair of devices, except cables have been run in such a way that if one device failed, you could not remove it from the rack without removing cables from the remaining device. /facepalm
- kazen44 4y ago
  this is the reason why (ideally) you should have three racks in a redundant setup: 2 for redundant component placement (with redundant power obviously). 1 for interconnections between the racks, having prepared cabling and MPO panels installed to make interconecting devices easily, without having to change anything in the equipment racks compared or vice versa. Makes replacing hardware a breeze, and prevents a lot of issues with cable mess inside equipment racks because those cables are very short to MPO panels.
ilyt 4y ago
> In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacements (velcro, tie wraps, etc). There is a fine line between cleanliness and usability. We have 7 racks that went thru some changes and never in the history (12+ years) needed to replace any broken ones. It just doesn't happen. What did happen were re-wires when some hardware were replaces but smaller ones were small enough that whether it was nicely bundled "permanently" or not didn't matter much, and bigger ones were "take everything out and recable" job anyway so again, what was in the rack (and there were some spaghettified ones too) also didn't matter. Yeah routing it properly the first time helps but making it "permanent" by tying it down isn't really a problem, just put another bundle for next 5 servers.
- JohnClark1337 4y ago
  It just doesn't happen Depends on the datacenter. I work in a large testing environment, so equipment is being moved and parts are being swapped out all the time. The cabling needs to be nice enough that you have good airflow, can see all of the components, and to make swapping parts easier. Perfection isn't necessary but cleanliness makes life easier.
eitland 4y ago
I have been in some data centers over the years know one data center I visited multiple times where one blade chassis (their newest acquisition I think) stood on an old (wobbly IIRC) desk a meter or so from the closest rack and not near anything that could support it. It was connected via a way too long fiber optic cable, of which most was placed on the end of a broomstick on the nearest server rack, and then the end of it was either wound around or tied to (I can't remember) the part of the broom stick that stuck out from the top of the rack towards the desk with the blade chassis before it fell down to the back of the chassis. I'm thankful I never stumbled into it.

by rtp4me 4y ago

vidarh 4y ago

Couldn't agree more. And the latter (tied down) quite often devolve to the former over time exactly because it's so much effort to maintain that suddenly an urgent repair happens and ties get cut and cables torn out and something new just thrown in, and the effort of re-doing the original setup means it gets deferred. I stopped tieing cables together years ago for that reason. The more the perceived effort in getting what needs to be done out of the way + fixing the cabling afterwards, the less chance it gets done properly. Instead I'd aim to set an example of stretching the cables out of the way, labelling and getting length roughly right that is simple enough to work with in a rush and trivial to "fix up" afterwards. In large environments where crises are more likely to involved routing traffic away from entire racks or data centres, you can afford being a stickler for these things. Most places are not that large, and being too strict creates more problems than it solves.

jonhohle 4y ago

Is there any good resources for learning more about this? I have a lot of legacy AV equipment mixed with modern digital components and networking and despite best efforts, I typically fall into either a rats nest or something that’s too constrained to repair/modify without undoing everything.

Zircom 4y ago
I mean honestly there's not too much expertise involved, it's mostly common sense. It's just that it takes so much longer to do it right at the beginning to ensure its maintainable and ordered rather than just doing it the easy way and hoping that it's someone else's problem by the time someone actually needs to find the other end or replace a cable. Making your own cables that are the exact length they need to be instead of using preterminated ones and just stuffing the slack into the side of the rack, routing them down the side and through the floor or the cabling trays on the ceiling, standardized naming schemes for labeling devices/ports on either end of cables, documented rack diagrams, etc.
- chasd00 4y ago
  i once had to make up about a 100 ethernet cables of varying lengths at once. Man, untwisting, sorting, and separating the individual wires to insert into the connector before crimping made for some very sore fingers.
rtp4me 4y ago
Not sure of any specific resources, just years of running cables :-) Some of my lessons learned: * Whatever you do, be very mindful of future events (repairs, replacements, etc). Spending an extra 5mins upfront when you are in a hurry will save you lots of time later. * Label each end of the cable if possible. It is a PIA in the beginning but will be a real life-saver when you need to do maintenance * Replace old (fat) cables with new, thinner cables of the proper length when possible. I typically add 1ft extra to any cable for a little slack. * Think of airflow. Lots of cables bundled in the back of a rack can seriously cause airflow issues. Try to run all cables along the sides of the rack. * NEVER run cables across the back of gear - especially for multi-node chassis that require rear access for maintenance * Velcro is your friend. * Stick with a cable color standard if possible. For me, I use white for IPMI, blue for data, red for external access * Keep an updated diagram in the rack for the next guy/gal who needs to do work * Always remove old/un-needed cables after a maintenance job. * Keep the rack clean!
- dontbesquare 4y ago
  Velcro is your friend. So much this!
- systems_glitch 4y ago
  Abandoned-in-place wiring is the bane of every tech's and installer's existence. It's impossible to have worked on legacy equipment and not spent some time reterminating abandoned cable :/ Not to mention having to weed through it when tracing problems. We're in an ongoing battle with abandoned-in-place wiring in our current building. There's data wiring back to ARCnet (93 ohm coax), and power wiring going back to knob and tube days.
  - h2odragon 4y ago
    > 93 ohm coax ooo nifty. be on the lookout for vampire taps. they never truly die.
    - rtp4me 4y ago
      Wow - vampire taps. That brings back some memories :-)
jcrawfordor 4y ago
"Finger duct" is often the easiest way to get relatively clean results in a situation like AV where cabling is sort of poorly standardized (connections front and back, inline DC supplies, etc). It's fairly cheap and often run vertically on rack posts and horizontally above and below cable-dense areas like patch panels and switches, but in an AV situation you probably only need one vertical channel to get a big improvement in results. Finger duct can get pretty messy without planning but I think it's usually the best results to effort ratio.
- systems_glitch 4y ago
  Often used inside industrial equipment as well. Routes pneumatic lines pretty good.

Zircom 4y ago

My first real IT job was doing all the grunt work (and whatever sysadmin tasks I could talk the greybeards into trusting the new kid with) in our two DCs, one at the corporate office and the other at our manufacturing plant. Our DC at the corporate office was OK aside from the patch panel which was a fucking mess, that and when they originally built the thing however long ago they installed the switches at the top of the racks BACKWARDS so the ports where at the FRONT of the rack instead of the back with the server ports and they just ran with it instead of taking 20 minutes to flip them around, so instead of neat orderly cables going up and down between the ports you had these giant bundles on either side of the servers going from front to back I had to unbind and untangle anytime one needed replacing >.> This[0] is the what I managed to get our patch panels there reduced too and slightly cleaned too after removing all of these[1] cables that were still plugged into dead ports that my predecessor just never bothered to take out after retiring or recabling servers that had been on the other end. I'm 99% sure he was just running new cables and configuring a new port anytime someone called to complain about an unreachable server instead of trying to find the other end of the relevant cable. [0] - https://i.imgur.com/PIPhAal.jpeg https://i.imgur.com/PIPhAal.jpeg [1] - https://i.imgur.com/PIPhAal.jpeg https://i.imgur.com/PIPhAal.jpeg Wish I still had some pictures of the plant DC though because holy cow it was even worse, no patch panels anywhere, but they did have a drop floor so I could pull the tiles up and run cables down there if I was so inclined. My predecessor, not so much. He seemed to prefer running them between the tops of the racks, with random cables crisscrossing above your head, always drooping down and getting snagged on the step ladder as I carried it between racks, or getting caught in the doors constantly. And they would never let us have any downtime to clean up the cabling, nor could I talk them into just switching the network configs to use a different damn port on the actual servers so I could just run new fucking cables ;_; I had to settle for obsessively tracing and labeling each end of the cable with the port# and device that was on the other end of the cable, because otherwise every call about checking on a server turned into an hour of either tracing cables either under the floor, pulling tiles one by one to follow it to the other end, or a jungle safari sorting through the tangled mess of cables on top of the racks following the one in question as it arced over multiple aisles to it's final destination. Which of course I had to do anyways to get them all labeled, but at least this way I only had to do it once and it was done forever. Good times.

walrus01 4y ago

one of the key things to do in a 'serious' datacenter is separate the fiber from the power. this is why serious facilities have a dedicated channel of yellow plastic overhead fiber tray which contains ONLY 1.6mm or 2.0mm jacketed 9/125 singlemode fiber, with nice smooth corners and spigots on it coming down above racks. and then the AC power is in conduit and long rectangular metal enclosures. and the -48VDC cabling on steel ladder rack, often with waxed string lacing if it was done by people who are being serious about it. on a per rack cabinet level it's also really important to pick one vertical side for your power cables, and data cables vertically up the OTHER side, and stick to it consistently.

systems_glitch 4y ago
We often end up with AC power down both sides for A and B busses. Prefer the "zero U" type solutions for that.
deleted 4y ago
[deleted]
gtfoutttt 4y ago
Why is this? Does power interfere with the fiber?
- walrus01 4y ago
  No, not at all, it's for proper organization, management, labeling, flexibility to add and remove things, etc.
- deleted 4y ago
  [deleted]
ilyt 4y ago
Then you get a bunch of boxes where "connectivity" and "power" is on opposite side than the other boxes ;/
kazen44 4y ago
what is also very common is having power ran across different power rails under the floor of the DC (different feeds obviously), which is then terminated upwards to a PDU into the customers rack. Fiber comes in from the top, with a MPO or patchpanel at the top of rack for customer termination.

gertrunde 4y ago

A particularly frustrating variant of the 'piano wire' issue: A redundant pair of devices, except cables have been run in such a way that if one device failed, you could not remove it from the rack without removing cables from the remaining device. /facepalm

kazen44 4y ago
this is the reason why (ideally) you should have three racks in a redundant setup: 2 for redundant component placement (with redundant power obviously). 1 for interconnections between the racks, having prepared cabling and MPO panels installed to make interconecting devices easily, without having to change anything in the equipment racks compared or vice versa. Makes replacing hardware a breeze, and prevents a lot of issues with cable mess inside equipment racks because those cables are very short to MPO panels.

ilyt 4y ago

> In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacements (velcro, tie wraps, etc). There is a fine line between cleanliness and usability. We have 7 racks that went thru some changes and never in the history (12+ years) needed to replace any broken ones. It just doesn't happen. What did happen were re-wires when some hardware were replaces but smaller ones were small enough that whether it was nicely bundled "permanently" or not didn't matter much, and bigger ones were "take everything out and recable" job anyway so again, what was in the rack (and there were some spaghettified ones too) also didn't matter. Yeah routing it properly the first time helps but making it "permanent" by tying it down isn't really a problem, just put another bundle for next 5 servers.

JohnClark1337 4y ago
It just doesn't happen Depends on the datacenter. I work in a large testing environment, so equipment is being moved and parts are being swapped out all the time. The cabling needs to be nice enough that you have good airflow, can see all of the components, and to make swapping parts easier. Perfection isn't necessary but cleanliness makes life easier.

eitland 4y ago

I have been in some data centers over the years know one data center I visited multiple times where one blade chassis (their newest acquisition I think) stood on an old (wobbly IIRC) desk a meter or so from the closest rack and not near anything that could support it. It was connected via a way too long fiber optic cable, of which most was placed on the end of a broomstick on the nearest server rack, and then the end of it was either wound around or tied to (I can't remember) the part of the broom stick that stuck out from the top of the rack towards the desk with the blade chassis before it fell down to the back of the chassis. I'm thankful I never stumbled into it.