3 ms·

I'd think that a robot that could neatly wire up racks with cabling (ethernet, power, & fiber) would be quite doable and useful. Does anyone here have experienc

I'd think that a robot that could neatly wire up racks with cabling (ethernet, power, & fiber) would be quite doable and useful. Does anyone here have experience with such a project?

sithadmin 5y ago
Having wasted way more hours of my life than I would like on rack-and-stack...I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. Which isn't going to happen any time soon, given that the current standards have so much momentum behind them. As things are: even in well-managed data centers, the racks themselves are always somewhat finnicky, with varying levels of precision in assembly from rack to rack that require odd workarounds for equipment installation more often than one would expect. And that's not to mention how incredibly variable the rack enclosures themselves can be, which has big implications for cable routing. And nevermind the fact that there's basically no standard for port placement on rackmount systems. Rack-and-stack labor is dirt cheap, or easily foisted off on your sysadmins for small deployments. I don't see a robot for this being competitive from a cost standpoint unless that robot is extremely general purpose and able to fulfill other roles.
- bluedino 5y ago
  >> I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. I think the same thing when people talking about sending a 'plumbing robot' to your house.
- donalhunt 5y ago
  This. Contingent workforces are cheap (minimum wage essentially). It's also cheaper to use connecters that mate on insertion than to use connectors that humans are accustomed to.
- xyzzy_plugh 5y ago
  Indeed. Even the largest DCs are putting so few racks up per day (still many, many racks) that throughput is rarely the problem. Actually getting the parts on time has been more of an issue in my experience, or finding out that your entire rack is forfeit because it fell over at the docking bay. I can count on one hand the number of times I've seen rack-and-stack go too slow for the consumers to actually use the hardware. Usually the hardware is sitting waiting to be provisioned for eons. The real labor is pulling bad drives/boards and rejiggering the network after the racks are in. So many outages are caused by magic hands making an error and pulling the wrong cord.
- Robotbeat 5y ago
  Basically, a giant blade server is what a server rack optimized for automated server install and replacement would look like. Something like a cross between a vertical warehouse robot and a tape jukebox is what would service it. Of course, at that point it you’re adding a bunch of cost and may be better off just designing the servers to last for like 10 years with enough redundancy (and hot/warm/cold spares) to not need any physical swapping, then replace the whole thing with a forklift at end of life.
  - wmf 5y ago
    Submer seems to be working on exactly that. https://www.youtube.com/watch?v=zIB_BIEttFo https://www.youtube.com/watch?v=zIB_BIEttFo (the robot is at the end)
  - eb0la 5y ago
    A friend worked in Live (now called Bing) some years ago between 2004-2007 I guess... He told me they had a robot like the ones from storagetek but for unplugging faulty blade servers and plug back new ones. Makes sense for large scale installations.
jjeaff 5y ago
I think it would be a lot more likely that the next step would be a different "smart rack" where connections, or a single, huge bandwidth network cable would come in to the rack in a single place. And then have a single connection per device in the rack. Then software would handle the routing.
- deleted 5y ago
  [deleted]
panic_on_oops 5y ago
Maybe not rack and stack, this will definitely not fully replace traditional datacenter operations but complement it and make it more streamline. Thinking about how we currently do disk and cable swaps,memory or even system boards misses the point. Of course it can't be automated because there's no unified standard. There needs to be one for such tasks to be fully automated with robots, the rack, server, network switch and even hot/cold aisles need to be redesigned to work with a standard that supports robotics arms instead of human ones. No more wires, maybe conductive rails. This will take infinity to accomplish (barring disruption in compute/storage) because - * Remote human hands are ridiculously cheap, and they can perform functions robots can't without extra pay or a paid for software upgrade (think shipment handling, etc). Only real use case that comes to mind is one without economic justification, but of necessity. Think datacenters in space or deep underwater, or hostile environments. So maybe innovation in this field will come from the government this time, and trickle to the private sector. Maybe. * The cost of such investment far outweighs the financial gains. In the very competitive cloud business, it's the services you offer that matters most, and the reliability of your systems. Sending a person with a code scanner to verify and do a quick disk or cable swap is not a risky endeavour. Unplugging the wrong cable and causing an outage never happens on properly designed systems, with properly set maintenances, so this is irrelevant (when was the last time you've heard that in an outage retrospect?)

I'd think that a robot that could neatly wire up racks with cabling (ethernet, power, & fiber) would be quite doable and useful. Does anyone here have experience with such a project?

sithadmin 5y ago

Having wasted way more hours of my life than I would like on rack-and-stack...I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. Which isn't going to happen any time soon, given that the current standards have so much momentum behind them. As things are: even in well-managed data centers, the racks themselves are always somewhat finnicky, with varying levels of precision in assembly from rack to rack that require odd workarounds for equipment installation more often than one would expect. And that's not to mention how incredibly variable the rack enclosures themselves can be, which has big implications for cable routing. And nevermind the fact that there's basically no standard for port placement on rackmount systems. Rack-and-stack labor is dirt cheap, or easily foisted off on your sysadmins for small deployments. I don't see a robot for this being competitive from a cost standpoint unless that robot is extremely general purpose and able to fulfill other roles.

bluedino 5y ago
>> I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. I think the same thing when people talking about sending a 'plumbing robot' to your house.
donalhunt 5y ago
This. Contingent workforces are cheap (minimum wage essentially). It's also cheaper to use connecters that mate on insertion than to use connectors that humans are accustomed to.
xyzzy_plugh 5y ago
Indeed. Even the largest DCs are putting so few racks up per day (still many, many racks) that throughput is rarely the problem. Actually getting the parts on time has been more of an issue in my experience, or finding out that your entire rack is forfeit because it fell over at the docking bay. I can count on one hand the number of times I've seen rack-and-stack go too slow for the consumers to actually use the hardware. Usually the hardware is sitting waiting to be provisioned for eons. The real labor is pulling bad drives/boards and rejiggering the network after the racks are in. So many outages are caused by magic hands making an error and pulling the wrong cord.
Robotbeat 5y ago
Basically, a giant blade server is what a server rack optimized for automated server install and replacement would look like. Something like a cross between a vertical warehouse robot and a tape jukebox is what would service it. Of course, at that point it you’re adding a bunch of cost and may be better off just designing the servers to last for like 10 years with enough redundancy (and hot/warm/cold spares) to not need any physical swapping, then replace the whole thing with a forklift at end of life.
- wmf 5y ago
  Submer seems to be working on exactly that. https://www.youtube.com/watch?v=zIB_BIEttFo https://www.youtube.com/watch?v=zIB_BIEttFo (the robot is at the end)
- eb0la 5y ago
  A friend worked in Live (now called Bing) some years ago between 2004-2007 I guess... He told me they had a robot like the ones from storagetek but for unplugging faulty blade servers and plug back new ones. Makes sense for large scale installations.

jjeaff 5y ago

I think it would be a lot more likely that the next step would be a different "smart rack" where connections, or a single, huge bandwidth network cable would come in to the rack in a single place. And then have a single connection per device in the rack. Then software would handle the routing.

deleted 5y ago
[deleted]

panic_on_oops 5y ago

Maybe not rack and stack, this will definitely not fully replace traditional datacenter operations but complement it and make it more streamline. Thinking about how we currently do disk and cable swaps,memory or even system boards misses the point. Of course it can't be automated because there's no unified standard. There needs to be one for such tasks to be fully automated with robots, the rack, server, network switch and even hot/cold aisles need to be redesigned to work with a standard that supports robotics arms instead of human ones. No more wires, maybe conductive rails. This will take infinity to accomplish (barring disruption in compute/storage) because - * Remote human hands are ridiculously cheap, and they can perform functions robots can't without extra pay or a paid for software upgrade (think shipment handling, etc). Only real use case that comes to mind is one without economic justification, but of necessity. Think datacenters in space or deep underwater, or hostile environments. So maybe innovation in this field will come from the government this time, and trickle to the private sector. Maybe. * The cost of such investment far outweighs the financial gains. In the very competitive cloud business, it's the services you offer that matters most, and the reliability of your systems. Sending a person with a code scanner to verify and do a quick disk or cable swap is not a risky endeavour. Unplugging the wrong cable and causing an outage never happens on properly designed systems, with properly set maintenances, so this is irrelevant (when was the last time you've heard that in an outage retrospect?)