19 ms·
The slow rise of robots in the data center
- johnohara 5y agoGone are the days of tape operators on roller skates, but this article is correct in its assessment that change in the datacenter moves at a very deliberate pace. Oddly enough, I found the reading pace of this article similar to the change it describes. Nothing negative mind you, just gently rolling in its delivery.
- markus_zhang 5y agoHi do you have any book/video talking about skaters in data center? I checked Youtube and only found one for switch board. Thanks!
- donalhunt 5y agoFrom personal experience, health and safety officers prefer workers to walk around facilities. Accidents while rare can result in hospitalisation (broken bones) when using skates, scooters and bicycles. I was once in a datacenter that had a golf cart to get from one end to another (yes - it was that long).
- markus_zhang 5y agoWow that's fun time~~
- kjs3 5y agoI've been in 3 or 4 (very large) datacenters over the years where they used 3-wheeled bicycles to get around. Bonus of this was there's a pretty roomy basket between the back wheels to hauls stuff around. But yeah...OSHA (in the USA) applies to datacenters too and I can't imagine allowing roller skates.
- johnohara 5y agoNo. Sorry. My comment was from a conversation I had in the mid-80's with a couple of field service reps who worked the site and couldn't believe it themselves. The magtapes used to hang on row after row of metal racks. The operators would retrieve the tapes, hold them on their forearms, and bring them back to what were usually a centralized set of tape drives. Here's an offset into a video showing what it typically looked like. [0] The narrator refers to a "search tape" used to retrieve the person's information. The tape had to be retrieved from the racks. The sheer weight and overall length of the rows/racks meant the rooms were often lower level with vinyl flooring over concrete (raised tile later). Imagine walking up and down for an 8-hour 5pm-1am shift. Not difficult to imagine skates. Here's another video with a good representation of how it worked. [1] As I have come to accept, the mid-80's was pre-everything. Deliberate change takes time. [0]: https://youtu.be/Iddrm7mHPrY?t=111 https://youtu.be/Iddrm7mHPrY?t=111 [1]: https://www.youtube.com/watch?v=Cwj6pfhWBps https://www.youtube.com/watch?v=Cwj6pfhWBps
- markus_zhang 5y agoThanks a lot for the links!
- dtgriscom 5y agoA few seconds after your starting point in that first video, you can see a loop of tape vibrating back in forth in a rectangular channel. That's a vacuum column, with a (mild) vacuum pulling the tape loop to the right. This lets the servomotors that move the tape past the heads start and stop rapidly, while the heavy tape reels can spin up and down more slowly. See how the vibrating loop passes back and forth across a hole in the middle of the column? That's a pressure sensor. As the loop moves across the hole, the pressure in the hole changes, telling the drive whether it's time to reel in or dispense more tape. More info: https://en.wikipedia.org/wiki/9_track_tape https://en.wikipedia.org/wiki/9_track_tape
- johnohara 5y agoIIRC, once it had a stable hold on the tape the drive would begin reading in an effort to locate the start of the data. On old tapes this might take a number of retries with the drive rewinding, reading, rewinding, reading... etc. Sometimes it didn't work at all and you were stuck with a bad tape that needed special attention.
- johnohara 5y agoCouldn't resist. Here's another video showing a streamlined and "much more compact" computer room from around 1990. That 44GB of disk storage (DASD) probably cost $150,000 at the time. Somebody signed the P.O. and slept well that night. [0] Best part. You could roller skate to the music. [0]: https://www.youtube.com/watch?v=vlvUz3T4WTA https://www.youtube.com/watch?v=vlvUz3T4WTA
- sithadmin 5y agoNot sure if I really buy that the Switch security bot is a serious project, given that over-the-top security theater seems to be one of Switch's main marketing points. Their employees wear military tactical gear; their facility walls and gates are designed like they're meant to house a prison; they have racks of what appear to be real guns and riot gear displayed prominently behind the security folks that check you in. Factor in the Bond-villain architectural aesthetic and it's really just too much to take seriously. Bizarro security roleplaying aside, Switch is probably the most professional and competent data center operator I've worked with.
- formerly_proven 5y ago> Factor in the Bond-villain architectural aesthetic and it's really just too much to take seriously. -- Sith admin
- donalhunt 5y ago[deleted]
- foobarbazetc 5y agoPresumably the Fed/State government people who hand out the DC contracts like the security cosplay. :)
- dmurray 5y ago
- aperrien 5y agoI'd think that a robot that could neatly wire up racks with cabling (ethernet, power, & fiber) would be quite doable and useful. Does anyone here have experience with such a project?
- sithadmin 5y agoHaving wasted way more hours of my life than I would like on rack-and-stack...I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. Which isn't going to happen any time soon, given that the current standards have so much momentum behind them. As things are: even in well-managed data centers, the racks themselves are always somewhat finnicky, with varying levels of precision in assembly from rack to rack that require odd workarounds for equipment installation more often than one would expect. And that's not to mention how incredibly variable the rack enclosures themselves can be, which has big implications for cable routing. And nevermind the fact that there's basically no standard for port placement on rackmount systems. Rack-and-stack labor is dirt cheap, or easily foisted off on your sysadmins for small deployments. I don't see a robot for this being competitive from a cost standpoint unless that robot is extremely general purpose and able to fulfill other roles.
- bluedino 5y ago>> I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. I think the same thing when people talking about sending a 'plumbing robot' to your house.
- donalhunt 5y agoThis. Contingent workforces are cheap (minimum wage essentially). It's also cheaper to use connecters that mate on insertion than to use connectors that humans are accustomed to.
- xyzzy_plugh 5y agoIndeed. Even the largest DCs are putting so few racks up per day (still many, many racks) that throughput is rarely the problem. Actually getting the parts on time has been more of an issue in my experience, or finding out that your entire rack is forfeit because it fell over at the docking bay. I can count on one hand the number of times I've seen rack-and-stack go too slow for the consumers to actually use the hardware. Usually the hardware is sitting waiting to be provisioned for eons. The real labor is pulling bad drives/boards and rejiggering the network after the racks are in. So many outages are caused by magic hands making an error and pulling the wrong cord.
- pm90 5y agoIt would be fascinating if we had infinitely reconfigurable data center network topologies. Among other things, you could have eg Just In Time network rebalancers that could add more capacity between nodes that have a lot of traffic between them. The reconfiguration would have to be instant, or at least take less time than the actual data transfer to have any benefits.
- sithadmin 5y agoThings are already moving this direction with adoption of network virtualization solutions and overlays becoming the norm in corporate data centers.
- wmf 5y agoThere has been a bunch of research on reconfigurable optical datacenter networks to do this.
- kjs3 5y agoYou do that virtually, not physically.
- kijiki 5y agoPlexxi tried something along those lines. It was cool, but very expensive, so it didn't really go anywhere.
- donalhunt 5y agoMy sense is that robots are in use more than the article makes out. Anywhere there is a risk of human error or repetitive work that can be simplified, you'll find robots are a candidate. The main issues with robots in my experience is whether you can get the desired throughput while including planned and unplanned maintenance work, reconfigurations, etc. The upfront capital cost generally means you don't have the ability to have a drop-in replacement ready to go when a change is required. That results in downtime. Folk familiar with LEAN six sigma will know that hidden bottlenecks should be avoided or you build up large backlogs during downtime. Good news for datacenter engineers is that fixing robots may become part of your remit in the future (interesting niche given the mix of software and mechanical). More interesting work will always exist!
- dv_dt 5y agoAre there any interesting mathematical or other tools that you recommend to evaluate hidden bottlenecks systematically? I wonder for example, about Toyota making a decision to add inventory buffer for chips vs other auto makers. Great decision in hindsight, but how did they evaluate that in context with other possible actions.
- pm90 5y agoThe only way to know for sure is by experiments/simulations. The heart of any competent operations team, whether its in computer systems or plain old logistics, is drills. You can't simulate every possible scenario, but you try to analyze and get an initial set, and the build that up as you run into actual production issues. What you want to avoid is running scripts for the first time in the middle of a disaster.
- wiml 5y agoFrom what I remember this (at least used to) fall under the heading "operations research", which spawned a lot of mathematical tools used elsewhere like queueing theory. Related terms include supply chain engineering, logistics, industrial engineering, and the general ideas of scientific management, Taylorism, etc.
- MR4D 5y agoSponsored by CBRE.
- klodolph 5y agoThere are various quotes about robots at Google, but the picture labeled "Google" is of an ordinary tape library (looks like a StorageTek / Oracle library). If Google has built robots, that's not one of them.
- tablespoon 5y ago> “The server rack is more than 50 years old. There is no other piece of technology in data centers that has survived for so long," Zsolt Szabo told DCD back in 2016. Quick, someone tell him about floors, walls, ceilings, doors, and electric light.
- laurent92 5y agoTell him about the SR-71 (entered service in 1966, and it’s still the same instances that are flying). Most Cessnas flying are also 50+ years old, because if you purchase a more recent one, it has to comply with all the newer rules.
- tablespoon 5y agoYeah, but those aren't in data centers.
- lawrencema92 5y agoFor now
- 1-more 5y agoYeah high speed high altitude reconnaissance is simply not mission-appropriate. That's why Switch uses predator drones loitering in a tight loop pattern for surveillance and a quick strike.
- pacificmint 5y agoThe SR-71 hasn’t flown in 22 years, so that’s a bad example. There are of course planes that have flown for that long, and longer. The B-52s come to mind. But the SR-71 isn’t one of them.
- WalterBright 5y agoDC-3s are still flying commercial (not joyriding).
- 5y ago
- tabtab 5y agoBack in the 90's we half joked that we needed a remote controlled robot with 3 fingers to press Alt Ctrl Delete rather than drive down to the data center at 3am to reboot a jammed server. I sketched "R2Reboot".
- trhway 5y agorecent AWS Frankfurt event when the datacenter was flooded with gas to push oxygen out seems to make a good use case for the robots there.
- contingencies 5y agoWhy did they do that? Are they trying to kill sleeping sysadmins, vaguely threaten intruders, did they have a rat problem, or what? Seems ... inexplicable without the involvement of bureaucrats.
- trhway 5y agoit was a fire scare. AC failure (some control mistake) -> rise of temp -> some hardware started to smoke.
- contingencies 5y agoAha. I wonder what that costs each time it triggers.
- SilverRed 5y agoPresumably less than the value of the building and server racks.
- kjs3 5y agoSee: Halon fire extinguisher
- grouphugs 5y agothe slow rise of poverty without a matching process of socialization