4 ms·

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.) - At one point we temporarily laser-beamed bandwidth across the street to another building - There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.

lostlogin 2y ago
> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Pointing the fans in or out?
- sitkack 2y ago
  You want to point them in.
  - gr3ml1n 2y ago
    The datacenters I've been in with emergency cooling fans in the walls all exhaust out, not in. Easier to get portable CRACs inside and get a good draft going.
wgx 2y ago
I recall getting a DC tour of LON3 and being totally blown away by it all as a 20-something web dev. Good times.
- chasd00 2y ago
  When I was in college I’d call up datacenters pretending to be a prospective customer and schedule a tour. I was totally fascinated by them and knew enough to sound legit, it was like going to an amusement park for me.
  - kjs3 2y ago
    When I was in college, I got a job in the campus DC for the same reason. Best job ever for an undergraduate student.
latchkey 2y ago
We had a bird land on a transformer up on a pole and blew fuses. A couple years later, I toured the facility and the fried carcass was still there on the ground below it.
- cbg0 2y ago
  Left as a warning to other birds, no doubt.
- datavirtue 2y ago
  This is fine.
kryogen1c 2y ago
> Data center science has... well improved since the earlier days You say that, but... > There was one day we knocked out windows and purchased box fans because servers were literally catching on fire This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute. Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and windows open trying to keep things cold enough, but alas. As the CRAC goes, so goes the servers
- inopinatus 2y ago
  running European ISPs in summer we’d nick desk fans off the telco folks to cool down our walls of USR Sportsters, distracting them first with snarky remarks about ATM overhead absolutely do not miss those days
- Henchman21 2y ago
  I’ve worked in CH1 for years now. The glycol in the chillers froze. Thats how cold it was! It was also 115 degrees ambient temp inside CH1. Techs were dipping in and out 5-10 minutes at a time to avoid heat stroke
greenavocado 2y ago
When it comes to Internet service we're living in the early 2000s in the some parts of the manufacturing world
- deleted 2y ago
  [deleted]
- kjs3 2y ago
  Manufacturing is always about 25 years behind the times. I made good scratch in the '00s helping manufactures with their DEC PDP-11 and DG Novas (from the 70s).
drpixie 2y ago
> and purchased box fans because servers were literally catching on fire Ah yes, or a collection of R2D2 portable air conditioners, with the tails draped out through the window. Or a coolant leak that no one noticed until the sub-floor was completely full and the floor panels started to float!
inopinatus 2y ago
Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.
- mdaniel 2y ago
  > dropped on my foot during installation, ... spanning tree misconfiguration, ... was dead within two years. Yikes, that escalated quickly. I'm glad you escaped the Switch Grim Reaper and my condolences to the families of the rest :(
- dmd 2y ago
  > everyone else involved in that project was dead within two years wait, what?
  - inopinatus 2y ago
    the tech sector in the 90s got pretty wild
chasd00 2y ago
In the bad old days I had a server at blue host in Dallas. Went to the dc once and there extension cords accross the racks suspended about 1ft off the ground that I had to step over to get to my server. Hey at least it was cheap :)
paulnpace 2y ago
I attended an OCP lecture by someone involved in building a facebook DC. One of the stories was learning that stuff on top gets hotter than stuff on bottom. This is, like, basic stuff here, guys. I've never understood the hiring practices in these projects.
PaulRobinson 2y ago
Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences: - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later we restored connectivity - Our main (BGP-ified) router out of the DC - no, there wasn't a redundant device - kept rebooting. A quick check showed the temp in the DC was so high, cooling so poor, that the *inlet* fan had an air temp of over 60C. We pointed some fans at it as a temporary measure. - In a similar vein, a few weeks later the air con in another room just gave up and started spewing water over the Nortel DMS-100 (we were a dial-in ISP with our own switch). Wasn't too happy to be asked to help mop it up (thought the water could potentially be live), but what to do? After that experience I spent time on a small, remote island where main link to the internet was a 1MB/sec link vis GS satellite (ping times > 500ms), and where the locals dialled in over a microwave phone network rated to 9600 baud, but somehow 56k modems worked... One fix I realised I needed was a Solaris box was missing a critical .so, there were no local backups or install media and so I phoned my mate back in the UK and asked him to whack up a copy on an FTP server for me to get the box back online. And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times. It was all terrific fun, but I'm so glad I now only really do software.
- Bluecobra 2y ago
  > It was all terrific fun, but I'm so glad I now only really do software. I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.
  - PaulRobinson 2y ago
    Better than having to collect the fox urine first...
  - 2y ago
kolanos 2y ago
> Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time. Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.
- mdaniel 2y ago
  Pro: even more opportunities to spy on every user in the world Con: interacting with internal stakeholders is waaaaay different from doing the same for the general public paying you. See also: every mention of GCP that ever shows up in these threads Plus all their SDKs would be written in php :-P

lostlogin 2y ago

> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Pointing the fans in or out?

sitkack 2y ago
You want to point them in.
- gr3ml1n 2y ago
  The datacenters I've been in with emergency cooling fans in the walls all exhaust out, not in. Easier to get portable CRACs inside and get a good draft going.

wgx 2y ago

I recall getting a DC tour of LON3 and being totally blown away by it all as a 20-something web dev. Good times.

chasd00 2y ago
When I was in college I’d call up datacenters pretending to be a prospective customer and schedule a tour. I was totally fascinated by them and knew enough to sound legit, it was like going to an amusement park for me.
- kjs3 2y ago
  When I was in college, I got a job in the campus DC for the same reason. Best job ever for an undergraduate student.

latchkey 2y ago

We had a bird land on a transformer up on a pole and blew fuses. A couple years later, I toured the facility and the fried carcass was still there on the ground below it.

cbg0 2y ago
Left as a warning to other birds, no doubt.
datavirtue 2y ago
This is fine.

kryogen1c 2y ago

> Data center science has... well improved since the earlier days You say that, but... > There was one day we knocked out windows and purchased box fans because servers were literally catching on fire This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute. Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and windows open trying to keep things cold enough, but alas. As the CRAC goes, so goes the servers

inopinatus 2y ago
running European ISPs in summer we’d nick desk fans off the telco folks to cool down our walls of USR Sportsters, distracting them first with snarky remarks about ATM overhead absolutely do not miss those days
Henchman21 2y ago
I’ve worked in CH1 for years now. The glycol in the chillers froze. Thats how cold it was! It was also 115 degrees ambient temp inside CH1. Techs were dipping in and out 5-10 minutes at a time to avoid heat stroke

greenavocado 2y ago

When it comes to Internet service we're living in the early 2000s in the some parts of the manufacturing world

deleted 2y ago
[deleted]
kjs3 2y ago
Manufacturing is always about 25 years behind the times. I made good scratch in the '00s helping manufactures with their DEC PDP-11 and DG Novas (from the 70s).

drpixie 2y ago

> and purchased box fans because servers were literally catching on fire Ah yes, or a collection of R2D2 portable air conditioners, with the tails draped out through the window. Or a coolant leak that no one noticed until the sub-floor was completely full and the floor panels started to float!

inopinatus 2y ago

Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.

mdaniel 2y ago
> dropped on my foot during installation, ... spanning tree misconfiguration, ... was dead within two years. Yikes, that escalated quickly. I'm glad you escaped the Switch Grim Reaper and my condolences to the families of the rest :(
dmd 2y ago
> everyone else involved in that project was dead within two years wait, what?
- inopinatus 2y ago
  the tech sector in the 90s got pretty wild

chasd00 2y ago

In the bad old days I had a server at blue host in Dallas. Went to the dc once and there extension cords accross the racks suspended about 1ft off the ground that I had to step over to get to my server. Hey at least it was cheap :)

paulnpace 2y ago

I attended an OCP lecture by someone involved in building a facebook DC. One of the stories was learning that stuff on top gets hotter than stuff on bottom. This is, like, basic stuff here, guys. I've never understood the hiring practices in these projects.

PaulRobinson 2y ago

Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences: - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later we restored connectivity - Our main (BGP-ified) router out of the DC - no, there wasn't a redundant device - kept rebooting. A quick check showed the temp in the DC was so high, cooling so poor, that the *inlet* fan had an air temp of over 60C. We pointed some fans at it as a temporary measure. - In a similar vein, a few weeks later the air con in another room just gave up and started spewing water over the Nortel DMS-100 (we were a dial-in ISP with our own switch). Wasn't too happy to be asked to help mop it up (thought the water could potentially be live), but what to do? After that experience I spent time on a small, remote island where main link to the internet was a 1MB/sec link vis GS satellite (ping times > 500ms), and where the locals dialled in over a microwave phone network rated to 9600 baud, but somehow 56k modems worked... One fix I realised I needed was a Solaris box was missing a critical .so, there were no local backups or install media and so I phoned my mate back in the UK and asked him to whack up a copy on an FTP server for me to get the box back online. And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times. It was all terrific fun, but I'm so glad I now only really do software.

Bluecobra 2y ago
> It was all terrific fun, but I'm so glad I now only really do software. I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.
- PaulRobinson 2y ago
  Better than having to collect the fox urine first...
- 2y ago

kolanos 2y ago

> Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time. Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.

mdaniel 2y ago
Pro: even more opportunities to spy on every user in the world Con: interacting with internal stakeholders is waaaaay different from doing the same for the general public paying you. See also: every mention of GCP that ever shows up in these threads Plus all their SDKs would be written in php :-P