5 ms·

Speaking as a Rackspace customer, if any of these guys are Linux techs, and you're looking, hire them. With signing bonuses. Anecdote: A couple years ago, I ha

by danjoc 10y ago

Speaking as a Rackspace customer, if any of these guys are Linux techs, and you're looking, hire them. With signing bonuses. Anecdote: A couple years ago, I had one explain to me (in a way that made sense) how the battery on the raid array was probably the cause of some problems with https. And _he was right_. Maybe not everyone there is a certified genius, but that really blew my mind. They really know hardware. I haven't talked to one who couldn't save my tail in a pinch. Rackspace might seem a bit on the expensive side, but their support is absurdly good.

backwardm 10y ago
I agree. Their support is always really great. Was the battery causing the time to drift or something, so that the SSL stuff saw the wrong time & broke? Just wondering how a battery could mess that up!
- danjoc 10y ago
  I don't remember now to be honest. I should have written it down :D It was an early interaction with them, and I didn't seriously think he would be right.
keyboardhitter 10y ago
Former Racker here, while I wouldn't agree all of their techs are amazing, what I liked about working there was the vast amount of knowledge spread between teams - coupled with Texan kindness and hospitality. Generally speaking any answer I needed I could get from a phone call or a few IMs. Working there was a great learning experience, and I met a ton of super intelligent, motivated people from the Austin office. But, all good things come to an end. Early 2015 I noticed talent being pushed towards newer offerings (e.g. Azure support), which made the typical, aging, "Linux support for small to medium business on bare metal machines" not so 'fanatical' anymore. People started to jump ship because the force behind the changes. It was a tough transition and probably still is; so I'm not really surprised that the layoffs are continuing. In any case, I appreciate your compliment and I'm glad you have had a good time with the company.
neom 10y ago
+1 - We hired many Rackspace folks at DigitalOcean when I worked there and I can't speak highly enough about those I encountered.
20years 10y ago
I was a previous Rackspace customer for over 10 years and at one time had 9 dedicated servers with them. Their Linux techs were top notch. IMO you are smart to want to obtain some of them. Unfortunately stating in early 2015 the support started to deteriorate. That coupled with the cost we were paying for their dedicated servers/fanatical support stopped making since.
- crikli 10y ago
  My experience has been sadly similar. We used to have probably 30 servers at the peak across several client accounts. But yup, around the first part of 2015 what used to be a one ring call to connect with an expert became 5 minutes, then 15, and a couple weeks back I waited 45 minutes to talk to a Linux tech. Unfuckingacceptable, not as a "managed" customer paying an extra $100/mo and $0.12. We've slowly but surely moved clients to other hosts and I can count the servers still at Rackspace on one hand and still have fingers.
- rjbwork 10y ago
  Yep. I know a guy who was a top level tech guy there and ran a big hardware lab and did support for tough cases. He left, according to him, because management just didn't understand the tech side anymore and was implementing all kinds of new things the support and tech guys didn't like and talent had been leaving for a while.
mikeash 10y ago
So... how was that battery the cause of an https problem?
- duskwuff 10y ago
  Not the OP, but a guess: A lot of RAID cards will switch from writeback mode to writethrough when the battery is low, because they can't guarantee that the (battery-backed) cache will persist across a power failure. Writethrough mode makes writes significantly slower, which could cause problems higher up the stack.
  - bogomipz 10y ago
    Yep they will also go write through during a battery "learn cycle". The DELL PERC controllers are notorious for this. And there is poor visibility into this. You basically have to know about the awful MegaCli utility to gain any insight. What is the learn cycle? "The purpose of the learn cycle is to determine the condition of the battery. The learn cycle charges, fully discharges, and then recharges the battery in order to determine the condition and health of the battery. The battery full charge capacity degrades over time and a battery is deemed completely degraded when it can no longer hold a charge for 24 hours and must be replaced." Source: https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-Greenplum-DCA-PERC-Battery https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-G...
    - Dylan16807 10y ago
      What kind of trash-tier reliability-increasing hardware stops working for a day every 90 days? I'm really surprised they didn't put two or three batteries into it.
      - true_tuna 10y ago
        All of them. You learn to disable learn cycle, most people learn that the hard way. I switched to flash backed cache for this reason (along with being sick of replacing batteries)
gourneau 10y ago
Heya if you are a racker looking for a job in the bay area please email me :) josh@gourneau.com Our company does neat biotech stuff
- keithwhor 10y ago
  Relentless, Josh. This guy is super talented and worked for NASA, listen to him, he will not steer you wrong.
DrScump 10y ago
It's reminiscent of when Dell closed their Round Rock support operation.
dvnguyen 10y ago
Reading this makes me realize how naive I was. I thought I could rent a cheap dedicated server from OVH and leverage Docker to build nice infrastructure without paying too much. Better staying with Google/AWS then.
dawnerd 10y ago
I used them for email for a while and I couldn't get my mail client to connect (Airmail, first gen). The guy on the phone spent hours with me and even purchased a copy of the app just to see if it would work for him. I don't remember what ultimately fixed it but I got it working - and it wasn't even a rackspace problem if I recall correctly. Still impressed by how far they were willing to go to help a customer.
- HeadlessChild 10y ago
  Speaking as a Linux support technician I would think that we really enjoy fixing things and have a hard time not to.

by danjoc 10y ago

backwardm 10y ago

I agree. Their support is always really great. Was the battery causing the time to drift or something, so that the SSL stuff saw the wrong time & broke? Just wondering how a battery could mess that up!

danjoc 10y ago
I don't remember now to be honest. I should have written it down :D It was an early interaction with them, and I didn't seriously think he would be right.

keyboardhitter 10y ago

Former Racker here, while I wouldn't agree all of their techs are amazing, what I liked about working there was the vast amount of knowledge spread between teams - coupled with Texan kindness and hospitality. Generally speaking any answer I needed I could get from a phone call or a few IMs. Working there was a great learning experience, and I met a ton of super intelligent, motivated people from the Austin office. But, all good things come to an end. Early 2015 I noticed talent being pushed towards newer offerings (e.g. Azure support), which made the typical, aging, "Linux support for small to medium business on bare metal machines" not so 'fanatical' anymore. People started to jump ship because the force behind the changes. It was a tough transition and probably still is; so I'm not really surprised that the layoffs are continuing. In any case, I appreciate your compliment and I'm glad you have had a good time with the company.

neom 10y ago

+1 - We hired many Rackspace folks at DigitalOcean when I worked there and I can't speak highly enough about those I encountered.

20years 10y ago

I was a previous Rackspace customer for over 10 years and at one time had 9 dedicated servers with them. Their Linux techs were top notch. IMO you are smart to want to obtain some of them. Unfortunately stating in early 2015 the support started to deteriorate. That coupled with the cost we were paying for their dedicated servers/fanatical support stopped making since.

crikli 10y ago
My experience has been sadly similar. We used to have probably 30 servers at the peak across several client accounts. But yup, around the first part of 2015 what used to be a one ring call to connect with an expert became 5 minutes, then 15, and a couple weeks back I waited 45 minutes to talk to a Linux tech. Unfuckingacceptable, not as a "managed" customer paying an extra $100/mo and $0.12. We've slowly but surely moved clients to other hosts and I can count the servers still at Rackspace on one hand and still have fingers.
rjbwork 10y ago
Yep. I know a guy who was a top level tech guy there and ran a big hardware lab and did support for tough cases. He left, according to him, because management just didn't understand the tech side anymore and was implementing all kinds of new things the support and tech guys didn't like and talent had been leaving for a while.

mikeash 10y ago

So... how was that battery the cause of an https problem?

duskwuff 10y ago
Not the OP, but a guess: A lot of RAID cards will switch from writeback mode to writethrough when the battery is low, because they can't guarantee that the (battery-backed) cache will persist across a power failure. Writethrough mode makes writes significantly slower, which could cause problems higher up the stack.
- bogomipz 10y ago
  Yep they will also go write through during a battery "learn cycle". The DELL PERC controllers are notorious for this. And there is poor visibility into this. You basically have to know about the awful MegaCli utility to gain any insight. What is the learn cycle? "The purpose of the learn cycle is to determine the condition of the battery. The learn cycle charges, fully discharges, and then recharges the battery in order to determine the condition and health of the battery. The battery full charge capacity degrades over time and a battery is deemed completely degraded when it can no longer hold a charge for 24 hours and must be replaced." Source: https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-Greenplum-DCA-PERC-Battery https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-G...
  - Dylan16807 10y ago
    What kind of trash-tier reliability-increasing hardware stops working for a day every 90 days? I'm really surprised they didn't put two or three batteries into it.
    - true_tuna 10y ago
      All of them. You learn to disable learn cycle, most people learn that the hard way. I switched to flash backed cache for this reason (along with being sick of replacing batteries)

gourneau 10y ago

Heya if you are a racker looking for a job in the bay area please email me :) josh@gourneau.com Our company does neat biotech stuff

keithwhor 10y ago
Relentless, Josh. This guy is super talented and worked for NASA, listen to him, he will not steer you wrong.

DrScump 10y ago

It's reminiscent of when Dell closed their Round Rock support operation.

dvnguyen 10y ago

Reading this makes me realize how naive I was. I thought I could rent a cheap dedicated server from OVH and leverage Docker to build nice infrastructure without paying too much. Better staying with Google/AWS then.

dawnerd 10y ago

I used them for email for a while and I couldn't get my mail client to connect (Airmail, first gen). The guy on the phone spent hours with me and even purchased a copy of the app just to see if it would work for him. I don't remember what ultimately fixed it but I got it working - and it wasn't even a rackspace problem if I recall correctly. Still impressed by how far they were willing to go to help a customer.

HeadlessChild 10y ago
Speaking as a Linux support technician I would think that we really enjoy fixing things and have a hard time not to.