5 ms·

My company used Rackspace back in 2013. It was my first gig where I was "managing servers" and "deploying to production". The rest of the team was as inexperien

My company used Rackspace back in 2013. It was my first gig where I was "managing servers" and "deploying to production". The rest of the team was as inexperienced as I was. We paid extra for the support services that entitled us to unlimited 24/7 human support. Whenever I was stuck on an issue with deploying, scaling, crashing etc. my boss would say "try calling support." Initially you'd get a first-line operator but I learned quickly to "escalate to a sysadmin". The sysadmin was eager to help and would log into your VPS and do anything you'd ask, plus advise and implement on stuff they thought you'd need! Debug your running app and code? Sure. Fix broken packages or get something to compile? Of course. Custom scaling and deployment scripts in bash? Gladly. Some reps were more helpful than others and I got to know them on a first-name basis. In hindsight it seems totally absurd to have a phone support running bash on our VPS but it was super helpful at the time. I learned a lot and they saved my bacon a few times. I can only imagine the liability issues.

stingraycharles 4y ago
I’ve had a similar pleasure in the early 2000s. It may seem absurd, but from a business perspective it makes sense: so many support questions fall into a gray area of “this is really not our business because the customer doesn’t know X, but without solving it they’ll be unhappy”. It makes sense to just say “fuck it, we’ll help them with anything, as long as they’re paying”. It’s a similar approach as we take at my current employer. If you’re using our database products, even if you’re stuck doing basic python stuff, don’t know how docker works, or an absolute noob at AWS, we’ll help you. The time we spend is cheaper than losing business, and enterprises love it.
- bengale 4y ago
  We do something similar on the non-tech side. We offer a saas product but most of our questions are more around how the person should be doing their job rather than how to use our software so we have a small team of experts that can answer those questions, they can also help with content creating on the marketing side too which is handy. We constantly hear from our customers that this level of service is one of the key things they really value from us.
  - JALTU 4y ago
    +1 Build the customer relationship.
hotpotamus 4y ago
I probably know some of the guys who helped you out and there's a couple great ones still there. The legal team came up with the phrase "reasonable endeavor" (they didn't like best effort), and they wrote a disclaimer form they'd have customers fill out that what they were about to was beyond official support. As long as you signed that, support was usually pretty happy to "give it the old college try" as one told me. And I'll let you in on another little secret - a lot of those sysadmins were learning along with you. One of my very early projects was building out a precision time protocol service for a customer who asked for it - which ended up being a good learning experience for me.
bengale 4y ago
I'm pretty often surprised by how good AWS premium support is. Every time I've asked them a question they seem to have gone to quite the lengths to gather information even when the issue isn't really theirs.
garciasn 4y ago
We had Rackspace and used them for our managed servers at a prior job. We had a machine with 18-24 cores and something like 768GB of RAM that handled all of our VMs, including our SAS server. It had a bunch of native SSDs and it was connected to a 20TB JBOD of SSDs and a 100TB JBOD of slower spinning disks. They recommended this setup to my predecessor as it was WAY less money than a similar SAN configuration. So. One day the two JBOD enclosures just start failing. We lose EVERYTHING. Ok, that sucks, but Rackspace handles all of our backups for us through our SLA. Let's get that restored while they figure out the JBOD issue. Two things come out of this: 1. Their 'fanatical support' were 'backing us up' but they never actually tested the solution and...we lost everything. There were no backups. 2. They sold us the JBODs less than 18 months prior but they no longer supported them and thus we'd have to invest in a SAN--which was a crazy investment more. --- We had already been planning a transfer to a new local data center over the course of a year's time. We called up the new company and told them we needed to move over to their shit in 7 days. They had us up in 2. I'll never consider Rackspace for anything.
- cutemonster 4y ago
  About which year did that happen? (The data lots etc) We were using Rackspace too, at a former workplace, for a while (no crash though).
  - garciasn 4y ago
    I'm guessing late 2015. We went to the interim provider while they did a slow planned migration to 100% GCP which was completed after I left in late 2018.
- mgbmtl 4y ago
  Yikes. A backup solution that was sold, but not configured (at all! not to mentioned non-tested backups), was a very frequent occurrence whenever I inherited clients (in the 2000-2010s, before AWS took over the world, or simple VM snapshots at most other hosts). (I'm not defending anyone here, I think it's how horrible many hosting companies where was back then, or still are, whenever they have sales teams that only aim to up-sell stuff and do not bother getting it properly implemented)
brightball 4y ago
I was with a company back in 2007-08 who was using Rackspace. I'll never forget the day I was sitting at a command prompt looking at htop trying to figure out what was straining the server so much...when I got a message in the terminal from Rackspace telling me exactly what was happening and what we needed to change to fix it. Support person just chatting with me right there in the terminal, without me even having to call for help. It was incredible and I'm really sad to see them struggling now. That turned me into a fan. When they picked up Slicehost, I was initially really happy about it but the benefits never really seemed to materialize. Then Digital Ocean appeared as essentially "the new Slicehost" not too much later and progress continued from there. Just never adapted as a viable cloud option. For a while there, before GCP and Azure were prominent a lot of conversations were "Amazon or Rackspace".
- rrdharan 4y ago
  That seems pretty sketchy from a data access perspective.
  - brightball 4y ago
    At the time, I thought it was pretty cool. IIRC part of the pitch and the cost was that they were literally supplementing your IT department with the same types of agreements you'd have for an outsourced IT provider.
  - debarshri 4y ago
    You will be surprised how much digitalocean has access in your cloud infra. We had a support ticket, the support person could see all our pods, namespace and kubernetes objects, disk mounts etc.
    - deleted 4y ago
      [deleted]
    - julianlam 4y ago
      How is that a surprise? They quite literally have physical access to your VPS (even if not easily). All the things you mentioned are simple metadata available via query. If things went really sideways on one of my servers, and I could not manage to get back in (even with their 3-4 alternative methods), I fully expect DigitalOcean to be able to log in and restore access, and charge me for the privilege.
swozey 4y ago
I used to be one of those phone answering sysadmins for a specific customer revenue. I think that job gave me phone anxiety. Every single different call was some new destroyed system by someone who had no idea what they were doing, hacked systems, unplanned migrations, whiteboarding-in-your-brain architecture discussions, etc. The other webhosts I worked at would never put an admin on the phone with a customer directly unless they were seriously enterprise, like a portion of the datacenter. One time the CTO of a major company called me directly and I had to migrate a SQL database that was the backend of their public website while having him breathe down my neck. Lots of calls like that. We gave great support but at cost. That was tiring and we did it in shifts. I hated the days I had to do that. FWIW for people with different support experiences, support was based on revenue. The different tiers (smb/enterprise, etc) assigned you more senior admins, etc. It wasn't a massive pool of admins or anything. I quit eventually because I hated answering the phones. Great company while I was there, though. I recommended it to everyone who could deal with the phone calls and San Antonio.
- quags 4y ago
  Thank you for your service.

stingraycharles 4y ago

I’ve had a similar pleasure in the early 2000s. It may seem absurd, but from a business perspective it makes sense: so many support questions fall into a gray area of “this is really not our business because the customer doesn’t know X, but without solving it they’ll be unhappy”. It makes sense to just say “fuck it, we’ll help them with anything, as long as they’re paying”. It’s a similar approach as we take at my current employer. If you’re using our database products, even if you’re stuck doing basic python stuff, don’t know how docker works, or an absolute noob at AWS, we’ll help you. The time we spend is cheaper than losing business, and enterprises love it.

bengale 4y ago
We do something similar on the non-tech side. We offer a saas product but most of our questions are more around how the person should be doing their job rather than how to use our software so we have a small team of experts that can answer those questions, they can also help with content creating on the marketing side too which is handy. We constantly hear from our customers that this level of service is one of the key things they really value from us.
- JALTU 4y ago
  +1 Build the customer relationship.

hotpotamus 4y ago

I probably know some of the guys who helped you out and there's a couple great ones still there. The legal team came up with the phrase "reasonable endeavor" (they didn't like best effort), and they wrote a disclaimer form they'd have customers fill out that what they were about to was beyond official support. As long as you signed that, support was usually pretty happy to "give it the old college try" as one told me. And I'll let you in on another little secret - a lot of those sysadmins were learning along with you. One of my very early projects was building out a precision time protocol service for a customer who asked for it - which ended up being a good learning experience for me.

bengale 4y ago

I'm pretty often surprised by how good AWS premium support is. Every time I've asked them a question they seem to have gone to quite the lengths to gather information even when the issue isn't really theirs.

garciasn 4y ago

We had Rackspace and used them for our managed servers at a prior job. We had a machine with 18-24 cores and something like 768GB of RAM that handled all of our VMs, including our SAS server. It had a bunch of native SSDs and it was connected to a 20TB JBOD of SSDs and a 100TB JBOD of slower spinning disks. They recommended this setup to my predecessor as it was WAY less money than a similar SAN configuration. So. One day the two JBOD enclosures just start failing. We lose EVERYTHING. Ok, that sucks, but Rackspace handles all of our backups for us through our SLA. Let's get that restored while they figure out the JBOD issue. Two things come out of this: 1. Their 'fanatical support' were 'backing us up' but they never actually tested the solution and...we lost everything. There were no backups. 2. They sold us the JBODs less than 18 months prior but they no longer supported them and thus we'd have to invest in a SAN--which was a crazy investment more. --- We had already been planning a transfer to a new local data center over the course of a year's time. We called up the new company and told them we needed to move over to their shit in 7 days. They had us up in 2. I'll never consider Rackspace for anything.

cutemonster 4y ago
About which year did that happen? (The data lots etc) We were using Rackspace too, at a former workplace, for a while (no crash though).
- garciasn 4y ago
  I'm guessing late 2015. We went to the interim provider while they did a slow planned migration to 100% GCP which was completed after I left in late 2018.
mgbmtl 4y ago
Yikes. A backup solution that was sold, but not configured (at all! not to mentioned non-tested backups), was a very frequent occurrence whenever I inherited clients (in the 2000-2010s, before AWS took over the world, or simple VM snapshots at most other hosts). (I'm not defending anyone here, I think it's how horrible many hosting companies where was back then, or still are, whenever they have sales teams that only aim to up-sell stuff and do not bother getting it properly implemented)

brightball 4y ago

I was with a company back in 2007-08 who was using Rackspace. I'll never forget the day I was sitting at a command prompt looking at htop trying to figure out what was straining the server so much...when I got a message in the terminal from Rackspace telling me exactly what was happening and what we needed to change to fix it. Support person just chatting with me right there in the terminal, without me even having to call for help. It was incredible and I'm really sad to see them struggling now. That turned me into a fan. When they picked up Slicehost, I was initially really happy about it but the benefits never really seemed to materialize. Then Digital Ocean appeared as essentially "the new Slicehost" not too much later and progress continued from there. Just never adapted as a viable cloud option. For a while there, before GCP and Azure were prominent a lot of conversations were "Amazon or Rackspace".

rrdharan 4y ago
That seems pretty sketchy from a data access perspective.
- brightball 4y ago
  At the time, I thought it was pretty cool. IIRC part of the pitch and the cost was that they were literally supplementing your IT department with the same types of agreements you'd have for an outsourced IT provider.
- debarshri 4y ago
  You will be surprised how much digitalocean has access in your cloud infra. We had a support ticket, the support person could see all our pods, namespace and kubernetes objects, disk mounts etc.
  - deleted 4y ago
    [deleted]
  - julianlam 4y ago
    How is that a surprise? They quite literally have physical access to your VPS (even if not easily). All the things you mentioned are simple metadata available via query. If things went really sideways on one of my servers, and I could not manage to get back in (even with their 3-4 alternative methods), I fully expect DigitalOcean to be able to log in and restore access, and charge me for the privilege.

swozey 4y ago

I used to be one of those phone answering sysadmins for a specific customer revenue. I think that job gave me phone anxiety. Every single different call was some new destroyed system by someone who had no idea what they were doing, hacked systems, unplanned migrations, whiteboarding-in-your-brain architecture discussions, etc. The other webhosts I worked at would never put an admin on the phone with a customer directly unless they were seriously enterprise, like a portion of the datacenter. One time the CTO of a major company called me directly and I had to migrate a SQL database that was the backend of their public website while having him breathe down my neck. Lots of calls like that. We gave great support but at cost. That was tiring and we did it in shifts. I hated the days I had to do that. FWIW for people with different support experiences, support was based on revenue. The different tiers (smb/enterprise, etc) assigned you more senior admins, etc. It wasn't a massive pool of admins or anything. I quit eventually because I hated answering the phones. Great company while I was there, though. I recommended it to everyone who could deal with the phone calls and San Antonio.

quags 4y ago
Thank you for your service.