6 ms·

Rely your web startup on Rackspace Cloud? Think again

89 points by edwincheese 16y ago

kjksf 16y ago
What is missing from this article (and comments so far) is a more comprehensive analysis of available options. If I lease a server from linode or AWS or theplanet or serverbeach or ${your favorite hosting provider}, would the situation be any different? I understand the article's author frustration with Rackspace, but it's a single data point hence hardly enough to be a basis for an intelligent choice of hosting provider. I'm not even sure if I sympathise with him. You can argue whether 1 hour notice before disabling a server is enough or not but there is an obvious conflict of interest. The interest of the person hosting server, who can potentially be a phisher himself, is for the site to stay up as long as possible. The interest of the public is served by terminating the server as quickly as possible.
- lsc 16y ago
  Your last two lines sum up the problem nicely. Everywhere, the more you pay, the more effort they will put in to helping you clean up your messes rather than shutting you down right away, which makes sense, because helping you clean up your mess is an expensive business to be in.
  - chopsueyar 16y ago
    Which doesn't explain Rackpace's behavior, considering their premium cost for "fanatical" support. Perhaps Rackspace cares more about it's shareholders than customers?
    - lsc 16y ago
      I could be mistaken, but I think that the rackspace cloud is their cut-rate product which comes with less support.
      - chopsueyar 16y ago
        http://www.rackspacecloud.com/cloud_hosting_products/servers/support http://www.rackspacecloud.com/cloud_hosting_products/servers... "We talk AND we listen. We're big fans of transparency and feedback. Technology works so much better combined with a bit of humanity."
- chpapa 16y ago
  So far I only know from one of my frd doing another startup, saying they experience something like this with Linode and were given 24 hrs to respond. This align with the respond I got from Linode, so I am actually considering to migrate from rackspace cloud to linode shortly.
oomkiller 16y ago
Good to know I wasn't wrong when I switched to Linode (From Rackspace). While phishing and spam is damaging, owners need more than 1-2 hours sometimes, especially when they are in a different part of the world.
lsc 16y ago
this will happen at any responsible web host. If you are hosting phishing sites, expect to get taken down. This trickles up. if you run a hosting company, and you get enough complaints that you don't deal with, then yeah, you will get shut down or asked to leave. That said, I think especially for higher-priced services, a phone call would be nice. (Note: I don't call my customers, though this is a policy I've considered implementing.) I'd be interested in what other people think about other notification systems.
- oomkiller 16y ago
  I don't think the issue is that he was hosting an active phishing site. The main issue here is the amount of time he was given to fix the problem was too small. You think Rackspace's upstreams would shut the pipes down if there were a bunch of phishing sites that set up shop? Doubtful. Usually they only get involved when there is a MASSIVE DDoS.
  - lsc 16y ago
    Well, from my experience in industry, big providers of bandwidth do apply pressure (up to and including the threat of disconnect) to large hosting companies in an effort to get them to clean up their act, even when it comes to things like spamming and phishing that are less outright destructive to the network than DDoS activity. You are right that a million dollar customer will get a lot more slack than a twenty dollar user... but, uh, to me that should be expected. Dealing with abuse is very expensive. Some places I've worked that has been the majority of support costs.
- ntoshev 16y ago
  What do you think about granularity of filtering: do not take all your customers' servers down, just filter the offending pages/services at network level until they sort it out.
  - lsc 16y ago
    again it's a matter of cost. If you only have one IP, and say, many domains on that one IP, only one of which is a phishing site, it is a /whole lot/ of work for me (who really can only mess with things at the network level) to selectively take out one domain or one url. I mean, it can be done, but the equipment and expertise required to pull it off would drive up my costs. Now, if I was a 'fully managed' provider that had a login on all my customer boxes, in that case, assuming the problem was a customer of a customer phishing rather than a compromised server, it would be fairly easy to log in and fix it. But that's not how I'm set up.
chopsueyar 16y ago
I second this. Not a fan of Rackspace. Some may argue having your own hardware is more expensive to maintain, but there is a definite advantage to controlling your physical hardware. Rackspace is helpful until you have a real problem, and you are left to fend for yourself.
fido 16y ago
Rackspace (and others) need to call their clients on the phone in these situations. They feel it is so important that it must be taken care of in one hour, yet they use email. Not exactly fair to the client....
- lsc 16y ago
  Isn't the rackspace cloud their cut-rate service? would they have called if it was a slicehost (premium price) customer? Still, I think it is a good idea, and one I ought to implement in my own service. The problem is that we all hate dealing with the phone, but that sounds to me like a lazy answer. On the other hand, really, if you are an abuse problem, honestly, I don't want you as a customer.
  - epochwolf 16y ago
    Slicehost doesn't have phone support last I checked.
    - deleted 16y ago
      [deleted]
mgkimsal 16y ago
What should happen here is this... When a server is 'shut down' for phishing or spam, a firewall blocks all incoming/outgoing except for traffic to/from a pre-determined IP address along with the notice that the server is being quarantined. Site owner/admin can then access the server, perform any investigations or deletions necessary, notify data center, then data center opens traffic again. Alternatively, something like a web-based shell allowing access, but all other traffic denied, would be acceptable. I've had servers shut down for 'abuse' which was one complaint from someone at 1am local time for me. I supposedly got a call from xxxxxx at 2 am, notifying me that action would be taken, and my server was taken offline at 3am. I wasn't awake until 7am, and couldn't get things resolved until about 10am. I was told I needed to 'rectify the situation', but how can I do that when access to the server is blocked? It's a ridiculous execution of policy, and only serves to heighten everyone's frustration. A private web-shell or single IP in the firewall to allow access would resolve most of the ill-feelings site owners caught in this situation have had.
- lsc 16y ago
  My policy is that I shut down the server, I create a fresh server for the user and then I attach the old disk read-only to that server, in case the user needs to retrieve data. The problem is that there is no way to 'clean up' after a break in without booting from trusted media. Otherwise, there is no way to know if you have closed all the backdoors the attacker left.
fookyong 16y ago
How was this construed as "phishing"? Did the customer simply set up a form that asks for a user's email address? because if so that describes tons of other services out there... e.g. Wufoo, Google Documents (but I guess they are not hosted on Rackspace!)
- rickmak 16y ago
  FYI, my phishing form @wufoo: http://rickmak.wufoo.com/forms/phishing/ http://rickmak.wufoo.com/forms/phishing/ Look professional? ;) Similar form on pandaform will make pandaform shutdown entirely.
  - fookyong 16y ago
    it looks like it got taken down. what was it?
  - chopsueyar 16y ago
    Well, we know wufoo doesn't use Rackspace.
varikin 16y ago
All this talk about how Rackspace should be treating startups make me think, "Why?" I understand startups may not have the personel to react as fast as a larger company with dedicated personel, and I understand that there might be a very large percentage of startups on Rackspace which might account for a nice chunk of revenue. But what makes startups different from my personal website or a larger corporation? If Rackspace receives a complaint about a phishing site hosted on their servers, they should do what they can to correct that, regardless of which client is using the server that has the phishing attack. The startup should get the same treatment from Rackspace as the large corporation and the guy with some simple homepage.
- edwincheese 16y ago
  It is like asking why we need special facilities in building for the disables. Why didn't we treat everybody the same and don't provide any special facilities for disable? I do think that site owner of any scale IS responsible to abuse complains, but the scale of the company do make a different. The influence and harmfulness of a fake shop in Amazon is not same as a fake shop in a very-small-online-shop. Thus people expect Amazon to take action instantly. A start-up is not possibile to react as fast as large company, so why shouldn't we give them a reasonable time for them to do their job?
  - wrs 16y ago
    The people being phished don't care how big you are or how fast you can respond. What would they consider a reasonable time?
    - chopsueyar 16y ago
      If the people being phished know they are being phished, then why are they allowing this?
  - shadowsun7 16y ago
    So you're comparing your startup to a disabled person? Unlike the hypothetical disabled person, however, your startup, when used for phishing, is allowing real harm to happen to other people. Your attitude suggests that a) you care less about your clients than about what's convenient for you as a startup founder, because b) you're small. But that's a very bad excuse, and a dumb one, at that. Clients don't care about your size. All they care about is the results your products give them. And that leads to another question: why should anyone buy into your services if you don't/can't act promptly and responsibly in response to problems? And yet you seem to be getting angry about Rackspace for caring about the clients affected by such criminal activity further downstream from them. I'd think twice before buying into your products, with an attitude like that. Rackspace appears to care. You do not.
cedsav 16y ago
Welcome to the joys and sorrows of hosting user-generated content. If this makes you feel better,rackspace is just your first problem. You'll also find soon enough that abuses can get your domain blacklisted in chrome, firefox, and for opendns users.
matrix 16y ago
I'm thinking Rackspace might well have been in the right on this one. If the customer was in fact phishing, Rackspace was well within their rights to shut down the account. It's really up to the application creator to prevent that abuse. That said, it's good to have a reminder of the risks of outsourcing your hosting. I still think the tradeoff is worth it for experimental products where you don't want to invest too much upfront.
- nl 16y ago
  I agree. Hosting providers usually reserve the right to shutdown your server if it has been hacked, and phishing is often more directly harmful to people.
  - watchandwait 16y ago
    Agree. This wasn't a simple DMCA issue-- phishing is an active, criminal activity. Even one hour notice is generous.
    - A1kmm 16y ago
      Taking the contents of the article on faith, I think the point was that their customers were abusing the startup's service for phishing. This would be analogous to, for example, AWS taking reddit offline because a user posted a phising link. Nearly every web business which lets people put information on the web that others can see will face abuse issues at some point or the other - and cutting off a business and its legitimate customers because of one client misusing the service does not inspire confidence.
  - royuen 16y ago
    The problem here is Rackspace as a infrastructure provider judging on behalf of service provider. They give no explanation of the complaint details or why it is justified. Many comments here take the phishing as "a fact", their terms might grant them the power to shut any server down but this is a threat I think every startup should learn if they use Rackspace Cloud or consider to. And we learnt that. 24 hrs is not just for respond to remove the content, it is also for the server providers to verify the complaint and react responsibly.
malbiniak 16y ago
This may sound contrived and sarcastic, but I assure you it is not. Try your best to get Robert Scoble's attention on this. Rackspace has tasked him with being an evangelist primarily focused on startups and bringing them into the Rackspace services. It's my understanding that things like this should be his primary concern. I could be mistaken, but I'm used to that. http://twitter.com/malbiniak/status/22672201225 http://twitter.com/malbiniak/status/22672201225
- jon_dahl 16y ago
  Your middle paragraph is reasonable and a good idea. What's with the first and third?
  - malbiniak 16y ago
    first - didn't want to come off as "complain and your problem will be fixed." i never anticipated having it called reasonable. third - second paragraph is my interpretation of a conversation i had with some people from rackspace and it could be wrong. also, hello from minneapolis.
royuen 16y ago
@matrix, I think overall direction is correct, Rackspace should try their best to notify their customer if they received complaints. However the ridiculous point of this case is totally shutting down all servers of a startup in just 1 hour, which is really horrible. Instead I believe they should at least give 12 - 24 hours for a startup to react and investigate before taking down all their services? Is that really on the same direction that they want to serve startups?
- damncabbage 16y ago
  > they should at least give 12 - 24 hours for > a startup to react and investigate By which time the phisher has already done the damage and moved on.
mcknz 16y ago
To be fair, a service the size of Rackspace Cloud is bound to screw up at least a few times. Then again, no one ever got fired for switching to Linode.
- StavrosK 16y ago
  First of all, what is up with Linode's London datacenter? I barely press enter on commands and they've already been carried out. Seriously, it installs packages before the text from the server reaches me, and I get 15 ms latency on it (the SSH echo feels faster than my local PC). Is it just me, or are their other datacenters slower than their London one? It might be the latency, as I was actually in London, but the one I have in Georgia doesn't seem to have that fast disk accesses... Bottom line, though, the London datacenter is just time-traveling fast.
credo 16y ago
imo the biggest mistake here was in PandaForm trusting all of its user-generated content. It seems like they should have been doing some sort of verification of the UGC. Alternately, if they really wanted to go with a fully laissez-faire no-security-checks approach, they shouldn't have picked a hosting company (in this case, Rackspace) that requires customers to sign an agreement that has strict anti-phishing rules.
ohashi 16y ago
I am really not a fan of all the defending RackSpace here. They pulled the plug unreasonably without proper notification. If you're going to pull the plug or even thinking about it... email simply isn't going to cut it. You need someone to call the owner and make contact to explain what's going on or how to resolve it. I've had my servers compromised, I've had phishing content setup before, I have never had the plug pulled. I've had hosts contact me, give me appropriate amounts of time to handle it and some of them even offered to help secure my box or look into how it got compromised in the first place.
goosmurf 16y ago
Post is lacking information on whether the site was actually a phishing site and who (or which entity) submitted a complaint. I think verification of the legitimacy of a complaint should be a critical step before disabling a site, otherwise you're prone to DoS. It would also be good to know what steps the complainant took. Did s/he try to contact Pandaform, or immediately go to Rackspace as the owner of the IP? Without knowing whether the complaint was legitimate, and what steps Rackspace took to verify this (or not) its tough to say whether their actions were appropriate.
- chopsueyar 16y ago
  Their actions are inappropriate. The majority of commentors here assume guilty until proven innocent. In fact, the author of the article has still been unable to identify the "phishing" form. Was there even really a phishing form? With all this anti-phishing technology built into modern browsers, why is it Rackspace's responsibility to "protect" users from "alleged" phishing sites? For copyright, we have the DMCA (for better or worse). Perhaps we need some similar sort of due process for hosting providers or cloud providers. Most appalling is Rackspace's lack of transparency in handling this. Pandaform was never contacted by the complainant. The complainant only contacted Rackspace. Rackspace assumed, without investigation, the complaint was legitimate and gave the guy notice via email. Also, the real rub, is the fact the Rackspace would terminate his account if he got a second complaint. So, it is not necessary to have an actual phishing form on his site to have his Rackspace hosting terminated. Someone simply needs to allege this to Rackspace, and his account and data will be gone forever. That is fanatical? Again, I cannot recommend Rackspace.
- epochwolf 16y ago
  I'm also wondering if this person has a history of complaints.
Aegean 16y ago
We had a similar problem with another vps provider. Some php script was remotely exploited and sending spam. They disabled the VPS. Our website and email was down for 24 hours. They enabled the VPS the next day for us to look at it. What's worse I was asked to reinstall the VPS and it took me some convincing for them to enable it so I get my data from the server. Lesson learned? I think you should never have any operations where you don't have full control. This still holds true but it was a bit ironic because to have full control I ran everything on the VPS, effectively transferring full control to the VPS provider! It was the single point of failure. Now I will transfer each different service to a different provider. Email to rackspace, websites to a host, git services to github and so on.
lmz 16y ago
I wonder if this abuse monitoring service can be outsourced somehow. Pay a team of people around the world to monitor the abuse email address and hand them the kill switch for so they can respond immediately to take down one account only instead of the hosting provider taking down the whole server.
PonyGumbo 16y ago
Oy. This comes the day after I moved all of my sites to Rackspace Cloud.
- pinksoda 16y ago
  Oh no, get as far away as you can! Check out Linode or SoftLayer.
paolomaffei 16y ago
Linode isn't really any better. I sent an email campaign to an optin list from campaignmonitor - someone (1 person out of 1000) reported spam. Campaignmonitor asked us what was going on - while this is debatable they may have been damaged from the complaing But even linode asked us about it, just because the website where people signed up was on a linode hosting. 1 complain, nothing to do with hosting a part that the link is the here, and they asked us if there was something wrong. I wonder if you can just blast a spam email with a competitor website (hosted on linode) inside and get him offline...

89 points by edwincheese 16y ago

kjksf 16y ago

What is missing from this article (and comments so far) is a more comprehensive analysis of available options. If I lease a server from linode or AWS or theplanet or serverbeach or ${your favorite hosting provider}, would the situation be any different? I understand the article's author frustration with Rackspace, but it's a single data point hence hardly enough to be a basis for an intelligent choice of hosting provider. I'm not even sure if I sympathise with him. You can argue whether 1 hour notice before disabling a server is enough or not but there is an obvious conflict of interest. The interest of the person hosting server, who can potentially be a phisher himself, is for the site to stay up as long as possible. The interest of the public is served by terminating the server as quickly as possible.

lsc 16y ago
Your last two lines sum up the problem nicely. Everywhere, the more you pay, the more effort they will put in to helping you clean up your messes rather than shutting you down right away, which makes sense, because helping you clean up your mess is an expensive business to be in.
- chopsueyar 16y ago
  Which doesn't explain Rackpace's behavior, considering their premium cost for "fanatical" support. Perhaps Rackspace cares more about it's shareholders than customers?
  - lsc 16y ago
    I could be mistaken, but I think that the rackspace cloud is their cut-rate product which comes with less support.
    - chopsueyar 16y ago
      http://www.rackspacecloud.com/cloud_hosting_products/servers/support http://www.rackspacecloud.com/cloud_hosting_products/servers... "We talk AND we listen. We're big fans of transparency and feedback. Technology works so much better combined with a bit of humanity."
chpapa 16y ago
So far I only know from one of my frd doing another startup, saying they experience something like this with Linode and were given 24 hrs to respond. This align with the respond I got from Linode, so I am actually considering to migrate from rackspace cloud to linode shortly.

oomkiller 16y ago

Good to know I wasn't wrong when I switched to Linode (From Rackspace). While phishing and spam is damaging, owners need more than 1-2 hours sometimes, especially when they are in a different part of the world.

lsc 16y ago

this will happen at any responsible web host. If you are hosting phishing sites, expect to get taken down. This trickles up. if you run a hosting company, and you get enough complaints that you don't deal with, then yeah, you will get shut down or asked to leave. That said, I think especially for higher-priced services, a phone call would be nice. (Note: I don't call my customers, though this is a policy I've considered implementing.) I'd be interested in what other people think about other notification systems.

oomkiller 16y ago
I don't think the issue is that he was hosting an active phishing site. The main issue here is the amount of time he was given to fix the problem was too small. You think Rackspace's upstreams would shut the pipes down if there were a bunch of phishing sites that set up shop? Doubtful. Usually they only get involved when there is a MASSIVE DDoS.
- lsc 16y ago
  Well, from my experience in industry, big providers of bandwidth do apply pressure (up to and including the threat of disconnect) to large hosting companies in an effort to get them to clean up their act, even when it comes to things like spamming and phishing that are less outright destructive to the network than DDoS activity. You are right that a million dollar customer will get a lot more slack than a twenty dollar user... but, uh, to me that should be expected. Dealing with abuse is very expensive. Some places I've worked that has been the majority of support costs.
ntoshev 16y ago
What do you think about granularity of filtering: do not take all your customers' servers down, just filter the offending pages/services at network level until they sort it out.
- lsc 16y ago
  again it's a matter of cost. If you only have one IP, and say, many domains on that one IP, only one of which is a phishing site, it is a /whole lot/ of work for me (who really can only mess with things at the network level) to selectively take out one domain or one url. I mean, it can be done, but the equipment and expertise required to pull it off would drive up my costs. Now, if I was a 'fully managed' provider that had a login on all my customer boxes, in that case, assuming the problem was a customer of a customer phishing rather than a compromised server, it would be fairly easy to log in and fix it. But that's not how I'm set up.

chopsueyar 16y ago

I second this. Not a fan of Rackspace. Some may argue having your own hardware is more expensive to maintain, but there is a definite advantage to controlling your physical hardware. Rackspace is helpful until you have a real problem, and you are left to fend for yourself.

fido 16y ago

Rackspace (and others) need to call their clients on the phone in these situations. They feel it is so important that it must be taken care of in one hour, yet they use email. Not exactly fair to the client....

lsc 16y ago
Isn't the rackspace cloud their cut-rate service? would they have called if it was a slicehost (premium price) customer? Still, I think it is a good idea, and one I ought to implement in my own service. The problem is that we all hate dealing with the phone, but that sounds to me like a lazy answer. On the other hand, really, if you are an abuse problem, honestly, I don't want you as a customer.
- epochwolf 16y ago
  Slicehost doesn't have phone support last I checked.
  - deleted 16y ago
    [deleted]

mgkimsal 16y ago

What should happen here is this... When a server is 'shut down' for phishing or spam, a firewall blocks all incoming/outgoing except for traffic to/from a pre-determined IP address along with the notice that the server is being quarantined. Site owner/admin can then access the server, perform any investigations or deletions necessary, notify data center, then data center opens traffic again. Alternatively, something like a web-based shell allowing access, but all other traffic denied, would be acceptable. I've had servers shut down for 'abuse' which was one complaint from someone at 1am local time for me. I supposedly got a call from xxxxxx at 2 am, notifying me that action would be taken, and my server was taken offline at 3am. I wasn't awake until 7am, and couldn't get things resolved until about 10am. I was told I needed to 'rectify the situation', but how can I do that when access to the server is blocked? It's a ridiculous execution of policy, and only serves to heighten everyone's frustration. A private web-shell or single IP in the firewall to allow access would resolve most of the ill-feelings site owners caught in this situation have had.

lsc 16y ago
My policy is that I shut down the server, I create a fresh server for the user and then I attach the old disk read-only to that server, in case the user needs to retrieve data. The problem is that there is no way to 'clean up' after a break in without booting from trusted media. Otherwise, there is no way to know if you have closed all the backdoors the attacker left.

fookyong 16y ago

How was this construed as "phishing"? Did the customer simply set up a form that asks for a user's email address? because if so that describes tons of other services out there... e.g. Wufoo, Google Documents (but I guess they are not hosted on Rackspace!)

rickmak 16y ago
FYI, my phishing form @wufoo: http://rickmak.wufoo.com/forms/phishing/ http://rickmak.wufoo.com/forms/phishing/ Look professional? ;) Similar form on pandaform will make pandaform shutdown entirely.
- fookyong 16y ago
  it looks like it got taken down. what was it?
- chopsueyar 16y ago
  Well, we know wufoo doesn't use Rackspace.

varikin 16y ago

All this talk about how Rackspace should be treating startups make me think, "Why?" I understand startups may not have the personel to react as fast as a larger company with dedicated personel, and I understand that there might be a very large percentage of startups on Rackspace which might account for a nice chunk of revenue. But what makes startups different from my personal website or a larger corporation? If Rackspace receives a complaint about a phishing site hosted on their servers, they should do what they can to correct that, regardless of which client is using the server that has the phishing attack. The startup should get the same treatment from Rackspace as the large corporation and the guy with some simple homepage.

edwincheese 16y ago
It is like asking why we need special facilities in building for the disables. Why didn't we treat everybody the same and don't provide any special facilities for disable? I do think that site owner of any scale IS responsible to abuse complains, but the scale of the company do make a different. The influence and harmfulness of a fake shop in Amazon is not same as a fake shop in a very-small-online-shop. Thus people expect Amazon to take action instantly. A start-up is not possibile to react as fast as large company, so why shouldn't we give them a reasonable time for them to do their job?
- wrs 16y ago
  The people being phished don't care how big you are or how fast you can respond. What would they consider a reasonable time?
  - chopsueyar 16y ago
    If the people being phished know they are being phished, then why are they allowing this?
- shadowsun7 16y ago
  So you're comparing your startup to a disabled person? Unlike the hypothetical disabled person, however, your startup, when used for phishing, is allowing real harm to happen to other people. Your attitude suggests that a) you care less about your clients than about what's convenient for you as a startup founder, because b) you're small. But that's a very bad excuse, and a dumb one, at that. Clients don't care about your size. All they care about is the results your products give them. And that leads to another question: why should anyone buy into your services if you don't/can't act promptly and responsibly in response to problems? And yet you seem to be getting angry about Rackspace for caring about the clients affected by such criminal activity further downstream from them. I'd think twice before buying into your products, with an attitude like that. Rackspace appears to care. You do not.

cedsav 16y ago

Welcome to the joys and sorrows of hosting user-generated content. If this makes you feel better,rackspace is just your first problem. You'll also find soon enough that abuses can get your domain blacklisted in chrome, firefox, and for opendns users.

matrix 16y ago

I'm thinking Rackspace might well have been in the right on this one. If the customer was in fact phishing, Rackspace was well within their rights to shut down the account. It's really up to the application creator to prevent that abuse. That said, it's good to have a reminder of the risks of outsourcing your hosting. I still think the tradeoff is worth it for experimental products where you don't want to invest too much upfront.

nl 16y ago
I agree. Hosting providers usually reserve the right to shutdown your server if it has been hacked, and phishing is often more directly harmful to people.
- watchandwait 16y ago
  Agree. This wasn't a simple DMCA issue-- phishing is an active, criminal activity. Even one hour notice is generous.
  - A1kmm 16y ago
    Taking the contents of the article on faith, I think the point was that their customers were abusing the startup's service for phishing. This would be analogous to, for example, AWS taking reddit offline because a user posted a phising link. Nearly every web business which lets people put information on the web that others can see will face abuse issues at some point or the other - and cutting off a business and its legitimate customers because of one client misusing the service does not inspire confidence.
- royuen 16y ago
  The problem here is Rackspace as a infrastructure provider judging on behalf of service provider. They give no explanation of the complaint details or why it is justified. Many comments here take the phishing as "a fact", their terms might grant them the power to shut any server down but this is a threat I think every startup should learn if they use Rackspace Cloud or consider to. And we learnt that. 24 hrs is not just for respond to remove the content, it is also for the server providers to verify the complaint and react responsibly.

malbiniak 16y ago

This may sound contrived and sarcastic, but I assure you it is not. Try your best to get Robert Scoble's attention on this. Rackspace has tasked him with being an evangelist primarily focused on startups and bringing them into the Rackspace services. It's my understanding that things like this should be his primary concern. I could be mistaken, but I'm used to that. http://twitter.com/malbiniak/status/22672201225 http://twitter.com/malbiniak/status/22672201225

jon_dahl 16y ago
Your middle paragraph is reasonable and a good idea. What's with the first and third?
- malbiniak 16y ago
  first - didn't want to come off as "complain and your problem will be fixed." i never anticipated having it called reasonable. third - second paragraph is my interpretation of a conversation i had with some people from rackspace and it could be wrong. also, hello from minneapolis.

royuen 16y ago

@matrix, I think overall direction is correct, Rackspace should try their best to notify their customer if they received complaints. However the ridiculous point of this case is totally shutting down all servers of a startup in just 1 hour, which is really horrible. Instead I believe they should at least give 12 - 24 hours for a startup to react and investigate before taking down all their services? Is that really on the same direction that they want to serve startups?

damncabbage 16y ago
> they should at least give 12 - 24 hours for > a startup to react and investigate By which time the phisher has already done the damage and moved on.

mcknz 16y ago

To be fair, a service the size of Rackspace Cloud is bound to screw up at least a few times. Then again, no one ever got fired for switching to Linode.

StavrosK 16y ago
First of all, what is up with Linode's London datacenter? I barely press enter on commands and they've already been carried out. Seriously, it installs packages before the text from the server reaches me, and I get 15 ms latency on it (the SSH echo feels faster than my local PC). Is it just me, or are their other datacenters slower than their London one? It might be the latency, as I was actually in London, but the one I have in Georgia doesn't seem to have that fast disk accesses... Bottom line, though, the London datacenter is just time-traveling fast.

credo 16y ago

imo the biggest mistake here was in PandaForm trusting all of its user-generated content. It seems like they should have been doing some sort of verification of the UGC. Alternately, if they really wanted to go with a fully laissez-faire no-security-checks approach, they shouldn't have picked a hosting company (in this case, Rackspace) that requires customers to sign an agreement that has strict anti-phishing rules.

ohashi 16y ago

I am really not a fan of all the defending RackSpace here. They pulled the plug unreasonably without proper notification. If you're going to pull the plug or even thinking about it... email simply isn't going to cut it. You need someone to call the owner and make contact to explain what's going on or how to resolve it. I've had my servers compromised, I've had phishing content setup before, I have never had the plug pulled. I've had hosts contact me, give me appropriate amounts of time to handle it and some of them even offered to help secure my box or look into how it got compromised in the first place.

goosmurf 16y ago

Post is lacking information on whether the site was actually a phishing site and who (or which entity) submitted a complaint. I think verification of the legitimacy of a complaint should be a critical step before disabling a site, otherwise you're prone to DoS. It would also be good to know what steps the complainant took. Did s/he try to contact Pandaform, or immediately go to Rackspace as the owner of the IP? Without knowing whether the complaint was legitimate, and what steps Rackspace took to verify this (or not) its tough to say whether their actions were appropriate.

chopsueyar 16y ago
Their actions are inappropriate. The majority of commentors here assume guilty until proven innocent. In fact, the author of the article has still been unable to identify the "phishing" form. Was there even really a phishing form? With all this anti-phishing technology built into modern browsers, why is it Rackspace's responsibility to "protect" users from "alleged" phishing sites? For copyright, we have the DMCA (for better or worse). Perhaps we need some similar sort of due process for hosting providers or cloud providers. Most appalling is Rackspace's lack of transparency in handling this. Pandaform was never contacted by the complainant. The complainant only contacted Rackspace. Rackspace assumed, without investigation, the complaint was legitimate and gave the guy notice via email. Also, the real rub, is the fact the Rackspace would terminate his account if he got a second complaint. So, it is not necessary to have an actual phishing form on his site to have his Rackspace hosting terminated. Someone simply needs to allege this to Rackspace, and his account and data will be gone forever. That is fanatical? Again, I cannot recommend Rackspace.
epochwolf 16y ago
I'm also wondering if this person has a history of complaints.

Aegean 16y ago

We had a similar problem with another vps provider. Some php script was remotely exploited and sending spam. They disabled the VPS. Our website and email was down for 24 hours. They enabled the VPS the next day for us to look at it. What's worse I was asked to reinstall the VPS and it took me some convincing for them to enable it so I get my data from the server. Lesson learned? I think you should never have any operations where you don't have full control. This still holds true but it was a bit ironic because to have full control I ran everything on the VPS, effectively transferring full control to the VPS provider! It was the single point of failure. Now I will transfer each different service to a different provider. Email to rackspace, websites to a host, git services to github and so on.

lmz 16y ago

I wonder if this abuse monitoring service can be outsourced somehow. Pay a team of people around the world to monitor the abuse email address and hand them the kill switch for so they can respond immediately to take down one account only instead of the hosting provider taking down the whole server.

PonyGumbo 16y ago

Oy. This comes the day after I moved all of my sites to Rackspace Cloud.

pinksoda 16y ago
Oh no, get as far away as you can! Check out Linode or SoftLayer.

paolomaffei 16y ago

Linode isn't really any better. I sent an email campaign to an optin list from campaignmonitor - someone (1 person out of 1000) reported spam. Campaignmonitor asked us what was going on - while this is debatable they may have been damaged from the complaing But even linode asked us about it, just because the website where people signed up was on a linode hosting. 1 complain, nothing to do with hosting a part that the link is the here, and they asked us if there was something wrong. I wonder if you can just blast a spam email with a competitor website (hosted on linode) inside and get him offline...