5 ms·

Dear Pair Networks, I'd like to highlight this lesson in how to loose (or not gain) customers by randomly shutting down technology sites that serve the decisi

by sh1mmer 17y ago

Dear Pair Networks, I'd like to highlight this lesson in how to loose (or not gain) customers by randomly shutting down technology sites that serve the decision makers you wish to influence. Tom P.S. Well done Slicehost.

gojomo 17y ago
150K hits in 30 minutes is 83 hits/second. That's a lot to ask for from a shared-hosting account. When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual. The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?) The Pair message suggests end-of-day logs will be the way to tell for sure.
- sh1mmer 17y ago
  I was being tongue-in-cheek, but there is a real point here. As you say I doubt today was unusual for HN, it's the way Pair went from zero to shut-down. One of the things I like about the way that Joyent operate as a cloud host is that they allow you burst on shared boxes because of those times you need it. At the same time they'll let you know you need to think about buying more resources without just slamming on the brakes. Pair should do a much better job of noticing a soft limit earlier, so if a heavy traffic day had hit HN PG et al should have been already aware they were overusing their shared hosting and were planning a route out.
  - bwb 17y ago
    That technology is not out there... A static site might be fun at that level and not causing problems, where as a WordPress site with super cache is fine, but then a Drupal site can only handle 1/4th of that. It currently is not really possible to track that per user in a shared environment and see who is using what resources from an individual perspective of MySQL / Apache / CPU / Mem.
    - moe 17y ago
      That technology is not out there Ofcourse it is. Or how do you think they determined that it was pg's site causing the trouble?
    - InclinedPlane 17y ago
      Absolutely wrong. That technology exists and is being used millions of times a day at sites all over the world. DreamHost, for example, employs just such a technology for their shared hosting: http://wiki.dreamhost.com/index.php/CPU_minutes http://wiki.dreamhost.com/index.php/CPU_minutes
  - mixmax 17y ago
    Exactly - they missed an obvious opportunity to upsell. If Trevor had gotten a mail stating that he needed to upgrade he would most probably just have done so and been happy. It's not like he can't afford it :-)
- deleted 17y ago
  [deleted]
- neurotech1 17y ago
  Yes and no. I doubt it is a HN-Focused tool, as this affected the static content from www.ycombinator.com, and images and CSS are not the focus of bots usually. Assuming it was not a DoS attack, a smart host should have noticed the traffic load increasing over time, and offered to upgrade to a less-loaded server and recommend a dedicated server. By disabling the site, they have lost a customer, and lost on a up-sell to a dedicated server.
  - gojomo 17y ago
    You're assuming the load today wasn't extraordinary. That's not yet clear. PG doesn't know of any surge, probably based on figures he has access to on other hosts and analytics tools. But what if HN/YC static resources at Pair were newly being deep-linked, by a much more popular site? Only seeing the end-of-day logs from Pair, compared to previous days, can definitively answer this question.
    - zck 17y ago
      tlb says it was "gradual": http://news.ycombinator.com/item?id=1283439 http://news.ycombinator.com/item?id=1283439 .
      - gojomo 17y ago
        I'm sure that's what the general News.YC/YC stats show. But as the Pair.com tech mentioned, today's logs only become available at midnight ET. If there was a surge this morning -- to the static resources only, as if by offsite deep-linking -- Pair's logs are the only place it would be evident.
- BrandonM 17y ago
  So let's see... you're hosting a website that has gotten large; that is, it's grown to the point that it will need higher-cost services in order to meet demand. You have a chance to add a valuable customer to your client base. How best to handle this? a) anything b) except c) killing their service
  - bigboote 17y ago
    Exactly
  - ubernostrum 17y ago
    How best to handle this? Of course, the flip side is that leaving it running adopts an attitude of "screw all our other customers, they can eat crappy service while we kiss up to the popular guys who are chewing up everybody else's server resources". Which isn't what I'd look for in a hosting provider...
    - pmjordan 17y ago
      False dichotomy. The correct way to handle this would have been to temporarily move the shared server to hardware where it won't impact other customers and notify YC that they need to move their server to a bigger server or the site will have to be shut down. Presumably with an ultimatum of a week or whatever.
      - TallGuyShort 17y ago
        Agreed - the whole reason you pay a host is that there's some level of management and responsibility there. You're not just renting hardware.
      - BrandonM 17y ago
        This is exactly the kind of approach that a smart company which cares about business would take.
  - jacquesm 17y ago
    Companies like slicehost make their money on the customers that do not use their servers.
- bigboote 17y ago
  Todays's logs are similar to other days. I guess the traffic is high for a shared account, but I've never noticed it being slow so I didn't think about it. I would have been happy to respond to an email saying I should upgrade & pay more, but even after a few emails with tech support that option didn't come up. The only limit they advertise for that class of account is data transfer, and because it's mostly just serving uparrow.gif -type files we're well below that limit.
- InclinedPlane 17y ago
  The "ask" is not that a shared hosting account support 83 hits/second. It's that the hosting company demonstrate the bare minimum of respect and professionalism in contacting the hostee ASAP when their site is being shut down due to excess traffic. This isn't bitbucket or geocities, dollars are changing hands. If you don't like running a business that makes money then by all means, treat your paying customers like they are a burden to you, they will get the hint and go elsewhere.
axod 17y ago
FWIW, It's probably not worth their effort of working out exactly what websites are using cheap shared hosting and overusing resources. I don't get shared hosting at all. VPSs are dirt cheap these days.
- CGamesPlay 17y ago
  VPSes are still shared hosting and are not immune from the problem.
  - comex 17y ago
    But they are trivial to migrate between machines on demand.
  - pavs 17y ago
    They won't shut you down for over-using resources. As a matter of fact you can't over-use resources, you can only use the resources you are given to you. I know they are different type of VPS, I am referring to Slicehost/Linode type VPS which has hard limit and acts very much like dedicated single box. More important to me is that I am really really really surprised that Ycombinator was running on a shared account. This just blew my mind. I have no hard feelings towards pear, I would have shut the site down too; possibly long ago.
    - deleted 17y ago
      [deleted]
    - chrischen 17y ago
      They limit memory at Slicehost but not CPU. I think you can still burst CPU for too long and they'd eventually tell you to upgrade.
noonespecial 17y ago
Customers nothing. "I didn't know what that 'hacker news' thing was or if it should have that much traffic so I shut it off". Really? Please turn in your geek card at the door.

by sh1mmer 17y ago

gojomo 17y ago

150K hits in 30 minutes is 83 hits/second. That's a lot to ask for from a shared-hosting account. When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual. The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?) The Pair message suggests end-of-day logs will be the way to tell for sure.

sh1mmer 17y ago
I was being tongue-in-cheek, but there is a real point here. As you say I doubt today was unusual for HN, it's the way Pair went from zero to shut-down. One of the things I like about the way that Joyent operate as a cloud host is that they allow you burst on shared boxes because of those times you need it. At the same time they'll let you know you need to think about buying more resources without just slamming on the brakes. Pair should do a much better job of noticing a soft limit earlier, so if a heavy traffic day had hit HN PG et al should have been already aware they were overusing their shared hosting and were planning a route out.
- bwb 17y ago
  That technology is not out there... A static site might be fun at that level and not causing problems, where as a WordPress site with super cache is fine, but then a Drupal site can only handle 1/4th of that. It currently is not really possible to track that per user in a shared environment and see who is using what resources from an individual perspective of MySQL / Apache / CPU / Mem.
  - moe 17y ago
    That technology is not out there Ofcourse it is. Or how do you think they determined that it was pg's site causing the trouble?
  - InclinedPlane 17y ago
    Absolutely wrong. That technology exists and is being used millions of times a day at sites all over the world. DreamHost, for example, employs just such a technology for their shared hosting: http://wiki.dreamhost.com/index.php/CPU_minutes http://wiki.dreamhost.com/index.php/CPU_minutes
- mixmax 17y ago
  Exactly - they missed an obvious opportunity to upsell. If Trevor had gotten a mail stating that he needed to upgrade he would most probably just have done so and been happy. It's not like he can't afford it :-)
deleted 17y ago
[deleted]
neurotech1 17y ago
Yes and no. I doubt it is a HN-Focused tool, as this affected the static content from www.ycombinator.com, and images and CSS are not the focus of bots usually. Assuming it was not a DoS attack, a smart host should have noticed the traffic load increasing over time, and offered to upgrade to a less-loaded server and recommend a dedicated server. By disabling the site, they have lost a customer, and lost on a up-sell to a dedicated server.
- gojomo 17y ago
  You're assuming the load today wasn't extraordinary. That's not yet clear. PG doesn't know of any surge, probably based on figures he has access to on other hosts and analytics tools. But what if HN/YC static resources at Pair were newly being deep-linked, by a much more popular site? Only seeing the end-of-day logs from Pair, compared to previous days, can definitively answer this question.
  - zck 17y ago
    tlb says it was "gradual": http://news.ycombinator.com/item?id=1283439 http://news.ycombinator.com/item?id=1283439 .
    - gojomo 17y ago
      I'm sure that's what the general News.YC/YC stats show. But as the Pair.com tech mentioned, today's logs only become available at midnight ET. If there was a surge this morning -- to the static resources only, as if by offsite deep-linking -- Pair's logs are the only place it would be evident.
BrandonM 17y ago
So let's see... you're hosting a website that has gotten large; that is, it's grown to the point that it will need higher-cost services in order to meet demand. You have a chance to add a valuable customer to your client base. How best to handle this? a) anything b) except c) killing their service
- bigboote 17y ago
  Exactly
- ubernostrum 17y ago
  How best to handle this? Of course, the flip side is that leaving it running adopts an attitude of "screw all our other customers, they can eat crappy service while we kiss up to the popular guys who are chewing up everybody else's server resources". Which isn't what I'd look for in a hosting provider...
  - pmjordan 17y ago
    False dichotomy. The correct way to handle this would have been to temporarily move the shared server to hardware where it won't impact other customers and notify YC that they need to move their server to a bigger server or the site will have to be shut down. Presumably with an ultimatum of a week or whatever.
    - TallGuyShort 17y ago
      Agreed - the whole reason you pay a host is that there's some level of management and responsibility there. You're not just renting hardware.
    - BrandonM 17y ago
      This is exactly the kind of approach that a smart company which cares about business would take.
- jacquesm 17y ago
  Companies like slicehost make their money on the customers that do not use their servers.
bigboote 17y ago
Todays's logs are similar to other days. I guess the traffic is high for a shared account, but I've never noticed it being slow so I didn't think about it. I would have been happy to respond to an email saying I should upgrade & pay more, but even after a few emails with tech support that option didn't come up. The only limit they advertise for that class of account is data transfer, and because it's mostly just serving uparrow.gif -type files we're well below that limit.
InclinedPlane 17y ago
The "ask" is not that a shared hosting account support 83 hits/second. It's that the hosting company demonstrate the bare minimum of respect and professionalism in contacting the hostee ASAP when their site is being shut down due to excess traffic. This isn't bitbucket or geocities, dollars are changing hands. If you don't like running a business that makes money then by all means, treat your paying customers like they are a burden to you, they will get the hint and go elsewhere.

axod 17y ago

FWIW, It's probably not worth their effort of working out exactly what websites are using cheap shared hosting and overusing resources. I don't get shared hosting at all. VPSs are dirt cheap these days.

CGamesPlay 17y ago
VPSes are still shared hosting and are not immune from the problem.
- comex 17y ago
  But they are trivial to migrate between machines on demand.
- pavs 17y ago
  They won't shut you down for over-using resources. As a matter of fact you can't over-use resources, you can only use the resources you are given to you. I know they are different type of VPS, I am referring to Slicehost/Linode type VPS which has hard limit and acts very much like dedicated single box. More important to me is that I am really really really surprised that Ycombinator was running on a shared account. This just blew my mind. I have no hard feelings towards pear, I would have shut the site down too; possibly long ago.
  - deleted 17y ago
    [deleted]
  - chrischen 17y ago
    They limit memory at Slicehost but not CPU. I think you can still burst CPU for too long and they'd eventually tell you to upgrade.

noonespecial 17y ago

Customers nothing. "I didn't know what that 'hacker news' thing was or if it should have that much traffic so I shut it off". Really? Please turn in your geek card at the door.