5 ms·

Congrats on the anniversary. To provide a balanced perspective on this thread, let me share our experience. We (Visual Website Optimizer) are hosted on Linode s

Congrats on the anniversary. To provide a balanced perspective on this thread, let me share our experience. We (Visual Website Optimizer) are hosted on Linode since beginning but have experienced degradation of service lately particularly around network availability. There is at least one minor outage every month in their Newark data center (planning around network outage is much harder). While their support is responsive, these regular outages affect our business in a very significant manner. Every time we contact them, their staff just says these outages _happen_ and unavoidable and I cannot believe they are okay with it. Maybe it is to do with their increased customer base, but earlier these outages were much less frequent and we enjoyed the service. We own 25+ servers with them and have lately given thought of moving to some other provider. It's their responsive customer service and the pain of migrating everything that has kept us sticking to Linode.

thaumaturgy 15y ago
Huh. I have a number of our pre-launch hosting customers on a web server in the Newark center; I get a text message within a few seconds of an outage, and haven't received any notifications in quite a while (unless it was due to my own error). Maybe it's something on your particular rack or node? Have you tried setting up new Linodes in Newark and migrating to those?
callmeed 15y ago
If you have 25+ servers, aren't some of them in other data centers and configured in some sort of high-availability setup? What's your technology stack? (FYI, I'm asking purely out of curiosity)
- paraschopra 15y ago
  We have a backup set of servers at a different data center as well. But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue. The only real solution to this is BGP routing which is complicated to setup and VERY expensive (only big guys do it). Here's some excellent answers on SO: http://serverfault.com/questions/153576/global-high-availability-setup-question http://serverfault.com/questions/153576/global-high-availabi...
  - cbf 15y ago
    But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. All resolvers cache records for the TTL (or SOA minimum for negative lookups) specified by your authoritative servers. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue. If the clients are web browsers they will try the alternative addresses in a record set if the one they pick doesn't respond.
  - AgentConundrum 15y ago
    Forgive my ignorance, but isn't it normal to cache the DNS for 24 hours or so by default? I think it's typical (though I've never done it, so I'll apologize again if I'm talking out of my ass here) to drop the TTL down to basically nothing well ahead of the migration so that when you flip the switch, everyone will notice almost immediately, and then you can push the TTL back up to normal on the updated record.
robflynn 15y ago
Which data center do you use? I have not experienced any outages with my linode server(s).
jsprinkles 15y ago
My Linode in Newark has not had network issues since last year. Acquaintance has a large deployment there with 100.0% availability. Positive this is not problems with your stack or monitoring?
- paraschopra 15y ago
  No, it isn't. In addition to using Pingdom, we are continuously monitoring availability from multiple different locations (these scripts also do DNS switching). Every time we detect an issue, we contact Linode and many times they confirm network issues (sometimes internal, sometimes upstream). Just two days back they confirmed one internal outage. See this screenshot: http://imgur.com/jYzuD http://imgur.com/jYzuD Almost every time we contact them about an outage, we get this drup response: "Apologize for inconvenience" which is okay, but they should at least upgrade their infra or take steps to avoid it. Maybe your acquaintance is not monitoring the deployment with higher frequency. We monitor our stack multiple times every minute.
  - jsprinkles 15y ago
    Pingdom is notoriously unreliable and I base this on personal experience and experiences of others. False positives are the norm and it is garbage.
    - paraschopra 15y ago
      Maybe, but we do monitor using our scripts as well.
  - hartror 15y ago
    I am pretty happy with Linode for the most part and haven't experienced many issues. However when we have had issues, diagnosed the issue and filed a ticket we get always get told it is a known issue and "Apologize for inconvenience". Why do I have to contact them to be told they are having an issue? Why can't they let me know asap so I can mitigate any problems they are having. The last issue we had was their backup system was failing and by the time we discovered it, it had been broken for days.
code_duck 15y ago
I notice a few blips now and then, too - it's visible on my graphs. Sometimes they can be as long as an hour or two. But then, I'm not getting any user reports in these times which makes me just wonder about the monitoring. I was having some major problems with a server that would flip out under heavy load. Turns out it was the specific kernel I was using. Linode support was quite helpful once I actually contacted them about it.
- Dramatize 15y ago
  I also get the blips which are visible on my graphs.
nkassis 15y ago
How regular and how long? They don't provide 100% uptime guarantee. A few hours a month of downtime is what 99.99% uptime. That's not bad at all. I personally haven't had issues with the network in the Dallas DC.
- rmah 15y ago
  There are 30*24 = 720 hours a month. If a "few hours" is 2 hours, then that is 718/720 = 99.72% uptime. This means 0.28% downtime vs 0.01% downtime. IOW, almost 30x worse than a 99.99% uptime promise (if that's even what Linode promises). Math FTW!
  - nkassis 15y ago
    ok.... seems like I can't enter numbers into irb ;p you are right 99.99% would be about an hour a year.
- paraschopra 15y ago
  Network outage for us even for couple of seconds can be bad because we are serving about 1300 requests/second at peak which can mean a number of users getting affected.
  - commx 15y ago
    The Internet itself is inherently unreliable. Your users will traverse multiple hops (many networks) before connecting to your servers at any provider, as will any external monitoring systems you put in place. Brief issues along the paths between them and your servers can and will occur. While the Internet as a whole is designed to route around problems, this doesn't happen instantaneously in most cases, although it can happen very quickly. This is not to say that internal network issues won't happen at any provider on occasion as well, because they certainly will. No network is perfect, least of all the massive system of interconnected networks known as the Internet, and you need to accept and plan for this. To believe or expect otherwise is simply unrealistic.
steve___ 15y ago
We use the Newark data center and monitor our server with Nagios. In the past 90 days I have not received an alert because the connection was down.

thaumaturgy 15y ago

Huh. I have a number of our pre-launch hosting customers on a web server in the Newark center; I get a text message within a few seconds of an outage, and haven't received any notifications in quite a while (unless it was due to my own error). Maybe it's something on your particular rack or node? Have you tried setting up new Linodes in Newark and migrating to those?

callmeed 15y ago

If you have 25+ servers, aren't some of them in other data centers and configured in some sort of high-availability setup? What's your technology stack? (FYI, I'm asking purely out of curiosity)

paraschopra 15y ago
We have a backup set of servers at a different data center as well. But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue. The only real solution to this is BGP routing which is complicated to setup and VERY expensive (only big guys do it). Here's some excellent answers on SO: http://serverfault.com/questions/153576/global-high-availability-setup-question http://serverfault.com/questions/153576/global-high-availabi...
- cbf 15y ago
  But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. All resolvers cache records for the TTL (or SOA minimum for negative lookups) specified by your authoritative servers. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue. If the clients are web browsers they will try the alternative addresses in a record set if the one they pick doesn't respond.
- AgentConundrum 15y ago
  Forgive my ignorance, but isn't it normal to cache the DNS for 24 hours or so by default? I think it's typical (though I've never done it, so I'll apologize again if I'm talking out of my ass here) to drop the TTL down to basically nothing well ahead of the migration so that when you flip the switch, everyone will notice almost immediately, and then you can push the TTL back up to normal on the updated record.

robflynn 15y ago

Which data center do you use? I have not experienced any outages with my linode server(s).

jsprinkles 15y ago

My Linode in Newark has not had network issues since last year. Acquaintance has a large deployment there with 100.0% availability. Positive this is not problems with your stack or monitoring?

paraschopra 15y ago
No, it isn't. In addition to using Pingdom, we are continuously monitoring availability from multiple different locations (these scripts also do DNS switching). Every time we detect an issue, we contact Linode and many times they confirm network issues (sometimes internal, sometimes upstream). Just two days back they confirmed one internal outage. See this screenshot: http://imgur.com/jYzuD http://imgur.com/jYzuD Almost every time we contact them about an outage, we get this drup response: "Apologize for inconvenience" which is okay, but they should at least upgrade their infra or take steps to avoid it. Maybe your acquaintance is not monitoring the deployment with higher frequency. We monitor our stack multiple times every minute.
- jsprinkles 15y ago
  Pingdom is notoriously unreliable and I base this on personal experience and experiences of others. False positives are the norm and it is garbage.
  - paraschopra 15y ago
    Maybe, but we do monitor using our scripts as well.
- hartror 15y ago
  I am pretty happy with Linode for the most part and haven't experienced many issues. However when we have had issues, diagnosed the issue and filed a ticket we get always get told it is a known issue and "Apologize for inconvenience". Why do I have to contact them to be told they are having an issue? Why can't they let me know asap so I can mitigate any problems they are having. The last issue we had was their backup system was failing and by the time we discovered it, it had been broken for days.

code_duck 15y ago

I notice a few blips now and then, too - it's visible on my graphs. Sometimes they can be as long as an hour or two. But then, I'm not getting any user reports in these times which makes me just wonder about the monitoring. I was having some major problems with a server that would flip out under heavy load. Turns out it was the specific kernel I was using. Linode support was quite helpful once I actually contacted them about it.

Dramatize 15y ago
I also get the blips which are visible on my graphs.

nkassis 15y ago

How regular and how long? They don't provide 100% uptime guarantee. A few hours a month of downtime is what 99.99% uptime. That's not bad at all. I personally haven't had issues with the network in the Dallas DC.

rmah 15y ago
There are 30*24 = 720 hours a month. If a "few hours" is 2 hours, then that is 718/720 = 99.72% uptime. This means 0.28% downtime vs 0.01% downtime. IOW, almost 30x worse than a 99.99% uptime promise (if that's even what Linode promises). Math FTW!
- nkassis 15y ago
  ok.... seems like I can't enter numbers into irb ;p you are right 99.99% would be about an hour a year.
paraschopra 15y ago
Network outage for us even for couple of seconds can be bad because we are serving about 1300 requests/second at peak which can mean a number of users getting affected.
- commx 15y ago
  The Internet itself is inherently unreliable. Your users will traverse multiple hops (many networks) before connecting to your servers at any provider, as will any external monitoring systems you put in place. Brief issues along the paths between them and your servers can and will occur. While the Internet as a whole is designed to route around problems, this doesn't happen instantaneously in most cases, although it can happen very quickly. This is not to say that internal network issues won't happen at any provider on occasion as well, because they certainly will. No network is perfect, least of all the massive system of interconnected networks known as the Internet, and you need to accept and plan for this. To believe or expect otherwise is simply unrealistic.

steve___ 15y ago

We use the Newark data center and monitor our server with Nagios. In the past 90 days I have not received an alert because the connection was down.