9 ms·

Linode Outage in Dallas, Fremont, Atlanta, Newark, and Toronto Data Centers

152 points by boplicity 4y ago

eatonphil 4y ago
They were just bought by Akamai so unfortunately things like this are bound to happen if they are trying to integrate with Akamai the way they'd need to successfully keep growing. I'm not talking just software integration but process and people integration as well. We'll see I guess!
- teknopaul 4y ago
  I don't think many users will be affected by dhcp outage. Most linodes have fixed IPs. Mine don't seem to be affected
  - mritzmann 4y ago
    > Most linodes have fixed IPs I would expect the opposite. Doesn't DHCP make more sense, for example, if the user creates a snapshot and restores it with a new IP? I don't use Lenode, so I can't judge, but I would expect DHCP to be default.
    - kyrra 4y ago
      Default created Debian based linodes use a static IP. How they assign you the IP is they have an initial network boot script that injects that static IP into your config. I learned this because I wanted to test image recovery of deleted linode. When I restored it, I got a different IP address from linode (the static IP assigned to me), and I couldn't connect to it. Using the web console I found the IP address was hard coded to the old one still.
      - bogantech 4y ago
        Maybe this changed recently but I created a few Debian instances this year, both were configured with DHCP by default
        ryandrake 4y ago
        Just checked my tiny Nanode 1 GB image (Fremont) running Debian and it's indeed running dhclient. Still have access, but I need the instance this weekend for a virtual event. Kind of sucky timing for a potential outage, Linode.
  - exikyut 4y ago
    That being said, maybe check if you have `dhclient`, `dhcpcd`, `systemd-resolved`, or (less likely) `dnsmasq` or `bind` running (htop / ps aux). If you do, your instance may fall over after its DHCP lease expires and it then presumably fails to acquire a new one and goes all 169.254.x.x on you. The possible counterpoint / information-vacuum bit here is that I don't know what the DHCP lease time is, so the expiry may already have happened. IOW, apparently this started 4h ago, and so the last leases were presumably issued 4h ago, and maybe the lease length is 1h, so everything's already as dead as it's going to get, so maybe if you're not already dead, you're fine.
  - simmons 4y ago
    For Linode Ubuntu 22.04 installations, at least, the default seems to be DHCP, even if the IP address is fixed. And I certainly wouldn't have thought anything about this, since even when using fixed IP addresses, I'm in the habit of using DHCP to provide a central source of IP address truth for my networks, to ease the pain of the inevitable renumbering.
  - jpmattia 4y ago
    > Most linodes have fixed IPs. We run some Windows instances, which were DHCP out of laziness. <facepalm> [edit: It's a PITA to fire up a Windows instance on Linode. Once working, switching out of DHCP is near the bottom of the todo list.]
  - bombcar 4y ago
    Most have DHCP-fixed IPs. The IP lease never changes but it's assigned by DHCP - check your settings (mine was in /etc/netplan/01-netcfg.yaml dhcp4: yes
- ubermonkey 4y ago
  I'm absolutely certain Akamai will fuck Linode up, same as Rackspace ruined Slicehost -- which is what sent me to Linode in the first place. I'm waiting for a new Linode to pop up.
  - esskay 4y ago
    Out of curiosity what is a new competitor going to offer that the existing well established ones like DigitaoOcean, Vultr, etc arent?
    - ubermonkey 4y ago
      I would much rather deal with a human-scale firm than a giant one. That alone would get me to switch, even at greater cost.
    - mekster 4y ago
      DigitalOcean doesn't have much Asian presence, Linode's backup doesn't support non default file system like zfs, DO / Linode don't have HDD block storage like Vultr does (which I think is a hidden gem you can't realize until you go add one as it's not mentioned in the pricing page) that can be a very nice option to throw backup data in at 40GB/$. Vultr's CPU isn't as good as the other 2 by the look of sysbench, even for high frequency types. Those 3 vendors only have 2 or 3 backup slots which can be kind of unreliable if you didn't notice data loss over a weekend and weekly backup can be too far apart. Hetzner is almost perfect if they have a bit more location. Very cheap, very high performing, daily backup with plenty of slots but no HDD block storage. Sometimes Amazon Lightsail can be a decent option as they have 7 daily backup slots, free bandwidth is comparable to the others and prices aren't bad except their CPU suck. Small things but none are perfect.
      - tomxor 4y ago
        > DO / Linode don't have HDD block storage Linode have had block storage for ages, I use them for ZFS.
        4y ago
- bongobingo1 4y ago
  The Akamai acquisition has had me kind of stressed to be honest. Here in Oceania there aren't a whole lot of established "cloud" hosts to pick from that host locally. Linode is fantastic and I have loved them for 10+ years. Digital Ocean doesn't host here, nearest is Singapore. Vultr does but I have had poor experiences with them. The other options are AWS and Google which are potentially unbounded expenses which is hard to sell to clients that already see $12/m (aud) as a cost sink, no matter what the server+service will save them in seat time. There are of course local hosts but they're either oversold cpanel hosts or comparatively expensive (3-10x the price for the same hardware). I guess it just feeds from my existential dread of being a small fish where if any one of these upstream providers decide "oh, no we flagged that account because of [black box reason], sorry!", you can be out a lively hood with no real recourse. Guess it just kinda sucks to be at what feels like the end of the long tail of infrastructure.
  - prng2021 4y ago
    Just curious, what caused the poor experience with Vultr?
    - texasbigdata 4y ago
      Not OP but just a guess, vultr was founded 2014 and linode is 2003. Obviously not a proxy for quality, but it’s one data point as to potentially why.
      - doctor_eval 4y ago
        Was going to ask the same. I’ve used vultr for years and found them rock solid, and have had good support when needed. They also just opened a DC in Melbourne.
  - flakeoil 4y ago
    These guys are in Australia amongst other locations: https://www.ovhcloud.com/ https://www.ovhcloud.com/
nimbius 4y ago
to whomever is at the top of the automation repo commit log, i wish you a safe and speedy recovery. Enjoy this friday, that it may succor a distant memory of fridays past when the latest build didnt break production.
simmons 4y ago
Arrgh! :( Just this week I needed to do some emergency rearranging of personal infrastructure, and moved one nameserver to Linode Fremont, while the other nameserver is in Linode Dallas. I was fully aware that I needed a more diverse long-term solution, but I figured "What are the chances that both Fremont and Dallas will go down in the next week or two?" Sorry, folks. I hate that my dangerous flirting with Murphy's Law is hurting other people.
- simmons 4y ago
  For whatever it's worth, since it was a DHCP problem, I was able to manually configure the static IPs and get the hosts back up. The last incoming connections logged before the outage were around ~ 6:24:00 am MDT (about one hour ago). I guess that's when my server needed to renew the DHCP lease and couldn't.
tttd 4y ago
j45 4y ago
That’s too bad. Which other vps hosts are folks using these days besides aws/azure/gc? If anyones thinking of switching to digitalocean, I’m currently switching linode from digitalocean who had an unforgivable, catastrophic loss of vm. Digitalocean also prevebts customers from downloading disk image backups offsite for any kind of adult DR. Linode lets you download backup images right from the admin console.
- mxuribe 4y ago
  I was thinking on moving to Hetzner (from Digital Ocean) - now that they have/will have a data center in Virginia....but i have heard iffy things about hetzner (some people are happy, others not so much). I should clarify, i'm only thinking on moving due to costs (not because i have encountered any ops or support issues). The stuff that i host is not as essential, and could do with slightly lower costs. ;-)
- pbowyer 4y ago
  I'm using UpCloud who are good, but a bit more expensive than Linode or DigitalOcean.
  - jaachan 4y ago
    Normally I'd agree, but... Have you seen today's status ? https://status.upcloud.com/ https://status.upcloud.com/ 6 hour downtime and no updates
- samtho 4y ago
  For what it’s worth, I’ve been a Linode customer for about 12 years and have never had a problem. This current issue does not even affect me as my boxes are on static IPs. My setup consists of a number of Linode VPSes, a single Hetzner Box in Germany, and an S3 glacier bucket (the only Amazon service I use) for backups. It’s probably not optimal, especially for the business case, but it works for all my hobby and side projects.
  - j45 4y ago
    For sure. I’ve had linode for a very long time. Not leaving them because of how good their tech support is or was to help me early on in my career, but they can’t be a point of failure either. I’ll just get to see how easily a hybrid cloud might be orchestrated like yours. Things are drastically easier to learn to setup and maintain like you have said compared to 5 or 10 years ago. Docker is docker, and traefik is traefik.
tiffanyh 4y ago
Off topic: any updates on Linode's Bare Metal service? It's been 2-years since it's been announced as "Coming Soon" and I haven't seen any updates. https://web.archive.org/web/20201116142627/https://www.linode.com/products/bare-metal/ https://web.archive.org/web/20201116142627/https://www.linod...
- ehayes 4y ago
  https://www.linode.com/products/bare-metal/ https://www.linode.com/products/bare-metal/ —still prefaced with "coming soon" but it looks like you can "create your build".
- joecot 4y ago
  I gave up. I submitted an inquiry on their sales form for dedicated and got no response, I contacted their support and got "we'll let you know." They've at least added nvme storage for instances, but in at least their Dallas data center the iops suck. In Newark it's a bit better. If you've got a bunch of servers at a linode data center and want bare metal ones, I ended up just looking at what other hosts are peered with their nearest exchange. Our servers are in Dallas and Vultr, Equinix via IBM Cloud, and Servermania all had peering to them through a local exchange at infomart or nearby. We ended up moving database servers to those bare metal services and connecting over wan to our linode ones. Latency less than 5ms. Linode also contacted us to ask why our monthly usage halved. We told them, they offered no alternatives.
  - tiffanyh 4y ago
    Same experience, no response. I tried contacting them both a year ago and about 6 months ago and never heard back. I went with Vultr (as you mentioned as well) and couldn't be happier.
- dboreham 4y ago
  A bare metal data point, since I just went through the process of finding an alternative to Packet.net since they were acquired by Equinix : try Hivelocity. My experience with them has been so far so good.
- 4y ago
chasd00 4y ago
I can still login to the backend and to my vm through Lish but my websites are down. (I'm in Dallas) edit: followed the instructions in this link mentioned upthread and rebooted, that solved it for me https://www.linode.com/docs/guides/network-helper/#individual-compute-instance-setting https://www.linode.com/docs/guides/network-helper/#individua...
- deleted 4y ago
  [deleted]
- patrickdavey 4y ago
  We ended up rebooting our server, that brought everything back up again.
- bombcar 4y ago
  If it's DHCP and you know your IP (probably because it's your A record) try manually forcing the IP.
mistertrotsky 4y ago
My sites in Newark are up. The situation is not as alarming as the submission title would suggest. They've updated the status page to indicate that it is limited to an issue with DHCP.
windhamdavid 4y ago
Mine in Atlanta are up. Already had the 'Network Helper' enabled.
lukeqsee 4y ago
I think "Outage" as stated is a bit over-the-top. We run significant operations in most of these DCs, and we have seen zero impact. YMMV of course, but I think it's a big disingenuous to say this is a full outage.
- boplicity 4y ago
  It's a full outage for me -- wasting quite a lot of time. It's been 7 hours so far. Hopefully it won't drag on longer. Who knows how many people it's affecting?
  - squidbeak 4y ago
    For us, too.
- CoffeeOnWrite 4y ago
  Mine in Dallas became inaccessible around 7:20am PT and came back around 7:50am.
- wolrah 4y ago
  For what it's worth I have around 25 Linodes active, mostly in Newark but a few in Atlanta and Dallas as well. Of those I had two in Newark which actually had their leases expire before we were aware of what was going on and then found three more (two Atlanta, one Newark) once we knew what to look for which were failing to renew. All five of those machines already had network helper enabled as Linode suggests but were unable to get a lease even with manual operation of the DHCP client until they were rebooted.
linodecustomerm 4y ago
Linode London has been experiencing "greater than normal" intermittent packet loss to Akamai network which they use to peer with other player such as Cloudflare in LDN for nearly two days - before they peered with CF via Linx if I remember correctly. It isn't just isolated to UDP customers. Currently it's fairly stable so for most it's hard to notice but certainly uncomfortable how far they want to drag this problem on - see status.linode.com for how grim the LDN situation is - their support said we'll tell you in a few days what it was once we fixed it. i.e. it's not the rock solid of the last 7 years and their support are largely deaf to how unacceptable this is based on previous standards.
boplicity 4y ago
FYI: Linode says that most people can fix this by enabling "Network Helper." You'll then likely need to reboot your server. It worked for me! https://www.linode.com/docs/guides/network-helper/#individual-compute-instance-setting https://www.linode.com/docs/guides/network-helper/#individua...
- chasd00 4y ago
  Thank you! this worked for me too. I was totally down, went in and enabled Auto Configure Networking per your link, and then rebooted.
- apocalyptic0n3 4y ago
  Most of my Linodes are still accessible, but this has worked for the two that went down so far.
- Ologn 4y ago
  My Newark node was down for over six hours, this is how I fixed it. Uptime on the VPS had been over a year before this (don't remember why it was rebooted a year ago). Like others here said, I did not greet the news of Akamai buying them with joy, and things like this starting seem to bare my initial impressions out.
- zzzeek 4y ago
  this.... replaces the use of DHCP with a static file rewriter on boot, OK, why is this recommended in general, should we expect DHCP to have ongoing issues that their "network helper" would not ?
  - bombcar 4y ago
    For the vast majority of people, a Linode is just a virtual server that sits on a static IP like a fat toad. This network helper just assigns that static IP instead of assuming DHCP will find it. You could turn it off later I guess.
    - TheNewsIsHere 4y ago
rossdavidh 4y ago
In almost 20 years with Linode, I've had nearly zero problems. Akamai buys them, and within months we have an outage. Not encouraging... On the other hand, it's only hitting my personal website, and the customer I have on Linode seems unaffected, so that's nice at least.
- bragr 4y ago
  To be fair, I know from personal experience Fremont has gone down hard for hours before.
- tiffanyh 4y ago
  Others have had different experiences. https://news.ycombinator.com/item?id=14571234 https://news.ycombinator.com/item?id=14571234
- tomxor 4y ago
  I'm wondering how much of this is perception... The number of reported issues coming through has exploded since Akamai. Over the last week alone, almost every day a new networking issue for London is in my inbox. However - none of the recent ones have actually affected any one of my Linodes, even though it's possible since it falls under the service and datacenter - perhaps the change we are seeing is more transparency? I'm not saying no one is affected... as others in this thread clearly are, and I was affected by the datacenter wide outage in London earlier in the year - but it's possible that being more upfront in notifications of issues could change the perception of what before would have been considered a collection of isolated incidents.</speculation> [edit] I generally appreciate transparency, but when it comes to varying degrees of service disruption maybe there is a balance to be struck... right now it feels like the signal to noise ratio is low in terms of it's affect across all users even within an area. Right now I am pretty much ignoring the Linode issues filling my inbox, if a serious one came through I'd probably miss it.
- rapind 4y ago
  I’ve had a non trivial number of issues with them over the years. Enough that I moved all of my infrastructure to other (more expensive) providers. That being said, it was probably value for money spent while it lasted.
mcculley 4y ago
Amusingly, I am trying to log in to the dashboard to reboot my nodes, but it requires a verification code which was sent to my email, which is hosted on Linode.
- velcrovan 4y ago
  Consider signing up to a service like MXGuardian that your MX records can point to, and that then forward email to your server. This will give you an “emergency inbox” that you can use in cases like this (might also help with spam depending on how you are currently dealing with that).
  - arjvik 4y ago
    What happens when MXGuardian goes down?
mbesto 4y ago
Seems like more and more outages are happening every day...tsk tsk Shameless plug - we're helping orgs fix this and get better clarity: https://www.awareops.com/ https://www.awareops.com/ Shoot me a message mb @ awareops dot com or go ahead and sign up.
ErneX 4y ago
Downtime for me lasted almost 4 hours.
cmg 4y ago
I had 6 servers out of about 50 go down, all in Dallas/Atlanta even though most servers are in Newark. IPv6 connectivity seemed to remain up for all of them, though.
underdeserver 4y ago
3 minutes of monitoring? That's enough to declare the issue over?
markstos 4y ago
I found my Linode-hosted site was down this morning. I logged into the Linode console, rebooted and it came back up. Generally I've been very happy with Linode over several years.

152 points by boplicity 4y ago

They were just bought by Akamai so unfortunately things like this are bound to happen if they are trying to integrate with Akamai the way they'd need to successfully keep growing. I'm not talking just software integration but process and people integration as well. We'll see I guess!

teknopaul 4y ago
I don't think many users will be affected by dhcp outage. Most linodes have fixed IPs. Mine don't seem to be affected
- mritzmann 4y ago
  > Most linodes have fixed IPs I would expect the opposite. Doesn't DHCP make more sense, for example, if the user creates a snapshot and restores it with a new IP? I don't use Lenode, so I can't judge, but I would expect DHCP to be default.
  - kyrra 4y ago
    Default created Debian based linodes use a static IP. How they assign you the IP is they have an initial network boot script that injects that static IP into your config. I learned this because I wanted to test image recovery of deleted linode. When I restored it, I got a different IP address from linode (the static IP assigned to me), and I couldn't connect to it. Using the web console I found the IP address was hard coded to the old one still.
    - bogantech 4y ago
      Maybe this changed recently but I created a few Debian instances this year, both were configured with DHCP by default
      - ryandrake 4y ago
        Just checked my tiny Nanode 1 GB image (Fremont) running Debian and it's indeed running dhclient. Still have access, but I need the instance this weekend for a virtual event. Kind of sucky timing for a potential outage, Linode.
- exikyut 4y ago
  That being said, maybe check if you have `dhclient`, `dhcpcd`, `systemd-resolved`, or (less likely) `dnsmasq` or `bind` running (htop / ps aux). If you do, your instance may fall over after its DHCP lease expires and it then presumably fails to acquire a new one and goes all 169.254.x.x on you. The possible counterpoint / information-vacuum bit here is that I don't know what the DHCP lease time is, so the expiry may already have happened. IOW, apparently this started 4h ago, and so the last leases were presumably issued 4h ago, and maybe the lease length is 1h, so everything's already as dead as it's going to get, so maybe if you're not already dead, you're fine.
- simmons 4y ago
  For Linode Ubuntu 22.04 installations, at least, the default seems to be DHCP, even if the IP address is fixed. And I certainly wouldn't have thought anything about this, since even when using fixed IP addresses, I'm in the habit of using DHCP to provide a central source of IP address truth for my networks, to ease the pain of the inevitable renumbering.
- jpmattia 4y ago
  > Most linodes have fixed IPs. We run some Windows instances, which were DHCP out of laziness. <facepalm> [edit: It's a PITA to fire up a Windows instance on Linode. Once working, switching out of DHCP is near the bottom of the todo list.]
- bombcar 4y ago
  Most have DHCP-fixed IPs. The IP lease never changes but it's assigned by DHCP - check your settings (mine was in /etc/netplan/01-netcfg.yaml dhcp4: yes
ubermonkey 4y ago
I'm absolutely certain Akamai will fuck Linode up, same as Rackspace ruined Slicehost -- which is what sent me to Linode in the first place. I'm waiting for a new Linode to pop up.
- esskay 4y ago
  Out of curiosity what is a new competitor going to offer that the existing well established ones like DigitaoOcean, Vultr, etc arent?
  - ubermonkey 4y ago
    I would much rather deal with a human-scale firm than a giant one. That alone would get me to switch, even at greater cost.
  - mekster 4y ago
    DigitalOcean doesn't have much Asian presence, Linode's backup doesn't support non default file system like zfs, DO / Linode don't have HDD block storage like Vultr does (which I think is a hidden gem you can't realize until you go add one as it's not mentioned in the pricing page) that can be a very nice option to throw backup data in at 40GB/$. Vultr's CPU isn't as good as the other 2 by the look of sysbench, even for high frequency types. Those 3 vendors only have 2 or 3 backup slots which can be kind of unreliable if you didn't notice data loss over a weekend and weekly backup can be too far apart. Hetzner is almost perfect if they have a bit more location. Very cheap, very high performing, daily backup with plenty of slots but no HDD block storage. Sometimes Amazon Lightsail can be a decent option as they have 7 daily backup slots, free bandwidth is comparable to the others and prices aren't bad except their CPU suck. Small things but none are perfect.
    - tomxor 4y ago
      > DO / Linode don't have HDD block storage Linode have had block storage for ages, I use them for ZFS.
      - 4y ago
bongobingo1 4y ago
The Akamai acquisition has had me kind of stressed to be honest. Here in Oceania there aren't a whole lot of established "cloud" hosts to pick from that host locally. Linode is fantastic and I have loved them for 10+ years. Digital Ocean doesn't host here, nearest is Singapore. Vultr does but I have had poor experiences with them. The other options are AWS and Google which are potentially unbounded expenses which is hard to sell to clients that already see $12/m (aud) as a cost sink, no matter what the server+service will save them in seat time. There are of course local hosts but they're either oversold cpanel hosts or comparatively expensive (3-10x the price for the same hardware). I guess it just feeds from my existential dread of being a small fish where if any one of these upstream providers decide "oh, no we flagged that account because of [black box reason], sorry!", you can be out a lively hood with no real recourse. Guess it just kinda sucks to be at what feels like the end of the long tail of infrastructure.
- prng2021 4y ago
  Just curious, what caused the poor experience with Vultr?
  - texasbigdata 4y ago
    Not OP but just a guess, vultr was founded 2014 and linode is 2003. Obviously not a proxy for quality, but it’s one data point as to potentially why.
    - doctor_eval 4y ago
      Was going to ask the same. I’ve used vultr for years and found them rock solid, and have had good support when needed. They also just opened a DC in Melbourne.
- flakeoil 4y ago
  These guys are in Australia amongst other locations: https://www.ovhcloud.com/ https://www.ovhcloud.com/

nimbius 4y ago

to whomever is at the top of the automation repo commit log, i wish you a safe and speedy recovery. Enjoy this friday, that it may succor a distant memory of fridays past when the latest build didnt break production.

simmons 4y ago

Arrgh! :( Just this week I needed to do some emergency rearranging of personal infrastructure, and moved one nameserver to Linode Fremont, while the other nameserver is in Linode Dallas. I was fully aware that I needed a more diverse long-term solution, but I figured "What are the chances that both Fremont and Dallas will go down in the next week or two?" Sorry, folks. I hate that my dangerous flirting with Murphy's Law is hurting other people.

simmons 4y ago
For whatever it's worth, since it was a DHCP problem, I was able to manually configure the static IPs and get the hosts back up. The last incoming connections logged before the outage were around ~ 6:24:00 am MDT (about one hour ago). I guess that's when my server needed to renew the DHCP lease and couldn't.

tttd 4y ago

j45 4y ago

That’s too bad. Which other vps hosts are folks using these days besides aws/azure/gc? If anyones thinking of switching to digitalocean, I’m currently switching linode from digitalocean who had an unforgivable, catastrophic loss of vm. Digitalocean also prevebts customers from downloading disk image backups offsite for any kind of adult DR. Linode lets you download backup images right from the admin console.

mxuribe 4y ago
I was thinking on moving to Hetzner (from Digital Ocean) - now that they have/will have a data center in Virginia....but i have heard iffy things about hetzner (some people are happy, others not so much). I should clarify, i'm only thinking on moving due to costs (not because i have encountered any ops or support issues). The stuff that i host is not as essential, and could do with slightly lower costs. ;-)
pbowyer 4y ago
I'm using UpCloud who are good, but a bit more expensive than Linode or DigitalOcean.
- jaachan 4y ago
  Normally I'd agree, but... Have you seen today's status ? https://status.upcloud.com/ https://status.upcloud.com/ 6 hour downtime and no updates
samtho 4y ago
For what it’s worth, I’ve been a Linode customer for about 12 years and have never had a problem. This current issue does not even affect me as my boxes are on static IPs. My setup consists of a number of Linode VPSes, a single Hetzner Box in Germany, and an S3 glacier bucket (the only Amazon service I use) for backups. It’s probably not optimal, especially for the business case, but it works for all my hobby and side projects.
- j45 4y ago
  For sure. I’ve had linode for a very long time. Not leaving them because of how good their tech support is or was to help me early on in my career, but they can’t be a point of failure either. I’ll just get to see how easily a hybrid cloud might be orchestrated like yours. Things are drastically easier to learn to setup and maintain like you have said compared to 5 or 10 years ago. Docker is docker, and traefik is traefik.

tiffanyh 4y ago

Off topic: any updates on Linode's Bare Metal service? It's been 2-years since it's been announced as "Coming Soon" and I haven't seen any updates. https://web.archive.org/web/20201116142627/https://www.linode.com/products/bare-metal/ https://web.archive.org/web/20201116142627/https://www.linod...

ehayes 4y ago
https://www.linode.com/products/bare-metal/ https://www.linode.com/products/bare-metal/ —still prefaced with "coming soon" but it looks like you can "create your build".
joecot 4y ago
I gave up. I submitted an inquiry on their sales form for dedicated and got no response, I contacted their support and got "we'll let you know." They've at least added nvme storage for instances, but in at least their Dallas data center the iops suck. In Newark it's a bit better. If you've got a bunch of servers at a linode data center and want bare metal ones, I ended up just looking at what other hosts are peered with their nearest exchange. Our servers are in Dallas and Vultr, Equinix via IBM Cloud, and Servermania all had peering to them through a local exchange at infomart or nearby. We ended up moving database servers to those bare metal services and connecting over wan to our linode ones. Latency less than 5ms. Linode also contacted us to ask why our monthly usage halved. We told them, they offered no alternatives.
- tiffanyh 4y ago
  Same experience, no response. I tried contacting them both a year ago and about 6 months ago and never heard back. I went with Vultr (as you mentioned as well) and couldn't be happier.
dboreham 4y ago
A bare metal data point, since I just went through the process of finding an alternative to Packet.net since they were acquired by Equinix : try Hivelocity. My experience with them has been so far so good.
4y ago

chasd00 4y ago

I can still login to the backend and to my vm through Lish but my websites are down. (I'm in Dallas) edit: followed the instructions in this link mentioned upthread and rebooted, that solved it for me https://www.linode.com/docs/guides/network-helper/#individual-compute-instance-setting https://www.linode.com/docs/guides/network-helper/#individua...

deleted 4y ago
[deleted]
patrickdavey 4y ago
We ended up rebooting our server, that brought everything back up again.
bombcar 4y ago
If it's DHCP and you know your IP (probably because it's your A record) try manually forcing the IP.

mistertrotsky 4y ago

My sites in Newark are up. The situation is not as alarming as the submission title would suggest. They've updated the status page to indicate that it is limited to an issue with DHCP.

windhamdavid 4y ago

Mine in Atlanta are up. Already had the 'Network Helper' enabled.

lukeqsee 4y ago

I think "Outage" as stated is a bit over-the-top. We run significant operations in most of these DCs, and we have seen zero impact. YMMV of course, but I think it's a big disingenuous to say this is a full outage.

boplicity 4y ago
It's a full outage for me -- wasting quite a lot of time. It's been 7 hours so far. Hopefully it won't drag on longer. Who knows how many people it's affecting?
- squidbeak 4y ago
  For us, too.
CoffeeOnWrite 4y ago
Mine in Dallas became inaccessible around 7:20am PT and came back around 7:50am.
wolrah 4y ago
For what it's worth I have around 25 Linodes active, mostly in Newark but a few in Atlanta and Dallas as well. Of those I had two in Newark which actually had their leases expire before we were aware of what was going on and then found three more (two Atlanta, one Newark) once we knew what to look for which were failing to renew. All five of those machines already had network helper enabled as Linode suggests but were unable to get a lease even with manual operation of the DHCP client until they were rebooted.

linodecustomerm 4y ago

Linode London has been experiencing "greater than normal" intermittent packet loss to Akamai network which they use to peer with other player such as Cloudflare in LDN for nearly two days - before they peered with CF via Linx if I remember correctly. It isn't just isolated to UDP customers. Currently it's fairly stable so for most it's hard to notice but certainly uncomfortable how far they want to drag this problem on - see status.linode.com for how grim the LDN situation is - their support said we'll tell you in a few days what it was once we fixed it. i.e. it's not the rock solid of the last 7 years and their support are largely deaf to how unacceptable this is based on previous standards.

boplicity 4y ago

FYI: Linode says that most people can fix this by enabling "Network Helper." You'll then likely need to reboot your server. It worked for me! https://www.linode.com/docs/guides/network-helper/#individual-compute-instance-setting https://www.linode.com/docs/guides/network-helper/#individua...

chasd00 4y ago
Thank you! this worked for me too. I was totally down, went in and enabled Auto Configure Networking per your link, and then rebooted.
apocalyptic0n3 4y ago
Most of my Linodes are still accessible, but this has worked for the two that went down so far.
Ologn 4y ago
My Newark node was down for over six hours, this is how I fixed it. Uptime on the VPS had been over a year before this (don't remember why it was rebooted a year ago). Like others here said, I did not greet the news of Akamai buying them with joy, and things like this starting seem to bare my initial impressions out.
zzzeek 4y ago
this.... replaces the use of DHCP with a static file rewriter on boot, OK, why is this recommended in general, should we expect DHCP to have ongoing issues that their "network helper" would not ?
- bombcar 4y ago
  For the vast majority of people, a Linode is just a virtual server that sits on a static IP like a fat toad. This network helper just assigns that static IP instead of assuming DHCP will find it. You could turn it off later I guess.
  - TheNewsIsHere 4y ago

rossdavidh 4y ago

In almost 20 years with Linode, I've had nearly zero problems. Akamai buys them, and within months we have an outage. Not encouraging... On the other hand, it's only hitting my personal website, and the customer I have on Linode seems unaffected, so that's nice at least.

bragr 4y ago
To be fair, I know from personal experience Fremont has gone down hard for hours before.
tiffanyh 4y ago
Others have had different experiences. https://news.ycombinator.com/item?id=14571234 https://news.ycombinator.com/item?id=14571234
tomxor 4y ago
I'm wondering how much of this is perception... The number of reported issues coming through has exploded since Akamai. Over the last week alone, almost every day a new networking issue for London is in my inbox. However - none of the recent ones have actually affected any one of my Linodes, even though it's possible since it falls under the service and datacenter - perhaps the change we are seeing is more transparency? I'm not saying no one is affected... as others in this thread clearly are, and I was affected by the datacenter wide outage in London earlier in the year - but it's possible that being more upfront in notifications of issues could change the perception of what before would have been considered a collection of isolated incidents.</speculation> [edit] I generally appreciate transparency, but when it comes to varying degrees of service disruption maybe there is a balance to be struck... right now it feels like the signal to noise ratio is low in terms of it's affect across all users even within an area. Right now I am pretty much ignoring the Linode issues filling my inbox, if a serious one came through I'd probably miss it.
rapind 4y ago
I’ve had a non trivial number of issues with them over the years. Enough that I moved all of my infrastructure to other (more expensive) providers. That being said, it was probably value for money spent while it lasted.

mcculley 4y ago

Amusingly, I am trying to log in to the dashboard to reboot my nodes, but it requires a verification code which was sent to my email, which is hosted on Linode.

velcrovan 4y ago
Consider signing up to a service like MXGuardian that your MX records can point to, and that then forward email to your server. This will give you an “emergency inbox” that you can use in cases like this (might also help with spam depending on how you are currently dealing with that).
- arjvik 4y ago
  What happens when MXGuardian goes down?

mbesto 4y ago

Seems like more and more outages are happening every day...tsk tsk Shameless plug - we're helping orgs fix this and get better clarity: https://www.awareops.com/ https://www.awareops.com/ Shoot me a message mb @ awareops dot com or go ahead and sign up.

ErneX 4y ago

Downtime for me lasted almost 4 hours.

cmg 4y ago

I had 6 servers out of about 50 go down, all in Dallas/Atlanta even though most servers are in Newark. IPv6 connectivity seemed to remain up for all of them, though.

underdeserver 4y ago

3 minutes of monitoring? That's enough to declare the issue over?

markstos 4y ago

I found my Linode-hosted site was down this morning. I logged into the Linode console, rebooted and it came back up. Generally I've been very happy with Linode over several years.