4 ms·

Recently one of my production node landed on a bad physical host, DO reboot my node twice within 24 hours, I tweeted about it and DO support open a ticket for m

Recently one of my production node landed on a bad physical host, DO reboot my node twice within 24 hours, I tweeted about it and DO support open a ticket for me. Since the physical node have problem, I plan to take snapshot then spin up a new node using the image. However, as the physical node have problem, the snapshot take more than 7 hours and still failed to snapshot (No way to cancel snapshot task). DO support told me the snapshot is running, but I seriously think their support not honest about it and I ask they provide proof to show the snapshot is moving, they failed to do so. (From UI I can see JSON response show progress stuck at 3%, they lied) Yesterday, DO reboot again my node. Just my two cent, never ever use DO for serious production usage.

ghshephard 12y ago
Alternative perspective - Recognize any time you use a cloud provider, you need to be aware that not only are the nodes ephemeral, but the zone/data center you are deploying on may go down as well. Plan for your server to go down at any time, and, if you have actual $$$ in play, also plan for the zone to go down. Amazon encourages this thinking by having very low-latency links between AZ (Availability Zones) - and they also recommend deploying in two geographical regions should a regional disaster occur.
- paukiatwee 12y ago
  I know that all. The point is, DO support lied. After reboot, they still put my node in bad physical host. Also, DO reboot without notice (AWS have EC2 events).
- toomuchtodo 12y ago
  Amazon has a robust ecosystem of supporting services and APIs to enable you to orchestrate as you described. DigitalOcean does not.
WorldWideWayne 12y ago
Why would you tweet about it before opening a support ticket yourself?
curiously 12y ago
it works fine for production. although one time one of my none essential droplet had trojan and started launchign DDOS attacks and it was shut down. I don't know how it happened but the password was super weak. Strong passwords and always use a private key when SSHing.
- wyattjoh 12y ago
  Or no password and only private key :)
  - curiously 12y ago
    isn't the whole purpose of private key to avoid having to type in password?
    - AYBABTME 12y ago
      You don't want passwords, at all, so that you can't be bruteforced. On the otherside, your private key, on your local machine, should be password protected. But the node itself should not accept passwords over SSH.
alfg 12y ago
Same happened to one of my production nodes as well. I had to take a snapshot and migrate to another physical node as suggested by the support team. Had about 6 reboots without warning in the last 2 months. I love DO, but sometimes the reboots without warning can be annoying.
wmf 12y ago
Unfortunately, VM storage isn't a solved problem at scale. You can use remote storage (like EBS) to avoid being dependent on any particular server, but in practice it also has outages and performance anomalies.

ghshephard 12y ago

Alternative perspective - Recognize any time you use a cloud provider, you need to be aware that not only are the nodes ephemeral, but the zone/data center you are deploying on may go down as well. Plan for your server to go down at any time, and, if you have actual $$$ in play, also plan for the zone to go down. Amazon encourages this thinking by having very low-latency links between AZ (Availability Zones) - and they also recommend deploying in two geographical regions should a regional disaster occur.

paukiatwee 12y ago
I know that all. The point is, DO support lied. After reboot, they still put my node in bad physical host. Also, DO reboot without notice (AWS have EC2 events).
toomuchtodo 12y ago
Amazon has a robust ecosystem of supporting services and APIs to enable you to orchestrate as you described. DigitalOcean does not.

WorldWideWayne 12y ago

Why would you tweet about it before opening a support ticket yourself?

curiously 12y ago

it works fine for production. although one time one of my none essential droplet had trojan and started launchign DDOS attacks and it was shut down. I don't know how it happened but the password was super weak. Strong passwords and always use a private key when SSHing.

wyattjoh 12y ago
Or no password and only private key :)
- curiously 12y ago
  isn't the whole purpose of private key to avoid having to type in password?
  - AYBABTME 12y ago
    You don't want passwords, at all, so that you can't be bruteforced. On the otherside, your private key, on your local machine, should be password protected. But the node itself should not accept passwords over SSH.

alfg 12y ago

Same happened to one of my production nodes as well. I had to take a snapshot and migrate to another physical node as suggested by the support team. Had about 6 reboots without warning in the last 2 months. I love DO, but sometimes the reboots without warning can be annoying.

wmf 12y ago

Unfortunately, VM storage isn't a solved problem at scale. You can use remote storage (like EBS) to avoid being dependent on any particular server, but in practice it also has outages and performance anomalies.