4 ms·

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is: 1. I was never notified of the

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is: 1. I was never notified of the outage. I noticed it myself when attempting to log into one of my VMs and then started looking for status updates. Sadly, the best status updates I got were here on Hacker News. 2. When my servers did come back up, at least one of my IP addresses had changed, which meant I had to update all of the relevant DNS entries (which, as everyone here no doubt knows, can take up to 48 hours to propagate). I was never notified of this change in any way.

Maarten88 12y ago
2. Azure does not guarantee that you keep your ip address by default. You should configure a cname if you use Azure Websites or get a reserved ip address, available with Cloud Services
- badgersandjam 12y ago
  Also as I found out, set the TTL on your CNAME very low. Also this is a PITA if you use the @ entry in your DNS.
  - mnutt 12y ago
    Why should you have to keep the CNAME low if it always resolves to the same domain on Azure?
    - badgersandjam 12y ago
      Only because some caching DNS resolvers are broken and keep the referral cached for the TTL of the CNAME record.
- silverbax88 12y ago
  Actually, I am supposed to have a fixed IP. And I have CNAMES configured. I will review to determine if there is some other way I can set this up, but my issue is that I would never allow my products to be offline for hours without notifying my customers.
  - coreysa 12y ago
    First, I am really sorry about the impact the incident had on your service. Specifically, for your changing IP, are you currently using the reserved IP feature. You can reserve both your external IP and your internal IP. External IP: http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx Internal IP: http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx
shaydoc 12y ago
Do you having monitoring setup on your cloud vm, because you should have alerts triggering emails to notify you when this is happening. Secondly, you are using an IP address and expecting that to be static? The recommended approach is to use a CNAME so you don't hit that issue, alternatively, you can have up to 5 Reserved-IPs per subscription and attach that Reserved-IP to your VM : New-AzureReservedIP from powershell Edit : see http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addresses/ http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addre...
- silverbax88 12y ago
  Yes, I do. That's why I was checking my VMs. This still doesn't absolve my cloud server provider from notifying me when they have a global system failure.
hosay123 12y ago
Sadly this is a common situation. They appear to hold off making any public notice (including often on their own "service dashboards") until support forums are screaming with upset users. Google App Engine has had numerous outages like this, the only one I can find any public documentation for being a 6 hour outage in 2012: http://googleappengine.blogspot.co.uk/2012/10/about-todays-app-engine-outage.html http://googleappengine.blogspot.co.uk/2012/10/about-todays-a... (and let's not forget the old-style Datastore corruption incident, where every App Engine user got to manually merge split-brain database tables after a messed up failover)
- dewitt 12y ago
  Hi hosay123, The App Engine team has a proactive policy about posting about downtime: https://groups.google.com/forum/#!forum/google-appengine-downtime-notify https://groups.google.com/forum/#!forum/google-appengine-dow... Since the team highlights basically anything that looks like it is impacting customers, the issues don't always warrant a stand-alone blog post, but you'll notice that generally speaking the last post in each thread is a full public post-mortem with diagnosis and remediation. Let me know if there's more you think might be useful for you as a GAE customer. Thanks!
  - blablabla123 12y ago
    I'm not trying to be evil but I used to migrate a small website away from GAE because of continuing issues and very limited communication from Google. Azure isn't perfect but their support is way more responsive, at least to us eventhough we don't have any fancy support contract. Eventhough my GAE experience was 2-3 yrs ago, I have to say that Azure has way less issues.
yourad_io 12y ago
> which, as everyone here no doubt knows, can take up to 48 hours to propagate I think that's been largely dispelled. I can't find the link right now unfortunately but I remember a post looking into DNS propagation realities from either this or last year, and they found that overwhelming majority of DNS servers they tried (99%+) respected the TTLs set exactly as they should. * My personal rule of thumb is, if it hasn't propagated within an hour, I need to look at it again because I messed up. Tools like this [1]are invaluable when you're paranoid about whether your new record has propagated. [1] https://www.whatsmydns.net https://www.whatsmydns.net * ugh. Does anyone know which post I'm talking about? My google-fu is failing me hard.
- vijaykiran 12y ago
  Perhaps not the link you are looking for - but there was some discussion on here sometime ago - https://news.ycombinator.com/item?id=3397253 https://news.ycombinator.com/item?id=3397253

Maarten88 12y ago

2. Azure does not guarantee that you keep your ip address by default. You should configure a cname if you use Azure Websites or get a reserved ip address, available with Cloud Services

badgersandjam 12y ago
Also as I found out, set the TTL on your CNAME very low. Also this is a PITA if you use the @ entry in your DNS.
- mnutt 12y ago
  Why should you have to keep the CNAME low if it always resolves to the same domain on Azure?
  - badgersandjam 12y ago
    Only because some caching DNS resolvers are broken and keep the referral cached for the TTL of the CNAME record.
silverbax88 12y ago
Actually, I am supposed to have a fixed IP. And I have CNAMES configured. I will review to determine if there is some other way I can set this up, but my issue is that I would never allow my products to be offline for hours without notifying my customers.
- coreysa 12y ago
  First, I am really sorry about the impact the incident had on your service. Specifically, for your changing IP, are you currently using the reserved IP feature. You can reserve both your external IP and your internal IP. External IP: http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx Internal IP: http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx

shaydoc 12y ago

Do you having monitoring setup on your cloud vm, because you should have alerts triggering emails to notify you when this is happening. Secondly, you are using an IP address and expecting that to be static? The recommended approach is to use a CNAME so you don't hit that issue, alternatively, you can have up to 5 Reserved-IPs per subscription and attach that Reserved-IP to your VM : New-AzureReservedIP from powershell Edit : see http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addresses/ http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addre...

silverbax88 12y ago
Yes, I do. That's why I was checking my VMs. This still doesn't absolve my cloud server provider from notifying me when they have a global system failure.

hosay123 12y ago

Sadly this is a common situation. They appear to hold off making any public notice (including often on their own "service dashboards") until support forums are screaming with upset users. Google App Engine has had numerous outages like this, the only one I can find any public documentation for being a 6 hour outage in 2012: http://googleappengine.blogspot.co.uk/2012/10/about-todays-app-engine-outage.html http://googleappengine.blogspot.co.uk/2012/10/about-todays-a... (and let's not forget the old-style Datastore corruption incident, where every App Engine user got to manually merge split-brain database tables after a messed up failover)

dewitt 12y ago
Hi hosay123, The App Engine team has a proactive policy about posting about downtime: https://groups.google.com/forum/#!forum/google-appengine-downtime-notify https://groups.google.com/forum/#!forum/google-appengine-dow... Since the team highlights basically anything that looks like it is impacting customers, the issues don't always warrant a stand-alone blog post, but you'll notice that generally speaking the last post in each thread is a full public post-mortem with diagnosis and remediation. Let me know if there's more you think might be useful for you as a GAE customer. Thanks!
- blablabla123 12y ago
  I'm not trying to be evil but I used to migrate a small website away from GAE because of continuing issues and very limited communication from Google. Azure isn't perfect but their support is way more responsive, at least to us eventhough we don't have any fancy support contract. Eventhough my GAE experience was 2-3 yrs ago, I have to say that Azure has way less issues.

yourad_io 12y ago

> which, as everyone here no doubt knows, can take up to 48 hours to propagate I think that's been largely dispelled. I can't find the link right now unfortunately but I remember a post looking into DNS propagation realities from either this or last year, and they found that overwhelming majority of DNS servers they tried (99%+) respected the TTLs set exactly as they should. * My personal rule of thumb is, if it hasn't propagated within an hour, I need to look at it again because I messed up. Tools like this [1]are invaluable when you're paranoid about whether your new record has propagated. [1] https://www.whatsmydns.net https://www.whatsmydns.net * ugh. Does anyone know which post I'm talking about? My google-fu is failing me hard.

vijaykiran 12y ago
Perhaps not the link you are looking for - but there was some discussion on here sometime ago - https://news.ycombinator.com/item?id=3397253 https://news.ycombinator.com/item?id=3397253