9 ms·
Windows Azure Storage certificate expired?
- rwg 14y agoAt ex-work, SSL certificates expiring was probably #3 in the list of reasons for services being unavailable. It's a really easy thing to overlook in service monitoring.
- spamizbad 14y agoWhen I was a contractor, this happened to one of my customers that was under contract (I set up the cert initially). I fixed the issue overnight without major impact, but boy was I embarrassed by the bush-league mistake. Since then, in addition to service monitoring, I've taken to calendaring certificate expiration dates ~14 days before they expire. Low tech but I've never forgotten since.
- Ironlink 14y agoNote to self: Compulsively add every certificate expiry as an event on my calendar.
- koenigdavidmj 14y agoBetter yet: set up a Nagios (or your site's equivalent) watcher for the expiration. Example: http://exchange.nagios.org/directory/Plugins/Network-Protocols/HTTP/check_ssl_certificate/details http://exchange.nagios.org/directory/Plugins/Network-Protoco...
- yajoe 14y agoI thought this was a repost of the same story from last February related to expired certs: http://www.wired.com/wiredenterprise/2012/02/azure_outage/ http://www.wired.com/wiredenterprise/2012/02/azure_outage/ I'm pretty tolerant of mistakes (I make them all the time), but I have little patience for repeating mistakes.
- smarx 14y agoIn this case, an SSL certificate has expired. The leap year issue last year was a bug about date addition: 2/29/2012 + 1 year was calculated as 2/29/2013, which is a date that doesn't exist. The only similarity is that both involved certificates. (The invalid date was used internally as an expiration date for a certificate.)
- vyrotek 14y agoGlad to see you around these parts Smarx :) You maybe not remember me but I came to Redmond for an Azure bootcamp you hosted for a few companies a few years ago. (I'm sure you did lots of those) I was with IActionable. We've had a few 'encounters' on the MSDN forums too. Azure sure has changed since those days... Definitely for the better! I see you've been busy since you left MS. Somehow I missed all the news about Site44!
- vyrotek 14y agoDon't worry guys. Scott is on it! :) https://twitter.com/shanselman/status/305064837434724353 https://twitter.com/shanselman/status/305064837434724353
- varunsrin 14y agoThe Azure Dashboard is here, for anyone looking for updates on this: https://www.windowsazure.com/en-us/support/service-dashboard/ https://www.windowsazure.com/en-us/support/service-dashboard...
- photorized 14y agoI am not sure I trust that dashboard. Day before yesterday, when Azure SQL was down for the whole day, it took serval hours of 100% outages until the dashboard got updated with red "problem" icons. Before that, it was "information", "routine maintenance", etc. Amazon likes to downplay their AWS issues as well (their console is always green), but at least they provide a very detailed postmortem, and issue credits proactively. Have not seen that from MS yet, and it's been two days.
- jspaur 14y agoAny info on Sql Azure being down a whole day? We didn't see anything. This issue on the other hand, a real bummer.
- photorized 14y agoFrom what I saw, started as routine maintenance, and we began seeing SQL timeouts and errors on SQL East, and then all our DB servers on SQL East just vanished. Stayed gone for 5 hours at least. I took a screenshot of the status console once it got updated with the outage info (several hours into the outage): https://mobile.twitter.com/iTrendTV/status/303959900164341760/photo/1 https://mobile.twitter.com/iTrendTV/status/30395990016434176... Pretty unhappy about the whole experience. Luckily, we have our own colo in addition to Azure. If you want to see people's reactions as this event unraveled: http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/231c1231-c634-448d-966f-40980c32460d http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted...
- jspaur 14y agoOuch, we saw a similar thing in the South when things went sideways late last year. We've taken a similar approach to Sql Azure, use DataSync to replicate to offsite warm instances.
- gabbo 14y agoIf this outage is solely caused by an expired SSL certificate then this is really bad. It makes Azure look like amateur hour (even though it's definitely not). This may be acceptable for a smaller operation, but not Microsoft's cloud platform. There should be processes (manual and automated) to deal with this. On the bright side, after this you know they're not going to make the same mistake twice.
- niggler 14y ago"It makes Azure look like amateur hour (even though it's definitely not)" Why do you think Azure is not?
- gdc 14y agoDo you think that it is?
- niggler 14y agoYou can't deny that the current issue is certainly not something you would expect of a serious offering
- gdc 14y agoEvery major cloud offering on the market has had outages that were traced down to silly issues. I'm not sure if your dislike is targeted at just Azure, or the cloud in general.
- breck 14y agoWhy do SSL certificates expire? Seems like a suboptimal design decision.
- hwatson 14y agoTake a look at me.com, for example. Before Apple bought it, it was owned by SnappVille.com. If SSL certificates didn't expire, SnappVille could have continued using their certificates for me.com.
- paxswill 14y agoThat still doesn't fully explain why they expire, as CRLs and OCSP allow certificates to be revoked. I can't quite explain why having an expiration date is safer, I just feel it's a good practice, to protect against possible key compromise.
- tptacek 14y agoSSL certificate revocation is extremely fragile. http://www.imperialviolet.org/2012/02/05/crlsets.html http://www.imperialviolet.org/2012/02/05/crlsets.html
- CJefferson 14y agoBut, they would have to get the private certificate? And if someone got that even while Apple still owned me.com, lots of nasty man in the middle attacks could have been made.
- PeterisP 14y agoSometimes you forget about the certificates you have (such as in this case). In a 'default-deny' security mindset, things you forget should expire so that you don't get bitten by a security compromise for a certificate that you issued in 1999 but still are "trusting". Furthermore, private keys can be cracked, given enough time. Expiry dates reduce this risk with (a) less time to do it; (b) key is invalidated even if you don't know that it's been compromised, so every compromise ends sooner or later, and (c) renewed certificates can be made with longer keys as hardware gets more powerful.
- moonboots 14y agoI've been using the following script in my build process to avoid certificate expiration surprises: currentDateEpoch=$(date +%s) expirationDate=$(openssl x509 -in $my_cert -enddate -noout | sed 's/notAfter=//' | date -f -) expirationDateEpoch=$(date -d "$expirationDate" +%s) diff=$((expirationDateEpoch - currentDateEpoch)) oneMonthInSeconds=$((30 * 24 * 3600)) oneWeekInSeconds=$((7 * 24 * 3600)) if [ "$diff" -lt $oneWeekInSeconds ]; then printf "Certificate $my_cert needs to be renewed! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2 exit 3 fi if [ "$diff" -lt $oneMonthInSeconds ]; then printf "Certificate $my_cert expires in less than a month! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2 else printf "Note: certificate $my_cert expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2 fi
- smhinsey 14y agoIf your connection strings are set in .cscfg config files, remember that they are editable through the Management Portal and you can swap them to HTTP. Edit: Maybe not such a great idea. It looks like this relies on Storage under the hood as well. Now my instances are recycling. Later: It appears doing so actually corrupted the instance and required a redeploy, which is of course not currently possible. Awesome.
- rmason 14y agoLetting domains and SSL certs expire has been a consistent problem for Microsoft. A few years back they lost a key domain that took down Hotmail: http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2100-1023_3-234907.html http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2... Couldn't find it but I know they've had other embarrassing domain or certs expire over the years. You would think they'd fix it before they lose microsoft.com!
- bogrollben 14y agoDon't worry - they found the guy that did it: http://memegenerator.net/instance/35258710 http://memegenerator.net/instance/35258710
- InvisibleCities 14y agoLooks like somebody forgot to set up a share-point reminder.
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- Gargol 14y agoCan't logon to management dashboard. Staging instances went crazy. Live environment looks OK so far. Should I be worried and stay up all night monitoring this stuff ?
- niggler 14y agoDoes anyone use Azure for a production service? I'm interested in hearing people's thoughts ...
- alpb 14y agoiCloud is hosted on Azure Storage.
- niggler 14y agoI initially thought you were joking, but it appears that iCloud is hosted on Azure and Amazon Services: http://www.redmondpie.com/guess-what-icloud-uses-windows-azure-services-for-hosting-data/ http://www.redmondpie.com/guess-what-icloud-uses-windows-azu...
- jborden13 14y agoYes overall the service has been good for us. We are mostly down, with a few services partially up but throwing errors and causing more confusion. Now we are just sitting on our hands getting pelted by clients...
- jspaur 14y agoWe use Azure pretty seriously and overall, we really like it. We went with the MSFT stack due to some of our low level code to deal with printing. That being said, it's had it's share of outages, but honestly no less than if we were racking and stacking our own machines. The most frustrating thing though (with any cloud provider really) is that if it goes down all you can do is sit and wait...atleast if its our fault we feel like we're in control.
- mikeocool 14y agoAdded this tool to my suite of monitoring tools last time an outage caused by an expired cert popped up on hacker news: http://prefetch.net/articles/checkcertificate.html http://prefetch.net/articles/checkcertificate.html Runs on a daily cron job and emails if any of the certs it's monitoring are within 30 days of expiration.
- awa 14y agois there something similar for windows?
- taylorbuley 14y agoThe Microsoft employee who managed to let the hotmail.co.uk domain expire is suddenly looking a little less irresponsible. http://www.theregister.co.uk/2003/11/06/microsoft_forgets_to_renew_hotmail/ http://www.theregister.co.uk/2003/11/06/microsoft_forgets_to...
- dangrossman 14y agoFor those of us using Trello to manage projects, it's worth adding a list of reminders for your certificate renewals. Set a due date on each card and Trello will highlight when they're coming up soon.
- sriramk 14y agoI was involved with obtaining a previous version of the same certificate when I was at Windows Azure. There were several safeguards in place to stop exactly this scenario from happening. I'm wondering how they broke down.
- magic_haze 14y agoDev churn, probably. Could you expand a bit on the sort of safeguards that were in place? A couple of comments here mentioned trello, build scripts or just plain post-its, it'd be interesting to know how bigcorps do it...
- sriramk 14y agoThe most basic one was that really large sets of people that would get emailed a long time before expiry by the central crypto/cert management system. Microsoft has a very streamlined system internally for obtaining/managing cents since they do so much of it.
- YokoZar 14y agoPerhaps that's precisely the problem -- a large set of people were notified, rather than anyone in particular
- mpyne 14y agoAdmiral Hyman Rickover (the "Father of the nuclear Navy") had as one of his basic principles that "if you can't point your finger at the person responsible for something, then no one is responsible for it".
- runesoerensen 14y agoI'd like to invite you to try AppHarbor. We provide Windows application hosting on top of AWS, and strive to deliver a higher quality of service than Azure. We're standing by if you need help to migrate immediately. We do a number of things to keep our platform and your apps more available. For instance we set up reminders before our certificates expire...
- emp_ 14y agoThe thing that stops me starting to use apphb is the paid hostname, it is very backwards to feel like geocities all over again to host a simple Nancy blog. The service itself is amazing and congrats on the platform.
- runesoerensen 14y agoThanks for your input! I'm glad to hear you like the platform. I think it's worth noticing that a bunch of services (including a worker) is included on the free plan. The $10/month to run a full app with custom hostnames is actually pretty cheap when you take that into consideration. If your app/blog is focused on the .NET/Windows developer ecosystem you can apply for inclusion in the community application program (https://appharbor.com/page/community-application-program https://appharbor.com/page/community-application-program). If included we'll pay for your app.
- fmax30 14y agoI have used Appharbor in my class to host a Web Pages project (Basically the prof made the whole class host it). My experience is , that while the customer support is good. The Service I used ( the free one) was a bit slow. My site took quite sometime to get loaded ( around 5-10 secs where my page weight was around 300-400 KB ) , the same page took less than a second on my local service. But all in all I really liked AppHarbor :)
- runesoerensen 14y agoThe free worker is allocated the same amount of resources as paid ones, so that shouldn't be the problem. It definitely sounds like there must have been an issue somewhere if you experienced that kind of performance difference. I don't think it was related to the page weight (unless you're on a very slow connection), nor the platform for that matter. I hope we were able to figure out a solution or give you advice on how to resolve it :-)
- 727374 14y agoIt's the annual February Azure meltdown. Last year they had a leap year bug that decimated Compute in multiple regions. http://www.wired.com/wiredenterprise/2012/03/azure-leap-year-bug/ http://www.wired.com/wiredenterprise/2012/03/azure-leap-year...
- Scorpion 14y agoIt might have taken out Xbox Live cloud storage as well.
- vyrotek 14y agoYour comment reminded me of an article that mentioned that Halo 4 would be 'backed' by Azure [1]. I wonder if that's down as well. [1] http://mashable.com/2012/10/31/halo-4-windows-azure/ http://mashable.com/2012/10/31/halo-4-windows-azure/
- LogicX 14y agoA few services to check your SSL certs: http://checkmyssl.com/ https://sslcheck.globalsign.com/en_US Email notifications: https://www.sslshopper.com/ssl-checker http://www.serverexec.com/ (does domain expiration too) Script to run yourself, does email notification: http://www.prefetch.net/articles/checkcertificate.html Any others to recommend?
- pasbesoin 14y agoSomeone over there might want to fire up Project and put in some hard deadlines.