6 ms·

Availability sets on Windows Azure for an MVP

36 points by cpard 12y ago

znep 12y ago
Keep in mind that although availability sets can help reduce the chances that two VMs will go down at once, you do have to deal with the maintenance cycles that Azure has where they will reboot every one of your VMs over the course of x hours. This means that with a master/slave setup, a simple failover from a master to a slave is not sufficient, you need to be able to fail back automatically as well. Depending on the exact technologies you are using and how you are using them, this can have a lot of tricky corner cases that can end up much worse. If you don't do this, then when your master comes back up and they decide to reboot your slave next, which is a common scenario, you lose availability. This differs significantly from a scenario where VM failures/reboots aren't correlated so closely in time; if a VM failure is more random, it is more reasonable to require manual intervention to fail back to the master.
- dbarlett 12y ago
  Seconded. You're lucky to get clean ACPI shutdowns, half the time Azure "pulls the plug". The closest thing to an official explanation I've found (note date and chorus of comments): http://blogs.msdn.com/b/wats/archive/2013/09/24/windows-azure-virtual-machine-restarted-or-shutdown-with-out-any-notification.aspx http://blogs.msdn.com/b/wats/archive/2013/09/24/windows-azur...
- cpard 12y ago
  true, that's why we went for a master-master setup with the db.
- ultimape 12y ago
  Would a Quorum ( http://en.wikipedia.org/wiki/Quorum_(distributed_computing) http://en.wikipedia.org/wiki/Quorum_(distributed_computing) ) method - like done with Ceph's monitoring system agents - be able to handle this? I mean, you still may end up with the split-brain problem in the long run, but if that happens within a data-center would be the least of your worries. (Something like netflix's chaos army comes to mind for dealing with that).
NicoJuicy 12y ago
I'm kinda curious why you picked MariaDB instead of SQL Server ? What are the benefits?
- cpard 12y ago
  because of cost. MariaDB helps us to reduce cost, keep in mind that our storage needs are minimal. In any case if you can afford SQL server, you should go for it.
  - NicoJuicy 12y ago
    Any downsides of using MySQL instead of SQL Server in the development stage?
    - cpard 12y ago
      I'd say that the main downside is the time/effort to setup and maintain, especially as you scale up. At the end you should always judge based on your app and its requirements.
  - spo81rty 12y ago
    SQL Azure starts at $5 a month. Hosting a couple linux VMs for MariaDB would be a lot more expensive. SQL Azure is extremely affordable even as you scale up and it is fully managed. For basic SQL usage it is awesome and the value is unbeatable.
    - cpard 12y ago
      if you are in the situation to need dedicaded VMs for MariaDB then you should probably go for SQL Azure. Here this is not the case, the requirements for the DB are minimal and we already have 2 VMs running for the availability set. As I said, at the end you need to judge based on the specific needs of your app/product.
    - bad_user 12y ago
      Actually they are retiring the "Web and Business" pricing plans, which were charging for storage size and introducing a new pricing plan that charges for both storage size and "Database Throughput Units", which kind of obscures what they are charging for - something related to transactions volume and based on my experience with other "cloud hosted" databases, I can guarantee you that it gets pretty freaking expensive. The smartest thing to do is to never rely on a particular service of a cloud provider. For example we once made the choice of using Amazon's DynamoDB, which is great in the beginning, but at some points along with scale issues the costs went through the roof (DynamoDB also charges for "throughput capacity") and we just switched to Cassandra - but we planned for this migration ever since the beginning. And this is the problem with cloud providers - the pricing schemes on these services are designed to hook you with affordable prices in the beginning, but later on as your business grows they get much more expensive than hosting your own stuff. So use whatever makes you most productive, but careful about lock-in.
pling 12y ago
This could also be titled "how to enslave your MVP to increasing costs in the future". Azure isn't cheap by any means when you need to scale up our out.
- cpard 12y ago
  pling sure, but if you avoid using azure specific technologies it's quite easy to move somewhere else.
- ssiddharth 12y ago
  Simply decouple yourself from Azure specific tech and you're good to go, no?
  - pling 12y ago
    It is very hard to decouple yourself from the tooling and the platform specific features. The PaaS offering literally requires half of the additional services ad the basic deployment is simple and the IaaS offering is expensive and you need the latter to decouple effectively. Either way you also hang yourself with a vendor tied product and the ball and chain of licensing. The latter of which escalates pretty quick when your business grows. Our 8 man startup in 2005 now spends £200k a year on licensing because we're that tied to the damn platform.
    - cpard 12y ago
      pling, any chance to post/share how you are using the azure platform and what makes the migration hard? that would be really useful.
      - pling 12y ago
        We're not using it other than for internal failovers. We tried to build version two of our product on it. The costs increased rapidly until it was cheaper to buy a 42U and stuff it with HP kit and pay for a fat pipe. The storage was expensive, the per transaction costs are expensive, the bandwidth is expensive, the IaaS machines are expensive and the support is expensive when it goes wrong which it does occasionally and violently. This was all financial data aggregation and archival stuff. In test it was 23% more expensive than our own kit over 3 years.
NicoJuicy 12y ago
I'm wondering.. You said, build MVP on a budget... But your licensing costs are insanely high (200k pounds)... Isn't that contradictive? (or was that the licensing costs before the current situation?)
- cpard 12y ago
  That's not our cost, the 200k pounds are the cost pling mentioned for their case, 200k and MVP are indeed contradictory terms. We are actually participating on bizpark so we have some credits to use on the platform for free. Of course we try to keep our costs as low as possible.
  - pling 12y ago
    What I'm saying is that your product is a fit for the platform when it is an MVP. When you grow, you're locked into an expensive platform. In the short term there is no option of escape because the cost to port it becomes increasingly more expensive. Then before you know it, you're at that 200k. I've seen it happen at so many companies: "We'll build it on the MS stack because they gave us a great deal on MSDN or we used the PAP to get cheap licenses." ... after a few years ... "We got screwed by SQL 2012 and had to buy £50k of SQL licenses because the terminology between cores and CPUs changed on a whim and it's the ass-end of the SQL 2005 so we've got to make a big jump." I'm not saying it's an invalid approach (hell I make my living out of keeping these sorts of cogs spinning) but it's not a great foundation to run a startup on. Unpredictable costs or sudden price-jacking after a partial buyout really sours the shit out of the deal. Especially when you ask for a budget you can't get from your new investors who now are worried that their short term return is going out of the window on your platform and software license costs.
    - cpard 12y ago
      that's why we try to use the Azure IaaS and avoid microsoft specific technologies. In any case as we grow we'll have to also move to other enviroments because of the nature of our product so we need to be flexible on that. But I totally agree with you that you need to be carefull on how much locked you get on a specific environment that you dont have total control over it.
campbellmorgan 12y ago
Good article and particularly useful on the Galera replication - if you google replication, the tutorials that come up first are native mysql Master-Master replication which are difficult to setup and very likely to fail. I've been on azure for almost a year now and it is essential that you use their availability sets as their emails about "planned maintenance" often come sporadically after they do the maintenance. In one upgrade they somehow managed to shutdown and not restart my instances, so i also recommend having an externally hosted monitoring server if you have mission-critical stuff on azure.
- cpard 12y ago
  thank you! The whole point of the article was to offer a mashup of easy to follow steps to achieve replication over azure availability sets. I'm glad that you find it useful.

36 points by cpard 12y ago

znep 12y ago

Keep in mind that although availability sets can help reduce the chances that two VMs will go down at once, you do have to deal with the maintenance cycles that Azure has where they will reboot every one of your VMs over the course of x hours. This means that with a master/slave setup, a simple failover from a master to a slave is not sufficient, you need to be able to fail back automatically as well. Depending on the exact technologies you are using and how you are using them, this can have a lot of tricky corner cases that can end up much worse. If you don't do this, then when your master comes back up and they decide to reboot your slave next, which is a common scenario, you lose availability. This differs significantly from a scenario where VM failures/reboots aren't correlated so closely in time; if a VM failure is more random, it is more reasonable to require manual intervention to fail back to the master.

dbarlett 12y ago
Seconded. You're lucky to get clean ACPI shutdowns, half the time Azure "pulls the plug". The closest thing to an official explanation I've found (note date and chorus of comments): http://blogs.msdn.com/b/wats/archive/2013/09/24/windows-azure-virtual-machine-restarted-or-shutdown-with-out-any-notification.aspx http://blogs.msdn.com/b/wats/archive/2013/09/24/windows-azur...
cpard 12y ago
true, that's why we went for a master-master setup with the db.
ultimape 12y ago
Would a Quorum ( http://en.wikipedia.org/wiki/Quorum_(distributed_computing) http://en.wikipedia.org/wiki/Quorum_(distributed_computing) ) method - like done with Ceph's monitoring system agents - be able to handle this? I mean, you still may end up with the split-brain problem in the long run, but if that happens within a data-center would be the least of your worries. (Something like netflix's chaos army comes to mind for dealing with that).

NicoJuicy 12y ago

I'm kinda curious why you picked MariaDB instead of SQL Server ? What are the benefits?

cpard 12y ago
because of cost. MariaDB helps us to reduce cost, keep in mind that our storage needs are minimal. In any case if you can afford SQL server, you should go for it.
- NicoJuicy 12y ago
  Any downsides of using MySQL instead of SQL Server in the development stage?
  - cpard 12y ago
    I'd say that the main downside is the time/effort to setup and maintain, especially as you scale up. At the end you should always judge based on your app and its requirements.
- spo81rty 12y ago
  SQL Azure starts at $5 a month. Hosting a couple linux VMs for MariaDB would be a lot more expensive. SQL Azure is extremely affordable even as you scale up and it is fully managed. For basic SQL usage it is awesome and the value is unbeatable.
  - cpard 12y ago
    if you are in the situation to need dedicaded VMs for MariaDB then you should probably go for SQL Azure. Here this is not the case, the requirements for the DB are minimal and we already have 2 VMs running for the availability set. As I said, at the end you need to judge based on the specific needs of your app/product.
  - bad_user 12y ago
    Actually they are retiring the "Web and Business" pricing plans, which were charging for storage size and introducing a new pricing plan that charges for both storage size and "Database Throughput Units", which kind of obscures what they are charging for - something related to transactions volume and based on my experience with other "cloud hosted" databases, I can guarantee you that it gets pretty freaking expensive. The smartest thing to do is to never rely on a particular service of a cloud provider. For example we once made the choice of using Amazon's DynamoDB, which is great in the beginning, but at some points along with scale issues the costs went through the roof (DynamoDB also charges for "throughput capacity") and we just switched to Cassandra - but we planned for this migration ever since the beginning. And this is the problem with cloud providers - the pricing schemes on these services are designed to hook you with affordable prices in the beginning, but later on as your business grows they get much more expensive than hosting your own stuff. So use whatever makes you most productive, but careful about lock-in.

pling 12y ago

This could also be titled "how to enslave your MVP to increasing costs in the future". Azure isn't cheap by any means when you need to scale up our out.

cpard 12y ago
pling sure, but if you avoid using azure specific technologies it's quite easy to move somewhere else.
ssiddharth 12y ago
Simply decouple yourself from Azure specific tech and you're good to go, no?
- pling 12y ago
  It is very hard to decouple yourself from the tooling and the platform specific features. The PaaS offering literally requires half of the additional services ad the basic deployment is simple and the IaaS offering is expensive and you need the latter to decouple effectively. Either way you also hang yourself with a vendor tied product and the ball and chain of licensing. The latter of which escalates pretty quick when your business grows. Our 8 man startup in 2005 now spends £200k a year on licensing because we're that tied to the damn platform.
  - cpard 12y ago
    pling, any chance to post/share how you are using the azure platform and what makes the migration hard? that would be really useful.
    - pling 12y ago
      We're not using it other than for internal failovers. We tried to build version two of our product on it. The costs increased rapidly until it was cheaper to buy a 42U and stuff it with HP kit and pay for a fat pipe. The storage was expensive, the per transaction costs are expensive, the bandwidth is expensive, the IaaS machines are expensive and the support is expensive when it goes wrong which it does occasionally and violently. This was all financial data aggregation and archival stuff. In test it was 23% more expensive than our own kit over 3 years.

NicoJuicy 12y ago

I'm wondering.. You said, build MVP on a budget... But your licensing costs are insanely high (200k pounds)... Isn't that contradictive? (or was that the licensing costs before the current situation?)

cpard 12y ago
That's not our cost, the 200k pounds are the cost pling mentioned for their case, 200k and MVP are indeed contradictory terms. We are actually participating on bizpark so we have some credits to use on the platform for free. Of course we try to keep our costs as low as possible.
- pling 12y ago
  What I'm saying is that your product is a fit for the platform when it is an MVP. When you grow, you're locked into an expensive platform. In the short term there is no option of escape because the cost to port it becomes increasingly more expensive. Then before you know it, you're at that 200k. I've seen it happen at so many companies: "We'll build it on the MS stack because they gave us a great deal on MSDN or we used the PAP to get cheap licenses." ... after a few years ... "We got screwed by SQL 2012 and had to buy £50k of SQL licenses because the terminology between cores and CPUs changed on a whim and it's the ass-end of the SQL 2005 so we've got to make a big jump." I'm not saying it's an invalid approach (hell I make my living out of keeping these sorts of cogs spinning) but it's not a great foundation to run a startup on. Unpredictable costs or sudden price-jacking after a partial buyout really sours the shit out of the deal. Especially when you ask for a budget you can't get from your new investors who now are worried that their short term return is going out of the window on your platform and software license costs.
  - cpard 12y ago
    that's why we try to use the Azure IaaS and avoid microsoft specific technologies. In any case as we grow we'll have to also move to other enviroments because of the nature of our product so we need to be flexible on that. But I totally agree with you that you need to be carefull on how much locked you get on a specific environment that you dont have total control over it.

campbellmorgan 12y ago

Good article and particularly useful on the Galera replication - if you google replication, the tutorials that come up first are native mysql Master-Master replication which are difficult to setup and very likely to fail. I've been on azure for almost a year now and it is essential that you use their availability sets as their emails about "planned maintenance" often come sporadically after they do the maintenance. In one upgrade they somehow managed to shutdown and not restart my instances, so i also recommend having an externally hosted monitoring server if you have mission-critical stuff on azure.

cpard 12y ago
thank you! The whole point of the article was to offer a mashup of easy to follow steps to achieve replication over azure availability sets. I'm glad that you find it useful.