3 ms·

I'm also here because my work is blocked by the inability to do anything on Azure right now. I looked at their past status histories, and they have provided pr

I'm also here because my work is blocked by the inability to do anything on Azure right now. I looked at their past status histories, and they have provided pretty detailed postmortems for much more minor incidents. I don't have any reason to believe that they won't do that this time. I'm new to Azure, so this kind of outage really looks bad in my mind, but it's par for the course with Cloud stuff. They'll fix it and there will be one less thing to break in the future.

spondyl 6y ago
As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions. Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus. The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.
- NDizzle 6y ago
  Just wait until you see their made up rules for using their Redis implementation. You have to roll your own Redis to do anything interesting.
  - zanecodes 6y ago
    Their managed PostgreSQL offering also has fun quirks, such as that you have to connect using `username@db_name`, but your actual username in the database is still just `username`, so any third party software needs to support using a different username to connect than it uses to perform user-related queries (they have some sort of application-level load balancer in front of it that uses `db_name` to route the connection).
    - bombcar 6y ago
      This is ... something - wow - they couldn't just make the actual username "username@db_name"
      - zanecodes 6y ago
        Nope. Their load balancer seems to strip the `@db_name` from connections, and anyway some software doesn't like having an @ in Postgres usernames (which is probably also a sign it's vulnerable to SQL injections)
    - llama052 6y ago
      Oh that solution is a garbage fire. We hit the 4tb storage limit that they didn't tell us about on it when the documentation said 16tb. They told us we were on the older storage tier and that we needed to rebuild a huge analytics database on our time because they couldn't migrate us to a backend storage tier. So we had a database server that was in read only mode until we could migrate off of it. Then we had random restarts and outages with their proxy that handles all of the connections. Which is why you actually need the username@db-name, because without the @db-name it doesn't know where to route you, because the postgres connection string is actually just mapping to one IP that they manage for multiple databases! Fun trick is you can make your url anything that maps to the IP they use for their proxy and it'll take you to whatever the @db-name is in your connection string. Their solution to the random connections dropping for minutes at a time was to "implement retry logic" on our applications. Our solution was to migrate away from it.
    - thrixton 6y ago
      We were well into implementation of a SaaS that relied on PG and used their cloud offering before we found that the latency was just abysmal. The best we could get was ~45ms using pgbench vs ~10ms local and ~18 using AWS Aurora. Their solution, wait for the flexible server offering which is not available in AUS but when tested in another region showed ~30ms so still not fit for purpose. We eventually had to switch to running PG in AKS which has its pros and cons. The funniest thing was, running Azure VM to AWS PG was faster than running Azure VM to Azure PG.
asabla 6y ago
Been an active user of Azure for almost five or six years now. Things were a lot more rough in then (especially if you'd seen the old portal), unlike what it is now. People always talk about cloud as an entity which can/shouldn't have problems at all. When it's more about convenience then anything else. So I wouldn't worry too much about this if I were you.
- deathanatos 6y ago
  I don't particularly expect the cloud to be problem free. I do expect it to be possible to, when an issue is caused by an outage or a service issue, get that escalated to actual engineers. That has not been easy, IMO. Even obvious stuff, like "we sent X request around Y time and got Z 500 Internal Server Error from service S" have required multiple round-trips.¹ Messaging around transient issues helps. E.g., AWS would often email us to tell us "hey, sorry, such & such VM is being hosed by the underlying hardware, it is migrating but you'll see a forced reboot". That's all I need to know: I expect some interruption to that portion of the system, and I expect to see it heal within a certain time frame, and if it does, great, no support ticket required. Azure sort of has that with "Resource health"; but, e.g., one of our recent support tickets is that we had a VM reboot unexpectedly due to a resource health event with its disks. And then, a few hours later, another VM, same resource health event, same issue, same reboot. And then again, a few hours later. And that pattern, with no further communication, requires me to write the (obvious) ticket of, "So… what's happening?" (That ticket ended with such and such service was experiencing an incident. Never made the status page. It did make the internal "Service alerts" page.) Part of issuing a PM is also to help show & convince me, the customer, that I won't be filing support issues for the rest of my life. But this last 6 months has just felt like support ticket after support ticket, and it's sort of depressing, since I'd rather be coding. I do think, last spring or so, it was no nearly this bad. ¹Every place I've worked at… we page on 5xx codes sent to a client. If Azure is internally doing that, it sure is hard to tell from where I sit.

spondyl 6y ago

As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions. Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus. The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.

NDizzle 6y ago
Just wait until you see their made up rules for using their Redis implementation. You have to roll your own Redis to do anything interesting.
- zanecodes 6y ago
  Their managed PostgreSQL offering also has fun quirks, such as that you have to connect using `username@db_name`, but your actual username in the database is still just `username`, so any third party software needs to support using a different username to connect than it uses to perform user-related queries (they have some sort of application-level load balancer in front of it that uses `db_name` to route the connection).
  - bombcar 6y ago
    This is ... something - wow - they couldn't just make the actual username "username@db_name"
    - zanecodes 6y ago
      Nope. Their load balancer seems to strip the `@db_name` from connections, and anyway some software doesn't like having an @ in Postgres usernames (which is probably also a sign it's vulnerable to SQL injections)
  - llama052 6y ago
    Oh that solution is a garbage fire. We hit the 4tb storage limit that they didn't tell us about on it when the documentation said 16tb. They told us we were on the older storage tier and that we needed to rebuild a huge analytics database on our time because they couldn't migrate us to a backend storage tier. So we had a database server that was in read only mode until we could migrate off of it. Then we had random restarts and outages with their proxy that handles all of the connections. Which is why you actually need the username@db-name, because without the @db-name it doesn't know where to route you, because the postgres connection string is actually just mapping to one IP that they manage for multiple databases! Fun trick is you can make your url anything that maps to the IP they use for their proxy and it'll take you to whatever the @db-name is in your connection string. Their solution to the random connections dropping for minutes at a time was to "implement retry logic" on our applications. Our solution was to migrate away from it.
  - thrixton 6y ago
    We were well into implementation of a SaaS that relied on PG and used their cloud offering before we found that the latency was just abysmal. The best we could get was ~45ms using pgbench vs ~10ms local and ~18 using AWS Aurora. Their solution, wait for the flexible server offering which is not available in AUS but when tested in another region showed ~30ms so still not fit for purpose. We eventually had to switch to running PG in AKS which has its pros and cons. The funniest thing was, running Azure VM to AWS PG was faster than running Azure VM to Azure PG.

asabla 6y ago

Been an active user of Azure for almost five or six years now. Things were a lot more rough in then (especially if you'd seen the old portal), unlike what it is now. People always talk about cloud as an entity which can/shouldn't have problems at all. When it's more about convenience then anything else. So I wouldn't worry too much about this if I were you.

deathanatos 6y ago
I don't particularly expect the cloud to be problem free. I do expect it to be possible to, when an issue is caused by an outage or a service issue, get that escalated to actual engineers. That has not been easy, IMO. Even obvious stuff, like "we sent X request around Y time and got Z 500 Internal Server Error from service S" have required multiple round-trips.¹ Messaging around transient issues helps. E.g., AWS would often email us to tell us "hey, sorry, such & such VM is being hosed by the underlying hardware, it is migrating but you'll see a forced reboot". That's all I need to know: I expect some interruption to that portion of the system, and I expect to see it heal within a certain time frame, and if it does, great, no support ticket required. Azure sort of has that with "Resource health"; but, e.g., one of our recent support tickets is that we had a VM reboot unexpectedly due to a resource health event with its disks. And then, a few hours later, another VM, same resource health event, same issue, same reboot. And then again, a few hours later. And that pattern, with no further communication, requires me to write the (obvious) ticket of, "So… what's happening?" (That ticket ended with such and such service was experiencing an incident. Never made the status page. It did make the internal "Service alerts" page.) Part of issuing a PM is also to help show & convince me, the customer, that I won't be filing support issues for the rest of my life. But this last 6 months has just felt like support ticket after support ticket, and it's sort of depressing, since I'd rather be coding. I do think, last spring or so, it was no nearly this bad. ¹Every place I've worked at… we page on 5xx codes sent to a client. If Azure is internally doing that, it sure is hard to tell from where I sit.