6 ms·

As a network engineer writing code to automate aws and azure infrastructure I'm amazed at the number of pitfalls we've had with azure, they just keep popping up

As a network engineer writing code to automate aws and azure infrastructure I'm amazed at the number of pitfalls we've had with azure, they just keep popping up. Then the whole basic/standard options for every service. Feels like an order of magnitude higher in complexity for no reason, part of it is the abysmal documentation.

yoyohello13 5y ago
I find this to be a common theme with Microsoft products. I work heavily with SSIS and MS SQL. The Microsoft docs are so convoluted they are borderline useless for learning the products.
- oneplane 5y ago
  Same here, and not even GCP is that bad with the 'release' vs. 'beta' APIs that intermingle constantly. Most of the time Azure feels like just another 'we have virtual machines and a crappy API' service. Kinda like a pretend-cloud where a lot of products come almost together but never finish. Almost similar to the way Windows and backwards compatibility means you end up with 100 libraries, frameworks, languages and versions all sitting side-by-side and not really working together, just differently in parallel. Not useful for automation at all, which makes it not useful at scale. (Except when scaling means: we want to run windows VMs with AD and go from 10 to 100 and change no parameters at all)
  - spaetzleesser 5y ago
    My theory is that all MS products are ultimately just a layer based on SharePoint :)
    - oneplane 5y ago
      That reminds me of a relatively old twitter thread (that I can't find of course) about systems that have become operating systems and computers upon themselves. It went something like this: Code used to be running on a processor, then it ran on an OS on a processor. Then it became on a runtime on an OS on a processor. Then it an abstraction layer was sandwiched in between. Then a filesystem. Then a compatibility layer. Then a database. Then a browser. Then we went and created a runtime in the browser to run code again. We have come full circle. This probably applies to SharePoint but also the real-time OS on your GPU, the MSSQL database which has its own OS facilities you can run stuff on. Add too many features and the application becomes the very thing we wanted to get away from...
- FridgeSeal 5y ago
  My friend and I have a theory that this is because their docs are written by people so intimately familiar with how MS does things, they can no longer conceive of a situation where someone doesn’t already understand and think in the same way. It’s like the business equivalent of the “once you understand a monad you can no longer explain it” meme.
  - sli 5y ago
    It's Microsoft's modern, less-obvious version of EEE. You sell to the managers and equip them with hollow buzzwords because you know they're gonna override the engineers on every decision anyway. By then, MS has your firm's money and all you can do is deal with it.
  - imwillofficial 5y ago
    Or the ole “I tried to explain K8s to a friend, now we both don’t understand it.”
- habeebtc 5y ago
  I find the problems with Microsoft docs is less how convoluted they are (you kind of get used to the convention at some point - not to say it's good or sensical), it's the fact that Azure documentation is not of the same quality that the WinAPI docs were. Used to be a function doc would tell you: 1. All the parameters and their types 2. What they did, explained 3. All possible exceptions raised by the function 4. All possible return values 5. Supplementary documentation on the object or structs passed in or passed back 6. Some examples of it in use Now, the current docs sometimes have examples for Azure CLI/SDK stuff, but there is one convention that drives me bonkers that is as bad now as it ever was. For examples, often times you'll get the most important part replaced with a [insert your thing here]. The format of the thing you fill in there is often left as an exercise to the reader to intuit or guess.
  - billti 5y ago
    Writing one good set of docs that are suitable for all users (experts and newbies) is really hard. Structuring complex topics in a logical and progressive manner is also really hard. Having tried to write the docs on a few software projects, I honestly found writing the docs harder than writing the code (though maybe it gets easier with practice). I find the Azure docs to be decent. They are consistent across services for the most part, so you learn how to work with them after a bit of experience on the platform.
jiggawatts 5y ago
This has been my life for the last year: - Customer asks for something trivial to be deployed. I say, "no problem" and start beavering away on a Bicep template or whatever to deploy their stuff. - I hit some small but showstopper issue with a service that I expected to work, but it doesn't. I open a support ticket. - Inevitably, it turns out to be some stupid, stupid limitation caused by unfathomable laziness of the Azure developers. No workaround, no mitigation, but we're "working on it" with no ETA offered. - Literally years later a trivial fix for the glaring issue goes into "PREVIEW" for 9 months, long after the original project was closed. I no longer care... Networking is especially bad, with endless limitations that make no sense, like: IPv6 is incompatible with everything. Turning it on for anything anywhere in a vNET will permanently block unrelated features like IPv4 NAT. But of course, you can't "protocol translate" from IPv6 on the outside to IPv4 on the inside, so you end up painted into a corner. No bring-your-own-subnet, which means many lift & shift scenarios are impossible (we have customers using a class B pubic range internally). Azure forces NAT on IPv6, which makes no sense at all. All Azure PaaS services have firewalls that are IPv4 only. The built-in firewalls (e.g.: Azure SQL Database) do not support service tags, only CIDRs. I could go on and on...
- Hikikomori 5y ago
  Yes, seemingly trivial changes turns into huge and disruptive changes. Latest one we encountered was that you cannot add more CIDR ranges to a VNET without removing all VNET peering first. The feature for this has been in preview for like 2 years. I mean, AWS doesn't automatically handle this for you, but at least you can solve it easily enough (just use the API to create routes on the other side (their API/boto3 library is just amazing)).
  - jiggawatts 5y ago
    Availability Sets decrease availability, because they force "all or nothing" deallocate-reallocate cycles instead of one-at-a-time changes. You can't move a VM to a different subscription without also moving the entire vNet along with it. (I bet this worked great in the developer's lab with one test VM.) You can't move a VM to a different backup resource without deleting all backups, going back years. VMs can't be rolled back to VM snapshots. ExpressRoute bandwidth can be increased non-disruptively, but the only way to decrease it is to recreate it -- incurring a ~30 minute outage. The Activity Log (and most other audit logs) don't log the identity of the administrator that triggered the action about 10-50% of the time, depending on what kind of activities are going on. This makes it 100% useless as an audit log. There is no other way to obtain audits. Most network resources are Zone Redundant, except for NAT Gateway, which is now required in some scenarios. It's Zonal only, which means you need 3 subnets, one for each zone. Upgrading an internal load balancer from Basic to Standard cuts off Internet access, for "reasons". I'm told these are "security reasons". Uh-huh... App Gateway doesn't use a "user-agent" header for its monitors, which makes it incompatible with a surprisingly wide range of CotS software. This includes a bunch of Microsoft Software. I could go on and on...
deathanatos 5y ago
Yeah. Azure is a second- or third-tier cloud provider; they are not in the same league as AWS. AKS suffers, AFAICT, constant API server outages. We tried to escalate into a ticket, but we just get motte & bailey'd between "you're putting too much load on the API server" — okay, what load? how can I see that, control it? — "here is the top consumers" — they're all AKS itself? — "well, there's too much load on the API server" gah! (Yes, we pay for the "SLA".) You can't add IPv6 anywhere in a vnet, it will break unrelated things. We tried to add a managed PSQL server on IPv4 (b/c IPv6 is not supported): it "failed" (the API call to create it timed out with an internal server error … after 2 hours or so!) because something unrelated in the vnet used IPv6. ACR has a 20 TiB limit, no way to prune containers, we've had to work around IDK how many 500s, the API is slow as dirt (response bandwidths of ~50 kbps — bits — and their team does not think that's a problem. It can take 10 minutes to enumerate a few megabytes of metadata…) Undelete-able manifests that I guess we will just pay for indefinitely? I feel like I could build ACR on top of Azure Blobstorage and it would be more reliable with better performance… VMs shipping with buggy kernels. (Support wanted to know what weird thing we were running to hit kernel bugs. "Docker"?) Global outages. Known outages often don't get mentioned on the status page intentionally. I still don't know what the difference between an availability set and a VMSS is. Everything is in preview. Everything. Audit logs fail to load some times. Beyond complicated interactions/differences between "service principals", "applications", "enterprise applications". 2FA app now requires authenticating twice, per login, because each tenant acts as a separate yet not separate login. So. much. auth. Role assignments that don't know what principal is being granted permission, b/c ARM & AAD don't do referential integrity. Docs that are outdated. Requests for updated docs closed without update, because "we don't have the data <from some other internal team, I think, but so?>"?. Docs that describe API calls badly. "foo: the foo query parameter" No docs, at least, that I'm aware of, about what permissions does the API call require. The docs conflate "permission" and "role" (different in Azure) all. the. time. Azure doesn't know what permissions some calls require, and simply says "give it Contributor" (close to all permissions)… and yeah that works but I want to show auditors we're doing PoLP? Support… The SLA often isn't met, we're assigned reps in China (n.b., this isn't a language barrier problem, it's, how is someone who is literally sleeping during my business hours because that's timezones for you supposed to even meet a support SLA that wants a 8/4/2 hour response? And AFAICT from response times, they're not a night shift…?), the first response is worthless (doesn't answer the inquiry, requests information present in the original request, often isn't technically proficient, etc.), the writing is broken and sloppy. Half the time a simple "reread what you're about to send. Does it solve their problem?"… we literally got a blank email back. We've had tickets where the first response we get is "we haven't heard back from you" because sometimes their responses fail to get linked into the ticket in the portal. They lack any formal bug reporting mechanism, and support is not equipped to handle bugs. My God, it's full of bugs.

yoyohello13 5y ago

I find this to be a common theme with Microsoft products. I work heavily with SSIS and MS SQL. The Microsoft docs are so convoluted they are borderline useless for learning the products.

oneplane 5y ago
Same here, and not even GCP is that bad with the 'release' vs. 'beta' APIs that intermingle constantly. Most of the time Azure feels like just another 'we have virtual machines and a crappy API' service. Kinda like a pretend-cloud where a lot of products come almost together but never finish. Almost similar to the way Windows and backwards compatibility means you end up with 100 libraries, frameworks, languages and versions all sitting side-by-side and not really working together, just differently in parallel. Not useful for automation at all, which makes it not useful at scale. (Except when scaling means: we want to run windows VMs with AD and go from 10 to 100 and change no parameters at all)
- spaetzleesser 5y ago
  My theory is that all MS products are ultimately just a layer based on SharePoint :)
  - oneplane 5y ago
    That reminds me of a relatively old twitter thread (that I can't find of course) about systems that have become operating systems and computers upon themselves. It went something like this: Code used to be running on a processor, then it ran on an OS on a processor. Then it became on a runtime on an OS on a processor. Then it an abstraction layer was sandwiched in between. Then a filesystem. Then a compatibility layer. Then a database. Then a browser. Then we went and created a runtime in the browser to run code again. We have come full circle. This probably applies to SharePoint but also the real-time OS on your GPU, the MSSQL database which has its own OS facilities you can run stuff on. Add too many features and the application becomes the very thing we wanted to get away from...
FridgeSeal 5y ago
My friend and I have a theory that this is because their docs are written by people so intimately familiar with how MS does things, they can no longer conceive of a situation where someone doesn’t already understand and think in the same way. It’s like the business equivalent of the “once you understand a monad you can no longer explain it” meme.
- sli 5y ago
  It's Microsoft's modern, less-obvious version of EEE. You sell to the managers and equip them with hollow buzzwords because you know they're gonna override the engineers on every decision anyway. By then, MS has your firm's money and all you can do is deal with it.
- imwillofficial 5y ago
  Or the ole “I tried to explain K8s to a friend, now we both don’t understand it.”
habeebtc 5y ago
I find the problems with Microsoft docs is less how convoluted they are (you kind of get used to the convention at some point - not to say it's good or sensical), it's the fact that Azure documentation is not of the same quality that the WinAPI docs were. Used to be a function doc would tell you: 1. All the parameters and their types 2. What they did, explained 3. All possible exceptions raised by the function 4. All possible return values 5. Supplementary documentation on the object or structs passed in or passed back 6. Some examples of it in use Now, the current docs sometimes have examples for Azure CLI/SDK stuff, but there is one convention that drives me bonkers that is as bad now as it ever was. For examples, often times you'll get the most important part replaced with a [insert your thing here]. The format of the thing you fill in there is often left as an exercise to the reader to intuit or guess.
- billti 5y ago
  Writing one good set of docs that are suitable for all users (experts and newbies) is really hard. Structuring complex topics in a logical and progressive manner is also really hard. Having tried to write the docs on a few software projects, I honestly found writing the docs harder than writing the code (though maybe it gets easier with practice). I find the Azure docs to be decent. They are consistent across services for the most part, so you learn how to work with them after a bit of experience on the platform.

jiggawatts 5y ago

This has been my life for the last year: - Customer asks for something trivial to be deployed. I say, "no problem" and start beavering away on a Bicep template or whatever to deploy their stuff. - I hit some small but showstopper issue with a service that I expected to work, but it doesn't. I open a support ticket. - Inevitably, it turns out to be some stupid, stupid limitation caused by unfathomable laziness of the Azure developers. No workaround, no mitigation, but we're "working on it" with no ETA offered. - Literally years later a trivial fix for the glaring issue goes into "PREVIEW" for 9 months, long after the original project was closed. I no longer care... Networking is especially bad, with endless limitations that make no sense, like: IPv6 is incompatible with everything. Turning it on for anything anywhere in a vNET will permanently block unrelated features like IPv4 NAT. But of course, you can't "protocol translate" from IPv6 on the outside to IPv4 on the inside, so you end up painted into a corner. No bring-your-own-subnet, which means many lift & shift scenarios are impossible (we have customers using a class B pubic range internally). Azure forces NAT on IPv6, which makes no sense at all. All Azure PaaS services have firewalls that are IPv4 only. The built-in firewalls (e.g.: Azure SQL Database) do not support service tags, only CIDRs. I could go on and on...

Hikikomori 5y ago
Yes, seemingly trivial changes turns into huge and disruptive changes. Latest one we encountered was that you cannot add more CIDR ranges to a VNET without removing all VNET peering first. The feature for this has been in preview for like 2 years. I mean, AWS doesn't automatically handle this for you, but at least you can solve it easily enough (just use the API to create routes on the other side (their API/boto3 library is just amazing)).
- jiggawatts 5y ago
  Availability Sets decrease availability, because they force "all or nothing" deallocate-reallocate cycles instead of one-at-a-time changes. You can't move a VM to a different subscription without also moving the entire vNet along with it. (I bet this worked great in the developer's lab with one test VM.) You can't move a VM to a different backup resource without deleting all backups, going back years. VMs can't be rolled back to VM snapshots. ExpressRoute bandwidth can be increased non-disruptively, but the only way to decrease it is to recreate it -- incurring a ~30 minute outage. The Activity Log (and most other audit logs) don't log the identity of the administrator that triggered the action about 10-50% of the time, depending on what kind of activities are going on. This makes it 100% useless as an audit log. There is no other way to obtain audits. Most network resources are Zone Redundant, except for NAT Gateway, which is now required in some scenarios. It's Zonal only, which means you need 3 subnets, one for each zone. Upgrading an internal load balancer from Basic to Standard cuts off Internet access, for "reasons". I'm told these are "security reasons". Uh-huh... App Gateway doesn't use a "user-agent" header for its monitors, which makes it incompatible with a surprisingly wide range of CotS software. This includes a bunch of Microsoft Software. I could go on and on...

deathanatos 5y ago

Yeah. Azure is a second- or third-tier cloud provider; they are not in the same league as AWS. AKS suffers, AFAICT, constant API server outages. We tried to escalate into a ticket, but we just get motte & bailey'd between "you're putting too much load on the API server" — okay, what load? how can I see that, control it? — "here is the top consumers" — they're all AKS itself? — "well, there's too much load on the API server" gah! (Yes, we pay for the "SLA".) You can't add IPv6 anywhere in a vnet, it will break unrelated things. We tried to add a managed PSQL server on IPv4 (b/c IPv6 is not supported): it "failed" (the API call to create it timed out with an internal server error … after 2 hours or so!) because something unrelated in the vnet used IPv6. ACR has a 20 TiB limit, no way to prune containers, we've had to work around IDK how many 500s, the API is slow as dirt (response bandwidths of ~50 kbps — bits — and their team does not think that's a problem. It can take 10 minutes to enumerate a few megabytes of metadata…) Undelete-able manifests that I guess we will just pay for indefinitely? I feel like I could build ACR on top of Azure Blobstorage and it would be more reliable with better performance… VMs shipping with buggy kernels. (Support wanted to know what weird thing we were running to hit kernel bugs. "Docker"?) Global outages. Known outages often don't get mentioned on the status page intentionally. I still don't know what the difference between an availability set and a VMSS is. Everything is in preview. Everything. Audit logs fail to load some times. Beyond complicated interactions/differences between "service principals", "applications", "enterprise applications". 2FA app now requires authenticating twice, per login, because each tenant acts as a separate yet not separate login. So. much. auth. Role assignments that don't know what principal is being granted permission, b/c ARM & AAD don't do referential integrity. Docs that are outdated. Requests for updated docs closed without update, because "we don't have the data <from some other internal team, I think, but so?>"?. Docs that describe API calls badly. "foo: the foo query parameter" No docs, at least, that I'm aware of, about what permissions does the API call require. The docs conflate "permission" and "role" (different in Azure) all. the. time. Azure doesn't know what permissions some calls require, and simply says "give it Contributor" (close to all permissions)… and yeah that works but I want to show auditors we're doing PoLP? Support… The SLA often isn't met, we're assigned reps in China (n.b., this isn't a language barrier problem, it's, how is someone who is literally sleeping during my business hours because that's timezones for you supposed to even meet a support SLA that wants a 8/4/2 hour response? And AFAICT from response times, they're not a night shift…?), the first response is worthless (doesn't answer the inquiry, requests information present in the original request, often isn't technically proficient, etc.), the writing is broken and sloppy. Half the time a simple "reread what you're about to send. Does it solve their problem?"… we literally got a blank email back. We've had tickets where the first response we get is "we haven't heard back from you" because sometimes their responses fail to get linked into the ticket in the portal. They lack any formal bug reporting mechanism, and support is not equipped to handle bugs. My God, it's full of bugs.