8 ms·

Here's a fun fact about Azure Kubernetes: 1. Deploy your Linux service on k8s with redundant nodes 2. Create a k8s VolumeClaim and mount it on your nodes to g

Here's a fun fact about Azure Kubernetes: 1. Deploy your Linux service on k8s with redundant nodes 2. Create a k8s VolumeClaim and mount it on your nodes to give your application some long-lived or shared disk storage (i.e. for processing user-uploaded files). 3. Wait until the subtle bugs start to appear in your app. Because persistent k8s volumes on Azure are provided by Azure disk storage service behind the scenes, lots of weird Windows-isms apply. And this goes beyond stuff like case insensitivity for file names. For example, if a user tries to upload a file called "COM1" or "PRN1", it will blow up with a disk write error. Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!

AaronFriel 8y ago
Is this with the managed-disk volume (which IIRC is formatted ext4) or with `AzureFiles`, which is essentially SMB/CIFS?
colemickens 8y ago
The only way this would make any sense is if you were using Azure Files, rather than Azure Disks. There's virtually never a time when it makes sense to use Azure Files over Azure Disks, and even when it does, a change in the application would be better advised than using Azure Files. >Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018! This is hyperbolic bordering on flatly false. This is more reasonable and accurate: "Azure is the only cloud vendor that serves their Samba product from Windows boxes, and thus leak Win/NTFS-isms into their Samba shares [that shouldn't be used anyway]." How would an ext4 filesystem, mounted under Linux, attached as a block device to a VM, be subjected to Windows-isms? What you're implying doesn't even make sense.
- parasubvert 8y ago
  I really would have to disagree with the statement one should never use Azure Files over Azure disks. 1. Most Azure VM types have very stringent limits on attached disks; a K8s worker can easily blow past this limit. 2. You have tremendous complexity to deal with: pick Azure managed disks vs unmanaged disks on storage accounts (you can’t mix them on the same cluster). You have to understand the trade of of standard vs premium storage and how they bill (premium rounds up and charges by capacity, not consumption). And you need the right VM types for premium. 3. Managed disks each create a resource object in your resource group. A resource group last I checked had hard limits on the number of resources (like 4000?). Each VM is minimum 3 to 4 resources (with a NIC, image, and disk)... at scale this gets difficult. 4. Azure disks require significant time to create. , mount and remount. A StatefulSet pod failure will sometimes take 3-5 minutes for it’s PV to move to a different worker. And worse when your Azure region has allocation problems. Azure files are near instantaneous unmoubt/remount. 5. Azure disks are block storage and thus only ReadWriteOnce. Azure files are RWM. So, sure, if you’re running a cluster database with dedicated per node PVs and limited expected redeployments... use Azure disks. If you need a PV for any other reason... especially for application tiers that churn frequently.. use Azure files.
  - colemickens 8y ago
    This is all true, I sort of forget that I still have a sort of Azure Stockholm syndrome. I'd say it's good feedback but it's nothing Azure doesn't know about. Maybe Azure Files performance has improved to the point where it's more usable for storage scenarios. I suppose it probably comes down to the use case and application behavior. It would be good if Azure had someone testing out these scenarios and interfacing with the larger k8s community, maybe through the SIG, for these sorts of musings and questions.
manigandham 8y ago
You're not using Azure Disks because they are attached to your VM as a block device and have no knowledge of the file system. PVs in AKS using Azure Disks can only be attached to a single node, as clearly stated in the documentation: https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynam... >> An Azure disk can only be mounted with Access mode type ReadWriteOnce, which makes it available to only a single AKS node. If needing to share a persistent volume across multiple nodes, consider using Azure Files. So you must be using a file share across multiple nodes using Azure Files, which is a SMB file share service that may have compatibility issues with the Samba protocol as described in the (arguably hard to find) docs: https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-shares--directories--files--and-metadata https://docs.microsoft.com/en-us/rest/api/storageservices/na... >> Directory and file names are case-preserving and case-insensitive. >> The following file names are not allowed: LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, PRN, AUX, NUL, CON, CLOCK$, dot character (.), and two dot characters (..).

AaronFriel 8y ago

Is this with the managed-disk volume (which IIRC is formatted ext4) or with `AzureFiles`, which is essentially SMB/CIFS?

colemickens 8y ago

The only way this would make any sense is if you were using Azure Files, rather than Azure Disks. There's virtually never a time when it makes sense to use Azure Files over Azure Disks, and even when it does, a change in the application would be better advised than using Azure Files. >Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018! This is hyperbolic bordering on flatly false. This is more reasonable and accurate: "Azure is the only cloud vendor that serves their Samba product from Windows boxes, and thus leak Win/NTFS-isms into their Samba shares [that shouldn't be used anyway]." How would an ext4 filesystem, mounted under Linux, attached as a block device to a VM, be subjected to Windows-isms? What you're implying doesn't even make sense.

parasubvert 8y ago
I really would have to disagree with the statement one should never use Azure Files over Azure disks. 1. Most Azure VM types have very stringent limits on attached disks; a K8s worker can easily blow past this limit. 2. You have tremendous complexity to deal with: pick Azure managed disks vs unmanaged disks on storage accounts (you can’t mix them on the same cluster). You have to understand the trade of of standard vs premium storage and how they bill (premium rounds up and charges by capacity, not consumption). And you need the right VM types for premium. 3. Managed disks each create a resource object in your resource group. A resource group last I checked had hard limits on the number of resources (like 4000?). Each VM is minimum 3 to 4 resources (with a NIC, image, and disk)... at scale this gets difficult. 4. Azure disks require significant time to create. , mount and remount. A StatefulSet pod failure will sometimes take 3-5 minutes for it’s PV to move to a different worker. And worse when your Azure region has allocation problems. Azure files are near instantaneous unmoubt/remount. 5. Azure disks are block storage and thus only ReadWriteOnce. Azure files are RWM. So, sure, if you’re running a cluster database with dedicated per node PVs and limited expected redeployments... use Azure disks. If you need a PV for any other reason... especially for application tiers that churn frequently.. use Azure files.
- colemickens 8y ago
  This is all true, I sort of forget that I still have a sort of Azure Stockholm syndrome. I'd say it's good feedback but it's nothing Azure doesn't know about. Maybe Azure Files performance has improved to the point where it's more usable for storage scenarios. I suppose it probably comes down to the use case and application behavior. It would be good if Azure had someone testing out these scenarios and interfacing with the larger k8s community, maybe through the SIG, for these sorts of musings and questions.

manigandham 8y ago

You're not using Azure Disks because they are attached to your VM as a block device and have no knowledge of the file system. PVs in AKS using Azure Disks can only be attached to a single node, as clearly stated in the documentation: https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynam... >> An Azure disk can only be mounted with Access mode type ReadWriteOnce, which makes it available to only a single AKS node. If needing to share a persistent volume across multiple nodes, consider using Azure Files. So you must be using a file share across multiple nodes using Azure Files, which is a SMB file share service that may have compatibility issues with the Samba protocol as described in the (arguably hard to find) docs: https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-shares--directories--files--and-metadata https://docs.microsoft.com/en-us/rest/api/storageservices/na... >> Directory and file names are case-preserving and case-insensitive. >> The following file names are not allowed: LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, PRN, AUX, NUL, CON, CLOCK$, dot character (.), and two dot characters (..).