5 ms·

Microsoft is here to stay mainly because once you got your foot in the door at the big companies you will stay forever. The reasoning is indeed since we already

Microsoft is here to stay mainly because once you got your foot in the door at the big companies you will stay forever. The reasoning is indeed since we already have Microsoft guys Azure will fit us well but in practice none of these Microsoft guys will be able to help you on any Azure issue. So when Azure is involved in a big corp, accenture is usually not far behind. The experience with the solutions this doom duo comes up with are absolute hell to deal with. In upfront cost Azure looks better but in general that's rarely the case. All the azure API's seem half baked. Once you're doing anything more advanced you will run into issues, just look at the terraform azure provider issue tracker for a bunch of issues that people run into because it's not clear until you actually try out the apis. Here's another example if you want to use shared storage on kubernetes with any reasonable iops, the azurefile premium storage increases IOPS per Gigabyte allocated. So if you want any kind of reasonable experience/price you have to easier spin up your own nfs server, use azure netapp or allocate 10TB shared premium filesystem per share, which is something like 70k a year.

oblio 7y ago
I like AWS, but AWS EFS has the same problem. They've improved it a bit through some recent changes, but it's not much better. The way it would work: they gave you absolutely pitiful base IOPS credits for EFS and everything else was related to disk space used. So more disk space used (and paid), more IOPS. After that they'd completely detroy your IOPS if you used up all the credits. By destroy I mean IOPS at the level of a HDD from 1995. I set up a Jenkins using EFS and initially it went well. It barely had any activity and after about 2 weeks it used up all the credits. After that even the login page would take 20 seconds to load.
- sciurus 7y ago
  I think it's throughput credits that EFS gives you (e.g r/w MiB/s), not IOPS. AFAIK they don't document the IOPS available at all. In my experience the latency for an individual i/o operation on EFS is always at the "HDD from 1995" level regardless of available burst credits. Something that does lots of small random I/O like checking out got repos on Jenkins workers is basically worst case for EFS. https://docs.aws.amazon.com/efs/latest/ug/performance.html https://docs.aws.amazon.com/efs/latest/ug/performance.html
  - oblio 7y ago
    Throughput credits, you're right, my bad. The agents were in ECS with no persistent storage, so that wasn't the problem. I was just running the Jenkins master off of EFS, for the persistent configuration storage. And I don't think it's the latency that's killing EFS usage, it's the throughput. While the credits were there, everything went smoothly, once the credits ran out, the base throughput was fit for IO meant for the 90s.
    - pnutjam 7y ago
      We had the same issue with a pgsql server. Started out fine, but to get decent performance you pay out the nose for higher disk throughput. It looks competitive when your pricing things out and don't know you need to pay for that. When you find out it's a classic sunk cost fallacy and most companies just eat the cost.
  - angry_octet 7y ago
    It's NFS, so the bad latency isn't surprising. The problem is that they don't have anything faster -- it tops out at 2GBy/s or something, even with hundreds of TB, even with multiple clients. You have to share your data over multiple EFS volumes, or build your own virtual gluster, which are extremely shit options. Also makes any kind of bug data HPC impractical. Bezos, if you're listening, fire someone. You should have next generation pNFS or lustre like protocols by 2016.
    - sciurus 7y ago
      They actually do have https://aws.amazon.com/fsx/lustre/ https://aws.amazon.com/fsx/lustre/ Latency of EFS is much worse than running your own NFS in my experience.
      - angry_octet 7y ago
        Doh, how did I not find that? But also, how does the pricing work? It seems to be half the price of EFS? It almost seems like that are assuing S3 read or Direct Connect to populate the FSx volume.
spydum 7y ago
That sounds like exactly like AWS though doesn't it? EFS iops scale with data size allocation.
audleman 7y ago
To be fair, Microsoft has only decided on Kubernetes within the last year. Before that there was heated debate on whether to support K8 or ServiceFabric, their "competing" standard. Now that all efforts are on K8 we'll see it improve pretty quickly.

oblio 7y ago

I like AWS, but AWS EFS has the same problem. They've improved it a bit through some recent changes, but it's not much better. The way it would work: they gave you absolutely pitiful base IOPS credits for EFS and everything else was related to disk space used. So more disk space used (and paid), more IOPS. After that they'd completely detroy your IOPS if you used up all the credits. By destroy I mean IOPS at the level of a HDD from 1995. I set up a Jenkins using EFS and initially it went well. It barely had any activity and after about 2 weeks it used up all the credits. After that even the login page would take 20 seconds to load.

sciurus 7y ago
I think it's throughput credits that EFS gives you (e.g r/w MiB/s), not IOPS. AFAIK they don't document the IOPS available at all. In my experience the latency for an individual i/o operation on EFS is always at the "HDD from 1995" level regardless of available burst credits. Something that does lots of small random I/O like checking out got repos on Jenkins workers is basically worst case for EFS. https://docs.aws.amazon.com/efs/latest/ug/performance.html https://docs.aws.amazon.com/efs/latest/ug/performance.html
- oblio 7y ago
  Throughput credits, you're right, my bad. The agents were in ECS with no persistent storage, so that wasn't the problem. I was just running the Jenkins master off of EFS, for the persistent configuration storage. And I don't think it's the latency that's killing EFS usage, it's the throughput. While the credits were there, everything went smoothly, once the credits ran out, the base throughput was fit for IO meant for the 90s.
  - pnutjam 7y ago
    We had the same issue with a pgsql server. Started out fine, but to get decent performance you pay out the nose for higher disk throughput. It looks competitive when your pricing things out and don't know you need to pay for that. When you find out it's a classic sunk cost fallacy and most companies just eat the cost.
- angry_octet 7y ago
  It's NFS, so the bad latency isn't surprising. The problem is that they don't have anything faster -- it tops out at 2GBy/s or something, even with hundreds of TB, even with multiple clients. You have to share your data over multiple EFS volumes, or build your own virtual gluster, which are extremely shit options. Also makes any kind of bug data HPC impractical. Bezos, if you're listening, fire someone. You should have next generation pNFS or lustre like protocols by 2016.
  - sciurus 7y ago
    They actually do have https://aws.amazon.com/fsx/lustre/ https://aws.amazon.com/fsx/lustre/ Latency of EFS is much worse than running your own NFS in my experience.
    - angry_octet 7y ago
      Doh, how did I not find that? But also, how does the pricing work? It seems to be half the price of EFS? It almost seems like that are assuing S3 read or Direct Connect to populate the FSx volume.

spydum 7y ago

That sounds like exactly like AWS though doesn't it? EFS iops scale with data size allocation.

audleman 7y ago

To be fair, Microsoft has only decided on Kubernetes within the last year. Before that there was heated debate on whether to support K8 or ServiceFabric, their "competing" standard. Now that all efforts are on K8 we'll see it improve pretty quickly.