4 ms·

When I worked at Jet, a shopping website trying to compete with Amazon, we obviously did not want to give money to Amazon, so we used Azure. For the most part

When I worked at Jet, a shopping website trying to compete with Amazon, we obviously did not want to give money to Amazon, so we used Azure. For the most part it was just fine, until we started using CosmosDB (then called DocumentDB). DocumentDB, in its first incarnation, was utterly terrible. The pricing was extremely hard to predict, so we would end up with ridiculous bills at the end of the week, the provided .NET SDK for it was buggy and horrible, but the very worst part was the WebUI appeared to be directly tied to your particular instance of CosmosDB. Why is this bad? Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! We got into situations where we had to turn off the entire application, just to bump up the resources for Cosmos. It felt like it was a complete amateur hour from Microsoft. My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. If I end up getting some free credits or something, maybe I'll give Azure another go but I would definitely not recommend it right now.

Twirrim 2y ago
> Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!
- tombert 2y ago
  I think it was querying the database to get usage information, and it was blocking the loading of the page as a result. It's been a few years, but if I recall correctly, the .NET API for changing the database provisioning did work, so some people would hack up a quick script to change resources in an emergency. I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.
- vosper 2y ago
  This is how hosted Elasticsearch works today, too. Really mess up your ES? Good luck with the web UI!
- motorest 2y ago
  > What the ever loving heck... seriously?! What's so surprising about that? If your CRUD operations take longer to do, and you're doing those to drive a GUI, of course the GUI will lumber along.
  - BirdieNZ 2y ago
    It's good practice to separate out your control plane and data plane, so in this kind of scenario you can use the control plane freely to manage and scale up the data plane without worrying about the data plane underresourcing affecting your control plane operations. The reverse also applies; by separating them you can have issues with your control plane but not have the database go down.
  - UltraSane 2y ago
    Control plane and data planes should be running on different hardware.
  - Twirrim 2y ago
    It shouldn't lag the web interface. There are common ways to handle this, such as making all resource mutations asynchronous, or if you want to keep it synchronous, ensuring that control processes get guaranteed CPU resources. It is not a particularly challenging thing to do.
- FridgeSeal 2y ago
  Because that would be the smart and sensible way to design a service like that, and those words are unequivocally not associated with Azure in my experience.
motorest 2y ago
> My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. A couple of years ago I stumbled upon a Azure project which started off using the old timey Cosmos DB. Looking at the repository history from those days, I saw a bunch of Entity Framework configurations and navigations and arcane wizardry that would take an engineer months to really figure out. Then there was an update to CosmosSDK, and all that EF noise was replaced by single CRUD operations that took the unserialized object, id and partition key as input. That's it. Worlds of difference, and a simple key-value query takes ~10ms to do. Yes, it's worlds of difference.
- mort96 2y ago
  > Worlds of difference, and a simple key-value query takes ~10ms to do. Unless that query goes over the Internet to another continent, that's a really long time isn't it?
  - maccard 2y ago
    Yes it’s ages but not multi continent long. Us east 1 to us east 2 is about 10-15ms in my experience.
    - mort96 2y ago
      You're right, I exaggerated unnecessarily there. Sorry
      - maccard 2y ago
        No reason to apologise - I assumed and tried to give a better reference. It is an absolute eternity though. A KV lookup is fractions of a microsecond in a managed language like c#. A http request is in the microseconds range on localhost and a smidge more on a performant local network. A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network. It’s an absolute eternity, and when thinking about this stuff and scaling, think “is it worth slowing down the normal case by 1000x to introduce an external service”
noen 2y ago
A team in my org worked with Jet for 2+ years to help y’all scale. It was interesting seeing the biweekly status updates, they basically all started with “This is how Jet.com broke Azure core services this week”. As much as it sucks, this was a deliberate strategy all the way from Satya - every employee knew Azure was a joke, but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.
- naasking 2y ago
  > but the only want to actually fix shit was to get internet scale customers to break it daily and weekly. I don't get it. There's lots of distributed systems theory that could provide a more robust, analytical approach to a scalable architecture. If a system is regularly breaking like this, it sounds like it should be a "back to the drawing board" moment.
  - cinntaile 2y ago
    Back to the drawing board risks delaying your products with several years. Their strategy was probably the right one.
    - naasking 2y ago
      It sounds like their product has been considered crap for years, so they've had the time.
      - FridgeSeal 2y ago
        “We’ve had years to fix it, all the experience and knowledge we’d need, but we’ve tried basically none of it and we’re at a loss as to why our product is still garbage! It’s truly a mystery” - Azure leadership I assume.
- etruong42 2y ago
  My experience working on Google Cloud Platform is similar. Even when the solution is relatively technically simple, there are many organizational impediments and politics because the solution likely involves multiple fiefdoms. The only thing that seems to get the fiefdoms to work on an issue is if a big enough customer calls enough of an attention to the issue so that no leader/fiefdom wants to be blamed in such a "high profile" issue that everyone got their work in gear for. This leads to organizational dysfunction since now everyone understands that there's no point in trying to fix anything until there is enough attention on it, and that is mainly achieved when a big customer complains. I've even had projects that took months of engineering work that would resolve many user complaints about a lack of really basic functionality, and different elements of leadership would block the launch. I can only guess that some fiefdom did not like change and would not state their objection publicly, so I only got stonewalled, and I never got an explanation to throw away months of work. I am as certain as I can be that if a big customer complained loudly enough about the problem that such basic functionality was missing, the organization would demand that the fix be launched.
UltraSane 2y ago
You would assume that the UI process would be running on Very High priority.
GranPC 2y ago
> My understanding is that Cosmos has gotten a lot better For some values of "better", I guess. Performance is still terrible, their data visualization/inspection tools are shameful, their SQL dialect is finicky and has no error reporting beyond "something is wrong with the input value", and their official Python SDK has a race condition bug that can silently clear out your documents when under heavy load. I used to work at a Cosmos-heavy house and I would utter "fucking Cosmos" around 15 times a day.

Twirrim 2y ago

> Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!

tombert 2y ago
I think it was querying the database to get usage information, and it was blocking the loading of the page as a result. It's been a few years, but if I recall correctly, the .NET API for changing the database provisioning did work, so some people would hack up a quick script to change resources in an emergency. I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.
vosper 2y ago
This is how hosted Elasticsearch works today, too. Really mess up your ES? Good luck with the web UI!
motorest 2y ago
> What the ever loving heck... seriously?! What's so surprising about that? If your CRUD operations take longer to do, and you're doing those to drive a GUI, of course the GUI will lumber along.
- BirdieNZ 2y ago
  It's good practice to separate out your control plane and data plane, so in this kind of scenario you can use the control plane freely to manage and scale up the data plane without worrying about the data plane underresourcing affecting your control plane operations. The reverse also applies; by separating them you can have issues with your control plane but not have the database go down.
- UltraSane 2y ago
  Control plane and data planes should be running on different hardware.
- Twirrim 2y ago
  It shouldn't lag the web interface. There are common ways to handle this, such as making all resource mutations asynchronous, or if you want to keep it synchronous, ensuring that control processes get guaranteed CPU resources. It is not a particularly challenging thing to do.
FridgeSeal 2y ago
Because that would be the smart and sensible way to design a service like that, and those words are unequivocally not associated with Azure in my experience.

motorest 2y ago

> My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. A couple of years ago I stumbled upon a Azure project which started off using the old timey Cosmos DB. Looking at the repository history from those days, I saw a bunch of Entity Framework configurations and navigations and arcane wizardry that would take an engineer months to really figure out. Then there was an update to CosmosSDK, and all that EF noise was replaced by single CRUD operations that took the unserialized object, id and partition key as input. That's it. Worlds of difference, and a simple key-value query takes ~10ms to do. Yes, it's worlds of difference.

mort96 2y ago
> Worlds of difference, and a simple key-value query takes ~10ms to do. Unless that query goes over the Internet to another continent, that's a really long time isn't it?
- maccard 2y ago
  Yes it’s ages but not multi continent long. Us east 1 to us east 2 is about 10-15ms in my experience.
  - mort96 2y ago
    You're right, I exaggerated unnecessarily there. Sorry
    - maccard 2y ago
      No reason to apologise - I assumed and tried to give a better reference. It is an absolute eternity though. A KV lookup is fractions of a microsecond in a managed language like c#. A http request is in the microseconds range on localhost and a smidge more on a performant local network. A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network. It’s an absolute eternity, and when thinking about this stuff and scaling, think “is it worth slowing down the normal case by 1000x to introduce an external service”

noen 2y ago

A team in my org worked with Jet for 2+ years to help y’all scale. It was interesting seeing the biweekly status updates, they basically all started with “This is how Jet.com broke Azure core services this week”. As much as it sucks, this was a deliberate strategy all the way from Satya - every employee knew Azure was a joke, but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.

naasking 2y ago
> but the only want to actually fix shit was to get internet scale customers to break it daily and weekly. I don't get it. There's lots of distributed systems theory that could provide a more robust, analytical approach to a scalable architecture. If a system is regularly breaking like this, it sounds like it should be a "back to the drawing board" moment.
- cinntaile 2y ago
  Back to the drawing board risks delaying your products with several years. Their strategy was probably the right one.
  - naasking 2y ago
    It sounds like their product has been considered crap for years, so they've had the time.
    - FridgeSeal 2y ago
      “We’ve had years to fix it, all the experience and knowledge we’d need, but we’ve tried basically none of it and we’re at a loss as to why our product is still garbage! It’s truly a mystery” - Azure leadership I assume.
etruong42 2y ago
My experience working on Google Cloud Platform is similar. Even when the solution is relatively technically simple, there are many organizational impediments and politics because the solution likely involves multiple fiefdoms. The only thing that seems to get the fiefdoms to work on an issue is if a big enough customer calls enough of an attention to the issue so that no leader/fiefdom wants to be blamed in such a "high profile" issue that everyone got their work in gear for. This leads to organizational dysfunction since now everyone understands that there's no point in trying to fix anything until there is enough attention on it, and that is mainly achieved when a big customer complains. I've even had projects that took months of engineering work that would resolve many user complaints about a lack of really basic functionality, and different elements of leadership would block the launch. I can only guess that some fiefdom did not like change and would not state their objection publicly, so I only got stonewalled, and I never got an explanation to throw away months of work. I am as certain as I can be that if a big customer complained loudly enough about the problem that such basic functionality was missing, the organization would demand that the fix be launched.

UltraSane 2y ago

You would assume that the UI process would be running on Very High priority.

GranPC 2y ago

> My understanding is that Cosmos has gotten a lot better For some values of "better", I guess. Performance is still terrible, their data visualization/inspection tools are shameful, their SQL dialect is finicky and has no error reporting beyond "something is wrong with the input value", and their official Python SDK has a race condition bug that can silently clear out your documents when under heavy load. I used to work at a Cosmos-heavy house and I would utter "fucking Cosmos" around 15 times a day.