12 ms·

Discovering Azure's unannounced breaking change with Cosmos DB

188 points by jmartens 4y ago

jmartens 4y ago
We used our own product to learn about and debug the issue. Its rather wild that they'd roll out this change so incrementally, which my colleague outlines here.
- Scaevolus 4y ago
  Gradual rollouts are pretty typical to give the team a chance to do a rollback before they cause a complete outage. This particular usage pattern probably just didn't appear as a significant enough spike in error rates.
  - jmartens 4y ago
    Ya, that makes sense, it really isn't a normal use-case. I wish we kept tracking the other regions to see if they have had this change roll out to them yet, or if it's still slow rolling.
- aoetalks 4y ago
  Is a global outage better than a regional one? You can’t predict the impact of changes on a system as complex these.
twodave 4y ago
Funny, I was just last week having an argument with one of our team leads. I'd told him to create a specific container without a partition key (which I wouldn't recommend except in certain circumstances), and he said he couldn't. I assumed he was just doing it wrong.
- int0x2e 4y ago
  In a document store, what does it even mean to create a container without a partition key? The document store has to partition the data somehow, and doing so implicitly sounds dangerous to me since all you're doing is creating a hotspot on one of the partitions...
  - twodave 4y ago
    It essentially means everything goes on the same partition, which for the scenario in question was exactly what we wanted. Not everything needs to be hyper-scalable, and usually there’s an amount of overhead associated with making things so. In a demanding SLA sort of environment an extra hop to a new physical partition can mean the difference between getting paid or not.
DishyDev 4y ago
As someone whose job involves maintaining uptime of a critical system that's dependent on Cosmos DB this sort of thing is scary. Where there's been other reliability issues with Cosmos before we've not had an understanding customer base, and it feels very out of my control. I'm finding a lot of the reliability guarantees of Azure PaaS services are overblown or come with big caveats when you start to work with them in a serious way. For example I've had some bad reliability issues with Azure Functions not firing, or their premium function runtimes becoming unresponsive. And it seems like that's just the start of the outstanding issues with them https://github.com/Azure/azure-functions-host/issues https://github.com/Azure/azure-functions-host/issues I think people need to look more carefully at these PaaS guarantees and look at what that 99.999% reliability Microsoft are claiming actually means.
- rrdharan 4y ago
  Even as bad as their reliability issues are, I'd still be more worried about their security issues: https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-vulnerability-walkthrough https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v... https://msrc-blog.microsoft.com/2021/08/27/update-on-vulnerability-in-the-azure-cosmos-db-jupyter-notebook-feature/ https://msrc-blog.microsoft.com/2021/08/27/update-on-vulnera...
- xiwenc 4y ago
  Do you know what blew me off? When azure executes maintenance on for instance postgresql servers, there is no record of that activity in the activity logs or anything to note in service health. The service was unavailable during the maintenance. And stronger yet when the database is unusable due to an incident the cpu is maxed out and it doesnt allow any successful connection, nothing is detected. How can this be a premium iaas/paas? Azure feels like the MS teams of tele conference. Companies buy in because they are already in the MS world. Not because azure is better.
  - djbusby 4y ago
    Premium is a price point not a service level
johndfsgdgdfg 4y ago
- NicoJuicy 4y ago
  Which service did Google shutdown unannounced in their cloud offering?
  - metadat 4y ago
    Stadia, their cloud gaming platform. It was cancelled without warning two weeks ago: https://news.ycombinator.com/item?id=33022768 https://news.ycombinator.com/item?id=33022768
    - NicoJuicy 4y ago
      Cloud offering in the context of Azure should be related to GCE... Stadia isn't related to that.
      - metadat 4y ago
        You're being overly pedantic. Cloud is cloud. I'm 100% certain Stadia ran atop GCP. It is a cloud offering in every sense that matters.
        rrdharan 4y ago
        Last I checked you couldn’t sign an Enterprise Licensing Agreement for Stadia: https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/ea-portal-agreements https://learn.microsoft.com/en-us/azure/cost-management-bill... https://cloud.google.com/skus/sku-groups/enterprise-agreement https://cloud.google.com/skus/sku-groups/enterprise-agreemen... That’s a huge difference that matters much more than the service implementation detail.
        metadat 4y ago
        What about Google Keep? Does this come with an enterprise licensing agreement? I agree this is a good bar to test against, although I don't believe consumers are savvy enough to be aware of such matters. Nor should they be required to do so to get confidence what Google offers them today will remain tomorrow. With that said, should is an extremely arbitrary word.
dagss 4y ago
This rhymes with my overall impression of Cosmos. It took us a while to see through the smokescreen because when talking to Microsoft support and representatives it is the Best Thing Ever and they sound so confident about it. But it really is a beta demo product sold with an alpha premium price tag. If your traffic pattern is exactly right, and you always scale traffic up and never ever down and do not have spikes, I guess it is probably OK. The main problem is the docs are (or, at least were 2 years ago) not clear about all the caveats and restrictions but pretend it is a generic database that just works. So one has to discover all the caveats oneself. Microsoft thinks the exact workings of the partitioning is something that should work so well you don't need to know it in detail. But, if your usecase is slightly off you end up really needing to know. I know at least one team who routinely copy all their data from one Cosmos instance to another and switch over traffic to the copy just to get a partitioning reset; it is one thing to have to do it; another to discover in production yourself it has to be done with no prior warning.. Also: The ipython+portal+Cosmos security meltdown from 1 1/2 years ago alone should be reason to look elsewhere. (No, not a competitor, just have spent way way way too much engineering time moving first on and then off Cosmos and yes I am bitter)
- MrBuddyCasino 4y ago
  I used it a few months ago, it still is a half-baked piece of shit. Code quality of client libs even worse than AWS. It could have been easy. We could have used Postgres.
  - sofixa 4y ago
    > It could have been easy. We could have used Postgres. And then all that would have been left is just to make that Postgres in a globally distributed DB and manage it (with something like Citus). Postgres' native scaling out (not even talking about globally distributed) capabilities are basically nonexistent so you need third party tooling.
    - MrBuddyCasino 4y ago
      We didn't need a "globally distributed DB". This was an internal tool for a BigCorp.
speedgoose 4y ago
I believe it is easy for a well-made software to immediately detect and report what goes wrong. With Sentry, Elk, or whatever else. So, let's say I'm woken up in the middle of the night because my black box database as a service suddenly returns errors. If I'm not incompetent, I should have error messages and stacktraces available in a few seconds. If I'm a rich cloud customer, I can call the premium cloud support and ask for an explanation. If not, I would probably have to debug it myself. With your service, I understand that I can blame the cloud provider faster. Maybe it can make the debugging session slightly faster when your monitoring also returns errors. End users don't care whether it's my code or the cloud provider code crashing, so it's a developer tool for emergencies. Did I understand well?
- jmartens 4y ago
  You got it right, it's a developer tool. Its not hard to get alerted about an issue, or to suspect a cloud dependency. Verifying it, which is typically required to take action, is what can take 10-30 minutes.
dec0dedab0de 4y ago
This seems like an accident. Microsoft should treat it as a bug, and set the default on their backend to fix it.
dharmab 4y ago
Back around 2017-2018 unannounced breaking changes in Azure services were so common, my team coined a term "Cloud Monday" (echoing Patch Tuesday) because usually our integration tests would break between 8-10AM Pacific Time on Mondays. (They did eventually become far less frequent.)
- TurkTurkleton 4y ago
  > my team coined a term "Cloud Monday" Azure being a shade of blue, you should've called it "Blue Monday"[0]. Could've even rigged up something to play the song when integration tests mysteriously failed. How does it feel/ to treat me like you do?/ When you've laid your hands upon me/ and told me who you are?... [0]: https://www.youtube.com/watch?v=c1GxjzHm5us https://www.youtube.com/watch?v=c1GxjzHm5us
rroot 4y ago
- andrewstuart2 4y ago
  What are you doing here if you're not going to RTFA? The fifth paragraph pretty clearly describes the issue before they go into depth on how they determined that Azure did indeed publish a backwards-incompatible change without notice.
  - rroot 4y ago
    I don't think it's appropriate to ask personal questions on HN. But since we're going personal I'd like to note that "RTFA" is not a professional language. The paragraph you mention states the issue, but it's pretty far from being a summary. What an "executive summary" section would include in this case in a very short text on: * What the problem is. * How long to took to resolve for how many people. * How it effected the company/project. You could add more relevant stuff, depending on what the article says. There are some good resource on the internet how to write summaries. People who are looking for interesting stuff to read in their 10 minute coffee break might read the article after the summary instead of skipping it altogether.
- girvo 4y ago
nobodyandproud 4y ago
For new projects, why wouldn’t anyone use postgres?
- semicolon_storm 4y ago
  It's a lot easier to sell Microsoft products to management when working at a Microsoft shop.
  - isbvhodnvemrwvn 4y ago
    Oracle of 2010s/2020s.
- jen20 4y ago
  One reason is wanting zero-downtime failover.
- haolez 4y ago
  Postgres is more expensive than CosmosDB for someone who is just playing around and is constrained to managed systems for some reason.
- eurasiantiger 4y ago
  Why would anyone want to maintain their own postgres?
- pjmlp 4y ago
  Because the customers we work with care mostly about SQL Server and Oracle.
hupt 4y ago
Cosmos was originally created for hosting massive datasets internally within Microsoft. For example they use it for the OS telemetry sent in from customer machines, and raw data for threat intelligence. As part of Microsoft's move of everything hosted on-premise to their cloud, they decided to upon up Cosmos to other users of Azure. But the primary customer is and will likely always be Microsoft themselves. Which is probably why we see these breaking changes, it'll be in response to some internal ticket most likely.
- CurtHagenlocher 4y ago
  CosmosDB is not the same as the internal Cosmos system.
- int0x2e 4y ago
  Cosmos != CosmosDB. The two have nothing in common (and trust me, it sure is fun having to constantly make sure which of the two someone is actually referring to every time...).
whalesalad 4y ago
This is very typical Microsoft behavior, unfortunately.
prepend 4y ago
This reenforces my idea that no one uses Cosmos because it is utter garbage. It sounds cool, but I was surprised when after what I think should be the worst and dumbest security design flaw breach [0] there wasn’t much uproar. I thought maybe no one is using it so there wasn’t much impact. Pushing out breaking changes without telling your customers also gets explained by there not being any (or many since these folks found it) users. Could you image how big of a deal it would be if a breaking change or elevated privs bug were in actually used products. [0] https://www.techtarget.com/searchsecurity/news/252505973/Researchers-discover-critical-flaw-in-Azure-Cosmos-DB https://www.techtarget.com/searchsecurity/news/252505973/Res...
HorizonXP 4y ago
So, as someone who was in the midst of planning a migration of a multi-billion $ revenue platform to using CosmosDB... Alternatives? LOL Basically just looking for geo-redundant, high read & write throughput. Our intention was to leverage Azure Event Grid/Kafka Connect to have event streaming used to coordinate writes between Redis (cache), Cosmos (transactional DB), and our systems of record (legacy). Majority of read/writes would occur via our API, but some would occur via the systems of record, hence the use of a log-based architecture.
- rrdharan 4y ago
  DynamoDB or Bigtable?
  - HorizonXP 4y ago
    We're on Azure, otherwise those were my goto choices too.
- noahl 4y ago
  Spanner offers that on GCP, and I believe CockroachDB offers something similar cross-cloud. Do you have any specific requirements for which cloud provider you use, or any particular interface you really need?
  - HorizonXP 4y ago
    Azure is our cloud provider. Interface is flexible, since our current implementation leverages Prisma ORM connected to Postgres & SQL Server. We're going to have to rebuild it anyway.
    - skunkworker 4y ago
      Vitess maybe? from what I have demoed and for being open source that’s the one I would choose. Unfortunately there isn’t a Postgres compatible one that is as mature yet and certainly not free.
    - 4y ago
yellow_lead 4y ago
I find it pretty interesting that this company/product (Metrist) has created a monitoring tool for different cloud products, because their monitoring is so bad. Honestly a good idea, but a bit sad these companies can't do this themselves.
didip 4y ago
I feel like Azure should just give up on Cosmos DB and go all in on managed Citus DB.
- outside1234 4y ago
  They did the opposite and now that is PostgreSQL for CosmosDB. Can’t make this shit up.
grogers 4y ago
The error message from azure are literally just “One of the specified inputs is invalid”? I get annoyed at AWS error messages because they aren't really machine readable (unless you're okay parsing a string that is subject to change), but at least they are almost always human readable with all the relevant details...
redditor98654 4y ago
I am proud of my team at AWS that took backwards compatibility very seriously. Even introducing a temporary backwards incompatibility was a no-go in design reviews. We had a service that had a list API that was paginated. It returned a nextToken to specify the start of the next page of the results. Internally we were doing a database migration to a completely different system and migrating one customer at a time. The problem was that if a customer was in the middle of a list call and had the next token with them which was generated from the previous database system, after migrating to the new database the older token would not be able to start from exactly where it should had the customer not been migrated. This was because it would not have all the information of the service to resume at the exact offset. One option was to throw an error and let the customer retry the request; another option was to return some possibly duplicate items in the next page; none of these were good enough for both engineers and PMs and instead we decide to take up a bunch of additional work so that no customer would be impacted. This was 10+ weeks of additional work for the whole team but we did it because culturally it felt the right thing to do for the customer. Note that the impact would have been tiny if at all. A customer would have to be in the middle of a paginated request and out migration system would have had to migrate that particular customer at that exact time and the impact would have been a few possibly duplicate items. But we didn’t know the actual impact of those temporary duplicated for a single call and we all agreed breaking changes like this are unexpected and cause customer to lose trust with us.
- jiggawatts 4y ago
  I read stuff like this about AWS and I wonder what the guys over at Azure are smoking. I can deploy trivial architectures with ordinary resources and hit four or five “broken by design” issues and at least one or two Heisenbugs that aren’t reproducible in other identical setups due to fun things like “legacy” subscriptions — an invisible internal setting.
  - NickGerleman 4y ago
    I previously worked at Microsoft in an unrelated area, but had a friend who worked on CosmosDB, and later a different part of Azure. There are some Microsoft products I genuinely love, but some are terrible. To an extent it is a reflection of the inconsistency in internal teams. Culture, values, skill level, and quality bar are all over the place depending on who you talk to, even compared to other large companies. From what I had heard, CosmosDB was not a healthy team, and I would not consider using it as a product.
sublimefire 4y ago
Correct me if I'm wrong but the article does not mention which "outdated sdk" version was used. In addition to that every API call requires a version which is not mentioned in the article [1]. It is not clear to me if the issue was with an old SDK using the newest api version in calls or was it something else? [1] https://learn.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/2021-04-01-preview/sql-resources/create-update-sql-container?tabs=HTTP https://learn.microsoft.com/en-us/rest/api/cosmos-db-resourc...
- aoetalks 4y ago
  They said in the article they tried updating their old SDK.
atraac 4y ago
I have similar opinion to some other comments. Some Azure services - like Application Insights - I absolutely loved, some I hated, CosmosDB, being the latter. They needed years to finally introduce PATCH in CosmosDB, Request Units feel like they're obscure on purpose to hide insane cost of using this storage, being able to use Stored Procedures only on one Partition Key(while /id is being the default...), requests would often fail with 429 Too Many Requests when the container was set to Autoscale with obscene limits that were never hit. Just setup Marten with Postgres and get it over with for fraction of the cost.
CiTyBear 4y ago
Breaking changes are very common with Python Azure SDK. First version where not pep8 compliant so when they decided to respect it, everything broke. Azure servicebus in python went from version 0.50.3 to 7.0.0 with almost everything renamed, class moved and so on.
- aoetalks 4y ago
  This is different than breaking changes in package upgrades. This is a change without the package version even changing (even newest package didn’t work)

188 points by jmartens 4y ago

We used our own product to learn about and debug the issue. Its rather wild that they'd roll out this change so incrementally, which my colleague outlines here.

Scaevolus 4y ago
Gradual rollouts are pretty typical to give the team a chance to do a rollback before they cause a complete outage. This particular usage pattern probably just didn't appear as a significant enough spike in error rates.
- jmartens 4y ago
  Ya, that makes sense, it really isn't a normal use-case. I wish we kept tracking the other regions to see if they have had this change roll out to them yet, or if it's still slow rolling.
aoetalks 4y ago
Is a global outage better than a regional one? You can’t predict the impact of changes on a system as complex these.

twodave 4y ago

Funny, I was just last week having an argument with one of our team leads. I'd told him to create a specific container without a partition key (which I wouldn't recommend except in certain circumstances), and he said he couldn't. I assumed he was just doing it wrong.

int0x2e 4y ago
In a document store, what does it even mean to create a container without a partition key? The document store has to partition the data somehow, and doing so implicitly sounds dangerous to me since all you're doing is creating a hotspot on one of the partitions...
- twodave 4y ago
  It essentially means everything goes on the same partition, which for the scenario in question was exactly what we wanted. Not everything needs to be hyper-scalable, and usually there’s an amount of overhead associated with making things so. In a demanding SLA sort of environment an extra hop to a new physical partition can mean the difference between getting paid or not.

DishyDev 4y ago

As someone whose job involves maintaining uptime of a critical system that's dependent on Cosmos DB this sort of thing is scary. Where there's been other reliability issues with Cosmos before we've not had an understanding customer base, and it feels very out of my control. I'm finding a lot of the reliability guarantees of Azure PaaS services are overblown or come with big caveats when you start to work with them in a serious way. For example I've had some bad reliability issues with Azure Functions not firing, or their premium function runtimes becoming unresponsive. And it seems like that's just the start of the outstanding issues with them https://github.com/Azure/azure-functions-host/issues https://github.com/Azure/azure-functions-host/issues I think people need to look more carefully at these PaaS guarantees and look at what that 99.999% reliability Microsoft are claiming actually means.

rrdharan 4y ago
Even as bad as their reliability issues are, I'd still be more worried about their security issues: https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-vulnerability-walkthrough https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v... https://msrc-blog.microsoft.com/2021/08/27/update-on-vulnerability-in-the-azure-cosmos-db-jupyter-notebook-feature/ https://msrc-blog.microsoft.com/2021/08/27/update-on-vulnera...
xiwenc 4y ago
Do you know what blew me off? When azure executes maintenance on for instance postgresql servers, there is no record of that activity in the activity logs or anything to note in service health. The service was unavailable during the maintenance. And stronger yet when the database is unusable due to an incident the cpu is maxed out and it doesnt allow any successful connection, nothing is detected. How can this be a premium iaas/paas? Azure feels like the MS teams of tele conference. Companies buy in because they are already in the MS world. Not because azure is better.
- djbusby 4y ago
  Premium is a price point not a service level

johndfsgdgdfg 4y ago

NicoJuicy 4y ago
Which service did Google shutdown unannounced in their cloud offering?
- metadat 4y ago
  Stadia, their cloud gaming platform. It was cancelled without warning two weeks ago: https://news.ycombinator.com/item?id=33022768 https://news.ycombinator.com/item?id=33022768
  - NicoJuicy 4y ago
    Cloud offering in the context of Azure should be related to GCE... Stadia isn't related to that.
    - metadat 4y ago
      You're being overly pedantic. Cloud is cloud. I'm 100% certain Stadia ran atop GCP. It is a cloud offering in every sense that matters.
      - rrdharan 4y ago
        Last I checked you couldn’t sign an Enterprise Licensing Agreement for Stadia: https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/ea-portal-agreements https://learn.microsoft.com/en-us/azure/cost-management-bill... https://cloud.google.com/skus/sku-groups/enterprise-agreement https://cloud.google.com/skus/sku-groups/enterprise-agreemen... That’s a huge difference that matters much more than the service implementation detail.
        metadat 4y ago
        What about Google Keep? Does this come with an enterprise licensing agreement? I agree this is a good bar to test against, although I don't believe consumers are savvy enough to be aware of such matters. Nor should they be required to do so to get confidence what Google offers them today will remain tomorrow. With that said, should is an extremely arbitrary word.

dagss 4y ago

This rhymes with my overall impression of Cosmos. It took us a while to see through the smokescreen because when talking to Microsoft support and representatives it is the Best Thing Ever and they sound so confident about it. But it really is a beta demo product sold with an alpha premium price tag. If your traffic pattern is exactly right, and you always scale traffic up and never ever down and do not have spikes, I guess it is probably OK. The main problem is the docs are (or, at least were 2 years ago) not clear about all the caveats and restrictions but pretend it is a generic database that just works. So one has to discover all the caveats oneself. Microsoft thinks the exact workings of the partitioning is something that should work so well you don't need to know it in detail. But, if your usecase is slightly off you end up really needing to know. I know at least one team who routinely copy all their data from one Cosmos instance to another and switch over traffic to the copy just to get a partitioning reset; it is one thing to have to do it; another to discover in production yourself it has to be done with no prior warning.. Also: The ipython+portal+Cosmos security meltdown from 1 1/2 years ago alone should be reason to look elsewhere. (No, not a competitor, just have spent way way way too much engineering time moving first on and then off Cosmos and yes I am bitter)

MrBuddyCasino 4y ago
I used it a few months ago, it still is a half-baked piece of shit. Code quality of client libs even worse than AWS. It could have been easy. We could have used Postgres.
- sofixa 4y ago
  > It could have been easy. We could have used Postgres. And then all that would have been left is just to make that Postgres in a globally distributed DB and manage it (with something like Citus). Postgres' native scaling out (not even talking about globally distributed) capabilities are basically nonexistent so you need third party tooling.
  - MrBuddyCasino 4y ago
    We didn't need a "globally distributed DB". This was an internal tool for a BigCorp.

speedgoose 4y ago

I believe it is easy for a well-made software to immediately detect and report what goes wrong. With Sentry, Elk, or whatever else. So, let's say I'm woken up in the middle of the night because my black box database as a service suddenly returns errors. If I'm not incompetent, I should have error messages and stacktraces available in a few seconds. If I'm a rich cloud customer, I can call the premium cloud support and ask for an explanation. If not, I would probably have to debug it myself. With your service, I understand that I can blame the cloud provider faster. Maybe it can make the debugging session slightly faster when your monitoring also returns errors. End users don't care whether it's my code or the cloud provider code crashing, so it's a developer tool for emergencies. Did I understand well?

jmartens 4y ago
You got it right, it's a developer tool. Its not hard to get alerted about an issue, or to suspect a cloud dependency. Verifying it, which is typically required to take action, is what can take 10-30 minutes.

dec0dedab0de 4y ago

This seems like an accident. Microsoft should treat it as a bug, and set the default on their backend to fix it.

dharmab 4y ago

Back around 2017-2018 unannounced breaking changes in Azure services were so common, my team coined a term "Cloud Monday" (echoing Patch Tuesday) because usually our integration tests would break between 8-10AM Pacific Time on Mondays. (They did eventually become far less frequent.)

TurkTurkleton 4y ago
> my team coined a term "Cloud Monday" Azure being a shade of blue, you should've called it "Blue Monday"[0]. Could've even rigged up something to play the song when integration tests mysteriously failed. How does it feel/ to treat me like you do?/ When you've laid your hands upon me/ and told me who you are?... [0]: https://www.youtube.com/watch?v=c1GxjzHm5us https://www.youtube.com/watch?v=c1GxjzHm5us

rroot 4y ago

andrewstuart2 4y ago
What are you doing here if you're not going to RTFA? The fifth paragraph pretty clearly describes the issue before they go into depth on how they determined that Azure did indeed publish a backwards-incompatible change without notice.
- rroot 4y ago
  I don't think it's appropriate to ask personal questions on HN. But since we're going personal I'd like to note that "RTFA" is not a professional language. The paragraph you mention states the issue, but it's pretty far from being a summary. What an "executive summary" section would include in this case in a very short text on: * What the problem is. * How long to took to resolve for how many people. * How it effected the company/project. You could add more relevant stuff, depending on what the article says. There are some good resource on the internet how to write summaries. People who are looking for interesting stuff to read in their 10 minute coffee break might read the article after the summary instead of skipping it altogether.
girvo 4y ago

nobodyandproud 4y ago

For new projects, why wouldn’t anyone use postgres?

semicolon_storm 4y ago
It's a lot easier to sell Microsoft products to management when working at a Microsoft shop.
- isbvhodnvemrwvn 4y ago
  Oracle of 2010s/2020s.
jen20 4y ago
One reason is wanting zero-downtime failover.
haolez 4y ago
Postgres is more expensive than CosmosDB for someone who is just playing around and is constrained to managed systems for some reason.
eurasiantiger 4y ago
Why would anyone want to maintain their own postgres?
pjmlp 4y ago
Because the customers we work with care mostly about SQL Server and Oracle.

hupt 4y ago

Cosmos was originally created for hosting massive datasets internally within Microsoft. For example they use it for the OS telemetry sent in from customer machines, and raw data for threat intelligence. As part of Microsoft's move of everything hosted on-premise to their cloud, they decided to upon up Cosmos to other users of Azure. But the primary customer is and will likely always be Microsoft themselves. Which is probably why we see these breaking changes, it'll be in response to some internal ticket most likely.

CurtHagenlocher 4y ago
CosmosDB is not the same as the internal Cosmos system.
int0x2e 4y ago
Cosmos != CosmosDB. The two have nothing in common (and trust me, it sure is fun having to constantly make sure which of the two someone is actually referring to every time...).

whalesalad 4y ago

This is very typical Microsoft behavior, unfortunately.

prepend 4y ago

This reenforces my idea that no one uses Cosmos because it is utter garbage. It sounds cool, but I was surprised when after what I think should be the worst and dumbest security design flaw breach [0] there wasn’t much uproar. I thought maybe no one is using it so there wasn’t much impact. Pushing out breaking changes without telling your customers also gets explained by there not being any (or many since these folks found it) users. Could you image how big of a deal it would be if a breaking change or elevated privs bug were in actually used products. [0] https://www.techtarget.com/searchsecurity/news/252505973/Researchers-discover-critical-flaw-in-Azure-Cosmos-DB https://www.techtarget.com/searchsecurity/news/252505973/Res...

HorizonXP 4y ago

So, as someone who was in the midst of planning a migration of a multi-billion $ revenue platform to using CosmosDB... Alternatives? LOL Basically just looking for geo-redundant, high read & write throughput. Our intention was to leverage Azure Event Grid/Kafka Connect to have event streaming used to coordinate writes between Redis (cache), Cosmos (transactional DB), and our systems of record (legacy). Majority of read/writes would occur via our API, but some would occur via the systems of record, hence the use of a log-based architecture.

rrdharan 4y ago
DynamoDB or Bigtable?
- HorizonXP 4y ago
  We're on Azure, otherwise those were my goto choices too.
noahl 4y ago
Spanner offers that on GCP, and I believe CockroachDB offers something similar cross-cloud. Do you have any specific requirements for which cloud provider you use, or any particular interface you really need?
- HorizonXP 4y ago
  Azure is our cloud provider. Interface is flexible, since our current implementation leverages Prisma ORM connected to Postgres & SQL Server. We're going to have to rebuild it anyway.
  - skunkworker 4y ago
    Vitess maybe? from what I have demoed and for being open source that’s the one I would choose. Unfortunately there isn’t a Postgres compatible one that is as mature yet and certainly not free.
  - 4y ago

yellow_lead 4y ago

I find it pretty interesting that this company/product (Metrist) has created a monitoring tool for different cloud products, because their monitoring is so bad. Honestly a good idea, but a bit sad these companies can't do this themselves.

didip 4y ago

I feel like Azure should just give up on Cosmos DB and go all in on managed Citus DB.

outside1234 4y ago
They did the opposite and now that is PostgreSQL for CosmosDB. Can’t make this shit up.

grogers 4y ago

The error message from azure are literally just “One of the specified inputs is invalid”? I get annoyed at AWS error messages because they aren't really machine readable (unless you're okay parsing a string that is subject to change), but at least they are almost always human readable with all the relevant details...

redditor98654 4y ago

I am proud of my team at AWS that took backwards compatibility very seriously. Even introducing a temporary backwards incompatibility was a no-go in design reviews. We had a service that had a list API that was paginated. It returned a nextToken to specify the start of the next page of the results. Internally we were doing a database migration to a completely different system and migrating one customer at a time. The problem was that if a customer was in the middle of a list call and had the next token with them which was generated from the previous database system, after migrating to the new database the older token would not be able to start from exactly where it should had the customer not been migrated. This was because it would not have all the information of the service to resume at the exact offset. One option was to throw an error and let the customer retry the request; another option was to return some possibly duplicate items in the next page; none of these were good enough for both engineers and PMs and instead we decide to take up a bunch of additional work so that no customer would be impacted. This was 10+ weeks of additional work for the whole team but we did it because culturally it felt the right thing to do for the customer. Note that the impact would have been tiny if at all. A customer would have to be in the middle of a paginated request and out migration system would have had to migrate that particular customer at that exact time and the impact would have been a few possibly duplicate items. But we didn’t know the actual impact of those temporary duplicated for a single call and we all agreed breaking changes like this are unexpected and cause customer to lose trust with us.

jiggawatts 4y ago
I read stuff like this about AWS and I wonder what the guys over at Azure are smoking. I can deploy trivial architectures with ordinary resources and hit four or five “broken by design” issues and at least one or two Heisenbugs that aren’t reproducible in other identical setups due to fun things like “legacy” subscriptions — an invisible internal setting.
- NickGerleman 4y ago
  I previously worked at Microsoft in an unrelated area, but had a friend who worked on CosmosDB, and later a different part of Azure. There are some Microsoft products I genuinely love, but some are terrible. To an extent it is a reflection of the inconsistency in internal teams. Culture, values, skill level, and quality bar are all over the place depending on who you talk to, even compared to other large companies. From what I had heard, CosmosDB was not a healthy team, and I would not consider using it as a product.

sublimefire 4y ago

Correct me if I'm wrong but the article does not mention which "outdated sdk" version was used. In addition to that every API call requires a version which is not mentioned in the article [1]. It is not clear to me if the issue was with an old SDK using the newest api version in calls or was it something else? [1] https://learn.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/2021-04-01-preview/sql-resources/create-update-sql-container?tabs=HTTP https://learn.microsoft.com/en-us/rest/api/cosmos-db-resourc...

aoetalks 4y ago
They said in the article they tried updating their old SDK.

atraac 4y ago

I have similar opinion to some other comments. Some Azure services - like Application Insights - I absolutely loved, some I hated, CosmosDB, being the latter. They needed years to finally introduce PATCH in CosmosDB, Request Units feel like they're obscure on purpose to hide insane cost of using this storage, being able to use Stored Procedures only on one Partition Key(while /id is being the default...), requests would often fail with 429 Too Many Requests when the container was set to Autoscale with obscene limits that were never hit. Just setup Marten with Postgres and get it over with for fraction of the cost.

CiTyBear 4y ago

Breaking changes are very common with Python Azure SDK. First version where not pep8 compliant so when they decided to respect it, everything broke. Azure servicebus in python went from version 0.50.3 to 7.0.0 with almost everything renamed, class moved and so on.

aoetalks 4y ago
This is different than breaking changes in package upgrades. This is a change without the package version even changing (even newest package didn’t work)