4 ms·

Same story with Cosmos. On an IoT pipeline I set up with event hub, Cosmos needed $15k/month to keep up with the flow without generating 429s (i.e., causing th

Same story with Cosmos. On an IoT pipeline I set up with event hub, Cosmos needed $15k/month to keep up with the flow without generating 429s (i.e., causing the ingestion function to drop events). The RUs shouldn't have needed to be that high, but due to upstream providers there was a regular spike every five minutes that would exceed the average RU need. RUs are a hard per second cap; there's no average or windowing. I had to set my RU consumption cost at a level that guaranteed 40% unused capacity. So I tried setting up the same ingestion database with Kafka Connect and Mongo on a $200/month VM. It worked flawlessly, and Azure helpfully suggested I downsize that VM because it was underutilized based on CPU statistics. What incentive do the Cosmos engineers have make it more efficient, or to make the RU pricing model more reflective of actual usage? Zero. It's a money hose. Why would you turn that off?

jiggawatts 5y ago
I saw CosmosDB turn up in some recommended multi-region designs, and I had a customer with DR requirements so I looked into it. I started by spinning up a small one in my lab but when I saw the pricing I back-pedalled very, very fast. Deleted the whole Resource Group and never looked into it again.
Avalaxy 5y ago
Would it not be possible for you to stream the data to a data lake and then take it from there to either do bulk inserts, or smooth out the inserts at a predictable rate to remove the peaks?
- fatbird 5y ago
  There were a variety of ways for me to do so by invoking more Azure services, but I stopped trying at that point, because even after smoothing it out, Cosmos would still be 50x as expensive as a basic VM running Mongo. But also, every time I start stringing together cloud services, I experience two things: first, exploding complexity because now I'm adding points of failure, integrations, transformations, to keep it all running; and second, this sense of "the whole point of the cloud is to simplify things, to offer canned services and features that save me the trouble of doing this in code for myself." Once I'm using cloud features to work around cloud limitations, I bail out because if I'm going to spend that time (and money), I'm going to get the benefits of something much more direct.

jiggawatts 5y ago

I saw CosmosDB turn up in some recommended multi-region designs, and I had a customer with DR requirements so I looked into it. I started by spinning up a small one in my lab but when I saw the pricing I back-pedalled very, very fast. Deleted the whole Resource Group and never looked into it again.

Avalaxy 5y ago

Would it not be possible for you to stream the data to a data lake and then take it from there to either do bulk inserts, or smooth out the inserts at a predictable rate to remove the peaks?

fatbird 5y ago
There were a variety of ways for me to do so by invoking more Azure services, but I stopped trying at that point, because even after smoothing it out, Cosmos would still be 50x as expensive as a basic VM running Mongo. But also, every time I start stringing together cloud services, I experience two things: first, exploding complexity because now I'm adding points of failure, integrations, transformations, to keep it all running; and second, this sense of "the whole point of the cloud is to simplify things, to offer canned services and features that save me the trouble of doing this in code for myself." Once I'm using cloud features to work around cloud limitations, I bail out because if I'm going to spend that time (and money), I'm going to get the benefits of something much more direct.