3 ms·

Yes, but if I'm reading their blog post correctly, their individual messages are much smaller than 64K "Pub/Sub passed the test with flying colours. We publish

Yes, but if I'm reading their blog post correctly, their individual messages are much smaller than 64K "Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."

dmytton 11y ago
Yes, but that has no impact on the cost. It's a flat rate in 64kb blocks. Batching and compression would only affect the networking fees from Google, not the per API call fees. If you batch up 64 1kb messages into a single batch then Google still bills you for 64 publish API calls.
- sharpy 11y ago
  They are wrapping small events into fewer larger ones. Hence the word batching. I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!
  - foota 11y ago
    "Batched calls are accounted for as multiple operations." - https://cloud.google.com/pubsub/pricing https://cloud.google.com/pubsub/pricing
    - deleted 11y ago
      [deleted]
    - sharpy 11y ago
      Right. If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit. But rather than publishing { "messages" : [ { message 1 }, { message 2 } ] } You publish { "messages" : [ { container message containing both, still less than 64k } ] } Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is. If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.
      - foota 11y ago
        That makes sense. Thanks. Do you see anything saying they're doing this?
      - dmytton 11y ago
        The Spotify graphs from Google's monitoring support the numbers I quoted. They may well be batching but Google is seeing (and billing) the numbers I quoted.
    - Gigablah 11y ago
      He's talking about a single call containing multiple messages in their own serialization format.
- i0exception 11y ago
  If 2M requests result in 1 Gb network traffic, each message is 500 bits (~60 bytes). Theoretically, that's 1K messages per batch. Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.

dmytton 11y ago

Yes, but that has no impact on the cost. It's a flat rate in 64kb blocks. Batching and compression would only affect the networking fees from Google, not the per API call fees. If you batch up 64 1kb messages into a single batch then Google still bills you for 64 publish API calls.

sharpy 11y ago
They are wrapping small events into fewer larger ones. Hence the word batching. I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!
- foota 11y ago
  "Batched calls are accounted for as multiple operations." - https://cloud.google.com/pubsub/pricing https://cloud.google.com/pubsub/pricing
  - deleted 11y ago
    [deleted]
  - sharpy 11y ago
    Right. If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit. But rather than publishing { "messages" : [ { message 1 }, { message 2 } ] } You publish { "messages" : [ { container message containing both, still less than 64k } ] } Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is. If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.
    - foota 11y ago
      That makes sense. Thanks. Do you see anything saying they're doing this?
    - dmytton 11y ago
      The Spotify graphs from Google's monitoring support the numbers I quoted. They may well be batching but Google is seeing (and billing) the numbers I quoted.
  - Gigablah 11y ago
    He's talking about a single call containing multiple messages in their own serialization format.
i0exception 11y ago
If 2M requests result in 1 Gb network traffic, each message is 500 bits (~60 bytes). Theoretically, that's 1K messages per batch. Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.