4 ms·

I don't know your flow, but how about using a queue system to prevent the bursts of data? We're internally using pubsub (and nats) sending billions of messages

by doh 9y ago

I don't know your flow, but how about using a queue system to prevent the bursts of data? We're internally using pubsub (and nats) sending billions of messages every day. We don't use the push mode as Spotify [0] but pull from the queue instead which allows us to run on our pace. We do write into bigquery (and citus) and each components is doing it on their own pace based on what they're capable of process at the moment. [0] https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-consumers/ https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-cons...

bruth 9y ago
What workloads do you prefer pubsub over nats and vice versa?
- doh 9y ago
  We use nats mostly for synchronous tasks (where the response has to be returned to the exact client initializing the request) and pubsub for everything else. We’re slowly transitioning to nats as we have a better control over what’s happening there. Also we had no issues with stability which I can’t say about pubsub
dillondoyle 9y ago
Pub Sub is the same quota hits (or +1 /connection). I could use pub sub but I still have to accept the incoming POST before publishing a message.
- dillondoyle 9y ago
  To be clearer it's same quota problem using cloud functions. Moving to something without the same quota/scale problems like Compute Engine or just DIY containers might make sense but then I'm back to a complicated system (Snowplow) that we have in AWS that's too complicated for me alone
  - doh 9y ago
    Understood. I think there is a bit simpler setup that you could utilize, but it does require a custom deployment on GCE. Request -> load balancer -> auto scaled pubsub writer -> pubsub -> client (write to BQ, ...) If it doesn't have to be a post request, then you could just write into pubsub directly. Pubsub has a wide support of libraries [0] so you could do it from any language. Pubsub also scales well. We're sending in around 300k messages per sec and have very few problems with it. [0] https://cloud.google.com/pubsub/docs/reference/libraries https://cloud.google.com/pubsub/docs/reference/libraries
    - dillondoyle 9y ago
      Yup that's pretty much where I'm going although without pub/sub ideally. it's marketing analytics so if a few inserts fail it doesn't matter. But might not be able to get the 0 -> 1000s instant working without Google's abstraction messaging in front (similar setup on AWS w. kinesis but it's too complicated for me) I did think about doing something like transforming the POST in AWS Lambda (the emitter data is hosted there too) and transform into something I can ingest direct into pub/sub. I can't control the emitting data. if I did it would be a much easier problem, each POST request contains very little data if there were multiple entries per single POST it would solve most of my problems, instead of sending 5k tiny JSON posts /sec lol
    - cobookman 9y ago
      You can make it simpler. Request -> gae -> pubsub -> dataflow to bq template. https://cloud.google.com/dataflow/docs/templates/provided-templates https://cloud.google.com/dataflow/docs/templates/provided-te...

by doh 9y ago

bruth 9y ago

What workloads do you prefer pubsub over nats and vice versa?

doh 9y ago
We use nats mostly for synchronous tasks (where the response has to be returned to the exact client initializing the request) and pubsub for everything else. We’re slowly transitioning to nats as we have a better control over what’s happening there. Also we had no issues with stability which I can’t say about pubsub

dillondoyle 9y ago

Pub Sub is the same quota hits (or +1 /connection). I could use pub sub but I still have to accept the incoming POST before publishing a message.

dillondoyle 9y ago
To be clearer it's same quota problem using cloud functions. Moving to something without the same quota/scale problems like Compute Engine or just DIY containers might make sense but then I'm back to a complicated system (Snowplow) that we have in AWS that's too complicated for me alone
- doh 9y ago
  Understood. I think there is a bit simpler setup that you could utilize, but it does require a custom deployment on GCE. Request -> load balancer -> auto scaled pubsub writer -> pubsub -> client (write to BQ, ...) If it doesn't have to be a post request, then you could just write into pubsub directly. Pubsub has a wide support of libraries [0] so you could do it from any language. Pubsub also scales well. We're sending in around 300k messages per sec and have very few problems with it. [0] https://cloud.google.com/pubsub/docs/reference/libraries https://cloud.google.com/pubsub/docs/reference/libraries
  - dillondoyle 9y ago
    Yup that's pretty much where I'm going although without pub/sub ideally. it's marketing analytics so if a few inserts fail it doesn't matter. But might not be able to get the 0 -> 1000s instant working without Google's abstraction messaging in front (similar setup on AWS w. kinesis but it's too complicated for me) I did think about doing something like transforming the POST in AWS Lambda (the emitter data is hosted there too) and transform into something I can ingest direct into pub/sub. I can't control the emitting data. if I did it would be a much easier problem, each POST request contains very little data if there were multiple entries per single POST it would solve most of my problems, instead of sending 5k tiny JSON posts /sec lol
  - cobookman 9y ago
    You can make it simpler. Request -> gae -> pubsub -> dataflow to bq template. https://cloud.google.com/dataflow/docs/templates/provided-templates https://cloud.google.com/dataflow/docs/templates/provided-te...