4 ms·

I'm in the middle of trying to move a very simple ETL-light script to Google (want to use bigquery over redshift). The idea was to use Cloud Functions. I have

I'm in the middle of trying to move a very simple ETL-light script to Google (want to use bigquery over redshift). The idea was to use Cloud Functions. I have ran into scale problems very fast at Google and now am having to use App Engine and add more complication which I don't have personal engineering skill/capacity. Google support first bumped me up to 12000 max queries per 100 seconds and said that's the limit, but that's not close to enough. We have super bursty traffic (webhooks that aren't concatenated from source). Google then increased to 100,000 per 100 seconds after they claimed 12k was maximum. It's still not enough and I think their sampling might be funky because when I look at the invocation/traffic graphs it doesn't align with the stated maximum (still throws quota errors). Kind of frustrating. The idea of stateless functions without worrying about engineering is a dream! I don't get why Google can't get 'google scale' here. In the past we've used Snowplow on AWS when we had engineering help but that's a huge amount of work/engineering time. The actual accept POST, store to cloud storage, insert to DB is less than 100 rows. Response time is also pretty bad in the extreme cases. Even though I end the response before processing data I still see 95%/99% spikes when bursting on the far less bursty data source (as in only bursting to 10/second). Just today for this function I got 3764ms 99%!

daxfohl 9y ago
My feeling is that Google cloud will always be hit and miss because their biggest customer by far is Google. So they'll do unbelievable scale on the stuff they use, and just passable on everything else. It's also why I think that Facebook never joined the rat race. Most of what they do has no relevance to the average company. Amazon, an online retailer, was really the most natural place to start an outsourcing of banal technologies for the largest market segment. And has grown from there.
helpfulgoogler 9y ago
According to [1], the standard quota is 1,000,000 invocations per 100 seconds and can be increased. Maybe try asking again for increased quota? If the documentation is incorrect, please file an issue [2]. Counter-intuitively, ending the response before processing the data may actually be hurting your p99 response time. According to [3], instances which are not currently handling a request get very little CPU and generally don't make any progress. This means that this work will still be waiting to be done when the next request comes in and can slow it down. You may even end up with an instance trying to process several requests' data and also trying to handle another request on top of that. This last request is going to get an overloaded instance and likely be very slow. Further, the function isn't guaranteed to ever be run again, so that deferred work might not even ever happen. It probably will cause you to use more of your invocations per 100 seconds quota, but you really shouldn't start background processing tasks in a Google Cloud Function that continue after completing a request. [1] https://cloud.google.com/functions/quotas https://cloud.google.com/functions/quotas [2] https://issuetracker.google.com/issues/new?component=187195&template=0 https://issuetracker.google.com/issues/new?component=187195&... [3] https://cloud.google.com/functions/docs/bestpractices/tips https://cloud.google.com/functions/docs/bestpractices/tips
- dillondoyle 9y ago
  I have paid support. It's socket connections we're hitting super quick. Interesting and thanks I had no idea on the end response CPU time. It's just storing the json to Google Cloud Storage but it does take 100-300ms it seems!
doh 9y ago
I don't know your flow, but how about using a queue system to prevent the bursts of data? We're internally using pubsub (and nats) sending billions of messages every day. We don't use the push mode as Spotify [0] but pull from the queue instead which allows us to run on our pace. We do write into bigquery (and citus) and each components is doing it on their own pace based on what they're capable of process at the moment. [0] https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-consumers/ https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-cons...
- bruth 9y ago
  What workloads do you prefer pubsub over nats and vice versa?
  - doh 9y ago
    We use nats mostly for synchronous tasks (where the response has to be returned to the exact client initializing the request) and pubsub for everything else. We’re slowly transitioning to nats as we have a better control over what’s happening there. Also we had no issues with stability which I can’t say about pubsub
- dillondoyle 9y ago
  Pub Sub is the same quota hits (or +1 /connection). I could use pub sub but I still have to accept the incoming POST before publishing a message.
  - dillondoyle 9y ago
    To be clearer it's same quota problem using cloud functions. Moving to something without the same quota/scale problems like Compute Engine or just DIY containers might make sense but then I'm back to a complicated system (Snowplow) that we have in AWS that's too complicated for me alone
    - doh 9y ago
      Understood. I think there is a bit simpler setup that you could utilize, but it does require a custom deployment on GCE. Request -> load balancer -> auto scaled pubsub writer -> pubsub -> client (write to BQ, ...) If it doesn't have to be a post request, then you could just write into pubsub directly. Pubsub has a wide support of libraries [0] so you could do it from any language. Pubsub also scales well. We're sending in around 300k messages per sec and have very few problems with it. [0] https://cloud.google.com/pubsub/docs/reference/libraries https://cloud.google.com/pubsub/docs/reference/libraries

daxfohl 9y ago

My feeling is that Google cloud will always be hit and miss because their biggest customer by far is Google. So they'll do unbelievable scale on the stuff they use, and just passable on everything else. It's also why I think that Facebook never joined the rat race. Most of what they do has no relevance to the average company. Amazon, an online retailer, was really the most natural place to start an outsourcing of banal technologies for the largest market segment. And has grown from there.

helpfulgoogler 9y ago

According to [1], the standard quota is 1,000,000 invocations per 100 seconds and can be increased. Maybe try asking again for increased quota? If the documentation is incorrect, please file an issue [2]. Counter-intuitively, ending the response before processing the data may actually be hurting your p99 response time. According to [3], instances which are not currently handling a request get very little CPU and generally don't make any progress. This means that this work will still be waiting to be done when the next request comes in and can slow it down. You may even end up with an instance trying to process several requests' data and also trying to handle another request on top of that. This last request is going to get an overloaded instance and likely be very slow. Further, the function isn't guaranteed to ever be run again, so that deferred work might not even ever happen. It probably will cause you to use more of your invocations per 100 seconds quota, but you really shouldn't start background processing tasks in a Google Cloud Function that continue after completing a request. [1] https://cloud.google.com/functions/quotas https://cloud.google.com/functions/quotas [2] https://issuetracker.google.com/issues/new?component=187195&template=0 https://issuetracker.google.com/issues/new?component=187195&... [3] https://cloud.google.com/functions/docs/bestpractices/tips https://cloud.google.com/functions/docs/bestpractices/tips

dillondoyle 9y ago
I have paid support. It's socket connections we're hitting super quick. Interesting and thanks I had no idea on the end response CPU time. It's just storing the json to Google Cloud Storage but it does take 100-300ms it seems!

doh 9y ago

I don't know your flow, but how about using a queue system to prevent the bursts of data? We're internally using pubsub (and nats) sending billions of messages every day. We don't use the push mode as Spotify [0] but pull from the queue instead which allows us to run on our pace. We do write into bigquery (and citus) and each components is doing it on their own pace based on what they're capable of process at the moment. [0] https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-consumers/ https://labs.spotify.com/2017/11/20/autoscaling-pub-sub-cons...

bruth 9y ago
What workloads do you prefer pubsub over nats and vice versa?
- doh 9y ago
  We use nats mostly for synchronous tasks (where the response has to be returned to the exact client initializing the request) and pubsub for everything else. We’re slowly transitioning to nats as we have a better control over what’s happening there. Also we had no issues with stability which I can’t say about pubsub
dillondoyle 9y ago
Pub Sub is the same quota hits (or +1 /connection). I could use pub sub but I still have to accept the incoming POST before publishing a message.
- dillondoyle 9y ago
  To be clearer it's same quota problem using cloud functions. Moving to something without the same quota/scale problems like Compute Engine or just DIY containers might make sense but then I'm back to a complicated system (Snowplow) that we have in AWS that's too complicated for me alone
  - doh 9y ago
    Understood. I think there is a bit simpler setup that you could utilize, but it does require a custom deployment on GCE. Request -> load balancer -> auto scaled pubsub writer -> pubsub -> client (write to BQ, ...) If it doesn't have to be a post request, then you could just write into pubsub directly. Pubsub has a wide support of libraries [0] so you could do it from any language. Pubsub also scales well. We're sending in around 300k messages per sec and have very few problems with it. [0] https://cloud.google.com/pubsub/docs/reference/libraries https://cloud.google.com/pubsub/docs/reference/libraries