7 ms·

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them.

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

alasdair_ 8y ago
>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it. This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).
- iamgopal 8y ago
  So basically we want to have cake and eat it too.
  - ajmurmann 8y ago
    Or have it be adjustable within a reasonable time frame?
  - paranoidrobot 8y ago
    This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
    - badlucklottery 8y ago
      >that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
      - Dylan16807 8y ago
        The problem is that the user gets those alerts, tries to change the limit they set, and it doesn't work. That is definitely not as good as can be done.
- itcmcgrath 8y ago
  Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.
  - cheeze 8y ago
    This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly. Its a problem with most cloud providers, but Google seems to be notorious for it.
    - welanes 8y ago
      From my experience Firebase are different. They're now a part of Google but support has been excellent anytime I needed it - both before and after they were acquired.
    - deadbunny 8y ago
      In my experience none of those folks are paying for support.
    - Humdeee 8y ago
      It's quite the phenomenon. Reaching out via social media to garner a company's attention when all other avenues have been exhausted... what a world we live in. "Company ignoring you? Send out a tweet, that'll work!" And it blows my mind that it actually does. It's very sad.
- benguild 8y ago
  Yeah, that would happen with the Google Maps API before. 150k request limit was not enough for a front-paging, and there was no rollover.
- onion2k 8y ago
  Your app shouldn't just fail if an API is unavailable. You should code things to fail gracefully. In the case of a traffic spike replacing the front end with an email capture form and a message saying "things are really popular right now, how about we remind you about this tomorrow (with a discount code!)" works well. This is hard for some things, but your startup failing because you didn't want to do it is much harder in the end.
  - asadjb 8y ago
    Most applications have atleast 1 SPOF, the DB. Just because the DB is now the Firebase API doesn't change anything. It would be very difficult to build products in a reasonable amount of time if everything has to be coded defensively. I can build my app quicker, and remain sane, if I assume that the DB will always be available, and just fail if the DB isn't there. Same for things like S3 (which I think had only 1 large scale failure in recent history), Redis, etc... There are APIs which can be unavailable and you need to work around those. For me, these are mostly third party services that I don't control. But then again, I'm not building the next Netflix. I don't have enough engineers to build an app that works with a chaos monkey! Not the best approach for all applications, but has been good enough for most projects I've worked on. Just my 2 cents.
- stickfigure 8y ago
  This is a pretty common train of thought, but it's not really a question of deciding between "my bill will explode!" and "my app will go down!" You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't look like a customer feature. Without these sorts of automated scaled services, the traditional behavior is "your app goes down". This is a big improvement!
  - jliptzin 8y ago
    No, if your app is failing in a way that a page view now costs $1.00 instead of $0.0001 for example then the application should go down.
ShaneOG 8y ago
A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.
- magnetic 8y ago
  How did you get your Startup credits?
  - ShaneOG 8y ago
    My company was part of a partner acceleration program, however you can apply to be part of the Google Cloud for Startups program here[1], without needing to be in an accelerator. [1] https://cloud.google.com/developers/startups/ https://cloud.google.com/developers/startups/
    - magnetic 8y ago
      Thanks guys. Appreciated
  - purerandomness 8y ago
    Not OP, but when our startup was accepted into https://www.thefamily.co/ https://www.thefamily.co/ (which is basically a European YC), amongst other perks, we got a huge amount of AWS credits. I'm sure other incubators/accelerators do similar offers for GCE/AWS.
mickronome 8y ago
I would very much like to have something like Google authenticator, but for billing. With the ability so set it as alarms on my phone (and coworkers), preferably with some smarts to detect short usage spikes. In essence, settings and updating amount, rate and velocity ( speed of rate change ) caps on the fly. Then I can set whatever tight limit I want to, and not worry about burning through too much cash because of some simple coding, or config error. A bug almost cost us several tens of thousands of BigQuery costs when a dev accidentally repeated a big query every 5 seconds in an automated script, and while we still had budget warnings, it still cost us a fair bit of money. Even after this, I found it tricky to set/catch budgets for single services. I think I had to use stackdriver to be able to get any kind of warning. It was in the ’blinking lights and sirens’- territory fast!
kylec 8y ago
Are the caps on by default? If not, they should be.
tkrupicka 8y ago
Hey! This might not be the best place to contact you, but my team is currently in crisis from a nasty Android Firestore bug. When the user's device switches networks it loses the real-time event listeners and in some cases doesn't re-establish until the app is reinstalled. We submitted a bug report and source for an app with the issue but haven't heard anything in a week. We love the product otherwise, but I thought I'd bring it to your attention.
- puf 8y ago
  Hey there. Thanks for letting us know. We unfortunately don't have a fix for this at the moment. We've seen reports on network connections not being reestablished, but have been unable to reproduce it so far. If you've filed a bug report, the best way to help us reproduce it is by adding details to the report. That then also gives our team a way to get back in touch with you once they're able to reproduce (and then fix) the problem. Thanks again for the report!
snowwolf 8y ago
How about having default conservative alerts built in for new accounts (not caps, just alerts). That way people who forget to set them will be reminded the first time they get one. An account that goes from $0 spend to $30k in 72 hours should really trigger some kind of flag - even internally within google. What if they didn't have any kind of grant and weren't able to pay?

alasdair_ 8y ago

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it. This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).

iamgopal 8y ago
So basically we want to have cake and eat it too.
- ajmurmann 8y ago
  Or have it be adjustable within a reasonable time frame?
- paranoidrobot 8y ago
  This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
  - badlucklottery 8y ago
    >that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
    - Dylan16807 8y ago
      The problem is that the user gets those alerts, tries to change the limit they set, and it doesn't work. That is definitely not as good as can be done.
itcmcgrath 8y ago
Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.
- cheeze 8y ago
  This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly. Its a problem with most cloud providers, but Google seems to be notorious for it.
  - welanes 8y ago
    From my experience Firebase are different. They're now a part of Google but support has been excellent anytime I needed it - both before and after they were acquired.
  - deadbunny 8y ago
    In my experience none of those folks are paying for support.
  - Humdeee 8y ago
    It's quite the phenomenon. Reaching out via social media to garner a company's attention when all other avenues have been exhausted... what a world we live in. "Company ignoring you? Send out a tweet, that'll work!" And it blows my mind that it actually does. It's very sad.
benguild 8y ago
Yeah, that would happen with the Google Maps API before. 150k request limit was not enough for a front-paging, and there was no rollover.
onion2k 8y ago
Your app shouldn't just fail if an API is unavailable. You should code things to fail gracefully. In the case of a traffic spike replacing the front end with an email capture form and a message saying "things are really popular right now, how about we remind you about this tomorrow (with a discount code!)" works well. This is hard for some things, but your startup failing because you didn't want to do it is much harder in the end.
- asadjb 8y ago
  Most applications have atleast 1 SPOF, the DB. Just because the DB is now the Firebase API doesn't change anything. It would be very difficult to build products in a reasonable amount of time if everything has to be coded defensively. I can build my app quicker, and remain sane, if I assume that the DB will always be available, and just fail if the DB isn't there. Same for things like S3 (which I think had only 1 large scale failure in recent history), Redis, etc... There are APIs which can be unavailable and you need to work around those. For me, these are mostly third party services that I don't control. But then again, I'm not building the next Netflix. I don't have enough engineers to build an app that works with a chaos monkey! Not the best approach for all applications, but has been good enough for most projects I've worked on. Just my 2 cents.
stickfigure 8y ago
This is a pretty common train of thought, but it's not really a question of deciding between "my bill will explode!" and "my app will go down!" You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't look like a customer feature. Without these sorts of automated scaled services, the traditional behavior is "your app goes down". This is a big improvement!
- jliptzin 8y ago
  No, if your app is failing in a way that a page view now costs $1.00 instead of $0.0001 for example then the application should go down.

ShaneOG 8y ago

A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.

magnetic 8y ago
How did you get your Startup credits?
- ShaneOG 8y ago
  My company was part of a partner acceleration program, however you can apply to be part of the Google Cloud for Startups program here[1], without needing to be in an accelerator. [1] https://cloud.google.com/developers/startups/ https://cloud.google.com/developers/startups/
  - magnetic 8y ago
    Thanks guys. Appreciated
- purerandomness 8y ago
  Not OP, but when our startup was accepted into https://www.thefamily.co/ https://www.thefamily.co/ (which is basically a European YC), amongst other perks, we got a huge amount of AWS credits. I'm sure other incubators/accelerators do similar offers for GCE/AWS.

mickronome 8y ago

I would very much like to have something like Google authenticator, but for billing. With the ability so set it as alarms on my phone (and coworkers), preferably with some smarts to detect short usage spikes. In essence, settings and updating amount, rate and velocity ( speed of rate change ) caps on the fly. Then I can set whatever tight limit I want to, and not worry about burning through too much cash because of some simple coding, or config error. A bug almost cost us several tens of thousands of BigQuery costs when a dev accidentally repeated a big query every 5 seconds in an automated script, and while we still had budget warnings, it still cost us a fair bit of money. Even after this, I found it tricky to set/catch budgets for single services. I think I had to use stackdriver to be able to get any kind of warning. It was in the ’blinking lights and sirens’- territory fast!

kylec 8y ago

Are the caps on by default? If not, they should be.

tkrupicka 8y ago

Hey! This might not be the best place to contact you, but my team is currently in crisis from a nasty Android Firestore bug. When the user's device switches networks it loses the real-time event listeners and in some cases doesn't re-establish until the app is reinstalled. We submitted a bug report and source for an app with the issue but haven't heard anything in a week. We love the product otherwise, but I thought I'd bring it to your attention.

puf 8y ago
Hey there. Thanks for letting us know. We unfortunately don't have a fix for this at the moment. We've seen reports on network connections not being reestablished, but have been unable to reproduce it so far. If you've filed a bug report, the best way to help us reproduce it is by adding details to the report. That then also gives our team a way to get back in touch with you once they're able to reproduce (and then fix) the problem. Thanks again for the report!

snowwolf 8y ago

How about having default conservative alerts built in for new accounts (not caps, just alerts). That way people who forget to set them will be reminded the first time they get one. An account that goes from $0 spend to $30k in 72 hours should really trigger some kind of flag - even internally within google. What if they didn't have any kind of grant and weren't able to pay?