8 ms·

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap

• everdev 8y ago
  Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.
  • heavenlyblue 8y ago
    Well, a rogue line could potentially as much shut down your app hosted in DO.
    • tgsovlerkhgsel 8y ago
      That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill. When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down. When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.
      • heavenlyblue 8y ago
        Well, then the issue with the Amazon Lambda is not that of it's ability to scale, but that of it's default not to limit your spending.
  • marktangotango 8y ago
    What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?
    • Something1234 8y ago
      One bankrupts you, the other just has a minor hiccup with a few lost customers that day.
      • ithkuil 8y ago
        a (cost) cap is a feature designed to prevent you from going bankrupt but at the expense of not recovering from the surge after you deplete the cap until the end of the cap granularity. So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it). This problem is a function of the granularity. Imagine a system that let you say: "I want to spend max $n per second with a extra burst of $m per day/week" You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN. The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.
    • deleted 8y ago
      [deleted]
  • vbezhenar 8y ago
    Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.
    • Xorlev 8y ago
      For services that are pay per request, this is effectively a hard cap. For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.
• ghaff 8y ago
  From what I understand, AWS at least is pretty good about giving you at least one "get out of jail free" card if things go awry. Caps are tough though. I can certainly understand a use case that would want a hard circuit breaker that just kills everything it can once it hits a certain threshold. Sort of; you presumably don't want everything on S3, for example, to be deleted. On the other hand, moving up the scale of serious businesses, I can imagine it would be hard to specify circuit breakers (rather than just alerts) and you get into issues of terminating services that affect all sorts of other services across the entire account.
  • thieving_magpie 8y ago
    There's nothing guaranteed about a "get out of jail free" card, so factoring that into your decision making is probably a bad road to go down.
    • ghaff 8y ago
      I'm not suggesting that. For a lot of casual users I'm not sure that cloud services are necessarily the best approach for an externally-facing service. They may well be better with a VPS that has a hard resources/dollar limit. Perhaps cloud providers should have some sort of hard circuit breaker option (though it won't help for some things like storage) but it's probably not a priority as not a lot of businesses--their primary customers--would be OK with effectively hitting the power button for their entire cloud account if they exceed some dollar amount that someone or other configured a couple of years ago.
  • itcmcgrath 8y ago
    Caps are definitely hard, getting them right as both a feature as well as setting them as a customer is still an art. This holds true for 'flat rate' too, where your trading correct sizing vs downtime when either get code wrong or get unexcepted success. I've been involved in all these situations so I definitely emphasize. We had daily hard caps and budgets alerts, but it's still an area we can do better. (Disclaimer: Product Manager for Cloud Firestore)
• itcmcgrath 8y ago
  Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.
  • alasdair_ 8y ago
    >Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it. This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).
    • iamgopal 8y ago
      So basically we want to have cake and eat it too.
      • ajmurmann 8y ago
        Or have it be adjustable within a reasonable time frame?
      • paranoidrobot 8y ago
        This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
        • badlucklottery 8y ago
          >that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
• vuln 8y ago
  You only have to be 'terrified' if you don't write good code. Maybe people should actually test their code before pushing it to production? It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '
  • dang 8y ago
    Please don't be rude. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  • JoJoJoboop 8y ago
    Well, since Firebase has lowered the barrier of entry with their easy setups, I think it is justified to expect at least some developers to make mistakes. I don't blame Firebase at all though -- great product.
• tgsovlerkhgsel 8y ago
  How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)? That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.
  • bufferoverflow 8y ago
    You should rate-limit and cache the potentially expensive paths of your app. There's no real way to protect against a clever DDoS, someone will have to eat the costs.
    • zwily 8y ago
      You can’t really rate limit server-side with Firebase. If a client has read-permission on a data node, an attacker can just read from there as many times as they want. There are some ways to limit writing using the rules engine but you’ll still get charged for failed writes. :) Only real way to rate limit firebase that I know of is to put some sort of proxy service in between, but then you lose a lot of the advantage of using FB in the first place.
• superfrank 8y ago
  > This is why infinitely scaling pay-as-you-go cloud services terrify me. It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does. The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go higher even if they wanted to. When we brought this up, they told us they were technically unlimited because you could shard your data into different DBs if you needed more connections, which is like saying all restaurants are all you can eat because you can keep buying more food.
  • itcmcgrath 8y ago
    Note this is about Cloud Firestore, not Firebase RTDB. It's very different infra. We announced our beta limits increased to 1M concurrent connections at the Next 18 conference, and we'll continue to improve from there.
  • fefb 8y ago
    The link posted is talking about the Firestore and not the Real Time. Firestore is in beta state that has a beta limit of 100k sim. connections, probably off beta it will be able to handle more connections. The Real Time has 100K limit and is a centralized db.
• jliptzin 8y ago
  I will also stay far away from these services. As a child post points out, if you do set a cap, you run the risk of cutting off a legitimate spike in high quality traffic. How do you tell the system to shut off due to an error and stay on during a spike in normal traffic? You can’t, unless you have someone watching it at all hours. That’s why I prefer to just rent whole machines on AWS. If I accidentally have some code stuck in an infinite loop making some O(1) call, don’t charge me $10,000 for that when it costs you nothing. If it’s actually consuming electricity and significant resources, I’ll know quickly because my service will go down, as it should, not scale infinitely until my company is bankrupt.