26 ms·

A startup’s Firebase bill suddenly increased from $25 to $1750 per month

• Yizahi 9y ago
  Google is famous for not having any humans in support. One robot will write abuse on you and another robot will happily oblige and ban you with no appellation possible. Building whole business on one of such platforms takes some "courage" :) .
  • Ygg2 9y ago
    I assume a same way a man sticking his hand in a blender and turning it on, is courageous.
  • victorhooi 9y ago
    What you said is quite untrue. I know - because I am one of those humans in support grins. I work in the support organisation for GSuite (formerly Google Apps). And I work across the hall from the guys who support GCP (Google Cloud Platform). And there are an entire army of us, helping support many, many enterprise customers. Even our consumer products have real people manning support phones - I know, because I've had to call them before (mostly Google Play Music issues - this was before I joined Google, so I couldn't use my internal pull...lol). Regarding your second point - abuse is a real problem for any cloud provider. I know, because I happen to work in anti-abuse. It's a difficult problem to crack, and there's a lot of smart people (at many companies) trying to solve it. But please understand there are real humans at the other end, trying to provide a good service - and it's not very kind to take cheap shots at them. There usually are appeal processes available for nearly any kind of action taken - if you were actually personally affected by an issue, or feel unsure about how something was handled - please feel free to reach out to me. I can't guarantee I can wave a magic wand and fix it, but I'm certainly willing to reach out to see what can be done. Disclaimer: I work for Google, but the views expressed here are purely my own.
    • yeukhon 9y ago
      Apparently time after time stories arise with Googlr lack of support and are very slow at answering ans helping. In this very damn case no freakin' response. Often people have to get a Googler to initiate a Hello wothin the organization. So while we respect you doing your day work, we think your organization as a whole fail at customer support. Yeah I have read sicceyss stories too but many of them seen to be businesses with really good pocket or have deep connection with Google already or just being lucky. Your competitor AWS would refund in less than an hour with a billing support, without going through any phone calls. Your competitor has a support subscription service at least for people to purchase (expensive but worth it). Yours?
      • victorhooi 9y ago
        Often, these types of blog posts only present one side of the story - so I would caution a bit of patience, to see the full story. This applies no matter what company is at the other end - whether it's us, Amazon, Facebook etc. In this case, I know there's a few technical details missing - I won't comment further on this specific issue, as far more qualified people than me are already on this issue =). Many times, I've seen it's due to a misunderstandings, or crossed-wires regarding technical issues. And yes, our support team does provide refunds in many cases - I know, because these sometimes get escalated to me to approve (e.g. if they're borderline). I think one was actually offered already in this case, but there may have been a misunderstanding or mix-up around that. And yes, we have a "paid" support service - it's actually comes free with any GSuite subscription (even at the most basic level, which I think is like $5 a month). You can chat, email, or phone. I work as an engineer in the team that supports this. For GCP (our cloud offerings), the support offering is a paid offering: https://cloud.google.com/support/ https://cloud.google.com/support/
    • Top19 9y ago
      I hired a freelancer from the Philippines on Upwork to work with me on some Google App Engine issues because Google's documentation is incredibly unhelpful at times and as well as bizarrely organized. He was from the Philippines, and it turned out he was a contractor providing "support" for Google Cloud. So I guess Google is outsourcing their support? Isn't this something Dell tried to do in the 90's with disastrous results?
• staticelf 9y ago
  I think Firebase was kind of douchebags assuming the story is correct, if you change how people are billed you should give people time to adjust their software. Stuff like that makes me want to host everything myself as it will for sure be the cheapest option.
  • tmikaeld 9y ago
    Hosting by yourself is always the safest long-term solution, even if you are hosting it in the cloud, you at least protect yourself from this "Company reserves the right to change the TOS at any time..."
  • ceejayoz 9y ago
    Yeah, ideally they'd have simulated the change for a while, and given folks a few months of "your bill soon will be $xxx" emails.
• throwaway13337 9y ago
  Look to stories like this before you go 'serverless' in your organization. Building on top of a proprietary stack with no guarantees on pricing and future availability will lead here.
  • Nemo157 9y ago
    It doesn't sound like it's the 'serverless' hosting that is their issue, rather that they have no direct control over the endpoint that their difficult to upgrade client software is hitting. They actually fix it in future versions of the client software by putting controllable serverless endpoints in front of the firebase endpoints they're hitting, so if something like this occurs again they could just change their endpoint to not hit the third-party resource that is costing them so much money.
    • raesene6 9y ago
      The more general problem with serverless architectures that this post illustrates is that you're essentially running on a series of FAAS/SAAS style services which can change their billing at any time. So you either have to code a portability layer so you can move to another one in the event of an unwanted change (which could be quite a lot of work), or you accept the risk that a 3rd party could cost you a lot of money until you can change off their service (which could be quite technically challenging depending on the complexity of your use case and the availability of comparable alternatives) Whilst with things like container services you retain more control (ultimately you can self-host if need be) as soon as you step onto pure serverless you'll always be at the mercy of the providers of the services you use.
      • brazzledazzle 9y ago
        I think they make it pretty clear the problem is with their inability to change the endpoint (i.e. the URL). Switching from lambda to something else could be a pain in the ass but they'd be hitting their own domain and changing where that points is relatively trivial. Right now they have clients out there hitting a Firebase endpoint that they have no control over and that they can't update quickly.
• jakozaur 9y ago
  Yet another example that Google so far doesn't get the public cloud and may have hard time to catch up with Amazon Web Services. E.g. shutdown Prediction APIs, charge 70x, not enough support. I believe Google got much better tech than AWS, but the culture, support and strategy has blind spots.
  • _pmf_ 9y ago
    > I believe Google got much better tech than AWS I get the impression Google is doing a lot of things different for the sake of doing things differently. AWS is much more pragmatic and transparent.
  • steego 9y ago
    I think Google gets the public cloud. What they struggle with is lower margin, revenue driven, customer obsessed businesses. First, it's important to understand unless you're buying ads from Google, or renting their machines, you are not their customer. Users are inventory and they spend a lot of money trying to procure inventory (users) for their customers (advertisers). When you look at it through this lens, you can see how Google as an institution can tolerate experimenting with ideas that are disconnected from revenue models and dropping failed ideas like hot potatoes. It makes sense for them to behave that way. Amazon's services are almost always designed with either a concrete or direct path to revenue upfront and they have a huge customer support infrastructure they can tap into. They move a little slower in certain areas of R&D because customer obsessed businesses are tolerant of deprecating technology when it doesn't work out. That means they'll do more market research, take less risks and focus on tenaciously steady growth rather than trying to always hit it out of the park. When choosing a vendor, it's important to understand how their offerings affect the solvency of their business and whether it fits into their core competency. In this case, Firebase's prices were too low. That should have been a warning sign. The other warning sign is Firebase is a such a small percentage of Google's revenue, Google doesn't have a huge incentive to fix customer service issues.
    • ChicagoBoy11 9y ago
      *"customer obsessed business are <less> tolerant of deprecating technology..."
• dullgiulio 9y ago
  Using Firebase or similar is akin to opening a restaurant that serves take-away pizzas from a nearby Pizza Hut, after nicely putting them in a porcelain plate. If Pizza Hut changes the pricing, you either close shop or adapt. Who would open such a restaurant?
  • deleted 9y ago
    [deleted]
• shaqbert 9y ago
  Pricing mistake 101: You never ever change those old plans. Instead, you grandfather them. Especially so if you're still a young platform and have most growth ahead of you, the "loss" of not charging the new - and supposedly higher - pricing is gonna be trivial 2 years from now, with lots of new projects coming in. But the NPS hit from pissing your most loyal users off and the subsequent damage to your growth curve are huge. Also, I do love Bezos mantra of always becoming cheaper over time. That is the core of Amazon's brand. Whereas Oracle, IBM, and the old guard always make you pot committed and charge an arm and a leg, the Amazon guys will leave money on the table. That is how you earn trust.
  • manigandham 9y ago
    > the Amazon guys will leave money on the table What? they're a business like any other and they're not interested in minimizing profit, there's lot of thought that goes into their pricing and you can be sure that your opinion and trust is an outcome of their strategy to make more money.
    • philfrasty 9y ago
      „...you can be sure that your opinion and trust is an outcome of their strategy to make more money...“ Is exactly what the parent comment said...
      • thr0waway1239 9y ago
        ..making up in scale what you lose in margin. I think Amazon pulls this off because their prima donna is the paying customer, not the programmer who thinks talking directly to customers is the worst possible reflection of their ability to automate. Automation is great and all, but Google has a major problem when it comes to empathizing with the customers. Also, is there any possibility of legal action against Google for this bait and switch, which seems to be quite clearly the case here?
        • fovc 9y ago
• piokoch 9y ago
  I am so surprised why people buy into some SAAS software that does simple things instead of hiring 2 programmers from cheap location (somewhere between India and Poland western border), getting Hetzner VPS or dedicates server (even the second option is cheap, including a few administrator hours) and be done. It is not so hard to predict failure/buy out of some SAAS provider, it happened so many times. I am curious why Venture Capital investors are not looking at that problem closely.
  • stephenr 9y ago
    > Venture Capital investors Nothing in the entire VC Funded startup scene makes sense when you try to apply basic logic, much less a sane way to operate a sustainable business. But that's just it: These companies are funded by people who are simply in it to make a profit, (as opposed to people who want to actually achieve something with their efforts). Users become dollar signs, and it's all about how quickly you can get enough new users to monetise the bejeezus out of their attention/personal information/etc, before they dump you for the next new flavour of the week.
  • bolololo1 9y ago
    Still cheaper for a single dev starting a startup to hook up to Firebase/AWS/Azure than hiring 'cheap' devs from Poland.
    • camus2 9y ago
      > Still cheaper for a single dev starting a startup to hook up to Firebase/AWS/Azure than hiring 'cheap' devs from Poland. Yes but you gotta be smart here. It's not like Firebase is the only database out there. And once your product grows, and you hit Firebase limits which forces you in the "pay as you go" plan, whatever cheapness you thought you bought into quickly becomes a huge expanse.
      • bradynapier 9y ago
        Precisely where we find ourselves and the inspiration for the article.
• octo_t 9y ago
  I find it hard to reconcile this suggestion at the bottom of the article > Always build your architecture in a way that will avoid becoming trapped into a specific service. Amazon’s AWS Lambda sitting between any services and your app is a strongly recommended path! - Build without depending on external services - You can do this by depending on this external service
  • chaostheory 9y ago
    I don't feel that relying on an external service is always bad as long as there's no vendor lock-in with technologies that are open standards. Case in point the managed postgresql and mysql services that Amazon, MS, and Google all support. That said I'm extremely wary of Cloud Spanner and Azure Cosmos DB or any specialized service that the big 3 offer.
  • ceejayoz 9y ago
    Lambda functions can be built in a pretty portable manner. You can build and test them locally, after all.
• DomBlack 9y ago
  > Always build your architecture in a way that will avoid becoming trapped into a specific service. Amazon’s AWS Lambda sitting between any services and your app is a strongly recommended path! Isn't using AWS Lamba as your gateway, trapping you into using a specific service, just as much as firebase was?
  • IanCal 9y ago
    Sounds so to me, all that takes is lambda changing how they charge and suddenly your hard-coded app is stuck again.
    • beambot 9y ago
      Solution here is to make sure all api endpoints and URLs used in your code (eg into services such as lambda) get mapped through a domain that you can control / change via DNS. Then you can always cut off a service or redirect all requests to a different endpoint of your choosing (eg another serverless system or your own backend). This is better than hardcoding an unalterable endpoint / url / domain in your code that is beyond your own control, and it is doubly important if updating code on your product is difficult or impossible.
  • lultimouomo 9y ago
    This times and again. If you're hitting an API hosted on a domain you don't control, you're bound to get screwed. Otherwise they would just be able to deploy a simple Firebase API proxy (for the single call that they use) that uses TLS tickets and keep alive, change the DNS records and call it a day.
  • floatboth 9y ago
    You could then switch to regular servers running an open source Lambda reimplementation e.g. https://github.com/siscia/effe https://github.com/siscia/effe https://github.com/atlassian/localstack https://github.com/atlassian/localstack etc. Just make sure you can change the URLs everywhere.
  • moduspol 9y ago
    You wouldn't have clients talk to AWS Lambda directly. You'd put it behind API Gateway, which would allow you to use your own domain in front. Your clients would just know to send requests to https://myapi.mycorp.com https://myapi.mycorp.com. Since you control the domain, you could then stop something like this more quickly.
• flipp3r 9y ago
  Reminds me of getting Bait-and-Switch'd by Google App Engine in 2011. From 10's of euros to 1000's. > They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options. Sounds like Google did a great job with Firebase. Definitely gives you that Google-feel of unresponsive support and no hope when you have problems with any of their services. I bet they also have a "community" support forum or Google Group full of helpless people talking to a virtual wall. (Edit: Yep, they recommend Stack Overflow, Quora, and then their Google Group.. lol. Some things never change)
  • Aeolun 9y ago
    Jup, had the same issue with Google App Engine. Luckily I could contact support... "What do you mean, you retain a running instance of every previous instance I pushed?!" Fun times.
    • mosselman 9y ago
      "you retain a running instance of every previous instance I pushed?!" What? Seriously? Is this documented anywhere? Would you recommend against using GAE?
      • mstade 9y ago
        > Would you recommend against using GAE? I've been burned so hard by them, I'd recommend against using anything Google.
        • Andrenid 9y ago
          It's getting to that point for me too now. I'm really really close to removing them from every aspect of my life. Professional and private.
          • bbcbasic 9y ago
• betageek 9y ago
  The detail that amazed me is the lack of visibility of the issue in reporting tools - it's all very well changing the terms, but not giving your customer any ability to see what is increasing the bill seems massively unfair.
  • wowtip 9y ago
    "It costs this much because we say so"
  • Hedja 9y ago
    Exactly the reason why I decided not to pay for Firebase and instead went with gradually rolling my own solutions. Their Realtime Database is extremely convenient and pretty useful for web apps, but there's no way to limit the connections and track bandwidth usage in real time. It's basically an open door for anyone with only your Security Rules to block data access. There's no logs to even see if your rules are working as expected, nor where your connections are coming from. They didn't even have that simple blue line showing basic usage in OP's post until recently. The profiling tool mentioned in the OP is also recent and doesn't give a proper picture. It's just really pathetic. They sell themselves as a service company for startups, but one of the biggest things startups need to do is manage their costs, which Firebase doesn't provide in any form. And now since Google bought them, their support will be near non-existent as with all Google products. Having said all that, their Free tier is great for live prototypes and doesn't require adding payment information. Just make sure you're not tying yourself in if you want to progress. Make thin wrappers around their libraries and export your data regularly if it's not throwaway.
    • kefka 9y ago
      "Welll, we have this there chalkboard that we write what we think your price is. And some intern draws lines on a graph to match what we think it is. Oh, proof? Nahh, we dont do that. Oh, logs? Nope, you'll just have to trust us..."
  • tlarkworthy 9y ago
    The database usage tab in the web console tells you how much bandwidth you have consumed including SSL. Author's issue was with the commandline profiler. (A Firebase dev)
• factsaresacred 9y ago
  I use Firebase and find it easy, useful and constantly improving (cloud functions etc.). Their support, however, is a joke. 5 questions can be logged per year for one-on-one email support and then you're on your own. And this is for urgent requests. I can understand not wanting to be inundated with wasteful questions but when Firebase is changing their API and pricing plans and you're trying to scale a product, and paying them lots of money in the process, support should be forthcoming. They also don't respond on their Twitter handle, which is a further crime.
  • doczoidberg 9y ago
    they respond quite fast on the google group. I had never problems with firebase support
  • gman83 9y ago
    https://firebase.community/ https://firebase.community/
  • pier25 9y ago
    For technical questions Puf seems to be telepathically connected to StackOverflow. He has answered any of my questions within a an hour or two at the most.
  • danholevoet 9y ago
    Dan from Firebase here. I’d like to clarify a few details about the support options that are available. First, the limit of five questions per year applies only to technical troubleshooting questions (i.e. some indeterminate issue in your code that you’d like help with). Developers are always able to contact us an unlimited number of times for issues relating to identified bugs or to request features. Additionally, any questions relating to your account or bills (such as the issue in the original post) are also unlimited. With that said, it’s been a year since we introduced the five-a-year limit on technical troubleshooting questions. I’ll see if this is actually a useful limit to keep in place and remove it if not.
    • factsaresacred 9y ago
• alexfi 9y ago
  > They said in most cases it doesn’t make a big difference… unless you use the REST API. So in the end it was a coding problem, because you forgot to add Keep-Alive = true? If you use the provided SDK's it should be no problem then right?
  • Buge 9y ago
    I don't think Keep-Alive = true would be enough. The article also mentions TLS session tickets would be necessary. And it's hard to call this a coding problem if the requirements of the API never specified that session tickets and keep alive were necessary.
    • alexfi 9y ago
      Yeah it's no coding problem on purpose, but sth you could actually fix. But it's not good manner to not mention it from Firebase.
      • mining 9y ago
        It's absolutely good manners for Firebase to tell you "If you fix this thing, your costs will go down."
        • izacus 9y ago
          > It's absolutely good manners for Firebase to tell you "If you fix this thing, your costs will go down." I wonder how many HN startup founders go around telling their customers on how to pay less.
    • weddpros 9y ago
      With keep-alive you may not even need TLS session tickets (if the keep alive timeout is long enough). keep-alive keeps the HTTP/TLS socket open. TLS session tickets allow to close and "cheaply" reopen TLS session on a new socket for a new HTTP connection. Using both keep-alive and TLS session tickets is optimal.
• bradynapier 9y ago
  I authored the post and yes you are correct about the AWS Lambda being another service. I have since modified it based on a users suggestion to reflect. I guess the point was that by adding the middleman between it there was more control --- We are self-funded and small group of people. We can't afford to hire a bunch of people to know about containers and servers and every little detail. Thanks for your comments! Didn't expect so many views so quickly on this...
  • geggam 9y ago
    Investigate SRV records for load balancers as well.
• youngtaff 9y ago
  "This includes failed attempts which are blocked by their security rules." Ouch!
  • kogepathic 9y ago
    > Ouch! Yeah, no kidding. It means someone nefarious (or bored) can easily run up a huge bill by just directing a ton of invalid requests to their API. Many inexpensive hosting providers provide you with a lot of bandwidth (e.g. Scaleway offers unlimited bandwidth at ~200MBit/s). Assuming you can saturate the link, that's around 64TB per month of traffic. Quite the Firebase bill they'd have.
    • bryanrasmussen 9y ago
      well it's not like anybody has the business model of doing denial of service attacks that stop when the ransom is paid, because if someone did have that business model this seems like a great way to make sure people will pay you. I mean just do like enough to cost $500 and say pay me $1000 today or you will pay google $10000 tomorrow.
  • Jumziey 9y ago
    I'm guessing that spamming invalid request on all of firebases customers endpoints is the most efficient way to deal with the whole ordeal... If a large enough customer base gets the problem they'll have to act... Unless they can happily close up shop because Google owns them... Ah well :(
• deleted 9y ago
  [deleted]
  • deleted 9y ago
    [deleted]
• apatters 9y ago
  Wowow! In this position my first step would be to determine if there's any legal basis for disputing the charge (an informal conversation with a lawyer or even just someone who's been through a similar experience is a fine start). Even when a billing change is disclosed in advance and the customer actually uses the bandwidth, massive spikes in metered pricing are a tricky area and have attracted the attention of regulators in the past--for instance telecoms have come under fire by charging sky-high overage fees for mobile data and not warning you until you've already racked up a few extra zeroes worth of charges on your next bill. Also, immediately dispute the charge with your credit card company, as you want this on their radar. Their interests are aligned with yours to at least some degree (if Google's causing financial distress for one of their customers with thousands of dollars in unanticipated charges, how many others may get hit?). Aside from that, if things played out as described: major dick move, Google! Is "do no evil" a bad joke these days? Seriously, though, Google is not a dumb company and my guess is that if the right person was made aware of this case (e.g., anyone who understands that the most important word in "customer relationship" is the second one), they would make it right. I'm totally on the side of the author: when possible, it's worth a little extra elbow grease to build your business on top of free, self-hosted software instead of proprietary SaaS. Think of it as insurance against cases like this one.
  • imron 9y ago
    > immediately dispute the charge with your credit card company This will result in the shutdown of their service, and ruin their app.
    • stemuk 9y ago
      I would also doubt that it would be worth it even without keeping in mind the risk of loosing the service. Google is usually pretty clever when it comes to TOS Agreements.
  • pbhjpbhj 9y ago
    "Don't be evil." was dropped in 2015 or thereabouts IIRC. I'd guess when they realised it was becoming more of a joke than a motto: "Google, the company who's motto is 'don't be evil' angered customers again today when ...
• BestHackerOnHN 9y ago
  My sympathies to the OP. That really sucks. But was the lengthy and masturbatory introduction really necessary?
• dhawalhs 9y ago
  Something similar happened to me early this month. Embedly went from $99/year to $99/month. I barely use it for a few image manipulation calls.
• jondubois 9y ago
  Synchronizing data in real-time is computationally expensive (this applies to both Operational Transform and Differential Synchronization approaches). The price of Firebase was probably kept artificially low to attract new users but this was a ticking timebomb. Those people/companies who are using Firebase as their primary database are going to suffer now. It should really only be used for specific applications where datasync makes sense such as collaborative text editing. I wouldn't even recommend using it for chat because it's really easy for the complexity to get out of hand there and you can usually achieve similar results using a combination of pub/sub and REST.
  • josephg 9y ago
    > Synchronizing data in real-time is computationally expensive No, its not. I was responsible for implementing, deploying and managing the infrastructure at lever.co when we were a tiny fledgeling startup. The entire application is built on top of JSON OT. I took some measurements at one point when we had ~thousands of active browser sessions of our app. At the time we were seeing about 1-2 OT merges (transform + re-apply) per day. All the other non-concurrent operations can get sent straight to the database. I don't know what the numbers are now, but I'd be shocked if OT ever becomes the bottleneck. For text based OT you'll see more concurrent edits. But for text based OT, I have a little C library that can comfortably do 20 million simple text OT transforms/second on a single core of my old 2012 macbook air. Good luck making that a bottleneck.
    • jondubois 9y ago
      I guess it somewhat depends what kind of OT we're talking about (text or JSON). In any case, I don't think that the bottleneck would be the OT algorithm itself - More likely, the bottleneck would be the number of messages (HTTP requests or WebSocket frames) required to send each individual operation between the client and server. If you have text-based OT and you send an operation over the wire each time a user presses a key, and if you do this for every single input field in your app, it's going to add up. Not saying it's not feasible but it's not going to be as fast as making a plain REST call for those use cases that don't require synching. That said, I wouldn't be surprised if OT turns out to be much faster than differential transform in terms of raw algorithm speed but again I think the bottleneck will be the number of frames/requests that need to go over the wire.
• stemuk 9y ago
  Did a Firebase team member already comment on this? In a few hours Google will present some new Firebase features at io17, and I would expect them to avoid bad press now as much as possible...
  • kuschku 9y ago
    That's probably the purpose of this post — the best way to get support from Google, after all, is still by posting a story on reddit, twitter or HN. Which is pretty sad.
    • stemuk 9y ago
      I think it's kind of weird to see Google/Firebase team members swarming around in dozens on HN when they present a new feature, which seems to me like a coordinated marketing effort by Google. But once problems arise or they get criticized by their own users, no one seems to be available for a comment.
      • mysterydip 9y ago
        I was about to post the same thing. Of the two Google cloud stories I saw on here yesterday, it felt like one in three posts had "disclaimer: I work on the Google cloud team" or something to that effect. As of this writing, there's 100 comments over 4 hours and not a peep from any of them that I see.
        • stemuk 9y ago
          Sometimes I am wondering who is actually steering the discussion to a certain point. I wouldn't go so far to call it censorship, but if the Google employees take over the comment section, it is very hard to call the flow of information "unbiased". YCombinator shouldn't let companies use Hacker News as a marketing tool, at least not to the extent Google does.
      • redwood 9y ago
        I was thinking the same thing. There's a giant circle of patting each other on the back every time they roll something out, and always have that very precise disclaimer that they work for the big G. Then again, as a meta comment I will point out that the fact that both you and I so clearly notice this suggest that we are following these updates much more closely then most of the folks here.
• neftaly 9y ago
  We've also gone from ~$25/mo to $1000-2000/mo with the pricing change. While it's not the end of the world (we're near shipping replacements for FB stuff anyway), there's also been 3-4 incidents of downtime (> 1 hour) in the last 2 months, with no way to contact anyone except their stupid web form.
  • brianwawok 9y ago
    Do you not pay for support? It starts at $150..
    • ocdtrekkie 9y ago
      The idea of having to pay extra to talk to them about their screw up is pretty laughable.
      • fpoling 9y ago
        It all depends how dependant customers on a product. At one point I worked for a company that successfully charged customers for fixing obvious bugs in a software on top of expensive support contructs. The sad part it worked for some time.
      • brianwawok 9y ago
        Why? I like the fact I can pay one price for self support and another for 15 minute resolution. Why force people to pay for support if they don't need it?
        • ocdtrekkie 9y ago
          I think there's a major difference between say, support figuring out how to implement their product, which is a you problem, their service is working fine. And a wholly different one if they screw up and you can't talk to them about it without paying them. The former is an incentive to self-serve and investigate yourself, the latter is an incentive for the company to provide shoddy service to people pay to resolve it.
• themihai 9y ago
  This is nothing new to me. I'm still working on my GAE/datastore migration...Price spikes, platform changes and deprecation are real risks when you use proprietary services/platforms.
• geekme 9y ago
  Google is notorious when it comes to support.
• kbart 9y ago
  That's the same, old vendor lock-in approach. It has happened with the servers, databases, network equipment etc. now it happens with cloud -- nothing unexpected, prepare to see more of these when cloud market stabilizes and clear leaders emerge. That's why I always prefer open solutions, that can be moved/migrated/replaced easily and advise others to do the same.
  • moduspol 9y ago
    Eh, it looks like the service provider hadn't taken into account an oddball use case in the pricing / metering, and then suddenly did. They certainly should have communicated that better and their tools / charts should have a way of showing the billable data, but it's a bit of a stretch to call it vendor lock-in. It's not like everyone on Firebase is complaining and the author admits they made major mistakes.
    • Silhouette 9y ago
      Just "communicating better" is not a reasonable response to making such a dramatic change in how much money you charge someone. If I tell you I'm going to punch you in the face, that doesn't make the punch acceptable behaviour. Cloud services that pull this kind of bait and switch deserve to be ridiculed and lose lots of business over the negative PR. Cloud services that unintentionally pull this kind of bait and switch but then don't make it right or provide meaningful support deserve no special treatment.
      • moduspol 9y ago
        I didn't say it's acceptable. I said it's not likely to be a vendor lock-in strategy, which is counter to the parent comment's claim.
        • Silhouette 9y ago
          From the GGP and some of your other comments here, it seems like you are arguing that communication is the main problem here rather than the change itself. Sorry if that was a misunderstanding.
• Kiro 9y ago
  > Obviously our app being shut down and our users being cut off was not an option. I wonder how many companies shut down due to this. My company is in a similar situation (not a tech cost but still) and it feels unbelievable that we may need to shut down even though we have so many customers. I never hear about these stories though, as if all companies have a large buffer that can handle these increases.
  • jstanley 9y ago
    Wouldn't it be better to port your app to another platform than to shut down?
• archiepeach 9y ago
  Is there a large difference between using Firebase Database and another hosted DBaaS, such as MongoDB Atlas? I use Firebase Database but keep it arms length by avoiding any of the proprietary features it offers. I use it exclusively as a JSON store which I feel I can easily export at any time if I wanted to switch to another NoSQL DBaaS. A few people here are mentioning how it's crazy to depend on external services for an app, but as an 'indie' dev, I can't spend time maintaining a secure, high-availability database. Or a mail server when I can use SendGrid, or a logging platform when I can use Sentry, or any of the other service my app depends on.
  • camus2 9y ago
    > A few people here are mentioning how it's crazy to depend on external services for an app, but as an 'indie' dev, I can't spend time maintaining a secure, high-availability database Nobody is saying that you shouldn't rely on external services. What's wrong on the other hand is to depend on a proprietary database offered by a single vendor. Firebase is a proprietary database, which means you can't go to the competition without having to rewrite your application. It's called vendor lock-in. Plenty of SAAS provide MongoDB as a service and you can install MongoDB on your own machine, so if a SAAS is price gouging you, you can move to the competitor with a cheaper plan. If you are not using Firebase ACL or Firebase realtime features then you shouldn't even be using Firebase at all. Don't use MongoDB either. Postgres provide JSON types and queries and most Postgres SAAS are much much cheaper than MongoDB SAAS.
    • douche 9y ago
      Exactly. Learn how to use an actual relational database. It's not that hard.
  • benjaminl 9y ago
    Firebase is one of the very few products to offer controllable client data replication for full offline first functionality. For apps that have significant offline requirements, this functionality can save a significant amount of time and significantly reduce the amount of code that needs to be written and more importantly maintained. One of my clients just recently seriously considered going with Firebase. The choice came down to Firebase and Couchbase, which has an even better offline functionality. After reading this post I am glad that we didn't going with Firebase. Of course Couchbase has their own problems. The supported version of Couchbase is prohibitively expensive for small organizations and Couchbase themselves strongly recommended against deploying the community edition. (The community edition also artificially lags behind the supported edition 6 months. Couchbase even delays critical security fixes that same 6 months.)
• CodeWriter23 9y ago
  $1/GB AYFKM? That's ridiculous. Should be about $0.01/GB.
  • motoboi 9y ago
    It looks like a good pricing model. To me it seems they correlate bandwidth usage with system usage and charge you using bandwidth as a surrogate of computing time, hardware and lahour (plus the bandwidth).
• duncan_bayne 9y ago
  "They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options." I really don't understand why anyone expects anything different from Google these days. Or possibly ever. "Don't be evil" ironically turned out to be a particularly evil marketing ploy.
• CodeWriter23 9y ago
  So, they discovered they were under billing by measuring bandwidth via the server logs instead of at the switch. Then proceeded to not tell a soul they would be making a change that has the potential to greatly increase your bill. Do I have that right?
  • yunolisten 9y ago
    Pretty much :)
• cwmma 9y ago
  They said the issue was due to TLS tickets and not setting keep alive values. TLS tickets are await of resuming a TLS session without having to renegotiate it, considering that you are only polling for a boolean, that negotiation overhead could be much greater then the actual being passed. This isn't something specific to firebase so that's why no library would mention it. Keep alive is a flag that allows you to reuse a single connection instead of closing it, if it's closed then again you have to renegotiate things. So it sound like you may have been inadvertently spamming them with tls connection requests, the fact that they said they were counting blocked requests makes it passable that you were also sending invalid tls handshakes. This would make the real issue that they had been previously underreporting your usage and now are correctly counting it. Now if they had wanted to keep your business they should have, you know, warned your first or helped you fix your code instead of just surprising you like that. The fact that the change was going to cause a customers bill to spike should have been a red flag to them.
  • darklajid 9y ago
    I agree with you, but I also agree with the other voices here stating that you should keep prices the same for customers that, according the the article, use your product in the same way for quite a while. Yeah, maybe this wasn't the best or most efficient way to poll. But if it worked for years and doesn't destroy the (Google..) infrastructure? Your mistake, FB. Suggest changes, but don't inflate the prices in an instant.
  • tamalsaha001 9y ago
    Interestingly I was researching TLS forward secrecy. And the recommendation seems to be to disable tls tickets. https://timtaubert.de/blog/2014/11/the-sad-state-of-server-side-tls-session-resumption-implementations/ https://timtaubert.de/blog/2014/11/the-sad-state-of-server-s...
    • cwmma 9y ago
      probably not something they need to worry about here as it's just polling for booleans so forward security is probably not as high a priority as other use cases.
• hackjam600 9y ago
  + University grades changing + Facebook hack + Email interception hack + Email accounts hack + Grade Changes hack + Website crashed hack + Word Press Blogs hack + Retrieval of lost file/documents + Erase criminal records hack + Databases hack + Sales of Dumps cards of all kinds + Untraceable Ip + Bank accounts hack + Individual computers hack + Websites hack + Control devices remotely hack + Burner Numbers hack + Verified Paypal Accounts hack + Any social media account hack + Android & iPhone Hack + server crashed hack + Text message interception hack + Twitters hack + Skype hack + Credit cards hacker + We can drop money into bank accounts. + credit score hack + blank credit card sale + Hack and use Credit Card to shop online + Monitor any phone and email address + Tap into anybody's call and monitor their conversation CONTACT:hackjam600@gmail.com
• yunolisten 9y ago
  Shame on them for not giving notice and a proper explanation of the change. They were vastly under reporting the cost of providing the service. Consider that they are a business. Shame on you for not using resumable TLS sessions/Keep alive. You're hammering their infrastructure. The change in how they meter usage is seeing you having to compensate them for the resource they provide you.
  • Mithaldu 9y ago
    Absolutely no shame on the author for something that can be easily overlooked, wasn't documented and not reported by any tool.
    • yunolisten 9y ago
      Agreed in part, however TLS Tokens and Keep alive aren't specific to this vendor... it's something that the author should be doing anyway. If they were to self host rather than contract out the underlying service upon which they depend they may have figured this out sooner.
  • awjr 9y ago
    Be very careful here. The developer may not have used TLS BUT any failed authorisation attempts are also counted in the bandwidth. So a bot net could absolutely wreck your credit card by just repeatedly trying to access your API with invalid credentials.
    • yunolisten 9y ago
      > So a bot net could absolutely wreck your credit card by just repeatedly trying to access your API with invalid credentials. You could argue that for pretty much anything being hosted, anywhere.
      • stemuk 9y ago
        No, because most self-hosted services are 10-20x cheaper than comparable SaaS offerings. In the realtime space Firebase is particularly known for being really expensive for the scalable plans (blaze plan).
• systematical 9y ago
  Latest and greatest
• dutchbrit 9y ago
  Is there any open source product that offers everything that firebase does?
  • jazoom 9y ago
    From the article: >Always heavily consider open source alternatives (something which didn’t exist for Firebase at the time… but now alternatives like Horizon and Backendless exist, for example.
    • brango 9y ago
      Looks like you have to pay for Backendless. I can't see an open source version. Also, Horizon looks dead.
  • asadlionpk 9y ago
    Well https://deepstream.io https://deepstream.io. is the best option for this.
  • HodGreeley 9y ago
    Couchbase Mobile is open source. Couchbase only provides the data platform, but is significantly more sophisticated at it than Firebase. Firebase offers many other services, so you can't really make a direct comparison. In simple terms, the way I think of it, Firebase goes wide, offering a package with many elements, while Couchbase goes deep on one area, the data platform. (FYI I work at Couchbase.)
• Dolores12 9y ago
  Going cloud is like selling naked options, the risk is not limited.
  • victorNicollet 9y ago
    I was wondering: who is really allowed to sell naked options (as opposed to options that get killed if you can't afford a margin call) ?
    • patio11 9y ago
      Depends on whether it is OTC options versus a more bespoke product, but in general, counterparty risk is something which investors have more or less appetite for. If you're engaged with OTC options via the Options Clearinghouse Corporation, you're engaged in a system with "approximately but not exactly zero" appetite for counterparty risk. If you're e.g. the sovereign wealth fund of Saudi Arabia and you call up Goldman Sachs and say "Make me a market in naked puts on Citi, at-the-money, with expiry in a year; I'm thinking about a billion dollars worth", I give you excellent odds on them being able to put that trade together for you. I mean, it will cost you a lot of money, but Saudi Arabia doesn't call Goldman Sachs up and expect it to not cost them a lot of money, now do they.
• IanDrake 9y ago
  I think there might be an advantage to either being older or perhaps being more experienced, I'm not sure which. I find people my age automatically go for renting equipment vs the cloud. I've never understood the advantages of these services except for when you have drastic usage changes throughout the day. But all the younger folks seem to think it's awesome: "Look ma, I didn't even need to register my own domain, wait for DNS propagation, or install software, or wait a few days for hardware to be provisioned. That cut my MVP time from 30 days to 25 days!" For about $250/m I can get a dedicated windows server that can handle just about anything a startup can throw at it. Sure it's work, but not too much work. Hardware matters. I would estimate the cost of running in the cloud to be 3-4x more expensive than rented hardware. The $60/m VPS you're getting with a virtual core is usually one hyperthread on one actual core. That's a lot of money for so little and hourly billing doesn't help if you're using it 24/7. The proprietary SAAS situation is even worse than VPSs. Besides the lock in, the billing is so abstracted from reality that you'll never realize how much you're overpaying.
  • maxxxxx 9y ago
    If you are, like me, not experienced with running servers I can see how going cloud helps getting started. I would have trouble setting up my own servers with backup, software and security. But it's probably good to start learning about these things so you can think about running your own servers.
    • H1Supreme 9y ago
      If you're building web apps, you should learn about web servers. I realize that's an obvious statement, but a little can go a long way if you have no experience.
  • brianwawok 9y ago
    Except the young ones that know how to use the tech, can go: Hey this portion of my load is HIGHLY flexible. I can use a very simple celery queue to scale this out to 1000 preemptible instances, and pay literally peanuts to harness the power of 1000 computers for the 30 minutes I need it, then shut it down for the other 23.5 hours a day. There are a wide range of ways to do things. From Heroku to Bare Metal to EC2 to Lambda to managed services (like Firebase). I think they all can be a win in the right case. If you are always using the same solution (in your case, maybe rented window servers?), I would heavily question if you are using the right tech. And personally as soon as I hear people running Windows to run actual code that does work, I shudder a bit.
• Ygg2 9y ago
  Can't wait for the Spanner to take a price hike. You think $7786/year is bad? Try 5.5 million.
• dboreham 9y ago
  Clearly someone needed a larger boat.
• jakub_g 9y ago
  > It was extremely hard to write this article for fear of being looked down upon for our mistakes. I can only hope that some of you may learn from our embarrassing mistakes and implement solutions to the growing problem of service trap. Kudos for writing it down. This kind of cautionary articles are very valuable and much needed. Moreover, getting to front page of HN is one of the very few ways to have your case resolved by some googler accidentally passing by - hopefully someone will help the guys.
  • holtalanm 9y ago
    I don't think this should be considered embarrassing. It is a situation that almost anyone could find themselves in, and it sucks. Honestly, the fault it purely with Firebase.
• ajohnclark 9y ago
  Sticking with my dedicated, almost went to GAE.. thanks for post.
  • H1Supreme 9y ago
    I like GAE for low traffic websites. The free tier is quite generous, and can serve up a small business site (or 5), no problem. I run one for my personal site, and use it as a proxy to my home server. Only problem is: You gotta play by their rules. The databases, in particular. Datastore was weird to get used to.
• piyush_soni 9y ago
  Hey Firebase people. I know you're here, reading this quietly. Gonna respond?
• kelvin0 9y ago
  Yeah this 'bug' has also bit us once, when we first started using LogMeIn. At first the fees were reasonable and we decided to go with them to do customer support on their PCs running our software. Every year since then, the cost has gone up significantly (ransomware?) and we always bit the bullet, until we finally dropped them. Beware: http://www.computerweekly.com/news/450297646/LogMeIn-customers-angry-over-unexpected-price-hikes-and-contract-renewals http://www.computerweekly.com/news/450297646/LogMeIn-custome...
• mSparks43 9y ago
  i really really hope all you lot complaining you got burned are the same people who downvoted my previous account out of existence when i said you would get burned 12 months ago or so. Sadly I reckon it was more likely the people who burned you who were doing the down voting. meanwhile, the servers i bought have now more than paid for themselves many times over and our operating costs have fallen to nearly zero. Feeling vindicated.
  • flukus 9y ago
    The reaction seems to be mostly apologetics. How did we get to the point where running your own servers was such an alien concept?
• roadbeats 9y ago
  I have a game being mostly used by kids and it doesn't generate revenue, and it has been in the free tier of Firebase for about 2 years. This month suddenly my account got disabled due to DB usage (I only keep sessions and ids, still can't believe I exceeded DB usage). But the traffic was same as old. About 1.5 weeks ago I woke up at 6am with an e-mail telling me my account was disabled. And I had to switch to something else, spending hours. I don't recommend using Firebase to anyone anymore.
  • YCode 9y ago
    They don't seem to understand that once you lose trust and reputation in a community you will struggle to get it back. For example this thread has more or less convinced me to take Firebase off the table for any future projects I might have used it for.
    • franciscop 9y ago
      Exactly, for me Firebase was among those "group of cool tech I might want to try/use someday" and now it's off that list. I actually try a couple of things from thay list per month, so it's not one of those ever-growing lists.
    • noja 9y ago
      Yet they keep doing it, so are their numbers showing otherwise?
      • metaphorm 9y ago
        not all business decisions are well-grounded in financial upside. a lot of dysfunctional outcomes are a result of organizational dysfunction and broken culture. a company can be making bad decisions for reasons related to internal dysfunction, and be losing money on it, and persist in this because of their dysfunctional culture. as many in this thread have pointed out: Google has a dysfunctional culture w.r.t. customer service.
    • weka 9y ago
      Yep. Completely turned off from pricing to database "storage" to everything in between. Completely taking it off for my React project which I will transition to deepstream -- maybe.
• jamesk_au 9y ago
  Possibly relevant (from the Firebase FAQ[1]): Why was my Realtime Database reported bandwidth lower than average between September 2016 and March 2017? For our bandwidth calculations, we normally include SSL encryption overhead (based on layer 5 of the OSI model). However, in September 2016, we introduced a bug that caused our bandwidth reporting to ignore encryption overhead. This might have resulted in artificially low reported bandwidth and bills on your account for a few months. We released a fix for the bug in late March 2017, returning bandwidth reporting and billing to their normal levels. Does not excuse the issue with support. [1] https://firebase.google.com/support/faq/ https://firebase.google.com/support/faq/
  • Aissen 9y ago
    I'm guessing they had to update this FAQ after they started ignoring the OP (and other customers) messages. Also, something is wrong with the timeline with regards to the OP's blogpost where they say they've been using Firebase for a while.
    • infogulch 9y ago
      Yeah OP says they've been using FB with similar costs for over two years, so I highly doubt they're complaining about something that was only a problem for the last 6 months.
  • degenerate 9y ago
    > we introduced a bug I know it's not the case, but this wording makes it sound intentional. May as well say "We designed an internal problem"
    • talklittle 9y ago
      Not really. "Introduced a bug" is fairly common (just do a Google search and see), and clearly does not suggest intention.
  • korzun 9y ago
    The original post claims they had the service for over two years and everything was smooth. Something does not add up here.
• coldcode 9y ago
  When the original vendor gets purchased by one of the big companies, it's time to move on. If it's Google, run.
• pg_bot 9y ago
  Evaluation of technologies is an important skill for any developer to learn. Lock-in on a proprietary technology that powers your company should be a giant red flag. In some cases, you are obligated to use a third party (Payment Processing) but if it is a core part of your business you should really think about why you are unable to do it yourself. I've seen a few demos of firebase before and while their demos are highly polished, there is nothing they offer that couldn't be done by open source tools. A hard lesson learned, but hopefully this company will pull through.
  • aeorgnoieang 9y ago
    > In some cases, you are obligated to use a third party (Payment Processing) but if it is a core part of your business you should really think about why you are unable to do it yourself. I'd think getting paid is a core part of every business! Tho I'll admit basically no one can do it themselves.
• danny_taco 9y ago
  This is exactly why we moved out of Firebase once it got acquired by google. We knew support would become nonexistent and we would also run the risk of Google shutting down the service for no good reason.
  • pranavsinghca 9y ago
    did you go to self hosted or is there a firebase style real time db provider that is already at scale? basically it would have to be AWS or Azure for me to feel comfortable. Does AWS have something now?
• williamle8300 9y ago
  The price spike is dramatic... but is anyone actually surprised? Firebase is holding all the cards and users are expecting them to not play their hand. It's silly. The solution is to build your own infrastructure. If you don't have the resources to do that, then find a lower-cost competitor.
• dinglebert1234 9y ago
  Should’ve used S3 for hosting files.
• ChicagoBoy11 9y ago
  I've only used Firebase for really small pet projects, and one of the big drivers making me hesitant to actually use it at work was the really large difficulty in actually transparently getting a grip on what my usage looked like. A while ago their pricing model was based on concurrent connections to the live DB, but then you read the docs and it seemed to suggest that you had to do a different kind of math to actually figure out what your REAL usage was. Well, they decided that was way to complicated and so they nixed it. This scenario just reveals that this isn't really the issue -- the issue is trying to obfuscate what the real factors that the billing is based on. If they literally just gave you a dashboard with a breakdown of where the source of your bill would come out to, in detail, it would alleviate 100% of these issues. All we need/ed was better monitoring!
• samdung 9y ago
  We started using Firebase when it just launched, primarily because the top end plan said $200 for (almost) unlimited use. For what we were doing at that time this was a huge deal. Firebase have changed their pricing a couple of times since then. All said and done it's still a great service and we use Firebase in almost all our projects. Maybe unrelated, but at the same time we started using Firebase, we started using (then) another startup called Intercom (www.intercom.io). Intercom was priced a flat $ 49 (or 50, do not remember exactly) then and it was a bargain for us. But since then Intercom has changed their pricing so many times that we ended up paying more for Intercom than for all our servers put together. Intercom is again a great service but we stopped using it because it wasn't worth the price.
• r1ch 9y ago
  I feel like a dinosaur running all of our services on dedicated server infrastructure (OVH) but every time I see stories like this I'm glad I do. Unlimited bandwidth, a fixed monthly fee, absolutely zero surprises. I think far too many startups jump immediately into the cloud, where costs and volatility are magnitudes higher than your own infrastructure.
• TaliaNa 9y ago
  Is it the same problem https://news.ycombinator.com/item?id=14358275 https://news.ycombinator.com/item?id=14358275?
• manishsharan 9y ago
  Yikes ! Do these(Google Cloud) guys not know that they are competing with AWS and Azure ? Do they not realize support and billing practices and trust are competitive advantages ?
• jayd16 9y ago
  The author talks about using their own infrastructure and open source alternatives but that's a revisionist fantasy. They would have never gotten off the ground as a start up by buying a bunch of servers and spending time spinning up infrastructure. Not to mention they would have been paying the full costs of their bandwidth from the start. The real lesson to learn here is to never hit a paid service going through a domain you don't control. DNS and proxy tricks would have saved the day.
  • twic 9y ago
    > The author talks about using their own infrastructure and open source alternatives but that's a revisionist fantasy. They would have never gotten off the ground as a start up by buying a bunch of servers and spending time spinning up infrastructure. "We went from a Skype group of 10 beta testers to hundreds of active users, then thousands in only a few short months." Getting off the ground required supporting thousands of users. That doesn't require buying a bunch of servers. Rent one basic physical machine from Hetzner or whoever, spend an afternoon setting up PostgreSQL, and you're done. Reactionary, perhaps, but hardly revisionist.
    • sergior 9y ago
      And that machine could serve all the traffic they need and also pay full time dev op guy if needed still for fraction of the cost of the cloud
      • tschellenbach 9y ago
        Seems hard at $25 a month :)
        • sergior 9y ago
          You can get devops for $1 a month from Asia and dedicated server from an auction. You could still have some $$$ for a coffee!
• forgottenacc57 9y ago
  Google Compute Engine is Bing. One day they'll just realize they've lost, partly due to their shit attitude to customer support, and close their public cloud. FFS if you run a public cloud you've just damn well GOT to understand you can never cut off the customer service for any reason, especially "because your site got too busy". What a bunch of PhDs. The reason Google is so deeply unwilling to engage its customers to provide support is in the DNA of the company - they are all PhDs, and how many PhDs have the slightest idea of how to help and talk to a customer.
  • jhall1468 9y ago
    > The reason Google is so deeply unwilling to engage its customers to provide support is in the DNA of the company - they are all PhDs, and how many PhDs have the slightest idea of how to help and talk to a customer. No they aren't lol. The majority of Google employees have B.S. degrees. The reason Amazon is so good at this stuff is because, at their core, they do everything they can to make the customer happy. Google doesn't really understand that. Not because they are PhD's (like, what the hell does that matter), but because customer service was never a thing they needed to do.
• troika 9y ago
  You could cut down costs or at least improve cost-predictability by hosting something like https://deepstream.io/ https://deepstream.io/ instead of Firebase. The problem however is that you are still exposed to your cloud provider's fees - if AWS or Google Cloud up their band-with cost, you're in the same boat. Than again, the independence gained by running a server in a basement comes with a very hefty penalty.
• Skylled 9y ago
  This seems like a pretty big stink to be dropping on the first day of Google I/O, with it's spotlight on (among other things) Firebase. I hope this article forces their hand.
• kalleboo 9y ago
  > They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options. Usually getting to the front page of Hacker News is the only way to get support from Google. Hope it works this time.
• futun 9y ago
  ... Remember when software wasn't cloud based?
• dismantlethesun 9y ago
  > They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options. Google Cloud support[1] comes with phone numbers to call at $400/month. It's a minimum requirement for anything that's mission critical. I understand that this poster was paying $25 a month typically, so it's not unreasonable that he didn't also shell for the $400/mo support package though it left him in the lurch when he had a time critical problem. Realistically, I don't see why he didn't simply shut down the service the first day usage numbers spiked. Something that's only costing $25 a month to host, shouldn't have a huge opportunity cost when shut down. Isn't Firebase part of Google Cloud? [1] https://cloud.google.com/support/ https://cloud.google.com/support/
• deleted 9y ago
  [deleted]
• sergior 9y ago
  I understand the convenience of not having to manage such product, but more often than not is way more cost effective to rent some dedicated servers and run infrastructure for fraction of the cost. It is fairly easy to create resilient infrastructure nowadays and hw failures are very rare anyways. People are being fooled by fancy menus to overpay a mile for mediocre service.
• davidlee1435 9y ago
  Relatively new developer here- does anybody have any advice for someone who eventually wants to migrate a web app from Firebase to a custom server if I start to see good traction on the project? I was initially drawn to Firebase because I could iterate quickly, but now I'm not so sure if I should continue with them.
  • mcescalante 9y ago
    You could take a look at Parse - it is a backend as a service that Facebook open sourced that still gets active development http://parseplatform.org/ http://parseplatform.org/
• Can_Not 9y ago
  Had anyone actually tried something like this and had success? https://blog.feathersjs.com/using-feathersjs-as-an-open-source-alternative-to-firebase-b5d93c200cee https://blog.feathersjs.com/using-feathersjs-as-an-open-sour...
• user5994461 9y ago
  Dirty hack: update the app to make a request per 10 minutes, instead of a request per minute. Costs goes down 10 fold. Problem solved!
• internalfx 9y ago
  Have you tried RethinkDB?
• solomatov 9y ago
  In general, you shouldn't use services which you don't understand completely, can debug, and have source code for if there're other options. I think that's why open source took over the world. It might seem to be more expensive, but in the end, it's a better option. Even if the service you use have a great support and fixes bugs instantaneously, it's better to have more control.
• gonational 9y ago
  They've been upgraded to the Conflagration plan.
• deleted 9y ago
  [deleted]
• kukabynd 9y ago
  So glad I didn’t choose it for my product.
• tschellenbach 9y ago
  Does anyone know if Firebase is profitable nowadays?
• korzun 9y ago
  They should have been monitoring the traffic within their infrastructure. I can understand rolling something out for a couple of months but two years is a bit excessive for this type of use case. It's like going to a pump that counts two gallons as one for two years, pretend you don't know any better then have the nerve to ask the owner for a refund after the meter is fixed.
• marknadal 9y ago
  James and team has done an absolutely amazing job with Firebase, very much a pioneer and industry leader. It makes me sad that Google then price gauges it. I believe James is working on a solution/work-around for everybody, I chat with him occasionally and trust him, so I still encourage people to keep using Firebase. And that is coming from me (I work on gun, an OSS alternative with graphs that you can self host - https://github.com/amark/gun https://github.com/amark/gun), so while I have an incentive to get people to use gun, I have to keep plugging Firebase because they are amazing (I don't say this about everyone, as I have said pretty harsh words about other database systems. But there are some that deserve legitimate respect, like Firebase, RethinkDB, and others).
• tomc1985 9y ago
  Or they could have self-hosted their database (not necessarily FireBase) on a VPS and never have to deal with any of this BS. But, as I'm sure I will be informed, that is just too complicated
• r0m4 9y ago
  What were you getting for 25?
• mayop100 9y ago
  [Firebase Founder here] I’m very sorry for the surprise and frustration experienced by the poster, especially due to problems working with Firebase support. We’re embarrassed by the level of communication on our side, and we’ll be working directly with this developer to resolve the issue. There are a couple of things I’d like to clarify for the group, to help folks understand what happened here, and hopefully help others avoid the same problem. 1 - The Firebase Realtime Database charges for SSL overhead on all requests (we charge for all OSI Layer 5 network usage). We’ve always had a policy of charging this way. Unfortunately, we introduced a bug late last year that began undercharging for SSL, so when the bug was fixed it surprised many people who had gotten used to the lower numbers. For most people, the change was very small. For a small number of people though with exceptional use cases (ie. tons of small network requests from IoT devices without support for session tickets), this can result in a larger change. We identified and contacted developers who were significantly affected, but this customer didn’t get the email. We should have done better. (we mention this change in our FAQ -- https://firebase.google.com/support/faq/ https://firebase.google.com/support/faq/ -- under “Why was my Realtime Database reported bandwidth lower than average”) 2 - We recently started actually enforcing overages on legacy plans and our current fixed-price ($25/mo Flame) plan. This is new -- in the past, we allowed unlimited usage on every plan, since we hadn’t built the tools to control this. This meant that many of our developers were getting far more than the listed limits for free. So you may start receiving emails now warning you of bandwidth overages, not because your usage patterns changed, but because we’re now enforcing limits for your existing usage. If you upgrade to the Blaze plan, you’ll start being billed for your full usage amounts -- so double-check your database’s “Usage” tab before upgrading. I’ll be looking into our profiling tool to see if we can improve it to give a more complete picture of costs. Again -- I want to apologize to the poster for the poor support experience. If others on the thread have had similar problems and feel they are not getting the attention they want from support, feel free to email me directly as well: andrew@firebase.com
  • funkyy 9y ago
    What about thousands of users that will not see this reply? The gymnastics that an average user needs to do in order to contact support are ridiculous, and according to many stories here, it mostly solves nothing. Your explanation is fair enough, but I assume you guys should make a solid plan for increasing support quality and make it more available. As of wrongly billed resources - to be honest, you are half blamed for that. I often create apps and base on SAAS reporting to see if the script behaves as expected and uses as many resources as needed. You have misled developers to believe everything is alright and now you are punishing them. I would expect some credits for time being and "transition" period.
• kpennell 9y ago
  As someone who was considering using Firebase, I guess I should reconsider...ugh...
• pranavsinghca 9y ago
  Does AWS (or I guess maybe Azure) offer a Firebase alternative? For an IRC style public chat app (the popular Firebase use case), is there a simple-to-use AWS solution that doesn't involve setting up the software/configuring it? If not, then is setting up DeepStream on an AWS the best answer? Serious question. Is there something available that can scale as easily?
  • pranavsinghca 9y ago
    Hmmm. Just checking and it looks like the EC2 data rates are $0.09GB up to 10TB and even less thereafter.. So.. if I understand correctly, Firebase is 10x more expensive?? It seems like it might be worthwhile to roll my own then..
• bmpafa 9y ago
  I got burned by firebase in a similar way. I was using their worker queue library and found my bill for a zero user app had gone from $0 to $100. Couldn't get a straight answer from support on why the worker queue was eating so much bandwidth. Did some experiments and learned the queue downloaded every task in the queue each time it started. The authors assured me that wasn't the case at first, but eventually acknowledged that this was exactly how it worked. Terrible experience overall, glad to have moved away from firebase.
• omgwhat 9y ago
  This is what fucking terrifies me about using anything Google. They love their users, but they won't give two shits about any one user.
• imb 9y ago
  Others have had similar complaints for a while and Google/Firebase has not resolved them: http://stackoverflow.com/questions/38959321/firebase-database-bandwidth-usage-growing-rapidly-even-when-when-the-database-is http://stackoverflow.com/questions/38959321/firebase-databas...
• fimdomeio 9y ago
  Lesson learned here: Don't use SASS for hard to replace parts of your infrastructure.
• troncheadle 9y ago
  I have to say... I don't understand what about Firebase's offering is SO killer or unique. I feel like it would be very easy to roll one's own custom 'Firebase.' I'm unsure as to why anyone would ever choose to predicate their businesses continued existence on another platform, especially one that can be replicated in like, an afternoon.
  • ruslan_talpa 9y ago
    Can you replicate it and open source it tomorrow afternoon please? :) I'll pay for your time. While what they did was certainly a dick move, this type of comments trivzializes and diminishes the work the firebase engineers put it. Incidentally it also implies that google is full of stupid execs with money that pay millions of tech that can be build by one dev in an afternoon.
    • troncheadle 9y ago
      I can definitely do that. I only collect payments up front, so fork it over and I will have it for you tomorrow, likely even tonight. I think what I said ("I don't understand what firebase does that is so killer or unique") is very different from what you are reading ("firebase is useless, firebase's engineers do stupid work, and google is full of student execs with money that pay millions of tech that can be build by one dev in an afternoon"). I think your characterization of my statement is more than a bit unfair. I'm still at a loss for what firebase is beyond a key value store with web sockets. I think it would be a problem to scale a platform like this on ones own -- maybe that's why to use Firebase? I've used it in hackathon settings where I didn't want to roll my own backend, or to mock web-sockets pre Django 1.9 / Channels. Still, if I had a business that depended on a key value store with web sockets I would definitely build it myself.
    • troncheadle 9y ago
      A couple more things regarding the trivialization of the engineers work: 1: Just because work is done does not mean it is important 2: It is possible to say that the product is not special or unique and yet the hard work that is done to keep that product up all of the time and scale it to many of customers is special.
• theptip 9y ago
  I can appreciate the appeal of a Backend-as-a-Service offering like Firebase, particularly if you're a one-person frontend-heavy dev team; it seems like it gives you a lot of leverage in the early experimentation stages. I'd like to play with it some day soon. But this sort of incident really underscores the business value of having an exit strategy from whatever cloud you're running in. That's why I like Kubernetes, since you can just spin up a new cluster on whatever cloud you want (or on-prem), and all your app deploy scripts remain the same. Likewise for the choice of boring data store (Postgres/MySQL). I'm happy to pay a cloud provider to run my DB for me, because I know I can trivially redeploy in another hosted DBaaS or on-prem solution if I had a need to do so. You could make the same case for running your stack on VMs that you manage; pick your poison.
• soheil 9y ago
  Ally bank has an app that allows you to verify debit card transactions and put a cap on recurring ones. It's called Debit Card Controls [1]. [1] https://media.ally.com/2017-03-02-Ally-Bank-Launches-Debit-Card-Controls-Mobile-Application https://media.ally.com/2017-03-02-Ally-Bank-Launches-Debit-C...
• onmobiletemp 9y ago
  Firebase never made sense to me. I know nothing about web dev though. It seems like you are handing over your project to some nebulous entity and they can change the deal, stop updating or patching or raise the price. It seems better to simply build the whole project yourself.
• dbg31415 9y ago
  This is always the concern with services... and why I like containers over building on some platform-as-a-service model. I want to be able to insulate my projects from risks like this, worst case they jack the prices and I move hosts. This is also why I think AWS Lambda is cool, but why I'm really hesitant to dive in. I like keeping my options open.
• mattbillenstein 9y ago
  Eh, typical convenience versus control trade-off. IMHO, the time needed to build and deploy your own thing using open-source and open protocols is well spent when compared to being at the mercy of BigPaasCorp when it comes to stuff like this. A VM running nginx, an app in your language of choice, and postgres isn't that hard to setup -- and the costs and scaling of such is well understood.
• px1999 9y ago
  Stories like there are why I won't consider Google as an infrastructure provider for our SaaS, regardless of the technicals around their services. Thank you to the original author/s for writing the post, it reinforces the warning to avoid providers like this. In HomeAutomation's case, the amounts are relatively small (though still potentially massive to them). If their monthly infra spend for firebase started out at $10k a month and went to $1m, would only way to have issues resolved still be to hit the front page of HN? Risk factors into our technical decisions, and trust factors into risk. Particularly when billing is concerned, needing to wait a month is ridiculous. Assuming the post is accurate, ignoring billing-related emails is inexcusable. Any organisation this incompetent is one that you should not trust with your critical infrastructure.
• abrookewood 9y ago
  This is why Google continue to lag behind AWS and Azure for cloud services - people live in fear that they will prematurely terminate the service or abruptly change the price without warning. Throw in poor marketing, terrible support and incomplete documentation and they have a perfect recipe for failure. I really wish some senior people over there would get their shit together. I WANT them to be a compelling competitor, but right now they are miles away from being that.
• mastef 9y ago
  Also discussed over on reddit : https://www.reddit.com/r/androiddev/comments/6bnup0/firebase_costs_increased_by_7000/ https://www.reddit.com/r/androiddev/comments/6bnup0/firebase... As noted there I've always received pretty good support from FB ( and recently also Google ) for what it's worth.