17 ms·

How we spent $30k in Firebase in less than 72 hours

307 points by slyall 8y ago

melan13 8y ago
The company success is good, but the SELECT ALL query is a shame bigger...
- sitkack 8y ago
  It shouldn't be _possible_ to run select all in production.
t0mbstone 8y ago
This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for less than $100 a month on AWS?
- everdev 8y ago
  Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.
  - heavenlyblue 8y ago
    Well, a rogue line could potentially as much shut down your app hosted in DO.
    - tgsovlerkhgsel 8y ago
      That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill. When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down. When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.
      - heavenlyblue 8y ago
        Well, then the issue with the Amazon Lambda is not that of it's ability to scale, but that of it's default not to limit your spending.
  - marktangotango 8y ago
    What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?
    - Something1234 8y ago
      One bankrupts you, the other just has a minor hiccup with a few lost customers that day.
      - ithkuil 8y ago
        a (cost) cap is a feature designed to prevent you from going bankrupt but at the expense of not recovering from the surge after you deplete the cap until the end of the cap granularity. So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it). This problem is a function of the granularity. Imagine a system that let you say: "I want to spend max $n per second with a extra burst of $m per day/week" You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN. The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.
    - deleted 8y ago
      [deleted]
  - vbezhenar 8y ago
    Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.
    - Xorlev 8y ago
      For services that are pay per request, this is effectively a hard cap. For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.
- ghaff 8y ago
  From what I understand, AWS at least is pretty good about giving you at least one "get out of jail free" card if things go awry. Caps are tough though. I can certainly understand a use case that would want a hard circuit breaker that just kills everything it can once it hits a certain threshold. Sort of; you presumably don't want everything on S3, for example, to be deleted. On the other hand, moving up the scale of serious businesses, I can imagine it would be hard to specify circuit breakers (rather than just alerts) and you get into issues of terminating services that affect all sorts of other services across the entire account.
  - thieving_magpie 8y ago
    There's nothing guaranteed about a "get out of jail free" card, so factoring that into your decision making is probably a bad road to go down.
    - ghaff 8y ago
      I'm not suggesting that. For a lot of casual users I'm not sure that cloud services are necessarily the best approach for an externally-facing service. They may well be better with a VPS that has a hard resources/dollar limit. Perhaps cloud providers should have some sort of hard circuit breaker option (though it won't help for some things like storage) but it's probably not a priority as not a lot of businesses--their primary customers--would be OK with effectively hitting the power button for their entire cloud account if they exceed some dollar amount that someone or other configured a couple of years ago.
  - itcmcgrath 8y ago
    Caps are definitely hard, getting them right as both a feature as well as setting them as a customer is still an art. This holds true for 'flat rate' too, where your trading correct sizing vs downtime when either get code wrong or get unexcepted success. I've been involved in all these situations so I definitely emphasize. We had daily hard caps and budgets alerts, but it's still an area we can do better. (Disclaimer: Product Manager for Cloud Firestore)
- itcmcgrath 8y ago
  Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.
  - alasdair_ 8y ago
    >Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it. This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).
    - iamgopal 8y ago
      So basically we want to have cake and eat it too.
      - ajmurmann 8y ago
        Or have it be adjustable within a reasonable time frame?
      - paranoidrobot 8y ago
        This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
        badlucklottery 8y ago
        >that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
- vuln 8y ago
  You only have to be 'terrified' if you don't write good code. Maybe people should actually test their code before pushing it to production? It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '
  - dang 8y ago
    Please don't be rude. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - JoJoJoboop 8y ago
    Well, since Firebase has lowered the barrier of entry with their easy setups, I think it is justified to expect at least some developers to make mistakes. I don't blame Firebase at all though -- great product.
- tgsovlerkhgsel 8y ago
  How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)? That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.
  - bufferoverflow 8y ago
    You should rate-limit and cache the potentially expensive paths of your app. There's no real way to protect against a clever DDoS, someone will have to eat the costs.
    - zwily 8y ago
      You can’t really rate limit server-side with Firebase. If a client has read-permission on a data node, an attacker can just read from there as many times as they want. There are some ways to limit writing using the rules engine but you’ll still get charged for failed writes. :) Only real way to rate limit firebase that I know of is to put some sort of proxy service in between, but then you lose a lot of the advantage of using FB in the first place.
- superfrank 8y ago
  > This is why infinitely scaling pay-as-you-go cloud services terrify me. It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does. The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go higher even if they wanted to. When we brought this up, they told us they were technically unlimited because you could shard your data into different DBs if you needed more connections, which is like saying all restaurants are all you can eat because you can keep buying more food.
  - itcmcgrath 8y ago
    Note this is about Cloud Firestore, not Firebase RTDB. It's very different infra. We announced our beta limits increased to 1M concurrent connections at the Next 18 conference, and we'll continue to improve from there.
  - fefb 8y ago
    The link posted is talking about the Firestore and not the Real Time. Firestore is in beta state that has a beta limit of 100k sim. connections, probably off beta it will be able to handle more connections. The Real Time has 100K limit and is a centralized db.
- jliptzin 8y ago
  I will also stay far away from these services. As a child post points out, if you do set a cap, you run the risk of cutting off a legitimate spike in high quality traffic. How do you tell the system to shut off due to an error and stay on during a spike in normal traffic? You can’t, unless you have someone watching it at all hours. That’s why I prefer to just rent whole machines on AWS. If I accidentally have some code stuck in an infinite loop making some O(1) call, don’t charge me $10,000 for that when it costs you nothing. If it’s actually consuming electricity and significant resources, I’ll know quickly because my service will go down, as it should, not scale infinitely until my company is bankrupt.
mirimir 8y ago
But wait, the image shows ~$600 million. So $30K is small. Maybe that ~$600 million isn't in USD?
- rosser 8y ago
  The first sentence in The Fine Article mentions Colombia, whose Peso uses "$" as its symbol, and currently has an exchange rate of ~2900:1 USD.
  - deleted 8y ago
    [deleted]
  - mirimir 8y ago
    Thanks. And I did miss the "COP" in front of the $ amount :(
- jsmthrowaway 8y ago
  No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million. The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a dot after it, because it is abbreviated there (mayo). Every programmer should understand locales even if they do not speak the language.
  - mirimir 8y ago
    No, it shows "$600,043,603". Where do you see "mil"? But OK, I didn't get that "COP" is Colombian Peso :( And there's another image that shows total collected as "USD $244,875". The ratio is 2450, which is close enough to the exchange rate.
    - jsmthrowaway 8y ago
      The Y axis of the graph which actually has relevant information. You are looking at a campaign page, and assuming those Colombian pesos are available to the author’s team. When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.
shaunpersad 8y ago
I can't say enough good things about Firebase and GCP in general, but I'm always cautious when using Firestore in particular. I usually avoid unbounded queries altogether, and treat it primarily as a key/value store to get by id. When I do use queries, it's always in places where the results have a well-defined limit (usually limit = 1), e.g. finding the most recent X or the highest X. With the above two, you get all the greatness of Firestore, but with a well-defined (low) cost that you can calculate ahead of time.
- itcmcgrath 8y ago
  We've also been rolling out updates to rules to enable you to enforce these types of things. There is performance implications to limit queries for the real-time update system at scale, but for most use cases this shouldn't be a problem. Definitely more we can improve here for control, and we're open to feedback.
  - fefb 8y ago
    Hey, thanks for listening. I believe in short time would be nice to have a way to use for the query the create, update and write time of a document. Now, I am doing the creat time manage inside of my document with Date.now(), but when I was running a bunch of promises to create the documents, the createTime between the documents that I was manipulating was in same case the same, so my pagination failed. Another things, like compound queries inside of subcollections should be nice. A way to export all the database for backup. A flag to alert the Firestore to return the document when I do an update in this document in the same response (one round trip (dynamoDb has it). I know I can reach this goal witht transaction, but I believe it is simpler than a transaction. A way to update a array without transaction. Thanks
  - degenerate 8y ago
    My open feedback is to get these rules in place as soon as possible. This story/blog makes me realize that GCP/Firebase does not currently play nice to dumb mistakes and 24h away from the dashboard can be disastrous. Multiple rules/filters need to exist to trigger SMS/Email alerts, or a pre-defined action, upon certain conditions.
deleted 8y ago
[deleted]
duxup 8y ago
>with every visitor to our site, we needed to call every document of payments in order to see the number of supports of a Vaki, or the total collected. On every page of our app! Oh that will do it.
deleted 8y ago
[deleted]
riquito 8y ago
> we were using Angular V.4 and we decided to upgrade everything to V.6. It was a huge risk and we wanted this campaign to be perfect, so we did it! In just a day and a half, our team had the first release ready in the new version. After some tests, it looked like the refactor helped the app’s speed, but it was not as fast as we wanted it. Our goal was to load in 3 seconds and it wasn’t working as we expected. You want it perfect and you cannot afford to put down the site but you're willing to take a "huge risk" based on (wrong) guesses, with clearly not enough time for proper QA. I sincerely suggest you to slow down and reflect on priorities and risk assessment, there's a reason if that firebase code slipped through. By the way I'm happy you avoided the worst case scenario, good luck with your project
- TAForObvReasons 8y ago
  > I sincerely suggest you to slow down That runs counter to the great Silicon Valley ethos of moving fast and breaking things
  - dboreham 8y ago
    The moving fast breaking things guy came from the east coast.
Something1234 8y ago
So what is their product? Can't read spanish, and don't use chrome for translate.
- whoisjuan 8y ago
  A GoFundMe kind of platform, that allows groups of people to put funds towards a common goal (like a party or whatever)...in this case they used it to donate to a politician who had a huge debt after losing an election.
sciurus 8y ago
When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws.
- sitkack 8y ago
  > Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an affordance for application maintenance and visibility. Differential analysis and graphing built directly into the execution substrate. Judge away!
  - cloakandswagger 8y ago
    Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.
    - dkoston 8y ago
      Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code. Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks. The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected. Kudos to the engineers for solving this issue under pressure.
franzwong 8y ago
But did you see the slow down in the request to firebase before going to upgrade frameworks?
paradite 8y ago
So it's a classic N+1 query?
- bsaul 8y ago
  Not really. N+1 is about « to many » relationships. You would need at least 2 collections to run into that problem. In sql equivalent, what happened would be more like doing a full table scan on each query instead of using an index ( and not even that, because pre-computing a total isn’t really like an index) This kind of « account balance » problems are typical of the problems where transactions are really useful. But also historically the kind of problem where nosql techs do a poor job ( they’re more built with « eventual consistency » in mind than atomic or transactional behaviors)
- alexpi 8y ago
  It's more like (in good old RDBMS world) 'SELECT * FROM payments' and calculate sum on client vs 'SELECT SUM(amount) FROM payments WHERE some_id = ?' though they already had this sum precalculated
  - jasonkester 8y ago
    That would be more forgivable. This is: SELECT * FROM payments where paymentID = 1 SELECT * FROM payments where paymentID = 2 SELECT * FROM payments where paymentID = 3 ... SELECT * FROM payments where paymentID = 14986 Each of those in its own API request over the wire, then sum them on the client.
deleted 8y ago
[deleted]
ConcernedCoder 8y ago
"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.
- noncoml 8y ago
  I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.
  - avargas 8y ago
    A bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.
    - whoisjuan 8y ago
      I agree with you. There are so many ways to make this infinitely more efficient. For starters, why are they re-calculating and re-rendering the value every time they get a new donation? Also, they could store those values in a more cold&cached-storage and just make the reads to update from Firestore. Don't use a freaking database that is charging you for every single read and write, to deal with mundane client-side renderings.
    - Timshel 8y ago
      Changing the currency will re-trigger all those conversion calls ...
  - dragonwriter 8y ago
    It's more like a naive database design which doesn't consider the access pattern and therefore requires an inefficient query, which is definitely an architectural issue.
ryandrake 8y ago
Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis management (What was the plan if they didn't miraculously find the bad line of code? When would they pull the plug on the site? Was there a contingency plan?) 7. Perfect being the enemy of good enough Looks like they were bailed out of the cost but what if that didn't happen?
- Aloha 8y ago
  Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.
  - boabab 8y ago
    updating the framework version is like.. literally the last try-something thing to do though.
  - nine_k 8y ago
    Yes, it's a good investigation tool, but definitely not a fix. That is, if "it started to work now, I don't know why", you're still in trouble, and should continue digging.
  - flukus 8y ago
    I think "try-anything" would be a better characterization in this case. They had smoke pouring out of the engine and tried changing the tire.
    - Aloha 8y ago
      No disagreement there. I don't understand how you can build a complex application like that without doing basic performance checks like, are we hitting the file system or database too often, our the image assets correctly sized, etc. I'm not a software engineer however.
whoisjuan 8y ago
I find ridiculous that their first solution was to go and upgrade to another Angular version, especially a non-beta version upgrade of a framework that is used in thousands of super high traffic websites with no problems. How clueless can you be?
- thatjsguy 8y ago
  > How clueless can you be? I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.
  - wmij 8y ago
    Isn't this why you do pull requests, code reviews and QA before deploying? You need to make sure that the more senior(s) on the team and proper testing catch these things before deploying.
  - sushid 8y ago
    Why are you so against junior devs? I've personally never met a junior dev who just starts going ahead and upgrading various dependencies, as you suggest. If anything, that shows a lack of proper hiring decision on you and your team's part. I do, however, agree that their practices are horrible (just look at their console, they're console.logging random things, running the dev mode of Firebase, and fetching some USD conversion call 10x on load with no caching) and they're lucky Google bailed them out at the last minute.
    - thatjsguy 8y ago
      > If anything, that shows a lack of proper hiring decision on you and your team's part. Hey, friend! I had no control over hiring for that gig.
pishpash 8y ago
"Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies." Doesn't sound like a future great company to me, especially when their lesson from this was Google will bail them out and "It is very important that tech teams debug every request to servers before release." rather than hiring less cavalier employees and putting in better process.
RobertRoberts 8y ago
Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt? This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them. This reminds me variable rate mortgages. With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I really don't understand the cloud at all for such high cost resources with literally unlimited/unpredictable pricing. Can someone explain this risk/reward scenario here?
- Spivak 8y ago
  You use dedicated hardware with manual or semi-automated scaling until you're big enough to want unlimited and green enough to eat a large bill knowing that you still made a profit.
- flukus 8y ago
  > How does anyone justify this risk? I'm more concerned about the risk mitigation strategies (capping) I'm seeing advocated. If your servers being pegged you've only got a few customers missing out while it's pegged, maybe even everyone getting service but sub-optimally. You can ride out the wave and everything goes back to normal. Putting caps in place is like pulling the plug out of the server after the CPU has been at 100% for 5 minutes and not plugging it in until the next billing cycle.
- chillacy 8y ago
  I don’t know what kind of load you expect but web servers don’t just slow down linearly forever: at some point when traffic exceeds capacity, queues fill up, memory maxes out, and all requests grind to a halt with enough load. The way out is scaling horizontally, and you want to do that at some point before you end up sending 503s to the load balancer.
bigbluedots 8y ago
This is another good reason to properly test before going live. If their site wasn't cloud hosted it would have just fallen over, most likely. Which means a failed crowdfunding effort. Maybe take off the cowboy boots, guys?
r3vrse 8y ago
I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to think critically when you can just 'npm update', fix 37 broken dependencies, and write a blog post about it? Right?
- onion2k 8y ago
  Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder. This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.
  - cube2222 8y ago
    Indeed, if your building anything, good metrics are one of the most important things. Didn't see them having any metrics apart of Google analytics.
  - projektir 8y ago
    And most companies not knowing how to measure experience, so they pass over experienced devs and can't understand their value...
  - bsaul 8y ago
    It comes with experience, but also with more senior devs explaining it to you. Seing someone use firebase to save paiements, then recompute a total from a collection, and as a consequence having its system explodes with less than one session per second, means everybody in the team drank the « let’s use this nosql google shiny tech, it’s so cool » cool aid. Even one conversation with any senior dev having some kind of experience with backend development would have asked about expected load, types of queries, data model, etc. And concluded that storing paiements was probably the least interesting scenario for using a tech like firebase.
oxguy3 8y ago
This reads less like a "mistake", and more like just not giving any thought to what your code is doing as you write it. A mistake would be if the programmer meant to implement X but instead implemented Y; this sounds more like the programmer just set out to implement Y without even considering other possibilities. Same for the framework change. An outdated framework might be a second or so slower, but a >30-second load time was never going to be fixed by updating. This is just bad problem-solving skills. When your app is taking 30 seconds to load, you don't just guess at what might make it faster -- you open your JS console and your log files, and you figure out where that time is being spent. Two minutes in the Performance tab of Chrome's developer tools, and you would have figured out the issue was on the back-end rather than the front-end.
- Dylan16807 8y ago
  They calculated a number the wrong way. That's not a hard mistake to make. And changing frameworks is silly but not really part of the problem.
thisismydevname 8y ago
The saddest thing here is that they seem proud of this "mistake" and how they solved it. This startup mindset is not always good.
newscracker 8y ago
It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed post mortem or root cause analysis to see what they could do to prevent it (the ending looked quit amateurish, by pointing to only one thing as a potential lesson) * wasted a lot of money that could've helped in the future (by instead using that to pay for en error) If I were in such a team, I would've honestly stated in such an article how deeply ashamed I am that we missed all these things, how "cowboy coding" and heroics must never be glorified, how we got very, very lucky in someone else waiving off charges (this is not a luxury that most startups or one person endeavors would have), and ended with asking for advice on what could be done to improve things (since it's obvious there were many more gaps than just how a few lines of code were written). To the team that wrote this code and this article — get some software development methodology adopted (any, actually) and some people who can help you follow any of those. Also read the rest of the comments here. You got very, very lucky in this instance. It may not be the same case again, and you may see your "life's work" get killed because you didn't really learn.
- Oras 8y ago
  If they had half of the above they wouldn't use Firebase. Don't get me wrong I think it is a great product and I would use it for MVP but the idea of using Firebase is to get something scalable easily and quickly by focusing on the frontend only. You can tell from their screenshots that they were only focusing on GA metrics and didn't check Firebase console
- megous 8y ago
  They just didn't profile before optimizing. That's all.
- segmondy 8y ago
  Oh you amazing programmer, please share the amazing systems that you are building so we can bow down and learn.
boulos 8y ago
Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days ago and see the chance to apply for the 100k grant on Google Cloud Services. We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant. > Now we could not be more grateful to Google, not only for having an awesome “Backend As A Service” like Firebase, but also for letting us have 2 million sessions, 60 supports per minute and billions of requests without letting our site go down. Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies.
- nstj 8y ago
  I don’t see why this is being downvoted. The title of the post should be changed to “We were given a $30k credit by GCP after not monitoring our account charges”
- ocdtrekkie 8y ago
  Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next week that Google would necessarily be willing to help, but I can absolutely guarantee you that a VM sitting on a Dell PowerEdge I've got lying around would suddenly obligate me to a $35,000 bill, no matter how bad my code. Ideally, I guess I'd want to see rather than a hard cap, some sort of smart alert that would go "holy crud, this is an unusual spike in the rate of requests" when the delta changed unusually, rather than waiting until I say, hit a high static cost bar or a hard cap that kills the site.
catchmeifyoucan 8y ago
Go serverless boys :)
- fefb 8y ago
  It won't resolve the problem
  - catchmeifyoucan 8y ago
    AWS Lambda costs about 20 cents per million requests. A billion requests would have put it just a little south of 200-300 dollars. Factor in a few additional read units for DynamoDB, but I don't see it going to total $30,000. The serverless architecture should scale as well.
    - fefb 8y ago
      They are using serverless architecture. Firestore is a "Daas". The latency for loading the page wasn't the real problem, it was the side-effect. The real problem resulted in 40bilion request, they were requesting all the collection of payments in each session. It reaches 16K documents downloaded /session, so they were paying 16K reads documents for session. Firestore gives you 50K reads/day for free, and ask $0.06 for 100K reads. So It was a really bad logical approaching from them. If they did in a best way , like update a document that holds the two accumulators : total money accumulated and total payments created, and read just this document in each session request their bill to show this information for 2M session should be less than $2, because you are reading 1 document for session instead of 16K documents. And pay attention that their system wasn't down, probably the 30s latency was a side-effect of downloading 16K document of each payment information in the client side.
wiradikusuma 8y ago
If there's one thing Google App Engine has taught me, over the course of 5+ years using it: Denormalize your data (or, optimize for read).
- bsaul 8y ago
  Just a precision in case a junior dev read this comment : denormalizing has nothing to do with the problem experienced by the OP. They just queried the whole collection of paiements instead of precomputing a total everytime a new paiement arrived. Denormalizing would have required having a relation between collections. Here there was just one (so there was nothing to denormalize). Optimizing for reads ( or simply think about read performance) doesn’t require denormalizing. It can also be a matter of creating an index, or precomputing values in a cache, like in OP case.
ericpauley 8y ago
I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?
- abhisuri97 8y ago
  As one of those amateur developers who publishes to Hackernoon, it's because they change titles to fit their "How to" type articles + grab attention.
- flukus 8y ago
  > but why are these posts suddenly getting so much attention? The same reason we slow down for car crashes, morbid curiosity. I don't think there is anything "sudden" about it though, we even have sites like thedailywtf dedicated to this level of idiocy.
  - gruez 8y ago
    thedailywtf: making fun of other dvelopers' incompetence hackernoon: author patting themselves on the back while everybody else is laughing at their incompetence
lifthrasiir 8y ago
I increasingly feel that modern pay-as-you-go services are so opaque to consumers that it takes the individual employee's empathy (highly subjective) or the publicity like this (highly subjective and you have to be lucky as well) to fix any significant problem. Every time a post like this with a "happy" ending crawls into the HN front page, there would be hundreds or maybe thousands of "unhappy" endings out there.
Apaec 8y ago
I feel depressed about this, on how the industry promotes and even supports extreme technical incompetence. Maybe is a consequence of "everyone should learn to code" campaigns and bootcamps.
- johnmarcus 8y ago
  Cheer up buddy, you were this incompetent at one time too, but now look at you, a full grown narcissistic asshole senior developer. You have come a long way and these kids will too!
betadreamer 8y ago
A lot of people here talks about lack of load testing and other "do it the right way" type of advice, but remember this is a startup. In my opinion, solid testing foundation will be such an overkill and the time is better spent on implementing more features. Also I bet they did some manual testing. They didn't catch it because this latency can only be seen by the account with a lot of followers. I agree that their first solution to upgrade is a bad idea... You should understand what caused the bug before trying to fix it. I highly encourage you to monitor the load/pay/request graph on a daily basis. Even better if you hang a screen on the office that displays these. The graphs are already provided by Firebase. That way you can catch these type of anomaly on day one. Also Firebase supports "Progressively roll out new features" https://firebase.google.com/use-cases/#new-features https://firebase.google.com/use-cases/#new-features
- whoisjuan 8y ago
  Lack of testing and poor code quality are completely different things though.
andybak 8y ago
Gosh. One for https://accidentallyquadratic.tumblr.com/ https://accidentallyquadratic.tumblr.com/ with the added spice of being on usage billing...
bartread 8y ago
"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work together to achieve a bunch of shared goals and doing an imperfect job of it. This isn't just rose-tinted: I'm quite sure they have their fair share of bad actors, and they certainly make decisions we don't all like (e.g., retiring products), but I don't think it's because the company is fundamentally evil.
- cm2187 8y ago
  Except that their core business is to build files on everyone of their users to sell ads... We could have a debate on whether Philip Morris is evil. I am sure most of their employees are pretty decent people.
  - Grazester 8y ago
    When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone?
    - cm2187 8y ago
      > When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone? Great logic. Philip Morris is evil because they sell products that cause cancer and death. Therefore Equifax, who extorts money from people to protect them against Equifax polluting their credit rating, and who leaks their data into the wild is not evil, because Equifax doesn't cause cancer nor death!
      - Grazester 8y ago
        When was the last time Google had a data breach and leaked my social security number they didn't have and didn't use to "extort" money from me?
TomK32 8y ago
Some have mentioned it here already, but I'd like to emphasis how important application logs are. How much trouble you can prevent by reading and understanding them. I've seen and fixed such bugs as described in the article, and before you start trying to upgrade anything a look in the log followed by a git bisect session is the first step. My rails apps have great logs, I get to see what views and partials are rendered, what queries are sent to the database and more important how often all that happens. If the log excerpt for a single request doesn't fit on my screen I know I have to do something.
carlsborg 8y ago
The blog authors should state up front that Google covered the costs with an additional startup grant, and that this was entirely due to a quadratically expensive query.
wazoox 8y ago
And that's why people write applications to understand and track your public cloud usage, like https://github.com/trackit/trackit https://github.com/trackit/trackit ... Most people simply can't follow through what's happening in their cloud apps.
shady-lady 8y ago
Words fail me as to how anybody could put something live without understanding how the application functioned as a whole. Unbelievable.
madmulita 8y ago
"It is very important that tech teams debug every request to servers before release..." No. You should know your application's profile, you wrote it. How many resources does you app need? That's something our developers believe is the "operations team"'s responsability. Well, now that you took the 'devops' role you can no longer keep ignoring this. Your new infrastructure provider will be more than happy to keep adding resources, one can only hope the pockets are deep enough. With attention to the profile this would have been caught at developing time, maybe testing time.
shusson 8y ago
| How we spent $30k in Firebase in less than 72 hours By not checking Google Billing after you launch your website. At the very least you should have a billing alert.
- snowwolf 8y ago
  At the very least Google (and really all cloud services) could have reasonable default billing alerts built in. And automatic spike detection. I'd rather have default alerts already configured that I can change, rather than none.
xtrapolate 8y ago
Simple stress tests should've revealed this. Basic profiling should've revealed this. This article makes it appear that they've went live without ever really testing the infrastructure under any kind of load whatsoever. The article refers to some mysterious "engineering team". It would appear very little actual engineering took place at that company.
manishsharan 8y ago
The issue with GCP and AWS providing limitless scaleability is that bad code gets a free ride and we all occasionally write bad code. If the OP had tested this on a database with 1 CPU server with 2GB RAM, they would have caught it quickly in early dev testing.
joeblau 8y ago
The thing I'm wondering about with the seniority of the dev team and what the code commit and code review process is. It seems like with a senior dev reviewing commits, someone would have caught that redundant work was being done. I worked at a startup in the Mission for a few months and I remember seeing a quadratic query that ran for every customer that was logged into our application. The CEO and team lead wondered why our app worked great in Dev (with only 5 users) but was terrible on premise (250 users). When I tried to explain the issue the two devs before me didn't really understand what I was talking about. It was a quick refactor and caching solution that fixed the problem, but it was clear that the development was still new.
danielhlockard 8y ago
My only issue - Where were the metrics around firebase request count? That could have been a simple dashboard.
doglover7869 8y ago
Wow, a complete shitshow which could've been avoided by consulting with 1 experienced developer good at debugging. This kind of negligence must be happening more than what we think so there are lots of opportunities out there.

307 points by slyall 8y ago

melan13 8y ago

The company success is good, but the SELECT ALL query is a shame bigger...

sitkack 8y ago
It shouldn't be _possible_ to run select all in production.

t0mbstone 8y ago

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for less than $100 a month on AWS?

everdev 8y ago
Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.
- heavenlyblue 8y ago
  Well, a rogue line could potentially as much shut down your app hosted in DO.
  - tgsovlerkhgsel 8y ago
    That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill. When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down. When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.
    - heavenlyblue 8y ago
      Well, then the issue with the Amazon Lambda is not that of it's ability to scale, but that of it's default not to limit your spending.
- marktangotango 8y ago
  What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?
  - Something1234 8y ago
    One bankrupts you, the other just has a minor hiccup with a few lost customers that day.
    - ithkuil 8y ago
      a (cost) cap is a feature designed to prevent you from going bankrupt but at the expense of not recovering from the surge after you deplete the cap until the end of the cap granularity. So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it). This problem is a function of the granularity. Imagine a system that let you say: "I want to spend max $n per second with a extra burst of $m per day/week" You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN. The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.
  - deleted 8y ago
    [deleted]
- vbezhenar 8y ago
  Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.
  - Xorlev 8y ago
    For services that are pay per request, this is effectively a hard cap. For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.
ghaff 8y ago
From what I understand, AWS at least is pretty good about giving you at least one "get out of jail free" card if things go awry. Caps are tough though. I can certainly understand a use case that would want a hard circuit breaker that just kills everything it can once it hits a certain threshold. Sort of; you presumably don't want everything on S3, for example, to be deleted. On the other hand, moving up the scale of serious businesses, I can imagine it would be hard to specify circuit breakers (rather than just alerts) and you get into issues of terminating services that affect all sorts of other services across the entire account.
- thieving_magpie 8y ago
  There's nothing guaranteed about a "get out of jail free" card, so factoring that into your decision making is probably a bad road to go down.
  - ghaff 8y ago
    I'm not suggesting that. For a lot of casual users I'm not sure that cloud services are necessarily the best approach for an externally-facing service. They may well be better with a VPS that has a hard resources/dollar limit. Perhaps cloud providers should have some sort of hard circuit breaker option (though it won't help for some things like storage) but it's probably not a priority as not a lot of businesses--their primary customers--would be OK with effectively hitting the power button for their entire cloud account if they exceed some dollar amount that someone or other configured a couple of years ago.
- itcmcgrath 8y ago
  Caps are definitely hard, getting them right as both a feature as well as setting them as a customer is still an art. This holds true for 'flat rate' too, where your trading correct sizing vs downtime when either get code wrong or get unexcepted success. I've been involved in all these situations so I definitely emphasize. We had daily hard caps and budgets alerts, but it's still an area we can do better. (Disclaimer: Product Manager for Cloud Firestore)
itcmcgrath 8y ago
Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.
- alasdair_ 8y ago
  >Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it. This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).
  - iamgopal 8y ago
    So basically we want to have cake and eat it too.
    - ajmurmann 8y ago
      Or have it be adjustable within a reasonable time frame?
    - paranoidrobot 8y ago
      This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
      - badlucklottery 8y ago
        >that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
vuln 8y ago
You only have to be 'terrified' if you don't write good code. Maybe people should actually test their code before pushing it to production? It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '
- dang 8y ago
  Please don't be rude. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- JoJoJoboop 8y ago
  Well, since Firebase has lowered the barrier of entry with their easy setups, I think it is justified to expect at least some developers to make mistakes. I don't blame Firebase at all though -- great product.
tgsovlerkhgsel 8y ago
How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)? That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.
- bufferoverflow 8y ago
  You should rate-limit and cache the potentially expensive paths of your app. There's no real way to protect against a clever DDoS, someone will have to eat the costs.
  - zwily 8y ago
    You can’t really rate limit server-side with Firebase. If a client has read-permission on a data node, an attacker can just read from there as many times as they want. There are some ways to limit writing using the rules engine but you’ll still get charged for failed writes. :) Only real way to rate limit firebase that I know of is to put some sort of proxy service in between, but then you lose a lot of the advantage of using FB in the first place.
superfrank 8y ago
> This is why infinitely scaling pay-as-you-go cloud services terrify me. It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does. The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go higher even if they wanted to. When we brought this up, they told us they were technically unlimited because you could shard your data into different DBs if you needed more connections, which is like saying all restaurants are all you can eat because you can keep buying more food.
- itcmcgrath 8y ago
  Note this is about Cloud Firestore, not Firebase RTDB. It's very different infra. We announced our beta limits increased to 1M concurrent connections at the Next 18 conference, and we'll continue to improve from there.
- fefb 8y ago
  The link posted is talking about the Firestore and not the Real Time. Firestore is in beta state that has a beta limit of 100k sim. connections, probably off beta it will be able to handle more connections. The Real Time has 100K limit and is a centralized db.
jliptzin 8y ago
I will also stay far away from these services. As a child post points out, if you do set a cap, you run the risk of cutting off a legitimate spike in high quality traffic. How do you tell the system to shut off due to an error and stay on during a spike in normal traffic? You can’t, unless you have someone watching it at all hours. That’s why I prefer to just rent whole machines on AWS. If I accidentally have some code stuck in an infinite loop making some O(1) call, don’t charge me $10,000 for that when it costs you nothing. If it’s actually consuming electricity and significant resources, I’ll know quickly because my service will go down, as it should, not scale infinitely until my company is bankrupt.

mirimir 8y ago

But wait, the image shows ~$600 million. So $30K is small. Maybe that ~$600 million isn't in USD?

rosser 8y ago
The first sentence in The Fine Article mentions Colombia, whose Peso uses "$" as its symbol, and currently has an exchange rate of ~2900:1 USD.
- deleted 8y ago
  [deleted]
- mirimir 8y ago
  Thanks. And I did miss the "COP" in front of the $ amount :(
jsmthrowaway 8y ago
No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million. The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a dot after it, because it is abbreviated there (mayo). Every programmer should understand locales even if they do not speak the language.
- mirimir 8y ago
  No, it shows "$600,043,603". Where do you see "mil"? But OK, I didn't get that "COP" is Colombian Peso :( And there's another image that shows total collected as "USD $244,875". The ratio is 2450, which is close enough to the exchange rate.
  - jsmthrowaway 8y ago
    The Y axis of the graph which actually has relevant information. You are looking at a campaign page, and assuming those Colombian pesos are available to the author’s team. When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.

shaunpersad 8y ago

I can't say enough good things about Firebase and GCP in general, but I'm always cautious when using Firestore in particular. I usually avoid unbounded queries altogether, and treat it primarily as a key/value store to get by id. When I do use queries, it's always in places where the results have a well-defined limit (usually limit = 1), e.g. finding the most recent X or the highest X. With the above two, you get all the greatness of Firestore, but with a well-defined (low) cost that you can calculate ahead of time.

itcmcgrath 8y ago
We've also been rolling out updates to rules to enable you to enforce these types of things. There is performance implications to limit queries for the real-time update system at scale, but for most use cases this shouldn't be a problem. Definitely more we can improve here for control, and we're open to feedback.
- fefb 8y ago
  Hey, thanks for listening. I believe in short time would be nice to have a way to use for the query the create, update and write time of a document. Now, I am doing the creat time manage inside of my document with Date.now(), but when I was running a bunch of promises to create the documents, the createTime between the documents that I was manipulating was in same case the same, so my pagination failed. Another things, like compound queries inside of subcollections should be nice. A way to export all the database for backup. A flag to alert the Firestore to return the document when I do an update in this document in the same response (one round trip (dynamoDb has it). I know I can reach this goal witht transaction, but I believe it is simpler than a transaction. A way to update a array without transaction. Thanks
- degenerate 8y ago
  My open feedback is to get these rules in place as soon as possible. This story/blog makes me realize that GCP/Firebase does not currently play nice to dumb mistakes and 24h away from the dashboard can be disastrous. Multiple rules/filters need to exist to trigger SMS/Email alerts, or a pre-defined action, upon certain conditions.

deleted 8y ago

[deleted]

duxup 8y ago

>with every visitor to our site, we needed to call every document of payments in order to see the number of supports of a Vaki, or the total collected. On every page of our app! Oh that will do it.

deleted 8y ago

[deleted]

riquito 8y ago

> we were using Angular V.4 and we decided to upgrade everything to V.6. It was a huge risk and we wanted this campaign to be perfect, so we did it! In just a day and a half, our team had the first release ready in the new version. After some tests, it looked like the refactor helped the app’s speed, but it was not as fast as we wanted it. Our goal was to load in 3 seconds and it wasn’t working as we expected. You want it perfect and you cannot afford to put down the site but you're willing to take a "huge risk" based on (wrong) guesses, with clearly not enough time for proper QA. I sincerely suggest you to slow down and reflect on priorities and risk assessment, there's a reason if that firebase code slipped through. By the way I'm happy you avoided the worst case scenario, good luck with your project

TAForObvReasons 8y ago
> I sincerely suggest you to slow down That runs counter to the great Silicon Valley ethos of moving fast and breaking things
- dboreham 8y ago
  The moving fast breaking things guy came from the east coast.

Something1234 8y ago

So what is their product? Can't read spanish, and don't use chrome for translate.

whoisjuan 8y ago
A GoFundMe kind of platform, that allows groups of people to put funds towards a common goal (like a party or whatever)...in this case they used it to donate to a politician who had a huge debt after losing an election.

sciurus 8y ago

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws.

sitkack 8y ago
> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an affordance for application maintenance and visibility. Differential analysis and graphing built directly into the execution substrate. Judge away!
- cloakandswagger 8y ago
  Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.
  - dkoston 8y ago
    Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code. Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks. The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected. Kudos to the engineers for solving this issue under pressure.

franzwong 8y ago

But did you see the slow down in the request to firebase before going to upgrade frameworks?

paradite 8y ago

So it's a classic N+1 query?

bsaul 8y ago
Not really. N+1 is about « to many » relationships. You would need at least 2 collections to run into that problem. In sql equivalent, what happened would be more like doing a full table scan on each query instead of using an index ( and not even that, because pre-computing a total isn’t really like an index) This kind of « account balance » problems are typical of the problems where transactions are really useful. But also historically the kind of problem where nosql techs do a poor job ( they’re more built with « eventual consistency » in mind than atomic or transactional behaviors)
alexpi 8y ago
It's more like (in good old RDBMS world) 'SELECT * FROM payments' and calculate sum on client vs 'SELECT SUM(amount) FROM payments WHERE some_id = ?' though they already had this sum precalculated
- jasonkester 8y ago
  That would be more forgivable. This is: SELECT * FROM payments where paymentID = 1 SELECT * FROM payments where paymentID = 2 SELECT * FROM payments where paymentID = 3 ... SELECT * FROM payments where paymentID = 14986 Each of those in its own API request over the wire, then sum them on the client.

deleted 8y ago

[deleted]

ConcernedCoder 8y ago

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

noncoml 8y ago
I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.
- avargas 8y ago
  A bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.
  - whoisjuan 8y ago
    I agree with you. There are so many ways to make this infinitely more efficient. For starters, why are they re-calculating and re-rendering the value every time they get a new donation? Also, they could store those values in a more cold&cached-storage and just make the reads to update from Firestore. Don't use a freaking database that is charging you for every single read and write, to deal with mundane client-side renderings.
  - Timshel 8y ago
    Changing the currency will re-trigger all those conversion calls ...
- dragonwriter 8y ago
  It's more like a naive database design which doesn't consider the access pattern and therefore requires an inefficient query, which is definitely an architectural issue.

ryandrake 8y ago

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis management (What was the plan if they didn't miraculously find the bad line of code? When would they pull the plug on the site? Was there a contingency plan?) 7. Perfect being the enemy of good enough Looks like they were bailed out of the cost but what if that didn't happen?

Aloha 8y ago
Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.
- boabab 8y ago
  updating the framework version is like.. literally the last try-something thing to do though.
- nine_k 8y ago
  Yes, it's a good investigation tool, but definitely not a fix. That is, if "it started to work now, I don't know why", you're still in trouble, and should continue digging.
- flukus 8y ago
  I think "try-anything" would be a better characterization in this case. They had smoke pouring out of the engine and tried changing the tire.
  - Aloha 8y ago
    No disagreement there. I don't understand how you can build a complex application like that without doing basic performance checks like, are we hitting the file system or database too often, our the image assets correctly sized, etc. I'm not a software engineer however.

whoisjuan 8y ago

I find ridiculous that their first solution was to go and upgrade to another Angular version, especially a non-beta version upgrade of a framework that is used in thousands of super high traffic websites with no problems. How clueless can you be?

thatjsguy 8y ago
> How clueless can you be? I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.
- wmij 8y ago
  Isn't this why you do pull requests, code reviews and QA before deploying? You need to make sure that the more senior(s) on the team and proper testing catch these things before deploying.
- sushid 8y ago
  Why are you so against junior devs? I've personally never met a junior dev who just starts going ahead and upgrading various dependencies, as you suggest. If anything, that shows a lack of proper hiring decision on you and your team's part. I do, however, agree that their practices are horrible (just look at their console, they're console.logging random things, running the dev mode of Firebase, and fetching some USD conversion call 10x on load with no caching) and they're lucky Google bailed them out at the last minute.
  - thatjsguy 8y ago
    > If anything, that shows a lack of proper hiring decision on you and your team's part. Hey, friend! I had no control over hiring for that gig.

pishpash 8y ago

"Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies." Doesn't sound like a future great company to me, especially when their lesson from this was Google will bail them out and "It is very important that tech teams debug every request to servers before release." rather than hiring less cavalier employees and putting in better process.

RobertRoberts 8y ago

Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt? This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them. This reminds me variable rate mortgages. With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I really don't understand the cloud at all for such high cost resources with literally unlimited/unpredictable pricing. Can someone explain this risk/reward scenario here?

Spivak 8y ago
You use dedicated hardware with manual or semi-automated scaling until you're big enough to want unlimited and green enough to eat a large bill knowing that you still made a profit.
flukus 8y ago
> How does anyone justify this risk? I'm more concerned about the risk mitigation strategies (capping) I'm seeing advocated. If your servers being pegged you've only got a few customers missing out while it's pegged, maybe even everyone getting service but sub-optimally. You can ride out the wave and everything goes back to normal. Putting caps in place is like pulling the plug out of the server after the CPU has been at 100% for 5 minutes and not plugging it in until the next billing cycle.
chillacy 8y ago
I don’t know what kind of load you expect but web servers don’t just slow down linearly forever: at some point when traffic exceeds capacity, queues fill up, memory maxes out, and all requests grind to a halt with enough load. The way out is scaling horizontally, and you want to do that at some point before you end up sending 503s to the load balancer.

bigbluedots 8y ago

This is another good reason to properly test before going live. If their site wasn't cloud hosted it would have just fallen over, most likely. Which means a failed crowdfunding effort. Maybe take off the cowboy boots, guys?

r3vrse 8y ago

I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to think critically when you can just 'npm update', fix 37 broken dependencies, and write a blog post about it? Right?

onion2k 8y ago
Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder. This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.
- cube2222 8y ago
  Indeed, if your building anything, good metrics are one of the most important things. Didn't see them having any metrics apart of Google analytics.
- projektir 8y ago
  And most companies not knowing how to measure experience, so they pass over experienced devs and can't understand their value...
- bsaul 8y ago
  It comes with experience, but also with more senior devs explaining it to you. Seing someone use firebase to save paiements, then recompute a total from a collection, and as a consequence having its system explodes with less than one session per second, means everybody in the team drank the « let’s use this nosql google shiny tech, it’s so cool » cool aid. Even one conversation with any senior dev having some kind of experience with backend development would have asked about expected load, types of queries, data model, etc. And concluded that storing paiements was probably the least interesting scenario for using a tech like firebase.

oxguy3 8y ago

This reads less like a "mistake", and more like just not giving any thought to what your code is doing as you write it. A mistake would be if the programmer meant to implement X but instead implemented Y; this sounds more like the programmer just set out to implement Y without even considering other possibilities. Same for the framework change. An outdated framework might be a second or so slower, but a >30-second load time was never going to be fixed by updating. This is just bad problem-solving skills. When your app is taking 30 seconds to load, you don't just guess at what might make it faster -- you open your JS console and your log files, and you figure out where that time is being spent. Two minutes in the Performance tab of Chrome's developer tools, and you would have figured out the issue was on the back-end rather than the front-end.

Dylan16807 8y ago
They calculated a number the wrong way. That's not a hard mistake to make. And changing frameworks is silly but not really part of the problem.

thisismydevname 8y ago

The saddest thing here is that they seem proud of this "mistake" and how they solved it. This startup mindset is not always good.

newscracker 8y ago

It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed post mortem or root cause analysis to see what they could do to prevent it (the ending looked quit amateurish, by pointing to only one thing as a potential lesson) * wasted a lot of money that could've helped in the future (by instead using that to pay for en error) If I were in such a team, I would've honestly stated in such an article how deeply ashamed I am that we missed all these things, how "cowboy coding" and heroics must never be glorified, how we got very, very lucky in someone else waiving off charges (this is not a luxury that most startups or one person endeavors would have), and ended with asking for advice on what could be done to improve things (since it's obvious there were many more gaps than just how a few lines of code were written). To the team that wrote this code and this article — get some software development methodology adopted (any, actually) and some people who can help you follow any of those. Also read the rest of the comments here. You got very, very lucky in this instance. It may not be the same case again, and you may see your "life's work" get killed because you didn't really learn.

Oras 8y ago
If they had half of the above they wouldn't use Firebase. Don't get me wrong I think it is a great product and I would use it for MVP but the idea of using Firebase is to get something scalable easily and quickly by focusing on the frontend only. You can tell from their screenshots that they were only focusing on GA metrics and didn't check Firebase console
megous 8y ago
They just didn't profile before optimizing. That's all.
segmondy 8y ago
Oh you amazing programmer, please share the amazing systems that you are building so we can bow down and learn.

boulos 8y ago

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days ago and see the chance to apply for the 100k grant on Google Cloud Services. We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant. > Now we could not be more grateful to Google, not only for having an awesome “Backend As A Service” like Firebase, but also for letting us have 2 million sessions, 60 supports per minute and billions of requests without letting our site go down. Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies.

nstj 8y ago
I don’t see why this is being downvoted. The title of the post should be changed to “We were given a $30k credit by GCP after not monitoring our account charges”
ocdtrekkie 8y ago
Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next week that Google would necessarily be willing to help, but I can absolutely guarantee you that a VM sitting on a Dell PowerEdge I've got lying around would suddenly obligate me to a $35,000 bill, no matter how bad my code. Ideally, I guess I'd want to see rather than a hard cap, some sort of smart alert that would go "holy crud, this is an unusual spike in the rate of requests" when the delta changed unusually, rather than waiting until I say, hit a high static cost bar or a hard cap that kills the site.

catchmeifyoucan 8y ago

Go serverless boys :)

fefb 8y ago
It won't resolve the problem
- catchmeifyoucan 8y ago
  AWS Lambda costs about 20 cents per million requests. A billion requests would have put it just a little south of 200-300 dollars. Factor in a few additional read units for DynamoDB, but I don't see it going to total $30,000. The serverless architecture should scale as well.
  - fefb 8y ago
    They are using serverless architecture. Firestore is a "Daas". The latency for loading the page wasn't the real problem, it was the side-effect. The real problem resulted in 40bilion request, they were requesting all the collection of payments in each session. It reaches 16K documents downloaded /session, so they were paying 16K reads documents for session. Firestore gives you 50K reads/day for free, and ask $0.06 for 100K reads. So It was a really bad logical approaching from them. If they did in a best way , like update a document that holds the two accumulators : total money accumulated and total payments created, and read just this document in each session request their bill to show this information for 2M session should be less than $2, because you are reading 1 document for session instead of 16K documents. And pay attention that their system wasn't down, probably the 30s latency was a side-effect of downloading 16K document of each payment information in the client side.

wiradikusuma 8y ago

If there's one thing Google App Engine has taught me, over the course of 5+ years using it: Denormalize your data (or, optimize for read).

bsaul 8y ago
Just a precision in case a junior dev read this comment : denormalizing has nothing to do with the problem experienced by the OP. They just queried the whole collection of paiements instead of precomputing a total everytime a new paiement arrived. Denormalizing would have required having a relation between collections. Here there was just one (so there was nothing to denormalize). Optimizing for reads ( or simply think about read performance) doesn’t require denormalizing. It can also be a matter of creating an index, or precomputing values in a cache, like in OP case.

ericpauley 8y ago

I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?

abhisuri97 8y ago
As one of those amateur developers who publishes to Hackernoon, it's because they change titles to fit their "How to" type articles + grab attention.
flukus 8y ago
> but why are these posts suddenly getting so much attention? The same reason we slow down for car crashes, morbid curiosity. I don't think there is anything "sudden" about it though, we even have sites like thedailywtf dedicated to this level of idiocy.
- gruez 8y ago
  thedailywtf: making fun of other dvelopers' incompetence hackernoon: author patting themselves on the back while everybody else is laughing at their incompetence

lifthrasiir 8y ago

I increasingly feel that modern pay-as-you-go services are so opaque to consumers that it takes the individual employee's empathy (highly subjective) or the publicity like this (highly subjective and you have to be lucky as well) to fix any significant problem. Every time a post like this with a "happy" ending crawls into the HN front page, there would be hundreds or maybe thousands of "unhappy" endings out there.

Apaec 8y ago

I feel depressed about this, on how the industry promotes and even supports extreme technical incompetence. Maybe is a consequence of "everyone should learn to code" campaigns and bootcamps.

johnmarcus 8y ago
Cheer up buddy, you were this incompetent at one time too, but now look at you, a full grown narcissistic asshole senior developer. You have come a long way and these kids will too!

betadreamer 8y ago

A lot of people here talks about lack of load testing and other "do it the right way" type of advice, but remember this is a startup. In my opinion, solid testing foundation will be such an overkill and the time is better spent on implementing more features. Also I bet they did some manual testing. They didn't catch it because this latency can only be seen by the account with a lot of followers. I agree that their first solution to upgrade is a bad idea... You should understand what caused the bug before trying to fix it. I highly encourage you to monitor the load/pay/request graph on a daily basis. Even better if you hang a screen on the office that displays these. The graphs are already provided by Firebase. That way you can catch these type of anomaly on day one. Also Firebase supports "Progressively roll out new features" https://firebase.google.com/use-cases/#new-features https://firebase.google.com/use-cases/#new-features

whoisjuan 8y ago
Lack of testing and poor code quality are completely different things though.

andybak 8y ago

Gosh. One for https://accidentallyquadratic.tumblr.com/ https://accidentallyquadratic.tumblr.com/ with the added spice of being on usage billing...

bartread 8y ago

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work together to achieve a bunch of shared goals and doing an imperfect job of it. This isn't just rose-tinted: I'm quite sure they have their fair share of bad actors, and they certainly make decisions we don't all like (e.g., retiring products), but I don't think it's because the company is fundamentally evil.

cm2187 8y ago
Except that their core business is to build files on everyone of their users to sell ads... We could have a debate on whether Philip Morris is evil. I am sure most of their employees are pretty decent people.
- Grazester 8y ago
  When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone?
  - cm2187 8y ago
    > When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone? Great logic. Philip Morris is evil because they sell products that cause cancer and death. Therefore Equifax, who extorts money from people to protect them against Equifax polluting their credit rating, and who leaks their data into the wild is not evil, because Equifax doesn't cause cancer nor death!
    - Grazester 8y ago
      When was the last time Google had a data breach and leaked my social security number they didn't have and didn't use to "extort" money from me?

TomK32 8y ago

Some have mentioned it here already, but I'd like to emphasis how important application logs are. How much trouble you can prevent by reading and understanding them. I've seen and fixed such bugs as described in the article, and before you start trying to upgrade anything a look in the log followed by a git bisect session is the first step. My rails apps have great logs, I get to see what views and partials are rendered, what queries are sent to the database and more important how often all that happens. If the log excerpt for a single request doesn't fit on my screen I know I have to do something.

carlsborg 8y ago

The blog authors should state up front that Google covered the costs with an additional startup grant, and that this was entirely due to a quadratically expensive query.

wazoox 8y ago

And that's why people write applications to understand and track your public cloud usage, like https://github.com/trackit/trackit https://github.com/trackit/trackit ... Most people simply can't follow through what's happening in their cloud apps.

shady-lady 8y ago

Words fail me as to how anybody could put something live without understanding how the application functioned as a whole. Unbelievable.

madmulita 8y ago

"It is very important that tech teams debug every request to servers before release..." No. You should know your application's profile, you wrote it. How many resources does you app need? That's something our developers believe is the "operations team"'s responsability. Well, now that you took the 'devops' role you can no longer keep ignoring this. Your new infrastructure provider will be more than happy to keep adding resources, one can only hope the pockets are deep enough. With attention to the profile this would have been caught at developing time, maybe testing time.

shusson 8y ago

| How we spent $30k in Firebase in less than 72 hours By not checking Google Billing after you launch your website. At the very least you should have a billing alert.

snowwolf 8y ago
At the very least Google (and really all cloud services) could have reasonable default billing alerts built in. And automatic spike detection. I'd rather have default alerts already configured that I can change, rather than none.

xtrapolate 8y ago

Simple stress tests should've revealed this. Basic profiling should've revealed this. This article makes it appear that they've went live without ever really testing the infrastructure under any kind of load whatsoever. The article refers to some mysterious "engineering team". It would appear very little actual engineering took place at that company.

manishsharan 8y ago

The issue with GCP and AWS providing limitless scaleability is that bad code gets a free ride and we all occasionally write bad code. If the OP had tested this on a database with 1 CPU server with 2GB RAM, they would have caught it quickly in early dev testing.

joeblau 8y ago

The thing I'm wondering about with the seniority of the dev team and what the code commit and code review process is. It seems like with a senior dev reviewing commits, someone would have caught that redundant work was being done. I worked at a startup in the Mission for a few months and I remember seeing a quadratic query that ran for every customer that was logged into our application. The CEO and team lead wondered why our app worked great in Dev (with only 5 users) but was terrible on premise (250 users). When I tried to explain the issue the two devs before me didn't really understand what I was talking about. It was a quick refactor and caching solution that fixed the problem, but it was clear that the development was still new.

danielhlockard 8y ago

My only issue - Where were the metrics around firebase request count? That could have been a simple dashboard.

doglover7869 8y ago

Wow, a complete shitshow which could've been avoided by consulting with 1 experienced developer good at debugging. This kind of negligence must be happening more than what we think so there are lots of opportunities out there.