3 ms·
Re-posting my Twitter update[1] here: We've concluded our analysis. 1. We're refunding the overages 2. We identified the root cause The root cause is that t
by Rauchg 4y ago
Re-posting my Twitter update[1] here:
We've concluded our analysis.
1. We're refunding the overages
2. We identified the root cause
The root cause is that the Astro bundle handed to the deployment process is monolithic. There was a top-level `await` for an RSS endpoint which called an API with `fetch`. The issue is that these two (and the rest of the app) were bundled together!
Therefore, any time the function was invoked, that top-level `await` was running for all endpoints. It never yielded. And it's fully autonomous, which means it'd keep running regardless of even a browser being open once the chain reaction started.
This is a Swiss Cheese[2] kind of failure. It required the top level await, the monolithic bundle, and the RSS function using `fetch` (i.e.: over the network) rather than `import`-ing the data layer API directly.
Most importantly, what we're doing: we are going to deploy a fix to ensure this doesn't happen again, across frameworks. I really appreciate Mike raising this and hopping on a zoom call with me while our team investigated.
As an addendum for HN: we're continuing to refine the tools and patterns to best "harness" the practically-infinite ability for Serverless and Edge functions to scale horizontally. It's an awesome property, but it's taught us valuable lessons. We've come a long way in adding guardrails and alerts, and this will be another value-added protection that future customers will enjoy.
[1] https://twitter.com/rauchg/status/1644099739959590912 https://twitter.com/rauchg/status/1644099739959590912
[2] https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- WXLCKNO 4y agoGood update. Good refund. Well handled.
- swyx 4y agowell, yes, but it wouldve been better if support had also been trained that this sort of issue gets a refund escalation. one should not have to rank on Twitter AND HN to get good service for a very reasonable failure.
- llambda 4y agoI'm currently stuck in their support hell and have been told: 1. My issue is not real 2. Okay, your issue is real, but because I'm not paying $$$ we're going to ignore you 3. I should do free work for Vercel and poll their community forums to see how widespread the issue is 4. Their support is only trained to handle frontend issues and because this is an issue with their CDN it's expected that they'll respond incompetently 5. They'll escalate with their CDN team and respond in one week (that was over a month ago, no follow up whatsoever) It's hard to take Vercel seriously. As a toy, it's probably fine. But I'll ultimately move this project off of their CDN product as soon as it reaches costly volume.
- deleted 4y ago[deleted]
- josevalerio 4y agoAwesome response, handled well! You guys are always on top of things community wise, especially Lee Robinson in any related thread :')
- wut42 4y agoI don't think we can say it's been "handled well" when the customer had to rant on social media to get CEO attention. If OP didn't had any followers or else, Vercel would probably had done nothing.
- josevalerio 4y agoFair points, hopefully this brings about the needed changes in these billing systems to prevent it happening in the future. It happens all the time on other providers as well. I'm very critical of the actual necessity of these infinitely scalable systems if you find my other comment in this thread
- lostdog 4y agoAnd what about billing limits or quotas?
- 5Qn8mNbc2FNCiVV 4y agoSince it runs on AWS, it has the same limit/quota lag as AWS has
- hoofhearted 4y agoIt’s really cool to see Guillermo actively on HN!