5 ms·
June 25th, 2023 Deno Deploy Postmortem
- bobnamob 3y agoDoes no one do preemptive load testing any more? This kind of metastability _should_ be detectable through fairly simple load till failure testing and recovery monitoring. The 10x load cited is a red herring when pulling static config during recovery is their bottleneck
- IMTDb 3y agoHindsight is 20/20. This specific attack configuration highlighted a weak point. What you don’t know about are all the other attacks that did not impact the service, because the way load testing was performed already lead to some mesures being taken.
- rvba 3y agoDDOS seems to be "the" standard attack though... the one you test against
- bobnamob 3y agoAlso, the post mortem fails to give any meaningful numbers detailing user impact :( Was this a global outage for 90+% of deno deploy hosted projects? “A large chunk” doesn’t give me much faith that it was isolated to any one region
- smarx007 3y agoAny recommendations of frameworks and learning materials? Ideally, something that could be integrated in the build process. I have mostly found some SaaS services (no-go) and Apache JMeter tutorials mostly aimed at QA (e.g., using GUI mode predominantly) so far.
- goliatone 3y agoI have used k6 in our CI/CD pipeline to do integration testing and then ran nightly actions to do smoke and load testing. It was easy to setup and it worked great. The framework provides a clean way to define expectations and to produce data that can then be visualized in grafana. https://k6.io/ https://k6.io/
- gillh 3y agoDeploy Aperture for runtime protection. https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture
- brasic 3y agoPeople inevitably convince themselves that DDOS-levels of traffic are unrealistic until they’ve actually been attacked a few times.
- catlifeonmars 3y agoReally? I’ve always found that DDoS is a constant specter in design/development due to how easy it is to execute.
- ZoF 3y ago[dead]
- treis 3y ago>Due to the attack volume, a large number of processing nodes had to be brought online at the same time. Each of these nodes require a small amount of metadata that gets pulled from a central database This is pretty interesting. Doesn't seem like the right scale for a small amount of metadata to topple over a DB. That's a pretty small load even if we're talking thousands of nodes.
- leesalminen 3y agoSounds like Postgres connection scaling issues to me. Just a wild guess but the fact that connections are heavy in pg is a foot gun that tends to rear its head during large scale-up events
- belter 3y agoA distributed system working against a single relational database, even if its just metadata, it's not only the ultimate anti-pattern but also sooo 2004...As this industry tends to shun the ’graybeards,’ there is never any institutional knowledge around. Every architecture seems to have to go through the same repeated adolescent struggles...
- mr90210 3y agoYour comment might be mixing up the Microservice pattern of a single database per bounded context. From the moment one runs a system in two nodes, one being the application server and the other the database server, that system is distributed. Two nodes is all it takes.
- belter 3y ago>"Due to the attack volume, a large number of processing nodes had to be brought online at the same time. Each of these nodes require a small amount of metadata that gets pulled from a central database"
- belter 3y agoWhat is the nature of your argument? That because as long as we have two nodes, we call that a distributed system, the anti-pattern at scale is not valid?
- Hixon10 3y agoThis Timeline of Events a bit scares me. It looks like, if someone hadn't decided to stop DDOS, deno would still had problems. I would expect to cut off all DDOS traffic on some gateway level, or so.
- inglor 3y agoThey addressed the issue by scaling up but the scale up itself ended up lagging behind the database. If that attack happens again presumably the bigger database + caching would mitigate it.
- yizhang7210 3y agoBut that would incur a proportionally large cost on the infra I assume? Being able to cutoff DDOS at a gateway level by some sort of filter criteria would still be quite useful imo.
- inglor 3y agoIt’s hard for them to have two projects (the open source Deno and a closed source Deploy runtime that’s not actually Deno). I’m wondering if this was open source rather than proprietary this sort of bug allowing a DDoS would have been caught before production. I know of multiple times in the past security issues were detected by the community and dealt in Node before they reached production. It must be a challenge for the Deno team competing against the likes of Amazon or Google without the andvantage/burden of open source.
- rmorey 3y ago> that’s not actually Deno You're saying Deno Deploy does not run Deno under the hood?
- inglor 3y agoCorrect. It runs a separate runtime.
- rmorey 3y agothank you. is this documented somewhere?
- inglor 3y agoNot that I know, my reference was meetings with people from the Deno team (many of whom are friends) - I'm not sure it's documented but it can easily be verified. Just try to do `Deno.connectDatagram` on deploy for example and see it's missing.
- mmastrac 3y agoI'm working at Deno, mostly on the open-source version so I can clarify a bit. The open-source code of Deno runs in Deploy, but some APIs are disabled and others are adapted for cloud use (ie: Kv works in CLI and Deploy, but is distributed on Deploy). Under the hood Deno Deploy is just Deno with scale built in. It's obviously not a simple soundbite but I recently built the new Deno.serve API implementation and the same code effectively runs in the CLI and Deploy modulo what's necessary to get it working in the Deploy environment.
- mr90210 3y agoMore sympathy wouldn’t hurt. The folks at Deno surely have learnt a few things from such an experience. Keep throwing rocks if you or your org haven’t overlooked something in the system you work on.
- deleted 3y ago[deleted]
- catlifeonmars 3y agoThis seems like a really basic problem that would have been uncovered by some basic load testing. Y’know, the kind of thing you do (in addition to penetration testing, operational prep, etc) before you let customers use it. Fundamental stuff.
- SkyPuncher 3y agoIt was probably a known issue. It’s really just whether it’s worth addressing as a startup. Most early adopters are also willing to deal with a bit of pain. Typically, it seems that the second major issue is when a company like deno needs together get serious about addressing known, but not profitable risks.
- ericbarrett 3y agoI appreciate this post-mortem. It's brief, direct, and shows they thought about their architecture rather than only scaling the DB: > The start-up payload will be cached across the globe
- endisneigh 3y agoSadly DDOS is still a thing. I can sympathize. What are the best ways of stopping this without using cloud infrastructure these days? I’ve become too used to using the cloud offerings so I’m out of touch.
- belter 3y agoDDOS are not only still a thing, but getting worst over time. Look at the bottom historical map in this diagram - https://www.digitalattackmap.com/ https://www.digitalattackmap.com/ You protect with a combination of services and some cloud providers will offer cost compensation for resources you might have to create to stay responsive during the attack.
- gillh 3y agoThe need an adaptive protection system like Aperture[0] to mitigate overloads. [0]: https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture
- gillh 3y agoLook at all these super knowledgeable “engineers” down voting this. Load shedding is exactly what would prevent this in the future, scaling up capacity and adding a cache is only going to help buy some time. Wish I could have a dollar for each time SREs duct tape rather than solve problems with systems thinking.
- ranguna 3y agoThis is a purely informative comment. I do not mean any harm. I believe you are getting downvotes because it's difficult to give a direct solution like you did without having more context on the matter. > Look at all these super knowledgeable “engineers” down voting this. Maybe people thought "look at this super knowledgeable 'engineer' shooting a one-off solution just by looking at an RCA"
- baq 3y agoSome people say small deployments don’t need pgbouncer. The issue is sometimes you don’t get to decide when you aren’t small anymore…
- ranguna 3y agoYou have just given my a good answer to the classical "you are not google" argument. Thank you.