11 ms·
Busiest hour ever in the history of Gov.uk 17M page requests just 21 5xx errors
- a2tech 6y agoI didn't see in the Twitter thread--can anyone tell me what was driving this huge spike in traffic on Gov.uk? I'm not in the UK so I'm not up to the minute with political announcements.
- mytailorisrich 6y agoNot only did the PM announced a new full lockdown in a TV address, as others have already said, but he also specifically gave a gov.uk URL as way to check details and rules. Huge audience plus that equalled massive traffic spike.
- dtf 6y agoYet, it seemed that the guidance the PM mentioned wasn't online at the moment that he mentioned it. Like many, I was hitting F5 repeatedly on this page waiting for a document that didn't seem to be there, eager to know restrictions on various aspects of life. But the /coronavirus page just contained stale advice from December. Eventually at 8:15pm, somebody found a link to a PDF of the new guidance and posted it on Twitter: "This has all been done with such crashing urgency that they haven’t even been able to transpose the PDF version of the guidance into the website yet, it is here:" https://twitter.com/AdamWagner1/status/1346188409300258816 https://twitter.com/AdamWagner1/status/1346188409300258816 [this is how it played out for me at least, did anyone else see this differently?]
- mytailorisrich 6y agoNice. I guess that this only increased the spike.
- EE84M3i 6y agoThere is a new lockdown: https://www.bbc.com/news/uk-55538937 https://www.bbc.com/news/uk-55538937 specifically, this would seem to coincide with the PM addressing the nation on TV.
- dingaling 6y agoThe sort of thing that multicast was intended to solve, instead of everyone requesting their own stream of the same data. But MC never gained traction even though most ISPs in the UK are now IPv6-enabled.
- AaronFriel 6y agoThat's because people want to start videos from the beginning and seek around, not just watch TV. Every time I see this comment about multicast, I think people are missing the point of the internet.
- mprovost 6y agoIPv6 doesn't have anything to do with multicast - it works fine on v4. For some definition of works - the only industry where I've heard of it being used extensively is finance. For live events like press conferences or sports this is already solved with broadcasting.
- iso1210 6y ago> the only industry where I've heard of it being used extensively is finance. > For live events like press conferences or sports this is already solved with broadcasting. 16 million people watched that press conference on BBC TV, broadcasting is very efficent at getting live pictures out. The BBC did experiment with multicast in the past, but CDNs seem to be a more scalable solution ironically. However in this specific case, the pictures actually left downing street via multicast (SDI form camera into encoder, MPEG-TS over multicast to the studio, back to SDI to get encoded on whichever output chain it is - Terrestial, Satelite, Cable, online) Indeed the standard to replace SDI (2110) is built around multicast, broadcast certainly uses multicast a fair bit.
- deleted 6y ago[deleted]
- parthdesai 6y agofrom the top active pages table, it looks it's due to the new lockdown in England.
- julian55 6y agoPrime Minister Boris Johnson announcing another lockdown.
- Turukawa 6y agoUK Prime Minister announced new COVID England-wide lockdown https://www.theguardian.com/world/2021/jan/05/covid-lockdown-in-england-likely-in-place-until-march-gove-warns https://www.theguardian.com/world/2021/jan/05/covid-lockdown...
- KineticLensman 6y agoI was one of the people whose first reaction was to look on gov.uk to understand the scope of the new lockdown, rather than some rehash of it in the media. It took me just two mouse clicks (open gov.uk favourite, click on a link [0] near the top of the page) to find this info. This is just awesome. [0] https://www.gov.uk/guidance/national-lockdown-stay-at-home https://www.gov.uk/guidance/national-lockdown-stay-at-home
- cbg0 6y agoIsn't this a little unimpressive, given that these are static pages served by a CDN?
- timsworkaccount 6y agoEh, it's the UK government. Any lack of fuckup is a surprise.
- superhuzza 6y agoOn the whole, the UK Gov is actually quite good at digital services. Much better than any of my experiences with services offered by the US, Canada or France.
- realusername 6y ago> Much better than any of my experiences with services offered by the US, Canada or France. That's not hard to be better in the case of France, half of the procedures require to send some physical letter and the rest are accessible on some atrocious websites.
- superhuzza 6y agoYou're absolutely right, there's still a very old-school mentality when it comes to French bureaucracy.
- skrebbel 6y agoThe UK government is often quoted as a leading example on how to run digital services well. A fuckup would be a surprise.
- iso1210 6y agoThe UK civil service is great - although it strikes me as someone really good was given the power and budget to make it happen. The UK government however...
- maniacalrobot 6y agogov.uk is a fantastic resource, the team behind it, GDS, should be proud of what they’ve accomplished with it
- jon-wood 6y agoThere are very few good things I have to say about the Conservative government, but the early days of GDS make it onto that short list. They intentionally structured GDS as much like a startup as possible, attracting a huge number of highly competent people, and giving them oversight of government digital projects. They're the inspiration for many other country's digital teams, including the USDS who are strongly modelled on GDS, and have spawned equivalent teams within other branches of the UK government such as the Ministry of Justice and NHS.
- gm3dmo 6y agoMany of the people who built GDS now work for public.digital who help governments around the world to build these services.
- lucasverra 6y agoGDS is /government-digital-service [1] [1] https://www.gov.uk/government/organisations/government-digital-service https://www.gov.uk/government/organisations/government-digit...
- raverbashing 6y agoSo from the graph aprox. 250 pages per second. Not a bad number, I wonder how it compares with a HN hug of death. Sure, caching helps but it's not the whole story (LB, configuring the cache, benchmarking, etc) Also what dashboard is that?
- danpalmer 6y agoAnecdotally, HN is probably ~1/100th of this. The "hug of death" is much more about bad servers and very low usage limits for things like shared hosting plans than it is about the volume of traffic.
- TeMPOraL 6y agoVarious hugs of death may be correlated. People re-post things found on one service to different ones. You can easily get a Reddit hug on top of HN hug.
- CraigRood 6y ago17M page Requests in a single 1 hour block would suggest significantly more than 250 pages per second. https://twitter.com/TheRealNooshu/status/1346419468151488512 https://twitter.com/TheRealNooshu/status/1346419468151488512
- raverbashing 6y agoYou're right, it seems that was the ramp-up, this one shows a more realistic value https://twitter.com/TheRealNooshu/status/1346187935088054276/photo/1 https://twitter.com/TheRealNooshu/status/1346187935088054276... 17M per hour is ~ 4700 per second.
- amyboyd 6y agoThe dashboard is Google Analytics.
- cagenut 6y agothats google analytics. specifically the "live" feature they added to commodify/undercut people who were moving to or splitting their spend with chartbeat (who's main thing was that live-on-site counter). fwiw, the HN hug of death rarely breaks 5-figures for that metric. Its plenty to destroy a wordpress blog on a two core VPS, but its nothing compared to something getting shared in a large facebook group or trending on twitter. News sites are several orders of magnitude larger than any other type of sites besides the mega-platforms (fb) and comms tools like gmail/slack. Its actually a core part of why the entire sector's business model failed together. Everyone chased "reach" (raw audience size) as their most important revenue generating metric (because it directly factored as higher CPMs). Thats how we wound up with dozens of "news" sites all catering to the exact same 100M people clicking awful headlines in facebook shares and then closing the tab to go back to facebook. They (we) solved for scale, but in a way that turned everyone into a commodified copy of each other with no meaningful connection or relationship to the audience. Then facebook just captured all of the ad revenue by gatekeeping/aggregating/"curating".
- prof-dr-ir 6y agoI have lived in five countries now, but not one (local or state) government had an online presence that was nearly as good as the gov.uk infrastructure. Their websites might not look the part, but they excel (again, in my experience) in their practicality, accessibility and ease of use. And that is exactly what you would expect from a government website.
- ignoranceprior 6y agoIn the US, 18F does a good job: https://18f.gsa.gov https://18f.gsa.gov
- systemvoltage 6y agoAgreed. I like the US design system more than anything else out there: https://designsystem.digital.gov/ https://designsystem.digital.gov/ It’s incredibly well thought out and designed.
- zucker42 6y agoWhat's wrong with their aesthetics? I've always found their simple design to be quite nice.
- skrebbel 6y agoI think he means they don't look particularly "high tech" or something. Which I'd say is a feature.
- mytailorisrich 6y agoIt's simple and clear. The aim is to convey information, not to be fancy and bloated, indeed.
- blackhaz 6y agoAnd gov.uk's clear design looks better than most of the wild wild web out there, to be honest.
- gadders 6y agoWell done for doing a victory lap but on the other hand they have unforced errors like this: https://www.independent.co.uk/news/uk/home-news/government-website-what-tier-down-crash-b1762243.html https://www.independent.co.uk/news/uk/home-news/government-w...
- XCSme 6y agoNote that 770k concurrent users reported by GA are not really concurrent. Each user times out after about 30mins only AFAIK, so it most likely means 770k in the last 30mins, not currently browsing.
- timthorn 6y agoYou can follow the tweets - it was 770k at 20:16, 615k at 20:13, 384k at 20:10, 256k at 20:07, 117k at 20:04, 39k at 20:02, and 45k at 19:55.
- buro9 6y agoWhen you serve traffic at this volume, it's no longer the requests per second that matter. You can easily have static content on lots of metals, but the new problem is saturation of peer links on egress, or unintentionally triggering DDoS mitigations along the path that the traffic takes (or on your own or the CDN services). That the content could be placed on enough metals to serve it is the easy part... but a nicely designed solution for serving the requests isn't as impressive as keeping the network solid and operating smoothly. This is also why in the tweet thread that performance, optimisation of resources, etc is called out explicitly... fail to do this and you kill your network. Fastly did great here, but the gov.uk people and GDS also did great in making the job for Fastly a lot easier.
- gonzo41 6y agoDon't forget Varnish the real hero. Or the magic behind the curtain more like it.
- mellosouls 6y agoDupe again, fwiw https://news.ycombinator.com/item?id=25638368 https://news.ycombinator.com/item?id=25638368
- lucasnortj 6y agoShows how many fools there are in the UK that anyone takes notice of our useless government. I for one pay not a single second of attention
- bigtones 6y agoAnd the UK Government is willingly sending details of every one of these page requests and website visitors to Google so Google can profit by better serving ads to the people of the UK. How nice. Also their content security policy in production exposes all their development and staging infrastructure and their AWS service endpoints.
- ChrisArchitect 6y agoyes yes GDS is legendary in web design/accessibility and influenced the US's 18f
- londons_explore 6y agoI have worked at a company where every 500 error is a reason to go check the logs and try to reproduce... We would literally page an engineer. I've also worked at a company where "as long as the error rate stays under 0.1%, it's fine". As long as each app server doesn't have more than 1 crash per week it's fine. I can tell you that engineers at the first company end up finding all kinds of super rare bugs, usually in the OS, platform, malloc library, load balancer, etc. They then commit fixes for those bugs which ends up helping the latter company...
- treve 6y agoIt's an interesting take. We definitely don't have the capacity & skills to be in the first company, although it would be awesome. I think most places I worked we had no choice but to be in the second company, although I think the rate is a bit below .1%. I'm grateful for all the people hunting down these bugs and making the world better for the rest of us. I remember around 2006, we still found PHP segfaults that were critical enough I reported at least a dozen. Everything feels so much more stable today (at least on that layer).
- shinji97 6y agocurious if you can share a little more on the first company you mentioned, what was their area of business?
- khaless 6y agoWhile it's nice to get paged, and look at every 5xx error; it doesn't really scale all that well once you get past a certain point, particularly if your application is gracefully degrading. That said, I love the wisdom in your comment that you find all sorts of super rare bugs, or conditions that could seriously effect performance, or availability if they become more common (which they often do). Past a point, I've found that an approach which works well is to encourage engineers/operators to drive by metrics, and pay close attention over time to p100's (max), as you've suggested with your 500 errors. Lots of goodies can be hidden behind them, just like you've found with the 500 errors.