10 ms·

Gov.uk site broke Google realtime analytics during lockdown announcement

101 points by whyleyc 6y ago

whyleyc 6y ago
Estimated that >1% of the UK population (769K) were concurrently on the site before GA topped out.
- taneq 6y ago
  That doesn’t sound like that much compared with the normal loads that GA handles every day, though? Gold star to the UK gov team for their bit holding up so well though.
  - hnlmorg 6y ago
    Its not. In a previous job I used to run fantasy sports games and one football World Cup we saw a spike sharper and faster than the one described on GOV.UK today due to two successive goals from a top player just as the half time whistle went. GA handled that spike just fine but if it wasn't for Akamai our site would have been toast. I'm not taking anything away from the engineers behind GOV.UK though. They do an excellent job.
1f60c 6y ago
Why did it break?
- g_p 6y ago
  Interesting question. I wonder if Google has a scale speed limit for an individual tenant's sites, or perhaps each URL? It seems that the underlying gov.uk page remained up throughout though.
- gowld 6y ago
  Usage quota for the service they pay for, presumably.
dazzawazza 6y ago
There are a lot of things to complain about in the UK but the digital services bit of the UK government is really well run. Congratulations on riding out the thundering herd.
- mnd999 6y ago
  Best not mention the COVID app.
  - zinekeller 6y ago
    Plus the search tool: https://www.theregister.com/2020/11/26/govuk_tier_postcode_lookup_crash/ https://www.theregister.com/2020/11/26/govuk_tier_postcode_l... (Worse of all, Auntie Beeb (aka the BBC) has operated a similar tool without issue.)
  - petepete 6y ago
    That didn't have anything to do with the GDS.
  - tehbeard 6y ago
    That got farmed out to a consultancy with the ear of one of our glorious politicians.. I will shit on gov IT any day of the week but I'm not blaming GDS for that particular fire.
- mkl95 6y ago
  Indeed. They also maintain a site dedicated to the gov.uk web design system, with free resources such as HTML components. https://design-system.service.gov.uk/ https://design-system.service.gov.uk/
  - abhinav22 6y ago
    Wow! The team in charge of digital at UK Govt are rockstars! Simply amazing for a govt agency to be this good!
    - 6y ago
deleted 6y ago
[deleted]
alexmcc81 6y ago
Am I the only one who thinks it's creepy that the largest advertising company in the world, and particularly one based outside the country, has so much data on what UK citizens access on their own government websites for health, tax, social welfare information etc?
- paxys 6y ago
  It should all be public information in the first place.
  - alexmcc81 6y ago
    General statistics on site usage would be interesting. But what about all the other data that could be extracted, both on a personal and national level? You could, for example, determine which areas are more likely to access HIV information on the NHS website. Insurance companies would like that but would it benefit the public or be in line with people's basic privacy expectations for government services? (I'm sure Google claim that they won't add a user looking at cancer information on the NHS site to their data profile but I wonder what other insights could be generated.)
    - DanBC 6y ago
      > determine which areas are more likely to access HIV information on the NHS website We can already do this. Have a look at the Public Health England "Fingertips" website. https://fingertips.phe.org.uk/ https://fingertips.phe.org.uk/ https://fingertips.phe.org.uk/profile/sexualhealth https://fingertips.phe.org.uk/profile/sexualhealth https://fingertips.phe.org.uk/profile/sexualhealth/data#page/0/gid/1938133286/pat/6/par/E12000004/ati/202/are/E06000015/cid/4/page-options/ovw-do-0 https://fingertips.phe.org.uk/profile/sexualhealth/data#page...
  - dastx 6y ago
    Not really, it will include who is accessing this data, which can be used to infer information on UK citizens.
  - 6y ago
jimnotgym 6y ago
If Boris had given out at least a basic summary of what the new rules mean I would have stayed off the website. A better speechwriter could have saved a huge amount of bandwidth
- mytailorisrich 6y ago
  The rules are: stay home, which is the take-away message and what he said. Some people will always go "yes, but..." and look for ways around the rules, as we've painfully seen in the UK since March. In fairness, some people have very good reasons not not stay home and the PM did also mention these reasons. Overall I found the message and level of details well calibrated for a short, straight to the point, speech. Going into an exhaustive list of all the rules and exceptions would have clouded the message and would not have been a good use of the format and of time.
  - LatteLazy 6y ago
    The only thing he announced, the only change, is that some (not all) schools are closing. The fact he spent 10 minutes and didn't quite manage to communicate which schools/kids were affected says it all really...
    - mytailorisrich 6y ago
      All schools are to close as he very clearly said. This is not the only change. Overall this is back to full lockdown as it was in March, which is quite stricter that the current situation (aka tier 4 in most of England) Edit: yes, children of key workers are still accepted although schools are closed, like in March. He said that, you guys clearly understood that. Let's not be disingenuous here...
      - jimnotgym 6y ago
        All schools...except nursery schools. And this is only England, Scotland, Wales and NI all have separate rules. Full lockdown like in March? I read that I should not go out more than once per day to exercise. In March I read that I must not.
random5634 6y ago
If folks think this load broke analytics please think again. Is there no critical thinking left out there in journalism? Do folks understand the scale analytics operates at? Some quick questions Did they actually pay for analytics 360? If not, did they do 10 million hits in a month for a property? if so - they may be over the limit of 10 million / month. More on differences here: https://blog.littledata.io/2018/02/28/google-analytics-360-versus-the-free-version/ https://blog.littledata.io/2018/02/28/google-analytics-360-v... Analytics 360 runs 12K+/month from memory, but someone probably has more recent info.
- fuzzy2 6y ago
  What journalism might you be referring to?
  - random5634 6y ago
    I'm starting with whoever wrote the headline here. Gov.uk site did not break google analytics. This is standard click-bait / shock value type headlines.
    - sMarsIntruder 6y ago
      Well, it actually broke Google Analytics Realtime, since basically the report stopped working.
      - random5634 6y ago
        False - they may just be using the FREE version for "individual or small businesses". This has limits. Google will stop analytics if you are on just the free account and start going over their free limits. This does not mean google analytics has broken. Is gov.uk an "individual or small business"? Or are they paying for 360?
        sMarsIntruder 6y ago
        As I said, the report broke. Stop. As the headline says. And yes you're right that they "may be just using the FREE version", but it's a "may be".
altrunox 6y ago
Is that much? He said they got around 88k RPS. I did a quick benchmark another day, I'm hosting my blog at the Digital Ocean App Plataform, in the free tier, used Locust with some VMs in Azure to stress test it, managed to hit 20k RPS and the only issue I had was that the my VMs at Azure were hitting memory limits... Doesn't look impressive at all, weird that GA broke.
2Gkashmiri 6y ago
This might be awkward for some but can governments be compelled to use Foss software and where a Foss software is lacking features, fund for development of that feature so that everyone benefits instead of lacing more and more reliance on proprietary tech which inturn forces citizens to use that proprietary junk because "compatibility"?
divbzero 6y ago
“Out of the 17M page requests made between 8–9pm (111M if you include assets: stylesheets, images, etc), we served 21 5xx errors.” [1] [1]: https://twitter.com/therealnooshu/status/1346419468151488512 https://twitter.com/therealnooshu/status/1346419468151488512

101 points by whyleyc 6y ago

whyleyc 6y ago

Estimated that >1% of the UK population (769K) were concurrently on the site before GA topped out.

taneq 6y ago
That doesn’t sound like that much compared with the normal loads that GA handles every day, though? Gold star to the UK gov team for their bit holding up so well though.
- hnlmorg 6y ago
  Its not. In a previous job I used to run fantasy sports games and one football World Cup we saw a spike sharper and faster than the one described on GOV.UK today due to two successive goals from a top player just as the half time whistle went. GA handled that spike just fine but if it wasn't for Akamai our site would have been toast. I'm not taking anything away from the engineers behind GOV.UK though. They do an excellent job.

1f60c 6y ago

Why did it break?

g_p 6y ago
Interesting question. I wonder if Google has a scale speed limit for an individual tenant's sites, or perhaps each URL? It seems that the underlying gov.uk page remained up throughout though.
gowld 6y ago
Usage quota for the service they pay for, presumably.

dazzawazza 6y ago

There are a lot of things to complain about in the UK but the digital services bit of the UK government is really well run. Congratulations on riding out the thundering herd.

mnd999 6y ago
Best not mention the COVID app.
- zinekeller 6y ago
  Plus the search tool: https://www.theregister.com/2020/11/26/govuk_tier_postcode_lookup_crash/ https://www.theregister.com/2020/11/26/govuk_tier_postcode_l... (Worse of all, Auntie Beeb (aka the BBC) has operated a similar tool without issue.)
- petepete 6y ago
  That didn't have anything to do with the GDS.
- tehbeard 6y ago
  That got farmed out to a consultancy with the ear of one of our glorious politicians.. I will shit on gov IT any day of the week but I'm not blaming GDS for that particular fire.
mkl95 6y ago
Indeed. They also maintain a site dedicated to the gov.uk web design system, with free resources such as HTML components. https://design-system.service.gov.uk/ https://design-system.service.gov.uk/
- abhinav22 6y ago
  Wow! The team in charge of digital at UK Govt are rockstars! Simply amazing for a govt agency to be this good!
  - 6y ago

deleted 6y ago

[deleted]

alexmcc81 6y ago

Am I the only one who thinks it's creepy that the largest advertising company in the world, and particularly one based outside the country, has so much data on what UK citizens access on their own government websites for health, tax, social welfare information etc?

paxys 6y ago
It should all be public information in the first place.
- alexmcc81 6y ago
  General statistics on site usage would be interesting. But what about all the other data that could be extracted, both on a personal and national level? You could, for example, determine which areas are more likely to access HIV information on the NHS website. Insurance companies would like that but would it benefit the public or be in line with people's basic privacy expectations for government services? (I'm sure Google claim that they won't add a user looking at cancer information on the NHS site to their data profile but I wonder what other insights could be generated.)
  - DanBC 6y ago
    > determine which areas are more likely to access HIV information on the NHS website We can already do this. Have a look at the Public Health England "Fingertips" website. https://fingertips.phe.org.uk/ https://fingertips.phe.org.uk/ https://fingertips.phe.org.uk/profile/sexualhealth https://fingertips.phe.org.uk/profile/sexualhealth https://fingertips.phe.org.uk/profile/sexualhealth/data#page/0/gid/1938133286/pat/6/par/E12000004/ati/202/are/E06000015/cid/4/page-options/ovw-do-0 https://fingertips.phe.org.uk/profile/sexualhealth/data#page...
- dastx 6y ago
  Not really, it will include who is accessing this data, which can be used to infer information on UK citizens.
- 6y ago

jimnotgym 6y ago

If Boris had given out at least a basic summary of what the new rules mean I would have stayed off the website. A better speechwriter could have saved a huge amount of bandwidth

mytailorisrich 6y ago
The rules are: stay home, which is the take-away message and what he said. Some people will always go "yes, but..." and look for ways around the rules, as we've painfully seen in the UK since March. In fairness, some people have very good reasons not not stay home and the PM did also mention these reasons. Overall I found the message and level of details well calibrated for a short, straight to the point, speech. Going into an exhaustive list of all the rules and exceptions would have clouded the message and would not have been a good use of the format and of time.
- LatteLazy 6y ago
  The only thing he announced, the only change, is that some (not all) schools are closing. The fact he spent 10 minutes and didn't quite manage to communicate which schools/kids were affected says it all really...
  - mytailorisrich 6y ago
    All schools are to close as he very clearly said. This is not the only change. Overall this is back to full lockdown as it was in March, which is quite stricter that the current situation (aka tier 4 in most of England) Edit: yes, children of key workers are still accepted although schools are closed, like in March. He said that, you guys clearly understood that. Let's not be disingenuous here...
    - jimnotgym 6y ago
      All schools...except nursery schools. And this is only England, Scotland, Wales and NI all have separate rules. Full lockdown like in March? I read that I should not go out more than once per day to exercise. In March I read that I must not.

random5634 6y ago

If folks think this load broke analytics please think again. Is there no critical thinking left out there in journalism? Do folks understand the scale analytics operates at? Some quick questions Did they actually pay for analytics 360? If not, did they do 10 million hits in a month for a property? if so - they may be over the limit of 10 million / month. More on differences here: https://blog.littledata.io/2018/02/28/google-analytics-360-versus-the-free-version/ https://blog.littledata.io/2018/02/28/google-analytics-360-v... Analytics 360 runs 12K+/month from memory, but someone probably has more recent info.

fuzzy2 6y ago
What journalism might you be referring to?
- random5634 6y ago
  I'm starting with whoever wrote the headline here. Gov.uk site did not break google analytics. This is standard click-bait / shock value type headlines.
  - sMarsIntruder 6y ago
    Well, it actually broke Google Analytics Realtime, since basically the report stopped working.
    - random5634 6y ago
      False - they may just be using the FREE version for "individual or small businesses". This has limits. Google will stop analytics if you are on just the free account and start going over their free limits. This does not mean google analytics has broken. Is gov.uk an "individual or small business"? Or are they paying for 360?
      - sMarsIntruder 6y ago
        As I said, the report broke. Stop. As the headline says. And yes you're right that they "may be just using the FREE version", but it's a "may be".

altrunox 6y ago

Is that much? He said they got around 88k RPS. I did a quick benchmark another day, I'm hosting my blog at the Digital Ocean App Plataform, in the free tier, used Locust with some VMs in Azure to stress test it, managed to hit 20k RPS and the only issue I had was that the my VMs at Azure were hitting memory limits... Doesn't look impressive at all, weird that GA broke.

2Gkashmiri 6y ago

This might be awkward for some but can governments be compelled to use Foss software and where a Foss software is lacking features, fund for development of that feature so that everyone benefits instead of lacing more and more reliance on proprietary tech which inturn forces citizens to use that proprietary junk because "compatibility"?

divbzero 6y ago

“Out of the 17M page requests made between 8–9pm (111M if you include assets: stylesheets, images, etc), we served 21 5xx errors.” [1] [1]: https://twitter.com/therealnooshu/status/1346419468151488512 https://twitter.com/therealnooshu/status/1346419468151488512