9 ms·

Increased error rate on Gov.uk on 5 January 2022

rahoulb 5y ago
That's an excellent incident report - especially given the target audience.
bilekas 5y ago
Nice to see things like this where they publicly release the info on the incident report. I did giggle a bit that they basically turned it off and on again though.
- blitzar 5y ago
  I assume they cleared the "is the computer pluged in to the power" step from technical support before attempting the olde power off / on routine.
marmarama 5y ago
So... a fix was released in 2017 for Ubuntu Xenial and the reverse proxy servers still weren't running the fix in 2022? Appreciate the honesty from central government, but it's not a great look.
- mariusmg 5y ago
  Sadly this is way more common than expected. People simply fear updates will "break" working software and simply "delay" them. It doesn't mmatter if there are security updates, some people have the updates == bad mentality.
  - RicoElectrico 5y ago
    To be fair, if you zoom out the timeline, Linux distros change userspace so much between releases I can't blame them. I know the new solutions are mostly for the better, but it's a moving target.
    - hericium 5y ago
      Linux is not macOS - you can patch/install security fix without forced UI redesign and configurability deprivation.
    - bravetraveler 5y ago
      Long term support releases like those from Red Hat and Ubuntu aim to address this by not introducing breaking functionality in their releases Version 8.x or 22.y that you get won't change in their minor releases in terms of behavior. They save the breakage for N+1, such as 9.x and 23.y They go through a ton of effort to backport the security fixes. This keeps things generally way more predictable. Problems do happen (eg: misaligned symbols from partial updates/restarts), but they're not as common as one would assume
      - mananaysiempre 5y ago
        My experience on desktop with Debian vs Arch is that I honestly can’t tell if the LTS approach makes things better or not. (I did run Red Hat once upon a time, but given that was even before it’s split into RHEL and Fedora, it was probably nothing like it is today in this respect.) Arch (or any other mostly-unpatched rolling distro, I expect—don’t know how, say, CentOS Stream does on the former point) requires constant tending, but every weekly or monthly update is pretty much painless. Sometimes one thing breaks, but usually with an accompanying warning, and if you put it there you know how to unbreak it. Upgrading and especially dist-upgrading Debian IME felt like you were running around putting out fires for a couple of hours every time, even if those upgrades were much rarer. (It’s like the old joke about regular paradise vs college paradise: in the regular one everything’s peachy except a man comes every evening and hammers a single nail up your arse, in the college one everything’s wonderful for months but then a man comes with a bucket of nails and tells you it’s the end of the semester.) I don’t know which ends up requiring more effort in the integral sense, and if that’s even the measure that matters, in particular on servers (even “pet” ones), I only know that I dread updating on a single Arch machine much less than I do a Debian one. I’m sure this won’t scale, but keeping a herd of like a homogeneous dozen servers on a rolling distro with a reproducible setup (Arch+aconfmgr, or hell, even NixOS unstable) doesn’t sound like an inherently stupid idea. (If you need ABI stability, e.g. if you’re running a proprietary database, none of this applies, but in such a case a rolling distro is probably not an option anyway for other reasons.)
  - KronisLV 5y ago
    > It doesn't mmatter if there are security updates, some people have the updates == bad mentality. I think that this point of view is an unfortunate reality due to the world that we live in. Everyone would like security updates, but only as long as they're guaranteed not to actually break anything else. I've had regular Debian updates break GRUB and prevent a server from rebooting: https://blog.kronis.dev/everything%20is%20broken/debian-and-grub-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-and-... (admittedly, i think i had the full unattended upgrades enabled, not just the security ones, but that's still a pretty catastrophic failure) Similarly, when i finally wanted to upgrade my install of GitLab to a newer version, it also failed spectacularly rendering the whole instance inoperable: https://blog.kronis.dev/everything%20is%20broken/gitlab-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/gitlab-upda... (there is no option to get "just" the security updates, you need a new major version once the old one is EOL) I actually did a writeup since, where i migrated over to Gitea, Nexus and Drone, because even if future updates will break them, the scope will be much more manageable: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-nexus-and-drone https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-... My takeaway there was pretty similar to what you're alluding to: > Also, something else that i just now realized was that my GitLab instance was so out of date, because updating it was a painful process - the data directories that i'd need to manually back up (in addition to automated backups which are done every other day) before an update would take a lot of space, the container images are large and download slowly, the changelogs are long and configuration changes plentiful (the Unicorn to Puma migration comes to mind) and as my other post shows, a lot can go wrong and it be a very frustrating process. > In short: if you want your users to update more often and stay safe, make updating an easier and less scary process. Of course, in the case of GitLab, i don't think that's something that's necessarily easy to do, since their Omnibus install is about as close as you can get to that, but even that had problems, at least in my particular case. Sometimes you just need an alternative or a few other pieces of software to do more or less the same thing, and that's okay. I've actually written about the difficulties with updates and how problematic they become when running a non-trivial amount of software before as well, in a slightly tongue in cheek manner: https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything Apologies if that's quite a few links, it's just that i think that you're offering only a part of the argument and it should be expanded to better reflect the reality for some people. Updates are feared because in practice they absolutely do bring problems and you cannot expect that ahead of time and will have to deal with it sooner or later, so not only do you need to evaluate how to upgrade everything (including configuration and other possibly breaking changes), but also have your backups and failover systems ready ahead of time. With how much software out there doesn't have proper test or staging environments and how often it's not easy to spin up a new environment, it's completely understandable that people are going to hold out for as long as they can, which leads to some pretty bad situations in practice. Of course, the need for any sort of a SLA and uptime only makes this much worse.
  - spacemanmatt 5y ago
    Everyone thinks they'll be so successful in the next 6 months that they call LTS distributions a 5-year plan.
- 46Bit 5y ago
  I don’t know the details of this incident, but the GOV.UK website uses relatively old infrastructure — they’re working to move onto a containerised platform, but still rely heavily on VMs and Puppet.
  - robin_reala 5y ago
    Here are the Puppet manifests if anyone is interested: https://github.com/alphagov/govuk-puppet https://github.com/alphagov/govuk-puppet
  - bravetraveler 5y ago
    I hope they intend on a pipeline of sorts, otherwise the containers will likely continue to sit down-rev They could choose to not specify tags and run latest - then a power outage might cause an update by consequence of a fresh pull
- anonu 5y ago
  Sometimes theres just as much reason to be an old infra then constantly upgrading to a new one. You know the old one works...
- ChrisRR 5y ago
  > it's not a great look. This government really doesn't care about its image
fodmap 5y ago
That report is brilliant. I'm impressed with the openness, clarity, and also the fact that javascript isn't mandatory to read it. Take note web devs!
- izietto 5y ago
  Take note web designers! I don't want to play the blame game, but if a page is designed in such a way that JS is required to implement it web devs (like me) can't do much about that!
  - marcellus23 5y ago
    I think it's the responsibility of web devs to push back against designs like that. Designers aren't even aware of this kind of stuff -- it's your job to make the case why it's bad. If you push back and you get overridden, fine. But I doubt that most devs ever do that.
- open-source-ux 5y ago
  I really like the efforts of GOV.UK to make their content clear, unambiguous and easy to understand. In particular, avoiding jargon unless it is needed. Of course, jargon has it's place when it addresses a technical or specialised audience. From a different domain, here is an excellent blog post from 2019 on how to write clear health information for people with varying levels of literacy. (And no, it doesn't mean "dumbing down" your writing.) Pee and poo and the language of health: https://digital.nhs.uk/blog/transformation-blog/2019/pee-and-poo-and-the-language-of-health https://digital.nhs.uk/blog/transformation-blog/2019/pee-and...
- pabs3 5y ago
  Their post about how they only use JS minimally: https://technology.blog.gov.uk/2016/09/19/why-we-use-progressive-enhancement-to-build-gov-uk/ https://technology.blog.gov.uk/2016/09/19/why-we-use-progres... https://news.ycombinator.com/item?id=12538144 https://news.ycombinator.com/item?id=12538144
  - 5y ago
breakingcups 5y ago
I'm envious of GOV.UK, my country is not doing bad but GOV.UK seems to be doing so much right when it comes to transparency, accessibility, etc. etc. Kudos to whoever was involved in making this outage report happen.
yaseer 5y ago
A pretty decent incident report by the standards of government websites. As a business + personal user over 15 years, I've noticed The UK's gov.uk websites have been improving at the fastest rate I've seen over the last 3-4 years in particular. Does anyone know why this is? (Or perhaps you have had a different experience).
bastawhiz 5y ago
I think this is an impressive example of technical writing for the average person. The metaphors are good, the writing style is clear and without jargon. Public incident reports are often written so that they can hide behind technical complexity. Writing clearly does a lot to build trust with the audience.

That's an excellent incident report - especially given the target audience.

Nice to see things like this where they publicly release the info on the incident report. I did giggle a bit that they basically turned it off and on again though.

blitzar 5y ago
I assume they cleared the "is the computer pluged in to the power" step from technical support before attempting the olde power off / on routine.

marmarama 5y ago

So... a fix was released in 2017 for Ubuntu Xenial and the reverse proxy servers still weren't running the fix in 2022? Appreciate the honesty from central government, but it's not a great look.

mariusmg 5y ago
Sadly this is way more common than expected. People simply fear updates will "break" working software and simply "delay" them. It doesn't mmatter if there are security updates, some people have the updates == bad mentality.
- RicoElectrico 5y ago
  To be fair, if you zoom out the timeline, Linux distros change userspace so much between releases I can't blame them. I know the new solutions are mostly for the better, but it's a moving target.
  - hericium 5y ago
    Linux is not macOS - you can patch/install security fix without forced UI redesign and configurability deprivation.
  - bravetraveler 5y ago
    Long term support releases like those from Red Hat and Ubuntu aim to address this by not introducing breaking functionality in their releases Version 8.x or 22.y that you get won't change in their minor releases in terms of behavior. They save the breakage for N+1, such as 9.x and 23.y They go through a ton of effort to backport the security fixes. This keeps things generally way more predictable. Problems do happen (eg: misaligned symbols from partial updates/restarts), but they're not as common as one would assume
    - mananaysiempre 5y ago
      My experience on desktop with Debian vs Arch is that I honestly can’t tell if the LTS approach makes things better or not. (I did run Red Hat once upon a time, but given that was even before it’s split into RHEL and Fedora, it was probably nothing like it is today in this respect.) Arch (or any other mostly-unpatched rolling distro, I expect—don’t know how, say, CentOS Stream does on the former point) requires constant tending, but every weekly or monthly update is pretty much painless. Sometimes one thing breaks, but usually with an accompanying warning, and if you put it there you know how to unbreak it. Upgrading and especially dist-upgrading Debian IME felt like you were running around putting out fires for a couple of hours every time, even if those upgrades were much rarer. (It’s like the old joke about regular paradise vs college paradise: in the regular one everything’s peachy except a man comes every evening and hammers a single nail up your arse, in the college one everything’s wonderful for months but then a man comes with a bucket of nails and tells you it’s the end of the semester.) I don’t know which ends up requiring more effort in the integral sense, and if that’s even the measure that matters, in particular on servers (even “pet” ones), I only know that I dread updating on a single Arch machine much less than I do a Debian one. I’m sure this won’t scale, but keeping a herd of like a homogeneous dozen servers on a rolling distro with a reproducible setup (Arch+aconfmgr, or hell, even NixOS unstable) doesn’t sound like an inherently stupid idea. (If you need ABI stability, e.g. if you’re running a proprietary database, none of this applies, but in such a case a rolling distro is probably not an option anyway for other reasons.)
- KronisLV 5y ago
  > It doesn't mmatter if there are security updates, some people have the updates == bad mentality. I think that this point of view is an unfortunate reality due to the world that we live in. Everyone would like security updates, but only as long as they're guaranteed not to actually break anything else. I've had regular Debian updates break GRUB and prevent a server from rebooting: https://blog.kronis.dev/everything%20is%20broken/debian-and-grub-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-and-... (admittedly, i think i had the full unattended upgrades enabled, not just the security ones, but that's still a pretty catastrophic failure) Similarly, when i finally wanted to upgrade my install of GitLab to a newer version, it also failed spectacularly rendering the whole instance inoperable: https://blog.kronis.dev/everything%20is%20broken/gitlab-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/gitlab-upda... (there is no option to get "just" the security updates, you need a new major version once the old one is EOL) I actually did a writeup since, where i migrated over to Gitea, Nexus and Drone, because even if future updates will break them, the scope will be much more manageable: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-nexus-and-drone https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-... My takeaway there was pretty similar to what you're alluding to: > Also, something else that i just now realized was that my GitLab instance was so out of date, because updating it was a painful process - the data directories that i'd need to manually back up (in addition to automated backups which are done every other day) before an update would take a lot of space, the container images are large and download slowly, the changelogs are long and configuration changes plentiful (the Unicorn to Puma migration comes to mind) and as my other post shows, a lot can go wrong and it be a very frustrating process. > In short: if you want your users to update more often and stay safe, make updating an easier and less scary process. Of course, in the case of GitLab, i don't think that's something that's necessarily easy to do, since their Omnibus install is about as close as you can get to that, but even that had problems, at least in my particular case. Sometimes you just need an alternative or a few other pieces of software to do more or less the same thing, and that's okay. I've actually written about the difficulties with updates and how problematic they become when running a non-trivial amount of software before as well, in a slightly tongue in cheek manner: https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything Apologies if that's quite a few links, it's just that i think that you're offering only a part of the argument and it should be expanded to better reflect the reality for some people. Updates are feared because in practice they absolutely do bring problems and you cannot expect that ahead of time and will have to deal with it sooner or later, so not only do you need to evaluate how to upgrade everything (including configuration and other possibly breaking changes), but also have your backups and failover systems ready ahead of time. With how much software out there doesn't have proper test or staging environments and how often it's not easy to spin up a new environment, it's completely understandable that people are going to hold out for as long as they can, which leads to some pretty bad situations in practice. Of course, the need for any sort of a SLA and uptime only makes this much worse.
- spacemanmatt 5y ago
  Everyone thinks they'll be so successful in the next 6 months that they call LTS distributions a 5-year plan.
46Bit 5y ago
I don’t know the details of this incident, but the GOV.UK website uses relatively old infrastructure — they’re working to move onto a containerised platform, but still rely heavily on VMs and Puppet.
- robin_reala 5y ago
  Here are the Puppet manifests if anyone is interested: https://github.com/alphagov/govuk-puppet https://github.com/alphagov/govuk-puppet
- bravetraveler 5y ago
  I hope they intend on a pipeline of sorts, otherwise the containers will likely continue to sit down-rev They could choose to not specify tags and run latest - then a power outage might cause an update by consequence of a fresh pull
anonu 5y ago
Sometimes theres just as much reason to be an old infra then constantly upgrading to a new one. You know the old one works...
ChrisRR 5y ago
> it's not a great look. This government really doesn't care about its image

fodmap 5y ago

That report is brilliant. I'm impressed with the openness, clarity, and also the fact that javascript isn't mandatory to read it. Take note web devs!

izietto 5y ago
Take note web designers! I don't want to play the blame game, but if a page is designed in such a way that JS is required to implement it web devs (like me) can't do much about that!
- marcellus23 5y ago
  I think it's the responsibility of web devs to push back against designs like that. Designers aren't even aware of this kind of stuff -- it's your job to make the case why it's bad. If you push back and you get overridden, fine. But I doubt that most devs ever do that.
open-source-ux 5y ago
I really like the efforts of GOV.UK to make their content clear, unambiguous and easy to understand. In particular, avoiding jargon unless it is needed. Of course, jargon has it's place when it addresses a technical or specialised audience. From a different domain, here is an excellent blog post from 2019 on how to write clear health information for people with varying levels of literacy. (And no, it doesn't mean "dumbing down" your writing.) Pee and poo and the language of health: https://digital.nhs.uk/blog/transformation-blog/2019/pee-and-poo-and-the-language-of-health https://digital.nhs.uk/blog/transformation-blog/2019/pee-and...
pabs3 5y ago
Their post about how they only use JS minimally: https://technology.blog.gov.uk/2016/09/19/why-we-use-progressive-enhancement-to-build-gov-uk/ https://technology.blog.gov.uk/2016/09/19/why-we-use-progres... https://news.ycombinator.com/item?id=12538144 https://news.ycombinator.com/item?id=12538144
- 5y ago

breakingcups 5y ago

I'm envious of GOV.UK, my country is not doing bad but GOV.UK seems to be doing so much right when it comes to transparency, accessibility, etc. etc. Kudos to whoever was involved in making this outage report happen.

yaseer 5y ago

A pretty decent incident report by the standards of government websites. As a business + personal user over 15 years, I've noticed The UK's gov.uk websites have been improving at the fastest rate I've seen over the last 3-4 years in particular. Does anyone know why this is? (Or perhaps you have had a different experience).

bastawhiz 5y ago

I think this is an impressive example of technical writing for the average person. The metaphors are good, the writing style is clear and without jargon. Public incident reports are often written so that they can hide behind technical complexity. Writing clearly does a lot to build trust with the audience.