7 ms·

British Library cyber incident review [pdf]

108 points by cnorthwood 3y ago

wara23arish 3y ago
I happened to be there while this attack was in progress (ocotober 23). And all there systems were really offline, POS didnt work, wifi didn’t work, literally anything connected to a computer didnt work. What’s unfortunate is that they flagged this vulnerability in 2022 and planned to review it in 2024 ??? Does it usually take this long to identify impact of users? They mentioned they paid for identity protection for their staff & ex-staff as well.
- alexriddle 3y ago
  I work in a related field (cyber insurance response) - typically takes a few months to identify exfiltrated data and then analyse it to understand what is in it. This might seem simple but there are usually in the region of hundreds of thousands to millions of files, and that may contain spreadsheets with tens of thousands of rows. This all has to be analysed, filtered and reduced to the point you have a list of PII which has been impacted, and can decide on what to do. Credit monitoring is usually offered as standard when a breach occurs, the UK is much less litigation friendly than the US so in the absence of any actual harm, that would discharge most of their obligations to protect you following an incident.
  - ooterness 3y ago
    Who decided credit monitoring was an adequate remedy for these breeches? I think I've accumulated three or four lifetimes of it by now, but it's never done anything but spew false alarms.
jefc1111 3y ago
"The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy" ... "this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA)" ¯\_(ツ)_/¯
- the8472 3y ago
  Occasionally malware groups do patch vulnerabilities to maintain exclusive control over the victim machines. But that wouldn't be my default expectation, so relying on virus software to provide security does not seem like a great idea.
- yard2010 3y ago
  There are many attack vectors to bypass MFA, especially sms based MFA
  - jefc1111 3y ago
    True, but if you don't have it enabled / required then you're giving off signals of negligence which may extend into other vulnerabilities.
nonrandomstring 3y ago
Good report. Well written incident summary useful for cyber-students to follow and learn. > The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy > increasing complexity of managing their access was flagged as a risk. > first detected unauthorised access to our network was identified at the Terminal Services server. This terminal server had been installed in February 2020 to facilitate efficient access for trusted external partners Sadly their response seems to be using more cloud infrastructure and outsourcing more. trusted != trustworthy The essential lesson - that good IT and security people within your company cost money. It is worth paying for vigilance, loyalty and care - has not been heeded.
- graemep 3y ago
  > Sadly their response seems to be using more cloud infrastructure and outsourcing more. CYA - it stops being their management's fault if its outsourced,
everfrustrated 3y ago
This report is a joke. No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence. They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical. Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.
- tokai 3y ago
  >They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. That doesn't fit their claims on page 7 about reviewing the lost data and contacting affected users.
  - toyg 3y ago
    They reviewed what the criminals later dumped on the dark web. They have no way to determine if the criminals kept more for themselves.
- clwg 3y ago
  I suspect they don't have the forensic evidence to determine the root cause. Chances are there are probably too many ways it could have happened, and the evidence was encrypted or simply wasn't being captured. At least they seem to have a plan moving forward that seems considered, though I think a lot of what they want to do is easier said than done effectively. I wish them the best of luck.
  - nonrandomstring 3y ago
    > I suspect they don't have the forensic evidence to determine the root cause. It said that. The terminal server entry point was completely scorched in the attack. Offsite rlogd would have helped.
- 3y ago
aiiotnoodle 3y ago
A lot of this sounds like they were under-resourced and the business increasingly adopted new technology with no ongoing support for their IT infrastructure. > These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls. > There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security. > Our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack. A lot of lines like the following, also indicate to me IT was increasingly were involved in fighting fires and maintining operational systems ("keeping the lights on") rather than deploying new infrastructure and automation, updating software etc. > Some of our older applications rely substantially on manual extract (...) which in a modern data management and reporting infrastructure would be encapsulated in secure, automated end-to end workflows. Modern business is IT, I know that I am preaching to the chior but this sounds a lot like their IT was seen as a cost.
- toyg 3y ago
  The British Library is closer to academia than business. Their IT provider is a state-adjacent entity: https://en.wikipedia.org/wiki/Jisc https://en.wikipedia.org/wiki/Jisc .
- clwg 3y ago
  I've known people who have worked in IT in national museum settings, and from what I heard it sounded like a mix of traditional IT support—ensuring the lights stayed on, printers could print, emails and phones worked, and a very simple website stayed online. Some aspects sounded quite interesting, but these weren't places pushing the envelope in any aspect of technology. I'm sure they were running outdated software and configurations on everything, but IT was closing their tickets and meeting their SLAs. And with no disrespect, these people weren't necessarily disruptors looking to shake up and modernize the museums' infrastructure and take it into the future either, they just did their job to the best of their ability and went home at the end of the day. To generalize I find that this usually holds true in a lot of non-tech industries, and IT is generally seen as a burdensome cost as opposed to enabler of business.
emmelaich 3y ago
I object to the word "utilises" instead of just plain "uses", especially from a library.
- jgrahamc 3y ago
  Yes. Horrible word. I actually banned utilise/utilize on the Cloudflare blog because use says the same thing (mostly), is easier to say, and shorter.
toyg 3y ago
> When alerted by the Library following discovery of the attack, Jisc (who provide the Library’s internet access and monitor movement of data across their networks) identified that an unusually high volume of data traffic (440GB) had left the Library’s estate at 1.30am on 28 October. "Jisc is the UK digital, data and technology agency focused on tertiary education, research and innovation." State-owned quango asleep at the wheel. Unsurprising.
- _tk_ 3y ago
  Could you elaborate why them being state owned was a contributing factor? We’ve seen countless similar incidents with private MSSPs as well.
  - toyg 3y ago
    Because the state (eh) of State-owned or state-adjacent anything, in modern Britain, is simply terrible. The dominant Thatcherite ideology ensures that state-provided services are almost invariably second-rate, thanks to systemic under-funding. In this case, it looks like Jisc was basically turned into a charity in 2011, so technically they're not even state-owned anymore.
- nonrandomstring 3y ago
  > State-owned quango asleep at the qwheel. Unsurprising. This used to be what we called JANET. Back in the day this was top banana and prestigious to work for like GCHQ etc. I expect they've died from a thousand cuts under the Tories. Every university I've been in the past 10 years have their ICT run by Microsoft, and is absolute rubbish.
- ta1243 3y ago
  The library's ISP said "yes, our monitoring shows you shifted an unusual amount of traffic" at that time. My ISP could do the same thing. How is that being asleep at the wheel?
Yeri 3y ago
[flagged]
physicsguy 3y ago
> The increasing use of third-party providers within our network, some of which has been due to capacity and capability constraints within Technology and elsewhere in the Library, was noted by the Library’s Corporate Information Governance Group (CIGG) in late 2022, and the increasing complexity of managing their access was flagged as a risk. A review of security provisions relating to the management of third parties was planned for 2024; and the tightening of access provisions that would be enabled by improvements to underlying computer and storage infrastructure and the migration of storage to the cloud, which is currently being implemented. Unfortunately, the attack occurred before these necessary pre-requisites for this work were completed. Price of everything and value of nothing. Outsource everything, underfund everything from systems renewal to staff salaries.
pheatherlite 3y ago
So Tom, Dick and Harry all have Terminal rdp access into the core infrastructure and they slept well knowing that they had - what was it? Ah, yes, - prevented clipboard copying as a hardening measure. That'll stop them pirates in their tracks. Nicely written post mortem. Though I can't help but notice the amount of committees and acronyms. Is it a British thing?
- KineticLensman 3y ago
  > the amount of committees and acronyms. Is it a British thing? Take a look at the US DoD, NASA, etc. They love acronyms, complicated internal organisation structures, just as much as the Brits do.
b800h 3y ago
"Our major software systems cannot be brought back in their pre-attack form, either because they are no longer supported by the vendor or because they will not function on the new secure infrastructure that is currently being rolled out." Ouch.
herodotus 3y ago
I have to applaud the library for releasing this report. In Canada, the most likely response to cyberattacks is mealy mouthed platitudes like "Please be assured that we take your privacy very seriously and are doing everything possible to recover the data and ensure that something like this does not happen again." and on and on. So refreshing.
suyash 3y ago
Nice job on publishing this detailed report, I wish after every attack all organizations disclosed in such detail so we can create future defence and counter measures in an open source way.
faceloss 3y ago
[dead]
pbhjpbhj 3y ago
A few naive questions: I see a few comments indicating that connecting Microsoft (? not mentioned anywhere in the report??) t Terminal Services to the internet was a wholly bad idea. Aside: is the report using "Terminal Services" generically, or do they mean that the server hasn't been updated since before 2009 (? when it seems Terminal Services became Remote Desktop Services (RDS))? Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure, or ...? RDP is default enabled on MS Windows installs (I always disable it), is that more of a problem than one might imagine? Do they say anywhere where the access was from (maybe only GCHQ know that). Presumably the firewall would only allow known connections - did they report on analysis of all the remote clients?
- EvanAnderson 3y ago
  > Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure... Exposing RDP to the Internet directly has been frowned-upon because of the attack surface being presented, there's no two factor "story" out-of-the-box, and you're opened up to brute force attempts on cruddy user passwords. Older versions of the Microsoft Remote Desktop Protocol had a much larger attack surface than current versions. The current versions with Network Level Authentication (starting in Windows Vista/Server 2008) present a smaller attacks surface. Older versions used "homegrown" Microsoft crypto, whereas current versions use TLS. Disclosure: I made a FLOSS fail2ban-like tool for RDP many years ago[0]. I had a situation where I was forced to expose RDP to the Internet and I didn't like having it open w/o some protection against brute force attacks. This tool happens to still work in Server 2022 and will slow the velocity of brute force attacks. I still highly recommend not exposing RDP directly to the Internet anyway. (The ts_block tool is missing some fairly essential functionality that I never got around to implementing. It works fine and is really easy to install but some things are sub-optimal.) [0] https://github.com/EvanAnderson/ts_block https://github.com/EvanAnderson/ts_block
- technion 3y ago
  The term has become a bit generic these days and people will use it in place of a range of things. Citrix or vmware are often just called "terminal server" by aome people. There is a huge difference between a port forward on port 3389, and publishing the gateway behind azure app proxy - the latter supporting mfa, account lockouts, and not actually requiring any open port to the internet. Much of the discussion online treats these as equal.
penguin_booze 3y ago
> This paper provides an overview of the cyber-attack on the British Library that took place in October 2023 and examines its implications for the Library’s operations, future infrastructure, risk assessment and lessons learned. For a report from British--and a library, no less--the lack of Oxford comma cocnerns me.
- seabass-labrax 3y ago
  Despite its name, use of the Oxford comma is more frequently promoted in the USA than it is in Britain. As a British person myself, I generally avoid it. N=1, but I wouldn't expect the London-based British Library to use a construction named after an Oxford University Press style guide.
gatvol 3y ago
Herein lies the kicker: > In common with other on-premise servers, this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA).
PatriciaBravo 3y ago
[dead]
ShaylaRaegan 3y ago
[dead]
racheljaneville 3y ago
[dead]
FayeLipscomb 3y ago
[dead]
lousiemerry 3y ago
[dead]
Dovelyntamren 3y ago
[dead]
egobiawa2 3y ago
[dead]

108 points by cnorthwood 3y ago

I happened to be there while this attack was in progress (ocotober 23). And all there systems were really offline, POS didnt work, wifi didn’t work, literally anything connected to a computer didnt work. What’s unfortunate is that they flagged this vulnerability in 2022 and planned to review it in 2024 ??? Does it usually take this long to identify impact of users? They mentioned they paid for identity protection for their staff & ex-staff as well.

alexriddle 3y ago
I work in a related field (cyber insurance response) - typically takes a few months to identify exfiltrated data and then analyse it to understand what is in it. This might seem simple but there are usually in the region of hundreds of thousands to millions of files, and that may contain spreadsheets with tens of thousands of rows. This all has to be analysed, filtered and reduced to the point you have a list of PII which has been impacted, and can decide on what to do. Credit monitoring is usually offered as standard when a breach occurs, the UK is much less litigation friendly than the US so in the absence of any actual harm, that would discharge most of their obligations to protect you following an incident.
- ooterness 3y ago
  Who decided credit monitoring was an adequate remedy for these breeches? I think I've accumulated three or four lifetimes of it by now, but it's never done anything but spew false alarms.

jefc1111 3y ago

"The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy" ... "this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA)" ¯\_(ツ)_/¯

the8472 3y ago
Occasionally malware groups do patch vulnerabilities to maintain exclusive control over the victim machines. But that wouldn't be my default expectation, so relying on virus software to provide security does not seem like a great idea.
yard2010 3y ago
There are many attack vectors to bypass MFA, especially sms based MFA
- jefc1111 3y ago
  True, but if you don't have it enabled / required then you're giving off signals of negligence which may extend into other vulnerabilities.

nonrandomstring 3y ago

Good report. Well written incident summary useful for cyber-students to follow and learn. > The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy > increasing complexity of managing their access was flagged as a risk. > first detected unauthorised access to our network was identified at the Terminal Services server. This terminal server had been installed in February 2020 to facilitate efficient access for trusted external partners Sadly their response seems to be using more cloud infrastructure and outsourcing more. trusted != trustworthy The essential lesson - that good IT and security people within your company cost money. It is worth paying for vigilance, loyalty and care - has not been heeded.

graemep 3y ago
> Sadly their response seems to be using more cloud infrastructure and outsourcing more. CYA - it stops being their management's fault if its outsourced,

everfrustrated 3y ago

This report is a joke. No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence. They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical. Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.

tokai 3y ago
>They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. That doesn't fit their claims on page 7 about reviewing the lost data and contacting affected users.
- toyg 3y ago
  They reviewed what the criminals later dumped on the dark web. They have no way to determine if the criminals kept more for themselves.
clwg 3y ago
I suspect they don't have the forensic evidence to determine the root cause. Chances are there are probably too many ways it could have happened, and the evidence was encrypted or simply wasn't being captured. At least they seem to have a plan moving forward that seems considered, though I think a lot of what they want to do is easier said than done effectively. I wish them the best of luck.
- nonrandomstring 3y ago
  > I suspect they don't have the forensic evidence to determine the root cause. It said that. The terminal server entry point was completely scorched in the attack. Offsite rlogd would have helped.
3y ago

aiiotnoodle 3y ago

A lot of this sounds like they were under-resourced and the business increasingly adopted new technology with no ongoing support for their IT infrastructure. > These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls. > There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security. > Our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack. A lot of lines like the following, also indicate to me IT was increasingly were involved in fighting fires and maintining operational systems ("keeping the lights on") rather than deploying new infrastructure and automation, updating software etc. > Some of our older applications rely substantially on manual extract (...) which in a modern data management and reporting infrastructure would be encapsulated in secure, automated end-to end workflows. Modern business is IT, I know that I am preaching to the chior but this sounds a lot like their IT was seen as a cost.

toyg 3y ago
The British Library is closer to academia than business. Their IT provider is a state-adjacent entity: https://en.wikipedia.org/wiki/Jisc https://en.wikipedia.org/wiki/Jisc .
clwg 3y ago
I've known people who have worked in IT in national museum settings, and from what I heard it sounded like a mix of traditional IT support—ensuring the lights stayed on, printers could print, emails and phones worked, and a very simple website stayed online. Some aspects sounded quite interesting, but these weren't places pushing the envelope in any aspect of technology. I'm sure they were running outdated software and configurations on everything, but IT was closing their tickets and meeting their SLAs. And with no disrespect, these people weren't necessarily disruptors looking to shake up and modernize the museums' infrastructure and take it into the future either, they just did their job to the best of their ability and went home at the end of the day. To generalize I find that this usually holds true in a lot of non-tech industries, and IT is generally seen as a burdensome cost as opposed to enabler of business.

emmelaich 3y ago

I object to the word "utilises" instead of just plain "uses", especially from a library.

jgrahamc 3y ago
Yes. Horrible word. I actually banned utilise/utilize on the Cloudflare blog because use says the same thing (mostly), is easier to say, and shorter.

toyg 3y ago

> When alerted by the Library following discovery of the attack, Jisc (who provide the Library’s internet access and monitor movement of data across their networks) identified that an unusually high volume of data traffic (440GB) had left the Library’s estate at 1.30am on 28 October. "Jisc is the UK digital, data and technology agency focused on tertiary education, research and innovation." State-owned quango asleep at the wheel. Unsurprising.

_tk_ 3y ago
Could you elaborate why them being state owned was a contributing factor? We’ve seen countless similar incidents with private MSSPs as well.
- toyg 3y ago
  Because the state (eh) of State-owned or state-adjacent anything, in modern Britain, is simply terrible. The dominant Thatcherite ideology ensures that state-provided services are almost invariably second-rate, thanks to systemic under-funding. In this case, it looks like Jisc was basically turned into a charity in 2011, so technically they're not even state-owned anymore.
nonrandomstring 3y ago
> State-owned quango asleep at the qwheel. Unsurprising. This used to be what we called JANET. Back in the day this was top banana and prestigious to work for like GCHQ etc. I expect they've died from a thousand cuts under the Tories. Every university I've been in the past 10 years have their ICT run by Microsoft, and is absolute rubbish.
ta1243 3y ago
The library's ISP said "yes, our monitoring shows you shifted an unusual amount of traffic" at that time. My ISP could do the same thing. How is that being asleep at the wheel?

Yeri 3y ago

[flagged]

physicsguy 3y ago

> The increasing use of third-party providers within our network, some of which has been due to capacity and capability constraints within Technology and elsewhere in the Library, was noted by the Library’s Corporate Information Governance Group (CIGG) in late 2022, and the increasing complexity of managing their access was flagged as a risk. A review of security provisions relating to the management of third parties was planned for 2024; and the tightening of access provisions that would be enabled by improvements to underlying computer and storage infrastructure and the migration of storage to the cloud, which is currently being implemented. Unfortunately, the attack occurred before these necessary pre-requisites for this work were completed. Price of everything and value of nothing. Outsource everything, underfund everything from systems renewal to staff salaries.

pheatherlite 3y ago

So Tom, Dick and Harry all have Terminal rdp access into the core infrastructure and they slept well knowing that they had - what was it? Ah, yes, - prevented clipboard copying as a hardening measure. That'll stop them pirates in their tracks. Nicely written post mortem. Though I can't help but notice the amount of committees and acronyms. Is it a British thing?

KineticLensman 3y ago
> the amount of committees and acronyms. Is it a British thing? Take a look at the US DoD, NASA, etc. They love acronyms, complicated internal organisation structures, just as much as the Brits do.

b800h 3y ago

"Our major software systems cannot be brought back in their pre-attack form, either because they are no longer supported by the vendor or because they will not function on the new secure infrastructure that is currently being rolled out." Ouch.

herodotus 3y ago

I have to applaud the library for releasing this report. In Canada, the most likely response to cyberattacks is mealy mouthed platitudes like "Please be assured that we take your privacy very seriously and are doing everything possible to recover the data and ensure that something like this does not happen again." and on and on. So refreshing.

suyash 3y ago

Nice job on publishing this detailed report, I wish after every attack all organizations disclosed in such detail so we can create future defence and counter measures in an open source way.

faceloss 3y ago

[dead]

pbhjpbhj 3y ago

A few naive questions: I see a few comments indicating that connecting Microsoft (? not mentioned anywhere in the report??) t Terminal Services to the internet was a wholly bad idea. Aside: is the report using "Terminal Services" generically, or do they mean that the server hasn't been updated since before 2009 (? when it seems Terminal Services became Remote Desktop Services (RDS))? Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure, or ...? RDP is default enabled on MS Windows installs (I always disable it), is that more of a problem than one might imagine? Do they say anywhere where the access was from (maybe only GCHQ know that). Presumably the firewall would only allow known connections - did they report on analysis of all the remote clients?

EvanAnderson 3y ago
> Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure... Exposing RDP to the Internet directly has been frowned-upon because of the attack surface being presented, there's no two factor "story" out-of-the-box, and you're opened up to brute force attempts on cruddy user passwords. Older versions of the Microsoft Remote Desktop Protocol had a much larger attack surface than current versions. The current versions with Network Level Authentication (starting in Windows Vista/Server 2008) present a smaller attacks surface. Older versions used "homegrown" Microsoft crypto, whereas current versions use TLS. Disclosure: I made a FLOSS fail2ban-like tool for RDP many years ago[0]. I had a situation where I was forced to expose RDP to the Internet and I didn't like having it open w/o some protection against brute force attacks. This tool happens to still work in Server 2022 and will slow the velocity of brute force attacks. I still highly recommend not exposing RDP directly to the Internet anyway. (The ts_block tool is missing some fairly essential functionality that I never got around to implementing. It works fine and is really easy to install but some things are sub-optimal.) [0] https://github.com/EvanAnderson/ts_block https://github.com/EvanAnderson/ts_block
technion 3y ago
The term has become a bit generic these days and people will use it in place of a range of things. Citrix or vmware are often just called "terminal server" by aome people. There is a huge difference between a port forward on port 3389, and publishing the gateway behind azure app proxy - the latter supporting mfa, account lockouts, and not actually requiring any open port to the internet. Much of the discussion online treats these as equal.

penguin_booze 3y ago

> This paper provides an overview of the cyber-attack on the British Library that took place in October 2023 and examines its implications for the Library’s operations, future infrastructure, risk assessment and lessons learned. For a report from British--and a library, no less--the lack of Oxford comma cocnerns me.

seabass-labrax 3y ago
Despite its name, use of the Oxford comma is more frequently promoted in the USA than it is in Britain. As a British person myself, I generally avoid it. N=1, but I wouldn't expect the London-based British Library to use a construction named after an Oxford University Press style guide.

gatvol 3y ago

Herein lies the kicker: > In common with other on-premise servers, this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA).

PatriciaBravo 3y ago

[dead]

ShaylaRaegan 3y ago

[dead]

racheljaneville 3y ago

[dead]

FayeLipscomb 3y ago

[dead]

lousiemerry 3y ago

[dead]

Dovelyntamren 3y ago

[dead]

egobiawa2 3y ago

[dead]