8 ms·

Maryland hasn't updated Covid case data for 15 days due to a security incident

112 points by zone411 5y ago

thegrif 5y ago
In the last 100 days, the MD Department of Information Technology (DoIT) has submitted $3,000,000 of ____emergency____ procurement requests for network monitoring, intrusion detection, and firewall assistance. All have been approved and funded. Sept 2021: - Splunk Emergency Professional Services - $800k - src: https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf Dec 2021: - Splunk Emergency Professional Services - $1.6m - Palo Alto Emergency Professional Services - $360k - Ciena Emergency Professional Services - $240k - src: https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf So...what do you think is going on at Maryland's Department of IT? ____TRULY SHAMEFUL INCIDENT COMMUNICATIONS____ Bad things happen in IT departments every day. Some can be anticipated, some even possibly prevented - but there will always exist the risk of data breach/attack. “Public officials have no higher responsibility than keeping the American people safe, and there is no greater threat to their safety than the cyber vulnerabilities of the systems that support our daily lives,” said Governor Hogan in November 2021. Secretary Leahy's commitment to the Governor and the people of Maryland is to lead the state's IT resources in taking all reasonable measures to protect against such risks. I do not yet know if this situation could have been reasonably prevented or if it is due to MD DoIT negligence/oversight. The flurry of emergency procurement requests suggests awareness and good intentions. And yet - here we are... What is clear to me, however, is that Secretary Leahy and his directs have ___failed to effectively manage communications___ throughout the response and remediation effort. Most damning, in my opinion, is the use of the incident report as a venue for high-fiving and taking victory laps: > Because of the state’s aggressive cybersecurity strategy, and the use of MD THINK and other cloud-based services, many of the department’s core functions were not affected. > src: https://health.maryland.gov/incidentupdate/Pages/default.aspx https://health.maryland.gov/incidentupdate/Pages/default.asp... Are you serious? It's been over two weeks. And it's clear from your report that you're nowhere close to understanding what happened, let alone fixing it. ____FUNDAMENTAL MISUNDERSTANDING OF INCIDENT IMPACT____ Whoever approved this for publication is oblivious to the downstream, rippling effects this incident continues to have: - At the time of writing, MD's local public health departments have been ___fighting a war blindfolded___ for 16 days. - The impact has rippled ___beyond MD's state borders___, impacting CDC data feeds as well. Hospitalizations (which are reported by each hospital network directly to the CDC) are trending upward while new cases (reported by each state's IIS to the CDC) have flatlined: https://imgur.com/gallery/k2gG5GS https://imgur.com/gallery/k2gG5GS. ____AND WHAT THE HECK IS MD THINK?____ MD Think is Maryland's cloud application platform. The state, along with Deloitte and an army of subcontractors, has been working on migrating legacy applications to this new infrastructure since 2017. The report points to the success of MD THINK in isolating the impact away from core MD Health Department services. Is MD Think really a success? Is this a case of what happens when legacy applications are left outside to rot in the sun? No. It turns out that MD Think has been a nightmare since it's inception. A series of program failures that jeopardized open enrollment for medicaid recipients led to a full audit of the MD DoIT in 2020. Here is what they had to say about MD THINK: > "...for one project (MD THINK), the estimated $314.1 million cost of completion had increased by $141.1 million (81.5 percent)...an explanation was not provided for the increase..." And on the topic of security: > "We identified 55 firewall rules that allowed traffic from any source to 71 unique network destinations as well as a small network segment within DoIT’s internal network without IDPS coverage..." > "...access to personally identifiable information (PII) for State vendors stored in the State’s Financial Management Information System (FMIS) was not adequately restricted, and the PII was accessible to thousands of State employees." > src: https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport?fileId=5eac668aa1ce580f64a4317e https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport... ____What About ImmuNET?____ A question that begs an answer is why ImmuNet, the state's immunization registry, is not included under this "titanium umbrella of protection?" And while there is a ton of publicly available paperwork about the scope of MD THINK, including the RFP, Deloitte's response, and transcripts of hearings to explore the audit report's findings, ImmuNet is not mentioned. Not once. Please keep in mind - ImmuNet is not some podunk, state government hobby project. It is __law__ that Maryland retain vaccination records for its residents. School systems, employers, health providers, and the CDC all rely on Immunization Information Systems (IIS) for ground truth vaccination records. It is the public health equivalent of DMV. ____PASSION DISCLAIMER____ The lion's share of my time this year has been spent working with state and local COVID vaccination programs within upstate New York. I have seen firsthand how important accurate, timely test and vaccination data is to how local public health departments coordinate response. It was infuriating to see the MD DoIT downplay disruptions to COVID related data feeds. We are going into year three of a war that demands accurate, timely data to inform decision making. What the hell have you guys been doing for two weeks?! The country's public health departments are filled with the true unsung heroes of the pandemic. They're underpaid, exhausted, afraid, and constantly under attack - yet they're our last (and in many ways, only) line of defense. SO... To the three people that made it all the way to the end of this post - I ask just one thing: IF YOU EVER FIND YOURSELF INVOLVED IN A FUCKUP THAT IMPACTS THE DEPARTMENT OF PUBLIC HEALTH: - RULE 1: Apologize and fix it. Now. With the kindness, urgency, and empathy a mission controller would show a stranded astronaut--Proceed to RULE 6. - RULE 2: If the problem is not your fault--return to RULE 1. - RULE 3: If the problem isn't even real--return to RULE 1. - RULE 4: If they are blaming you for a problem they created on their own--return to RULE 1. - RULE 5: If the problem is related to Microsoft Access, FoxPro, or any other technology you swore an oath to avoid--return to RULE 1. - RULE 6: The final rule. Do not downplay the importance of data and tools public health resources rely upon. And NEVER, EVER respond to the problem by boasting about all the great technical achievements made everywhere BUT the underfunded, under-appreciated public health department.
- briandrupieski 5y ago
  Thank you for posting this. As a Maryland resident I found this really enlightening. I wasn't aware of this and I'm pretty curious now what's going on. It makes me wonder how or if I can help, as an ordinary citizen. I read through those procurement requests. The reasons they cite in the remarks are interesting. Sept 2021, Splunk: > The staff providing the Splunk support services resigned as of March 31, 2021, and the contractor has been unable to replace them. The Department requested these support services through another contract vehicle, but that contractor was unable to source qualified and competent candidates in the appropriate labor categories and with the rates provided in the contract. Dec 2021, Palo Alto Firewalls: > Although the Department has two contract vehicles available to staff this need, neither contractor has been successful in staffing the role. One contractor who once provided the services lost three of their four resources in the past 12 months. The other contractor has been unable to source a candidate with the requisite skills and experience in the appropriate labor categories and rates provided in the contract. So people are quitting, and they can't find replacements. Wow, that makes me think we're seeing effects from the "Great Resignation". I wonder what the "rates provided in the contract" are.
kyrra 5y ago
Looks like we hugged the server to death? Won't load for me or archive.is.
- throaway46546 5y ago
  I feel sorry for their admins. First they get hacked and then they get DDoS'd.
- zone411 5y ago
  Sorry, it was up when I posted this. Here is a shorter version (without a FAQ): https://health.maryland.gov/newsroom/Pages/Maryland-Department-of-Health-Update-on-Investigation-of-Network-Security-Incident-.aspx https://health.maryland.gov/newsroom/Pages/Maryland-Departme..., here is their dashboard showing no updates since Dec. 3: https://coronavirus.maryland.gov/ https://coronavirus.maryland.gov/, and here is a news story: https://www.youtube.com/watch?v=cLTaIW24fxU https://www.youtube.com/watch?v=cLTaIW24fxU.
Reichhardt 5y ago
- lgrapenthin 5y ago
  Or just looking at death numbers.
- steve_mcdougall 5y ago
  There is no evidence Omicron is milder than Delta. https://www.aljazeera.com/news/2021/12/18/no-evidence-omicron-milder-than-delta-british-researchers-say https://www.aljazeera.com/news/2021/12/18/no-evidence-omicro...
  - XzetaU8 5y ago
    https://astralcodexten.substack.com/p/the-phrase-no-evidence-is-a-red-flag https://astralcodexten.substack.com/p/the-phrase-no-evidence...
    - jollybean 5y ago
      Here is the Oxford study [1] it's brutal. [1] https://www.imperial.ac.uk/news/232698/modelling-suggests-rapid-spread-omicron-england/ https://www.imperial.ac.uk/news/232698/modelling-suggests-ra...
      - walterbell 5y ago
        Past performance of UK models, https://data.spectator.co.uk/category/sage-scenarios https://data.spectator.co.uk/category/sage-scenarios
        jollybean 5y ago
        It's not a model, it's an observation. There is no clear evidence as of yet to suggest hospitalization or mortality rates of Omicron among our kind of population. We'll have a better idea in a few weeks.
jpe90 5y ago
This is actually pretty bad, their mobile app wasn't updated to explain the situation and was showing 0 new cases for the timeframe. I personally know people who took that at face value.
throwaway599281 5y ago
Counting cases and advertising them every time the news shows start is useless, it just fuels the paranoia. This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever.
- eru 5y ago
  Not quite every other virus. We eliminated a few handful of diseases. (And of course, lots and lots of viruses were eliminated naturally. Just by natural selection and competition. There are even some hints that coronaviruses are replacing influenca.)
- johnchristopher 5y ago
  > This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever. This statement mean different things to different people. Do we live with AIDS with or without condoms ? Do we live with Covid with or without masks and vaccines and other prophylactic measures ?
  - throwaway599281 5y ago
    - johnchristopher 5y ago
      I am not making myself clear enough: some use the statement "we have to live with covid" to mean "let covid cull the weak and the old, people die all the time anyway, vaccines and masks are only needed for the weak and the old so we shouldn't make them mandatory" and others to mean "covid is here to stay so are vaccines, masking and ventilation, etc.". > Have we ever imposed measures that limit the freedom of movement of the entire population, or subsects of it that have a higher infection rates per capita, such as drug users, prostitutes or homosexuals, as many countries have done? That's not the point. The point is that AIDS has changed some things that we haven't come back to yet. One of the most prominent one is condom usage. This change doesn't need to be proved/sanctified by laws to happen and yet it exists (but I'll add that some people have been sentenced to years in prison for not revealing their HIV/AIDS status when having sex with their partner). So: what does it mean to "live with covid" ?
vbphprubyjsgo 5y ago
covid isn't going to reach some new all time high infection rate because one monitoring system in one state was down for a few weeks (if that's the sentiment here. article doesn't load)
cyounkins 5y ago
Maryland Governor Larry Hogan tested positive today: https://twitter.com/GovLarryHogan/status/1472946270507397125 https://twitter.com/GovLarryHogan/status/1472946270507397125
- subsaharancoder 5y ago
  So did Elizabeth Warren, what's your point?
  - cyounkins 5y ago
    I just wanted to reference additional news pertaining to covid in Maryland.
throwanem 5y ago
...until today: https://coronavirus.maryland.gov/ https://coronavirus.maryland.gov/ now has data through 12/19, with daily updates to resume tomorrow per an update on that same page.
Goety 5y ago
Digital ID/health seems dead in the water.

112 points by zone411 5y ago

In the last 100 days, the MD Department of Information Technology (DoIT) has submitted $3,000,000 of ____emergency____ procurement requests for network monitoring, intrusion detection, and firewall assistance. All have been approved and funded. Sept 2021: - Splunk Emergency Professional Services - $800k - src: https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf Dec 2021: - Splunk Emergency Professional Services - $1.6m - Palo Alto Emergency Professional Services - $360k - Ciena Emergency Professional Services - $240k - src: https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf So...what do you think is going on at Maryland's Department of IT? ____TRULY SHAMEFUL INCIDENT COMMUNICATIONS____ Bad things happen in IT departments every day. Some can be anticipated, some even possibly prevented - but there will always exist the risk of data breach/attack. “Public officials have no higher responsibility than keeping the American people safe, and there is no greater threat to their safety than the cyber vulnerabilities of the systems that support our daily lives,” said Governor Hogan in November 2021. Secretary Leahy's commitment to the Governor and the people of Maryland is to lead the state's IT resources in taking all reasonable measures to protect against such risks. I do not yet know if this situation could have been reasonably prevented or if it is due to MD DoIT negligence/oversight. The flurry of emergency procurement requests suggests awareness and good intentions. And yet - here we are... What is clear to me, however, is that Secretary Leahy and his directs have ___failed to effectively manage communications___ throughout the response and remediation effort. Most damning, in my opinion, is the use of the incident report as a venue for high-fiving and taking victory laps: > Because of the state’s aggressive cybersecurity strategy, and the use of MD THINK and other cloud-based services, many of the department’s core functions were not affected. > src: https://health.maryland.gov/incidentupdate/Pages/default.aspx https://health.maryland.gov/incidentupdate/Pages/default.asp... Are you serious? It's been over two weeks. And it's clear from your report that you're nowhere close to understanding what happened, let alone fixing it. ____FUNDAMENTAL MISUNDERSTANDING OF INCIDENT IMPACT____ Whoever approved this for publication is oblivious to the downstream, rippling effects this incident continues to have: - At the time of writing, MD's local public health departments have been ___fighting a war blindfolded___ for 16 days. - The impact has rippled ___beyond MD's state borders___, impacting CDC data feeds as well. Hospitalizations (which are reported by each hospital network directly to the CDC) are trending upward while new cases (reported by each state's IIS to the CDC) have flatlined: https://imgur.com/gallery/k2gG5GS https://imgur.com/gallery/k2gG5GS. ____AND WHAT THE HECK IS MD THINK?____ MD Think is Maryland's cloud application platform. The state, along with Deloitte and an army of subcontractors, has been working on migrating legacy applications to this new infrastructure since 2017. The report points to the success of MD THINK in isolating the impact away from core MD Health Department services. Is MD Think really a success? Is this a case of what happens when legacy applications are left outside to rot in the sun? No. It turns out that MD Think has been a nightmare since it's inception. A series of program failures that jeopardized open enrollment for medicaid recipients led to a full audit of the MD DoIT in 2020. Here is what they had to say about MD THINK: > "...for one project (MD THINK), the estimated $314.1 million cost of completion had increased by $141.1 million (81.5 percent)...an explanation was not provided for the increase..." And on the topic of security: > "We identified 55 firewall rules that allowed traffic from any source to 71 unique network destinations as well as a small network segment within DoIT’s internal network without IDPS coverage..." > "...access to personally identifiable information (PII) for State vendors stored in the State’s Financial Management Information System (FMIS) was not adequately restricted, and the PII was accessible to thousands of State employees." > src: https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport?fileId=5eac668aa1ce580f64a4317e https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport... ____What About ImmuNET?____ A question that begs an answer is why ImmuNet, the state's immunization registry, is not included under this "titanium umbrella of protection?" And while there is a ton of publicly available paperwork about the scope of MD THINK, including the RFP, Deloitte's response, and transcripts of hearings to explore the audit report's findings, ImmuNet is not mentioned. Not once. Please keep in mind - ImmuNet is not some podunk, state government hobby project. It is __law__ that Maryland retain vaccination records for its residents. School systems, employers, health providers, and the CDC all rely on Immunization Information Systems (IIS) for ground truth vaccination records. It is the public health equivalent of DMV. ____PASSION DISCLAIMER____ The lion's share of my time this year has been spent working with state and local COVID vaccination programs within upstate New York. I have seen firsthand how important accurate, timely test and vaccination data is to how local public health departments coordinate response. It was infuriating to see the MD DoIT downplay disruptions to COVID related data feeds. We are going into year three of a war that demands accurate, timely data to inform decision making. What the hell have you guys been doing for two weeks?! The country's public health departments are filled with the true unsung heroes of the pandemic. They're underpaid, exhausted, afraid, and constantly under attack - yet they're our last (and in many ways, only) line of defense. SO... To the three people that made it all the way to the end of this post - I ask just one thing: IF YOU EVER FIND YOURSELF INVOLVED IN A FUCKUP THAT IMPACTS THE DEPARTMENT OF PUBLIC HEALTH: - RULE 1: Apologize and fix it. Now. With the kindness, urgency, and empathy a mission controller would show a stranded astronaut--Proceed to RULE 6. - RULE 2: If the problem is not your fault--return to RULE 1. - RULE 3: If the problem isn't even real--return to RULE 1. - RULE 4: If they are blaming you for a problem they created on their own--return to RULE 1. - RULE 5: If the problem is related to Microsoft Access, FoxPro, or any other technology you swore an oath to avoid--return to RULE 1. - RULE 6: The final rule. Do not downplay the importance of data and tools public health resources rely upon. And NEVER, EVER respond to the problem by boasting about all the great technical achievements made everywhere BUT the underfunded, under-appreciated public health department.

briandrupieski 5y ago
Thank you for posting this. As a Maryland resident I found this really enlightening. I wasn't aware of this and I'm pretty curious now what's going on. It makes me wonder how or if I can help, as an ordinary citizen. I read through those procurement requests. The reasons they cite in the remarks are interesting. Sept 2021, Splunk: > The staff providing the Splunk support services resigned as of March 31, 2021, and the contractor has been unable to replace them. The Department requested these support services through another contract vehicle, but that contractor was unable to source qualified and competent candidates in the appropriate labor categories and with the rates provided in the contract. Dec 2021, Palo Alto Firewalls: > Although the Department has two contract vehicles available to staff this need, neither contractor has been successful in staffing the role. One contractor who once provided the services lost three of their four resources in the past 12 months. The other contractor has been unable to source a candidate with the requisite skills and experience in the appropriate labor categories and rates provided in the contract. So people are quitting, and they can't find replacements. Wow, that makes me think we're seeing effects from the "Great Resignation". I wonder what the "rates provided in the contract" are.

kyrra 5y ago

Looks like we hugged the server to death? Won't load for me or archive.is.

throaway46546 5y ago
I feel sorry for their admins. First they get hacked and then they get DDoS'd.
zone411 5y ago
Sorry, it was up when I posted this. Here is a shorter version (without a FAQ): https://health.maryland.gov/newsroom/Pages/Maryland-Department-of-Health-Update-on-Investigation-of-Network-Security-Incident-.aspx https://health.maryland.gov/newsroom/Pages/Maryland-Departme..., here is their dashboard showing no updates since Dec. 3: https://coronavirus.maryland.gov/ https://coronavirus.maryland.gov/, and here is a news story: https://www.youtube.com/watch?v=cLTaIW24fxU https://www.youtube.com/watch?v=cLTaIW24fxU.

Reichhardt 5y ago

lgrapenthin 5y ago
Or just looking at death numbers.
steve_mcdougall 5y ago
There is no evidence Omicron is milder than Delta. https://www.aljazeera.com/news/2021/12/18/no-evidence-omicron-milder-than-delta-british-researchers-say https://www.aljazeera.com/news/2021/12/18/no-evidence-omicro...
- XzetaU8 5y ago
  https://astralcodexten.substack.com/p/the-phrase-no-evidence-is-a-red-flag https://astralcodexten.substack.com/p/the-phrase-no-evidence...
  - jollybean 5y ago
    Here is the Oxford study [1] it's brutal. [1] https://www.imperial.ac.uk/news/232698/modelling-suggests-rapid-spread-omicron-england/ https://www.imperial.ac.uk/news/232698/modelling-suggests-ra...
    - walterbell 5y ago
      Past performance of UK models, https://data.spectator.co.uk/category/sage-scenarios https://data.spectator.co.uk/category/sage-scenarios
      - jollybean 5y ago
        It's not a model, it's an observation. There is no clear evidence as of yet to suggest hospitalization or mortality rates of Omicron among our kind of population. We'll have a better idea in a few weeks.

jpe90 5y ago

This is actually pretty bad, their mobile app wasn't updated to explain the situation and was showing 0 new cases for the timeframe. I personally know people who took that at face value.

throwaway599281 5y ago

Counting cases and advertising them every time the news shows start is useless, it just fuels the paranoia. This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever.

eru 5y ago
Not quite every other virus. We eliminated a few handful of diseases. (And of course, lots and lots of viruses were eliminated naturally. Just by natural selection and competition. There are even some hints that coronaviruses are replacing influenca.)
johnchristopher 5y ago
> This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever. This statement mean different things to different people. Do we live with AIDS with or without condoms ? Do we live with Covid with or without masks and vaccines and other prophylactic measures ?
- throwaway599281 5y ago
  - johnchristopher 5y ago
    I am not making myself clear enough: some use the statement "we have to live with covid" to mean "let covid cull the weak and the old, people die all the time anyway, vaccines and masks are only needed for the weak and the old so we shouldn't make them mandatory" and others to mean "covid is here to stay so are vaccines, masking and ventilation, etc.". > Have we ever imposed measures that limit the freedom of movement of the entire population, or subsects of it that have a higher infection rates per capita, such as drug users, prostitutes or homosexuals, as many countries have done? That's not the point. The point is that AIDS has changed some things that we haven't come back to yet. One of the most prominent one is condom usage. This change doesn't need to be proved/sanctified by laws to happen and yet it exists (but I'll add that some people have been sentenced to years in prison for not revealing their HIV/AIDS status when having sex with their partner). So: what does it mean to "live with covid" ?

vbphprubyjsgo 5y ago

covid isn't going to reach some new all time high infection rate because one monitoring system in one state was down for a few weeks (if that's the sentiment here. article doesn't load)

cyounkins 5y ago

Maryland Governor Larry Hogan tested positive today: https://twitter.com/GovLarryHogan/status/1472946270507397125 https://twitter.com/GovLarryHogan/status/1472946270507397125

subsaharancoder 5y ago
So did Elizabeth Warren, what's your point?
- cyounkins 5y ago
  I just wanted to reference additional news pertaining to covid in Maryland.

throwanem 5y ago

...until today: https://coronavirus.maryland.gov/ https://coronavirus.maryland.gov/ now has data through 12/19, with daily updates to resume tomorrow per an update on that same page.

Goety 5y ago

Digital ID/health seems dead in the water.