7 ms·

UK Covid-19 data underreported due to file size limit exceeded

134 points by slashdotdash 6y ago

slashdotdash 6y ago
“The issue occurred because some files containing positive test results exceeded the maximum file size that takes these data files and loads then into central systems, officials said.”
- mytailorisrich 6y ago
  Latest news report suggest that this is Excel spreadsheets' size limit.
  - spuz 6y ago
    Where did you see this?
    - benaadams 6y ago
      https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.
      - zelos 6y ago
        >The problem was caused by an Excel spreadsheet reaching its maximum file size, which stopped new names being added in an automated process. > The files have now been split into smaller multiple files to prevent the issue happening again. Wow
tus88 6y ago
Are they using msdos or something?
- tonyedgecombe 6y ago
  I don't think they are even using grey matter.
mellosouls 6y ago
Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-blame-game-16-000-Covid-cases-missed-Excel-glitch.html https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct, I wish people would stop referencing it like it's a proven fact, or provide an authoritative source for the Excel claim. I hate to be that guy, but...
- riffraff 6y ago
  thank you for posting this, I could not understand why everybody kept referencing Excel while TFA didn't. I thought people had somehow done some math on #cases/sheets to arrive at an obvious-but-not-to-me fact :)
- zelos 6y ago
  I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?
  - s_dev 6y ago
    A normalised correctly structured SQL DB?
    - zelos 6y ago
      I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.
      - s_dev 6y ago
        A normalised SQL DB will have zero redundancy. It would also easily handle these data set sizes. The only reason Excel was chosen was because clearly some clerk just felt more familiar with the tool and didn't have the experience to know better. The reality is though for storing this kind of relational data -- it's a solved problem -- SQL would have been the correct tool to employ.

134 points by slashdotdash 6y ago

“The issue occurred because some files containing positive test results exceeded the maximum file size that takes these data files and loads then into central systems, officials said.”

mytailorisrich 6y ago
Latest news report suggest that this is Excel spreadsheets' size limit.
- spuz 6y ago
  Where did you see this?
  - benaadams 6y ago
    https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.
    - zelos 6y ago
      >The problem was caused by an Excel spreadsheet reaching its maximum file size, which stopped new names being added in an automated process. > The files have now been split into smaller multiple files to prevent the issue happening again. Wow

tus88 6y ago

Are they using msdos or something?

tonyedgecombe 6y ago
I don't think they are even using grey matter.

mellosouls 6y ago

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-blame-game-16-000-Covid-cases-missed-Excel-glitch.html https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct, I wish people would stop referencing it like it's a proven fact, or provide an authoritative source for the Excel claim. I hate to be that guy, but...

riffraff 6y ago
thank you for posting this, I could not understand why everybody kept referencing Excel while TFA didn't. I thought people had somehow done some math on #cases/sheets to arrive at an obvious-but-not-to-me fact :)
zelos 6y ago
I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?
- s_dev 6y ago
  A normalised correctly structured SQL DB?
  - zelos 6y ago
    I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.
    - s_dev 6y ago
      A normalised SQL DB will have zero redundancy. It would also easily handle these data set sizes. The only reason Excel was chosen was because clearly some clerk just felt more familiar with the tool and didn't have the experience to know better. The reality is though for storing this kind of relational data -- it's a solved problem -- SQL would have been the correct tool to employ.