4 ms·

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.u

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-blame-game-16-000-Covid-cases-missed-Excel-glitch.html https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct, I wish people would stop referencing it like it's a proven fact, or provide an authoritative source for the Excel claim. I hate to be that guy, but...

riffraff 6y ago
thank you for posting this, I could not understand why everybody kept referencing Excel while TFA didn't. I thought people had somehow done some math on #cases/sheets to arrive at an obvious-but-not-to-me fact :)
zelos 6y ago
I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?
- s_dev 6y ago
  A normalised correctly structured SQL DB?
  - zelos 6y ago
    I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.
    - s_dev 6y ago
      A normalised SQL DB will have zero redundancy. It would also easily handle these data set sizes. The only reason Excel was chosen was because clearly some clerk just felt more familiar with the tool and didn't have the experience to know better. The reality is though for storing this kind of relational data -- it's a solved problem -- SQL would have been the correct tool to employ.
- josephpmay 6y ago
  We’ve actually had a similar issue in the US with some Electronic Initial Case Report documents (to be clear, we mostly rely on Electronic Lab Results-ELR rather than eICR in the US to count COVID numbers, so this hasn’t resulted in an under-count) Rather than be a problem with the database having arbitrary file size limits, it actually was a problem with file size limits within network intermediaries. In most cases, the issue is less the limit, and more that the files themselves were too large because of how they were created. I don’t think this is the same issue as the UK (I don’t think they are using the eICR standard there yet), but it’s an example of how you could have a problem with file size limitations without storing data in Excel.
- bArray 6y ago
  A database running from a FAT disk partition for example.
- netsharc 6y ago
  It's probably more the web server configured to only allow x MB maximum for file uploads. Or since the government has been shown to be incompetent, maybe it's the maximum size for email attachments..
- bar8uyr 6y ago
  Originally I presumed this was an error importing Excel files and an upload file size limit..
- deleted 6y ago
  [deleted]
JonoBB 6y ago
Apparently each case was recorded in a column, and the number of columns had reached the maximum (16,384). https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880
- arethuza 6y ago
  The document referenced in that tweet doesn't seem to mention columns?
zelos 6y ago
Seems like other (slightly) more reputable sources are now reporting that it's Excel: https://www.standard.co.uk/news/uk/covid-testing-technical-issue-excel-spreadsheet-a4563616.html https://www.standard.co.uk/news/uk/covid-testing-technical-i... A Telegraph article mentions a Press Association report.
mellosouls 6y ago
BBC say they have 'confirmed' it's Excel, though still a bit mysterious on how they've done that: https://www.bbc.co.uk/news/uk-54422505 https://www.bbc.co.uk/news/uk-54422505 Particularly, they reckon they were using the older XLS rather than XLSX format with the lower 65000 row limit - although that seems more like a deflection from the fact functionality limits weren't properly handled, tested or designed for.
- dang 6y ago
  That article seems both more informative and more up-to-date, so we'll merge most of the comments from this thread into https://news.ycombinator.com/item?id=24689247 https://news.ycombinator.com/item?id=24689247. Thanks!

riffraff 6y ago

thank you for posting this, I could not understand why everybody kept referencing Excel while TFA didn't. I thought people had somehow done some math on #cases/sheets to arrive at an obvious-but-not-to-me fact :)

zelos 6y ago

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

s_dev 6y ago
A normalised correctly structured SQL DB?
- zelos 6y ago
  I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.
  - s_dev 6y ago
    A normalised SQL DB will have zero redundancy. It would also easily handle these data set sizes. The only reason Excel was chosen was because clearly some clerk just felt more familiar with the tool and didn't have the experience to know better. The reality is though for storing this kind of relational data -- it's a solved problem -- SQL would have been the correct tool to employ.
josephpmay 6y ago
We’ve actually had a similar issue in the US with some Electronic Initial Case Report documents (to be clear, we mostly rely on Electronic Lab Results-ELR rather than eICR in the US to count COVID numbers, so this hasn’t resulted in an under-count) Rather than be a problem with the database having arbitrary file size limits, it actually was a problem with file size limits within network intermediaries. In most cases, the issue is less the limit, and more that the files themselves were too large because of how they were created. I don’t think this is the same issue as the UK (I don’t think they are using the eICR standard there yet), but it’s an example of how you could have a problem with file size limitations without storing data in Excel.
bArray 6y ago
A database running from a FAT disk partition for example.
netsharc 6y ago
It's probably more the web server configured to only allow x MB maximum for file uploads. Or since the government has been shown to be incompetent, maybe it's the maximum size for email attachments..
bar8uyr 6y ago
Originally I presumed this was an error importing Excel files and an upload file size limit..
deleted 6y ago
[deleted]

JonoBB 6y ago

Apparently each case was recorded in a column, and the number of columns had reached the maximum (16,384). https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880

arethuza 6y ago
The document referenced in that tweet doesn't seem to mention columns?

zelos 6y ago

Seems like other (slightly) more reputable sources are now reporting that it's Excel: https://www.standard.co.uk/news/uk/covid-testing-technical-issue-excel-spreadsheet-a4563616.html https://www.standard.co.uk/news/uk/covid-testing-technical-i... A Telegraph article mentions a Press Association report.

mellosouls 6y ago

BBC say they have 'confirmed' it's Excel, though still a bit mysterious on how they've done that: https://www.bbc.co.uk/news/uk-54422505 https://www.bbc.co.uk/news/uk-54422505 Particularly, they reckon they were using the older XLS rather than XLSX format with the lower 65000 row limit - although that seems more like a deflection from the fact functionality limits weren't properly handled, tested or designed for.

dang 6y ago
That article seems both more informative and more up-to-date, so we'll merge most of the comments from this thread into https://news.ycombinator.com/item?id=24689247 https://news.ycombinator.com/item?id=24689247. Thanks!