4 ms·

https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database

https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

jiggawatts 6y ago
Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer). - US date formats mixed in with non-US date formats. - Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files. - Random comments or annotations in numeric fields (preventing graphing the data). We don't all need to be data scientists, or machine learning wizards, or quantum computing gods. But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff. Put rows in rows. Don't mix random notes into columns you might want to sum or graph. Don't mix units. Have headers. Use a sane date format. That is all. </rant>
- deleted 6y ago
  [deleted]
- cblconfederate 6y ago
  lets assume that local centers people do save them in rows, which is the most logical thing to do. But probably the software where they copy/paste them requires the data in columns, so maybe they transposed the data without realizing that it was truncated to max size. Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.
- hef19898 6y ago
  Oh, I saw excel files that contained screen shots of other excel files over cells containing data. Comments where in text fields, if you were lucky, or again screen shots, from word. And then the same people wondered why things based on this file didn't work.
- RobertKerans 6y ago
  They seem to have hit the row limit, not the column limit. They were using Excel to collate the data coming in from different test centres: when one of the CSVs was loaded in, the row limit was exceeded and Excel cut off the extra rows without the operator realising (I'm going off reporting by Alex Hearn at The Guardian here, and he's normally pretty good)
mellosouls 6y ago
Actually from The Daily Mail, and not clear where they are getting it from.
yodelshady 6y ago
Worth noting the primary source for that claim is the Daily Mail.
sails 6y ago
A question, was it possibly a csv? ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership. (Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)
- gridlockd 6y ago
  The story I heard was that it was indeed a CSV and that Excel (silently) drops anything beyond 1 million rows.
  - shadowgovt 6y ago
    More specifically: they were detouring through the old XLS file format, which has a 65k row limit.

jiggawatts 6y ago

Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer). - US date formats mixed in with non-US date formats. - Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files. - Random comments or annotations in numeric fields (preventing graphing the data). We don't all need to be data scientists, or machine learning wizards, or quantum computing gods. But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff. Put rows in rows. Don't mix random notes into columns you might want to sum or graph. Don't mix units. Have headers. Use a sane date format. That is all. </rant>

deleted 6y ago
[deleted]
cblconfederate 6y ago
lets assume that local centers people do save them in rows, which is the most logical thing to do. But probably the software where they copy/paste them requires the data in columns, so maybe they transposed the data without realizing that it was truncated to max size. Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.
hef19898 6y ago
Oh, I saw excel files that contained screen shots of other excel files over cells containing data. Comments where in text fields, if you were lucky, or again screen shots, from word. And then the same people wondered why things based on this file didn't work.
RobertKerans 6y ago
They seem to have hit the row limit, not the column limit. They were using Excel to collate the data coming in from different test centres: when one of the CSVs was loaded in, the row limit was exceeded and Excel cut off the extra rows without the operator realising (I'm going off reporting by Alex Hearn at The Guardian here, and he's normally pretty good)

mellosouls 6y ago

Actually from The Daily Mail, and not clear where they are getting it from.

yodelshady 6y ago

Worth noting the primary source for that claim is the Daily Mail.

sails 6y ago

A question, was it possibly a csv? ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership. (Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)

gridlockd 6y ago
The story I heard was that it was indeed a CSV and that Excel (silently) drops anything beyond 1 million rows.
- shadowgovt 6y ago
  More specifically: they were detouring through the old XLS file format, which has a 65k row limit.