6 ms·

While I am not in the habit of defending Microsoft, I think it should be pointed out that it wasn't using "Excel" that was the biggest problem, but using "13-ye

While I am not in the habit of defending Microsoft, I think it should be pointed out that it wasn't using "Excel" that was the biggest problem, but using "13-years obsolete format for Excel". Not that there was any reason to be using Excel for this, but it's a bit unfair to blame Microsoft for something they never claimed XLS could do, and provided a much better capacity in XLSX 13 years ago. Again, it's hard to cry too many tears for Microsoft, but it does seem a bit off-target to blame "Excel" for this...

ComodoHacker 6y ago
You get CSV files as input and have to upload them into the database. You don't need Excel at all, neither in XSL, nor in XSLX format. My bet is the biggest problem here is subcontracting this work to the lowest bidder, presumably from some developing country.
- baud147258 6y ago
  > subcontracting this work to the lowest bidder knowing a little how things (don't) work in the UK, it's likely subcontracted, but to a company belonging to a mate of the director in charge of the whole thing
- yawaramin 6y ago
  I'd take that bet ;-) From the article: > > And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.
kelnos 6y ago
Sure, and I think at least the headline ("ill-thought-out use of Excel") avoids blaming MS or Excel itself. Ultimately tools are built for particular things, and if you choose to use a tool for something it's not built for, and it breaks catastrophically, that's on you.
- sriram_malhar 6y ago
  No. Excel is definitely to blame. A simple error "cannot import/export more than 65k rows" would have done the trick. Not silently throw away rows.
  - corty 6y ago
    The premise of excel is very much "don't bug me, just do it". The highly praised UI simplifies away corner cases and errors intentionally. Summing over an empty cell? Most probably zero, why throw an error and confuse the user? The simplified UI makes correctness hard to impossible, so excel is just unsuitable for tasks where correctness is desired.
    - account42 6y ago
      Interpreting an empty cell as 0 is making an (often correct) assumption about what the user wanted. Throwing away data without warning is almost certainly never what the user wanted.
bobbylarrybobby 6y ago
The issue with is excel is that its failure mode is suboptimal. Programming languages will generally crash at runtime when an unrecoverable error occurs (index out of bounds, memory allocation failed, null pointer, etc). But excel just throws away your data?
- tomc1985 6y ago
  Indeed, you have to manually inspect results to verify correctness. If you're lucky maybe your error will result in one of those weird error values being shown instead, but way too often that is not the case Not to mention that complex formulae are still usually expressed as a bunch of gobbledygook in the cell value textbox, which is about as easy to parse as minified Javascript. And that's to technical users like ourselves.
  - logicOnly 6y ago
    I had Excel error out when I tried to open 10 million rows.
    - sueders101 6y ago
      Excel tops out a little more over a million rows without specifically handling of larger files. I’m not sure excusing it, just a heads up to all the junior analysts out there who haven’t dealt with the problem(not implying you’re a junior analyst, just wanted to point something out to those who aren’t familiar with the issue).
      - logicOnly 6y ago
        No worries, Excel is nice for small/medium sized data and has the UI/Visualization which is nice. The story I mentioned was because I wanted to look at the data before I started parsing it. I had full expectations to use either sqlite or Pandas.
- curiousllama 6y ago
  If it fails, it’s often silent. If not silent, hidden. If not hidden, misleading, if not misleading, opaque. If not opaque, trivial. Excel is a wonderfully powerful tool that’s very bad at handling errors clearly.
nabilhat 6y ago
Over-13-years old Excel already opened delimited flat files natively, and it threw warnings when you opened a .csv file that exceeded the worksheet row limit. Someone in the distant past was just clever enough to hack together the script (with no error handling) without being curious enough to know it wasn't necessary in the first place.
yawaramin 6y ago
Nobody is blaming Microsoft. From the article: > And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.
- tonyedgecombe 6y ago
  Blaming PHE seems to have become a regular occurance this year. I wonder if it's justified or not.
mcfedr 6y ago
If they had used the newer format, we'd be talking about this in a few weeks time when the larger limit was reached. This is a problem of bad coding, and using the wrong tool for the job. A defensive coding practice would have prevented this from going unseen. Using a database to store data would have prevented such arbitrary limits.
- rossdavidh 6y ago
  Yeah, something like "Missing Covid-19 test data was caused by using spreadsheet instead of database", would be more accurate.

ComodoHacker 6y ago

You get CSV files as input and have to upload them into the database. You don't need Excel at all, neither in XSL, nor in XSLX format. My bet is the biggest problem here is subcontracting this work to the lowest bidder, presumably from some developing country.

baud147258 6y ago
> subcontracting this work to the lowest bidder knowing a little how things (don't) work in the UK, it's likely subcontracted, but to a company belonging to a mate of the director in charge of the whole thing
yawaramin 6y ago
I'd take that bet ;-) From the article: > > And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.

kelnos 6y ago

Sure, and I think at least the headline ("ill-thought-out use of Excel") avoids blaming MS or Excel itself. Ultimately tools are built for particular things, and if you choose to use a tool for something it's not built for, and it breaks catastrophically, that's on you.

sriram_malhar 6y ago
No. Excel is definitely to blame. A simple error "cannot import/export more than 65k rows" would have done the trick. Not silently throw away rows.
- corty 6y ago
  The premise of excel is very much "don't bug me, just do it". The highly praised UI simplifies away corner cases and errors intentionally. Summing over an empty cell? Most probably zero, why throw an error and confuse the user? The simplified UI makes correctness hard to impossible, so excel is just unsuitable for tasks where correctness is desired.
  - account42 6y ago
    Interpreting an empty cell as 0 is making an (often correct) assumption about what the user wanted. Throwing away data without warning is almost certainly never what the user wanted.

bobbylarrybobby 6y ago

The issue with is excel is that its failure mode is suboptimal. Programming languages will generally crash at runtime when an unrecoverable error occurs (index out of bounds, memory allocation failed, null pointer, etc). But excel just throws away your data?

tomc1985 6y ago
Indeed, you have to manually inspect results to verify correctness. If you're lucky maybe your error will result in one of those weird error values being shown instead, but way too often that is not the case Not to mention that complex formulae are still usually expressed as a bunch of gobbledygook in the cell value textbox, which is about as easy to parse as minified Javascript. And that's to technical users like ourselves.
- logicOnly 6y ago
  I had Excel error out when I tried to open 10 million rows.
  - sueders101 6y ago
    Excel tops out a little more over a million rows without specifically handling of larger files. I’m not sure excusing it, just a heads up to all the junior analysts out there who haven’t dealt with the problem(not implying you’re a junior analyst, just wanted to point something out to those who aren’t familiar with the issue).
    - logicOnly 6y ago
      No worries, Excel is nice for small/medium sized data and has the UI/Visualization which is nice. The story I mentioned was because I wanted to look at the data before I started parsing it. I had full expectations to use either sqlite or Pandas.
curiousllama 6y ago
If it fails, it’s often silent. If not silent, hidden. If not hidden, misleading, if not misleading, opaque. If not opaque, trivial. Excel is a wonderfully powerful tool that’s very bad at handling errors clearly.

nabilhat 6y ago

Over-13-years old Excel already opened delimited flat files natively, and it threw warnings when you opened a .csv file that exceeded the worksheet row limit. Someone in the distant past was just clever enough to hack together the script (with no error handling) without being curious enough to know it wasn't necessary in the first place.

yawaramin 6y ago

Nobody is blaming Microsoft. From the article: > And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.

tonyedgecombe 6y ago
Blaming PHE seems to have become a regular occurance this year. I wonder if it's justified or not.

mcfedr 6y ago

If they had used the newer format, we'd be talking about this in a few weeks time when the larger limit was reached. This is a problem of bad coding, and using the wrong tool for the job. A defensive coding practice would have prevented this from going unseen. Using a database to store data would have prevented such arbitrary limits.

rossdavidh 6y ago
Yeah, something like "Missing Covid-19 test data was caused by using spreadsheet instead of database", would be more accurate.