10 ms·

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database?

• mytailorisrich 6y ago
  To be fair, there hasn't been any massive irretrievable data loss yet, so not too bad. https://www.youtube.com/watch?v=K_FrQnQv0Vw https://www.youtube.com/watch?v=K_FrQnQv0Vw (probably an accurate depiction of what's going on in Whitehall right now, though)
  • rgallagher27 6y ago
    At this point I think I'd rather have Peter Capaldi acting as Malcolm Tucker running Whitehall over the shower of incompetent people we have now. At least he might get something done in between the flaying
    • iso1631 6y ago
      I'd rather have David Brent, but we are where we are
• londons_explore 6y ago
  People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file. No database has the flexibility and flat learning curve that excel has. Until a database manages to meet these goals, it remains a good tool for certain usecases. We as engineers should be working towards making a real database thats just as easy to use as excel, and only then can we complain that people are using excel sheets for this kind of stuff.
  • chii 6y ago
    I'm surprised that MS Access didn't take this kind of niche. I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...
    • Closi 6y ago
      It depends which bits of functionality you want, but I think you are close to describing PowerQuery (Get & Transform) which is in Excel, which can ingest multiple data sources into effectively a relational database with hard datatypes and data-cleaning. In PowerQuery & PowerPivot there are also no practical limits on the number of rows of data you can have (other than the impact on processing time, clearly hundreds of millions of records might start to be a problem). It's not quite access - it doesn't actually persist/store the data, just aggregates it from other sources which is probably what is required here. Access is a little legacy and would have it's own issues. I think in reality it depends on how you are ingesting data (which is probably hospitals submitting excel templates or similar?). Maybe these could be automatically aggregated and put onto an analytics platform? (e.g. PowerBi). Hospitals could maybe report it on a portal, but I'm not convinced that the leadership would have wanted to change the established process (because of operational focus, change reduction & risk).
    • thu2111 6y ago
      For a very long time, it did. Weaknesses of Access in no particular order: very poor multi-user/multi-device-access features. No real concept of servers. Buggy, had a nasty habit of corrupting its custom file format causing data loss (Excel never does this). Does expect you to understand SQL, data normalisation, etc. VBA just about serviceable for a spreadsheet but not good enough for a 'real' database-backed app. Probably someone could try and make a better Access. There are surely lots of startups doing that already, along with hosting. All the "no code" tools that are out there, etc. Access had the advantage of coming with Office so lots of people had it already, whereas today's subscription based services that host your data have incremental cost.
      • 6y ago
• DanBC 6y ago
  I agree, but I'm genuinely glad they're not[1] using fax machines. [1] they almost certainly are using fax somewhere.
  • brigandish 6y ago
    It's probably Japan but that would be the least of the egregiousness. [1] Japan's COVID-19 Reports - 140KBs of Unadulterated Incompetence https://stdio.sangwhan.com/wtf-japan-covid-19-report/ https://stdio.sangwhan.com/wtf-japan-covid-19-report/ [2] The previous HN discussion https://news.ycombinator.com/item?id=22728674 https://news.ycombinator.com/item?id=22728674
    • poooogles 6y ago
      I get a cert warning on https://stdio.sangwhan.com/wtf-japan-covid-19-report/ https://stdio.sangwhan.com/wtf-japan-covid-19-report/ which really is a good start.
• avian 6y ago
  > Using an Excel sheet as a database How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".
  • benaadams 6y ago
    https://twitter.com/MaxCRoser/status/1313046638915706880 https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.
    • jiggawatts 6y ago
      Columns!? You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.) Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow: - Transposed data (new entries in columns) - Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer). - US date formats mixed in with non-US date formats. - Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files. - Random comments or annotations in numeric fields (preventing graphing the data). We don't all need to be data scientists, or machine learning wizards, or quantum computing gods. But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff. Put rows in rows. Don't mix random notes into columns you might want to sum or graph. Don't mix units. Have headers. Use a sane date format. That is all. </rant>
      • deleted 6y ago
        [deleted]
      • cblconfederate 6y ago
        lets assume that local centers people do save them in rows, which is the most logical thing to do. But probably the software where they copy/paste them requires the data in columns, so maybe they transposed the data without realizing that it was truncated to max size. Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.
• chimprich 6y ago
  > It's unlikely this will have had much impact since most reported cases were only a few days late I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error. This is safety-critical software. It should be stress tested and have sanity checks. It should be impossible for this to happen. For something like this system, part of a track and trace system costing billions, to fail because of a file size error is just unbelievable.
  • throwmeaway_pls 6y ago
    The WHO estimated 750 million cumulative infections on Friday. This error is not unbelievable because the system is just a bit of political theater. There’s nothing more we can do against an endemic human coronavirus. It’s time to move on.
    • hdjdbtbgwjsn 6y ago
      Yeah we should give up on controlling polio, measles etc as well. After all polio only kills 0.5% of cases as well. </sarcasm>
  • Mvandenbergh 6y ago
    I was being serious. Distributionally, most of the cases were 1-2 days before the error was spotted. To be clear though, when I said "not much impact", I didn't mean that people might not die because of it, only that it wouldn't affect the large scale functioning (such as it is) of the tracing operation.
    • fuzzy2 6y ago
      > Distributionally, most of the cases were 1-2 days before the error was spotted. That’s always true though if the virus is spreading exponentially (dunno whether it currently does in the UK).
      • 6y ago
• lini 6y ago
  Excel has a limit of ~1M rows per sheet. Total cases in the UK until now are ~500K. So it seems they use 2 rows per case in the sheet. Edit: oops, it seems they used columns for the cases and the ~16K column limit was hit when daily cases exceeded that.
• ajsnigrutin 6y ago
  Once you have a couple of thousands of rows in excell files, one file per day, possibly one per testing center.... possibly with inconsistent structure.... non standard date formats (even different in each row),... add to this the inconsistent namings ("Tested at" column with data: "Testing center 1, London", "TC1, London", "TC1 London", "TC1 LONDON",....), etc... Yes, anything-sql or whatever would be better, but deciding between putting all the data into sql by hand, or "uhm... i'll think of something and do it tomorrow", me, being lazy, I'd pick the second option,... ...until i'd get shamed on here. Maybe even a few days after.
  • hdjdbtbgwjsn 6y ago
    Cleaning data is a major part of data engineering there are plenty of professionals capable of helping with this task.
• learnstats2 6y ago
  It is absolutely standard that everyone doing office work outside of high-tech jobs uses Excel for data handling. It's not surprising that this reaches into government. Even if your canonical database is done properly, there will be a clash here as soon as you share (or give anyone the ability to generate) CSV files that are too large for Excel.
• makomk 6y ago
  Hey, at least the UK has automated handling of data! Parts of Germany managed to not tell people they'd tested positive for a week or two because their manual, Excel-based process for handling test results couldn't cope with the increased load from testing people returning to the country - and they're meant to be the competent ones we look up to, according to our press. (I get the impression that German experts have a rather different view.)
• FerretFred 6y ago
  > professionalism in data handling Nope. They appointed Dido Harding to cough manage it, and Matt Hancock oversees it. That should tell you all you need to know. I wonder if they were running Excel 2007?
• 231435359236 6y ago
  I think it's more interesting on what happens to any analysis done with this incorrect data. There was a news article last week suggesting that the growth in COVID-19 cases was "levelling off" and I wonder in retrospect would this incorrect data account for such a position. The article in question: https://www.bbc.co.uk/news/health-54387057 https://www.bbc.co.uk/news/health-54387057
• azalemeth 6y ago
  Excel always feels like a bit of a curse of the world for me, and one that makes me feel like I'm going mad. I work partly in partnership with the NHS within my university, and get to see it being applied, completely inappropriately, with no ability to comment. For me, I always feel like I'm living in some sort of parallel universe. Imagine living in a world where _you_, and a select group of others, know that there are better methods to drive a screw into a wall than punching it with your fist. However, 99% of people in the world punch screws into walls by hand, because they know how to use the only tool they have to hand (i.e. their hand). Their friends and neighbours do too. Their colleagues show them neat little tricks about how to fold your hand better to get the screw in with less pain, or more efficiently. The "proper" option -- drilling a pilot hole, using a self-tapping screw, understanding the difference between nails, bolts and screws, etc, or the concept of wall-plugs, is "too technical", "too complex", or requires "a large amount of infrastructure to fix a simple problem" [buying a drill & screwdriver, etc]. Sometimes people buy new walls with screws pre-installed because the thought of driving in those screws seems like an overwhelming technical burden. You, and a group of similarly-minded nerds, understand just how stupid this is at times. You personally then spend the rest of your life watching people occasionally have horrific hand injuries from punching screws into walls and refusing the offer of screwdrivers. Others profit from selling vertical hard surfaces with a whole array of convenient screws pre-installed; or get employed teaching others the _very_ basics of screw-driving, or sell products containing all sorts of klunky hacks to cope with the fact that screws manually installed are at odd angles and tend to fall over randomly. The supplier of a popular brand of screwdrivers happens to be the world's largest screw manufacturer, but never seems to promote screwdrivers to people who currently punch them in with their fists. Instead, they sell them a subscription to screws, with the option for a very expensive pre-screwed wall-delivery service, built to your specification (that they call Dynamics for some reason…). You, and a subset of your nerdy and educated friends recognise just how batshit insane the widespread use of punching-as-a-method-to-install-screws is, and occasionally laugh about it, mostly to distract yourself from the grim reality of buildings badly held together by punched-in-screws. Every so often a skyscraper-sized building collapses, [1] and it's realised in retrospect that the manually-driven-in-screws were not the right tool for the job. Still, people carry on banging in screws with their fists, and you increasingly realise that you have to -- somehow -- accept that in order to sleep at night without going insane. [1] https://www.businessinsider.com/excel-partly-to-blame-for-trading-loss-2013-2 https://www.businessinsider.com/excel-partly-to-blame-for-tr...
• gridlockd 6y ago
  > Using an Excel sheet as a database? You wouldn't (want to) believe how incredibly common that is.