6 ms·

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical an

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

pintxo 6y ago
Have you every worked in a large enterprise? I have personally tested an Excel based credit rating tool, to be rolled-out globally by a major financial institution. The fact that one should not do this, is by no means a reason not to do it.
- sails 6y ago
  Would be interested in hearing more about that Excel tool.
  - akho 6y ago
    Come work at a bank. Any bank.
    - strangeattractr 6y ago
      Can attest to this. Some of the things I've seen excel spreadsheets contorted to do horrify me. I once got spreadsheet dumped on me to debug because it wasn't working. A colleague used the spreadsheet to 'generate' interest rates that were then input into a mainframe. Dug into the VBA spaghetti mess, turns out this 15 year old script that took 15 minutes to run originally hit a bunch of Oracle/External APIs and performed calculations was now just copying the rates in from a csv file on a shared drive. The error was caused by an excel formula ticking over a new year causing it to look for a directory it did not need to access that didn't exist. I thought it was pretty funny until I heard that the entire asset finance business had been unable to write any loans for days because of this.
  - pintxo 6y ago
    It started very innocent: Some consulting company apparently was tasked to create a prototype of the credit rating algorithm. Quite appropriately, I would say, they did so in Excel. As you can easily tweak the logic, show input, intermediate results and output. And all can be manipulated by the client who does not need to know a thing about programming to do so. Great. From there it went downhill: Apparently someone up the hierarchy thought "Wow, that's 80% of what we need, let's just add a little UI and ship it" They added a UI, but not as a VBA UI as you might hope. No, they did the whole UI in different worksheets. Storing any intermediate data ... also on worksheets. Long story short, it was a mess, and a slow one for that. Oh, did I say this was a multi-lingual application? Production rollout was on Jan 2nd, Dec 31st around 4pm I found a bug in the other language, on the one machine which had the this language configured. I believe this was the only time I ever saw a programmer literally run down the office to that machine to debug this issue.
- nicoburns 6y ago
  I once created a very modern mobile offline-capable web app for a large enterprise company that was powered by APIs backed by excel. Luckily we didn't have to deal with those directly, but it was a bit of a pain for the poor guy doing th backend.
- nullsense 6y ago
  Fun fact: Toshiba has software that generates entire CRUD app skeletons driven 100% by excel spreadsheets. You fill out object definitions in excel spreadsheets with property names and data types etc. It's like an extremely uncool long lost cousin of rails -g Source: Got one of their engineers to show me after I heard about it and had to know if it was real. Anyone who's ever worked with or in Japan knows just how far they are willing to torture Excel spreadsheets to get them to do anything and everything.
  - chasd00 6y ago
    to play devil's advocate, i had a desktop support job in college and worked with an executive assistance. She was _very_ attractive and so i was at her desk "working on the printer" a lot, she knew Excel better than anyone i've ever seen (before or since). It was like watching a magician. In the right hands Excel is pretty amazing.
  - pintxo 6y ago
    I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is. It was one of my nicest testing gigs ever. A test session would take only minutes, all results documented to the t. And fiddling around with the test data was so easy. Would be interesting to know if this thing is still in use. Not sure the customer liked it that much, I regression tested the 6 previous - still running - versions of the service, something no one had cared to do for years. We found bugs both in the Spec and in the Code for nearly all old versions...
    - nullsense 6y ago
      >I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is. That's an upgrade on how Toshiba did testing. They did basically the same thing except it was Excel+Humans. No joke. Never saw that one with my own eyes but a coworker did. He also said they had doctors there and each doctor was responsible for X number of staff and every so often made them fill out a questionnaire of which the final question was "have you thought about killing yourself lately?" and if you answered yes you got a day off. Apparently people would ask for transfers to different offices where the doctor responsible for them wasn't physically present, so they could avoid even being asked the question for fear of being made to take time off work. One of the more senior guys on the project kinda just disappeared too. Fun times.
SkyBelow 6y ago
Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.
- TeMPOraL 6y ago
  Yup. What also helps is that the data literally sits on your drive, or your network drive. No SaaS vendor is holding it hostage.
  - helij 6y ago
    It doesn't feel like that. Even if I save the file (365 subscription and desktop apps) on my disk it steel feels 'cloudy'.
- codeulike 6y ago
  Yes exactly. An endless battle: IT want to stamp out spreadsheets and centralise everything but it all becomes an ossified bottleneck of development projects. Meanwhile people need to get stuff done so they use Excel.
indy 6y ago
I think you'd be shocked by how many of our financial and social institutions rely on emailing Excel spreadsheets to one another.
- cscharenberg 6y ago
  Don't forget healthcare! Excel and CSV for tons of data exchange.
  - sk5t 6y ago
    CSV is somewhat okay-ish. Excel, not so much (thanks Excel for discarding the important leading zeros in this field, and for lossily parsing the other as a date... oh, this Excel file is so big it got the "XLSB" treatment which is hard to undo)
  - chasd00 6y ago
    I think the healthcare industry is probably the only thing keeping fax machines alive. Interestingly, fax machine are tagged as HIPPA compliant. Lots of spreadsheets get printed and then fed to the fax machine for transmission instead of email/whatever. On the other side, and i'm being serious, someone gets the fax and creates a new spreadsheet and enters all the rows manually. fun fun.
- colinmhayes 6y ago
  I am absolutely not shocked by that. Most people do not know how to insert into a db yet alone spin one up and add tables. Everyone can excel.
WrtCdEvrydy 6y ago
Alternatively... "We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"
- zarq 6y ago
  "We should maybe start tracking some of this" "Where does the data come from?" "Dunno, a handful of hospitals... maybe?" "Right. Let's just use Excel, not gonna waste time on that now" ... six months later ... "Hey um..."
  - WrtCdEvrydy 6y ago
    From the species that brought you "The Game of Life" comes "The Game of Technical Debt"
- chasd00 6y ago
  >"We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it" you have no idea how right you are :)
  - WrtCdEvrydy 6y ago
    Been there, done that, got the sqlite t-shirt... Wasn't a bad choice, but once you load 20TB into it, you realize you done goofed.
libraryatnight 6y ago
In my org excel might have been the start, the intent being to move over to a DB once someone got it stood up. Then nobody would stand up the DB - because not urgent, we have the excel for now. The DB would be completely forgotten, and 3 years down the road everybody would be wondering wtf we have this huge excel file for. Then we would have six weeks of meetings to discuss the file until something else comes up and we forget it for another 3 years.
- TeMPOraL 6y ago
  That's why Excel wins. Because it's right there, infinitely adaptable to any change in workflow you may need to make. No need to send official requests through half a dozen layers of management to get the IT people to maybe, in half a year, add a column to the database. No need to involve 20 people for a year to create new software for a new workflow. You just agree with your co-workers on a change, swap a column, add some color, File->Save, done. It wins because it puts power in the hands of the people doing the work.
  - fivre 6y ago
    I mean, yeah, but pretend Excel is, idk, sword arms. You get a lot of work done by whirling around the office, spinning your swords, cutting off a variety of limbs and heads in the process. You get to the end, meet the few survivors, staring dismayed at the trail of blood and dismemberment behind you, and you yell, "BUT I GOT THE JOB DONE!" You're not wrong, but... I wish there were some alternative where we instead fixed our ossified, byzantine processes gradually over time, so that we didn't need to break out the sword arm tornado in the name of getting work done and then, in hindsight, say, "well, shame about the negative externalities, but there really just was no other way. Ah well, let's move on, everyone who isn't a pile of flesh and blood and bits on the floor tidy up. Gotta get everything clean for the next sword arm tornado."
    - TeMPOraL 6y ago
      But the negative externalities don't seem all that big, considering. For every problem like this, you get countless millions of person-hours and dollars saved in work that wouldn't have happened at all, or would have been delayed, because of the necessity and cost of creating complex IT systems to support it. If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant about sharp objects. The alternative then would be a central processing factory that takes several years and millions of dollars to built, and which has to be turned off for a month several times a year, to change the shape of the blades used by the automated cutters.
shadowgovt 6y ago
The common case for stories like this is the answer is "nobody." Time point 1: "We need a way to map the long-form epidemiology state. It's 2D data... Let's use Excel." "Will we hit limits?" "Pfft, no. Not unless we end up needing to track tens of thousands of patients, and that's not practical; we don't have enough people to do that tracking in the whole NHS." Time point 2: "This pandemic is an URGENT problem. We need to hire more people than ever before to do tracing. And be sure to keep the epidemeology state mapper updated so it can drive the summary dashboards!"
addicted 6y ago
Except it wasn’t the government. They outsourced the test and trace system to private companies.
- callamdelaney 6y ago
  Public Health England was responsible for collating the data which came in the form of CSV's - PHE is a government agency.
dgudkov 6y ago
Of course Excel. What else non-technical users can use? They simply don't have any other data tool.
manigandham 6y ago
Excel is perfectly fit for that and is nowhere near the actual issue. It's lightweight, fast, portable, compatible, and just about everyone can use it regardless of (technical) skill.

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

pintxo 6y ago

Have you every worked in a large enterprise? I have personally tested an Excel based credit rating tool, to be rolled-out globally by a major financial institution. The fact that one should not do this, is by no means a reason not to do it.

sails 6y ago
Would be interested in hearing more about that Excel tool.
- akho 6y ago
  Come work at a bank. Any bank.
  - strangeattractr 6y ago
    Can attest to this. Some of the things I've seen excel spreadsheets contorted to do horrify me. I once got spreadsheet dumped on me to debug because it wasn't working. A colleague used the spreadsheet to 'generate' interest rates that were then input into a mainframe. Dug into the VBA spaghetti mess, turns out this 15 year old script that took 15 minutes to run originally hit a bunch of Oracle/External APIs and performed calculations was now just copying the rates in from a csv file on a shared drive. The error was caused by an excel formula ticking over a new year causing it to look for a directory it did not need to access that didn't exist. I thought it was pretty funny until I heard that the entire asset finance business had been unable to write any loans for days because of this.
- pintxo 6y ago
  It started very innocent: Some consulting company apparently was tasked to create a prototype of the credit rating algorithm. Quite appropriately, I would say, they did so in Excel. As you can easily tweak the logic, show input, intermediate results and output. And all can be manipulated by the client who does not need to know a thing about programming to do so. Great. From there it went downhill: Apparently someone up the hierarchy thought "Wow, that's 80% of what we need, let's just add a little UI and ship it" They added a UI, but not as a VBA UI as you might hope. No, they did the whole UI in different worksheets. Storing any intermediate data ... also on worksheets. Long story short, it was a mess, and a slow one for that. Oh, did I say this was a multi-lingual application? Production rollout was on Jan 2nd, Dec 31st around 4pm I found a bug in the other language, on the one machine which had the this language configured. I believe this was the only time I ever saw a programmer literally run down the office to that machine to debug this issue.
nicoburns 6y ago
I once created a very modern mobile offline-capable web app for a large enterprise company that was powered by APIs backed by excel. Luckily we didn't have to deal with those directly, but it was a bit of a pain for the poor guy doing th backend.
nullsense 6y ago
Fun fact: Toshiba has software that generates entire CRUD app skeletons driven 100% by excel spreadsheets. You fill out object definitions in excel spreadsheets with property names and data types etc. It's like an extremely uncool long lost cousin of rails -g Source: Got one of their engineers to show me after I heard about it and had to know if it was real. Anyone who's ever worked with or in Japan knows just how far they are willing to torture Excel spreadsheets to get them to do anything and everything.
- chasd00 6y ago
  to play devil's advocate, i had a desktop support job in college and worked with an executive assistance. She was _very_ attractive and so i was at her desk "working on the printer" a lot, she knew Excel better than anyone i've ever seen (before or since). It was like watching a magician. In the right hands Excel is pretty amazing.
- pintxo 6y ago
  I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is. It was one of my nicest testing gigs ever. A test session would take only minutes, all results documented to the t. And fiddling around with the test data was so easy. Would be interesting to know if this thing is still in use. Not sure the customer liked it that much, I regression tested the 6 previous - still running - versions of the service, something no one had cared to do for years. We found bugs both in the Spec and in the Code for nearly all old versions...
  - nullsense 6y ago
    >I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is. That's an upgrade on how Toshiba did testing. They did basically the same thing except it was Excel+Humans. No joke. Never saw that one with my own eyes but a coworker did. He also said they had doctors there and each doctor was responsible for X number of staff and every so often made them fill out a questionnaire of which the final question was "have you thought about killing yourself lately?" and if you answered yes you got a day off. Apparently people would ask for transfers to different offices where the doctor responsible for them wasn't physically present, so they could avoid even being asked the question for fear of being made to take time off work. One of the more senior guys on the project kinda just disappeared too. Fun times.

SkyBelow 6y ago

Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.

TeMPOraL 6y ago
Yup. What also helps is that the data literally sits on your drive, or your network drive. No SaaS vendor is holding it hostage.
- helij 6y ago
  It doesn't feel like that. Even if I save the file (365 subscription and desktop apps) on my disk it steel feels 'cloudy'.
codeulike 6y ago
Yes exactly. An endless battle: IT want to stamp out spreadsheets and centralise everything but it all becomes an ossified bottleneck of development projects. Meanwhile people need to get stuff done so they use Excel.

indy 6y ago

I think you'd be shocked by how many of our financial and social institutions rely on emailing Excel spreadsheets to one another.

cscharenberg 6y ago
Don't forget healthcare! Excel and CSV for tons of data exchange.
- sk5t 6y ago
  CSV is somewhat okay-ish. Excel, not so much (thanks Excel for discarding the important leading zeros in this field, and for lossily parsing the other as a date... oh, this Excel file is so big it got the "XLSB" treatment which is hard to undo)
- chasd00 6y ago
  I think the healthcare industry is probably the only thing keeping fax machines alive. Interestingly, fax machine are tagged as HIPPA compliant. Lots of spreadsheets get printed and then fed to the fax machine for transmission instead of email/whatever. On the other side, and i'm being serious, someone gets the fax and creates a new spreadsheet and enters all the rows manually. fun fun.
colinmhayes 6y ago
I am absolutely not shocked by that. Most people do not know how to insert into a db yet alone spin one up and add tables. Everyone can excel.

WrtCdEvrydy 6y ago

Alternatively... "We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"

zarq 6y ago
"We should maybe start tracking some of this" "Where does the data come from?" "Dunno, a handful of hospitals... maybe?" "Right. Let's just use Excel, not gonna waste time on that now" ... six months later ... "Hey um..."
- WrtCdEvrydy 6y ago
  From the species that brought you "The Game of Life" comes "The Game of Technical Debt"
chasd00 6y ago
>"We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it" you have no idea how right you are :)
- WrtCdEvrydy 6y ago
  Been there, done that, got the sqlite t-shirt... Wasn't a bad choice, but once you load 20TB into it, you realize you done goofed.

libraryatnight 6y ago

In my org excel might have been the start, the intent being to move over to a DB once someone got it stood up. Then nobody would stand up the DB - because not urgent, we have the excel for now. The DB would be completely forgotten, and 3 years down the road everybody would be wondering wtf we have this huge excel file for. Then we would have six weeks of meetings to discuss the file until something else comes up and we forget it for another 3 years.

TeMPOraL 6y ago
That's why Excel wins. Because it's right there, infinitely adaptable to any change in workflow you may need to make. No need to send official requests through half a dozen layers of management to get the IT people to maybe, in half a year, add a column to the database. No need to involve 20 people for a year to create new software for a new workflow. You just agree with your co-workers on a change, swap a column, add some color, File->Save, done. It wins because it puts power in the hands of the people doing the work.
- fivre 6y ago
  I mean, yeah, but pretend Excel is, idk, sword arms. You get a lot of work done by whirling around the office, spinning your swords, cutting off a variety of limbs and heads in the process. You get to the end, meet the few survivors, staring dismayed at the trail of blood and dismemberment behind you, and you yell, "BUT I GOT THE JOB DONE!" You're not wrong, but... I wish there were some alternative where we instead fixed our ossified, byzantine processes gradually over time, so that we didn't need to break out the sword arm tornado in the name of getting work done and then, in hindsight, say, "well, shame about the negative externalities, but there really just was no other way. Ah well, let's move on, everyone who isn't a pile of flesh and blood and bits on the floor tidy up. Gotta get everything clean for the next sword arm tornado."
  - TeMPOraL 6y ago
    But the negative externalities don't seem all that big, considering. For every problem like this, you get countless millions of person-hours and dollars saved in work that wouldn't have happened at all, or would have been delayed, because of the necessity and cost of creating complex IT systems to support it. If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant about sharp objects. The alternative then would be a central processing factory that takes several years and millions of dollars to built, and which has to be turned off for a month several times a year, to change the shape of the blades used by the automated cutters.

shadowgovt 6y ago

The common case for stories like this is the answer is "nobody." Time point 1: "We need a way to map the long-form epidemiology state. It's 2D data... Let's use Excel." "Will we hit limits?" "Pfft, no. Not unless we end up needing to track tens of thousands of patients, and that's not practical; we don't have enough people to do that tracking in the whole NHS." Time point 2: "This pandemic is an URGENT problem. We need to hire more people than ever before to do tracing. And be sure to keep the epidemeology state mapper updated so it can drive the summary dashboards!"

addicted 6y ago

Except it wasn’t the government. They outsourced the test and trace system to private companies.

callamdelaney 6y ago
Public Health England was responsible for collating the data which came in the form of CSV's - PHE is a government agency.

dgudkov 6y ago

Of course Excel. What else non-technical users can use? They simply don't have any other data tool.

manigandham 6y ago

Excel is perfectly fit for that and is nowhere near the actual issue. It's lightweight, fast, portable, compatible, and just about everyone can use it regardless of (technical) skill.