We knew upfront that ML was going to be a proof of concept and that the job was going to be done manually either way. We are many things in the public sector, b

We knew upfront that ML was going to be a proof of concept and that the job was going to be done manually either way. We are many things in the public sector, but we’re not big risk takers, and this was a job that had to be done as fast as possible because the missing documents are required by law. So what we did was that we cloned the data, and let the business do it’s thing while we did ours. This of course presented us with unique data on the results. The things we measured were quality, speed, economy and employee satisfaction. Quality was measured by keeping track of casefiles, that were flagged as missing the document by each process. That gave us two lists, one with the casefiles found by the manual team and one found by the ML team. We then made a few random checks of casefiles that appeared on both lists, and we checked every casefile that was only on one list. The ML team flagged more casefiles correctly, it both found more and made fewer errors. Of course this doesn’t tell us how many casefiles we didn’t find, but it does show us that ML was better. Speed was relatively simple, it was start to finish and ML was faster. We did rent a lot of iron in Azure to achieve this, we could have never done it without a major enterprise cloud agreement. We needed Microsoft to allocate the stuff we needed, it wasn’t even a simple task of using the automatic systems. Economically it’s a bit of a touchy subject. I won’t go into details on that, but basically we know what work costs. Renting iron in Azure wasn’t expensive compared to having that many full time workers dedicated to the task. Employee satisfaction is hard, but we don’t have people on staff who’s job it is to go through half a million casefiles and look at millions of documents. We had to pull people away from their regular jobs to do so, and even with 7000 people on staff, it’s really hard to find people who actually want to do this kind of work. HR did a bunch of HR magic, and basically people would prefer ML to do this sort of thing in the future.