So I work for an APM vendor. COVID-based unemployment has been excellent for our public sector sales (as well for other APM vendors who are the competition). On

So I work for an APM vendor. COVID-based unemployment has been excellent for our public sector sales (as well for other APM vendors who are the competition). One of the sales people went to (I think) every unemployment insurance department in every state and pitched them our software. Quite a few were standing in a smoldering wreckage and they gladly accepted to see what they could do. That is how I personally got to look at the guts of unemployment insurance system of Kansas and Florida back in May. The KS system was ... well judge for yourself. It was run on 3 (yes that's three) servers - a 8 CPU 32GB web server hosting UI and IVR web service, a 4CPU 48GB service layer hosting a WCF service talking to mainframe, and some auxilary 2CPU 4GB server doing some little batch stuff. Yes, that's right, no redundancy. That's for the state with almost 4 million people. The bottlenecks were all in the mainframe which was reached by via screenscraping host key-sending software, which was doing GC.Collect on every end of the given session. It worked well when nobody was using the unemployment system but once half the state of KS lost their jobs, everything melted down. The vendor for the host-key fixed the library by commenting our GC.Collect. Don't know what happened afterwards. Then next week I got a look at FL which is the uFacts system built by Deloitte https://www2.deloitte.com/us/en/pages/public-sector/solutions/unemployment-insurance-services.html https://www2.deloitte.com/us/en/pages/public-sector/solution... mentioned in the article. There was no Deloitte there anymore and from what I can tell there hasn't been for 3 years. That one was is a standard 3 tier (web layer, services layer, database layer) hosted by more than enough hardware on web and services layer - although very much an inconsistent hodgepodge of IaaS machines from Azure, but it all ended in the big Oracle database that had some atrocious performance under heavy load. All timeouts were cranked up to the limit - I am talking WCF/WS timeouts north of 600 seconds, and Oracle DB query timeouts set to what I think is infinity - so things just backpressured out of the slow Oracle queries like there is no tomorrow. But optimizations were pretty obvious. They had an "Messages" table that was adding all the things that could happen to a case that was north of 100M records and suffered tremendously on the inserts. Pretty easy sharding or just archiving opportunity. Here I and couple of my colleagues were able to get to the team leads and developers in charge to present our findings. Even though the governor was yelling at them and the lawsuits were filed and people were struggling, the response back from the dev leads was remarkably devoid of urgency. "well that we already know about this, we'll hoping to be getting to this 3 sprints from now". We all shrugged and moved on.