tangent: one of the more satisfying patches of my programming career was improving system stability and roughly preserving performance while removing the use of

tangent: one of the more satisfying patches of my programming career was improving system stability and roughly preserving performance while removing the use of parallel processing the code grew like this: 1. some python tool is written to grab a bunch of data from a SQL DB through an ORM and do some kind of graph analysis on it. the code is annoyingly slow. 2. the problem is embarassingly parallel (tool performs analysis in terms of domain-specific paths, which can be processed independently for each region of the graph), so one of the devs chunks each problem into sub problems, distributes the work to a cluster, then aggregate the results. it's not lighting fast but a good speedup, good enough for now 3. the tool executes work in parallel but is still regarded by the business and the dev team as kind of slow. later on, another dev has time to do some profiling, sees the SQL queries generated are fairly daft and are failing to exploit ways to efficiently search the data, figures out some different queries. the tool is now running pretty fast. 4. there's an unrelated priority to uplift security. TLS is applied to the job queuing system used by the tool to execute work in parallel. the formerly-slow embarrassingly-parallel code is sending rather large volumes of intermediate data through the cluster, and alas, for technical reasons, when TLS is enabled the additional encryption overhead causes the queuing system to fail sometimes (it is so busy doing crypto on large messages that it delays sending heartbeats, which causes its clustered friends to infer a network partition, which causes the cluster to kick into panic-network-partition-recovery mode, which periodically breaks production and requires manual intervention to recover) 5. what's the fix? remember that we fixed the queries in step 3, meaning the original partial speedup from processing all the work in parallel is no longer required, so we delete all the parallel-chunking code to invert step 2, and the resulting system is fast enough when run sequentially, without using the queue to aggregate large intermediate results