I worked on a tool that processes a large-ish dataset using only data structures in memory, since this is much faster and simpler than using a system like Spark

I worked on a tool that processes a large-ish dataset using only data structures in memory, since this is much faster and simpler than using a system like Spark for example. Not only that, the nature of the processing algorithm (a reduce, in effect) makes it kind of pointless to run on a cluster of nodes. The dataset is 2-3B records with 5-12 64-bit values each, stored in a few dozen files using the Apache Arrow format. If we take the midpoint of this range that's 170 GB just with raw data. With the overhead of data structures, I was running the process with ~400 GiB of RAM and could have done more on a beefier machine. It took about 20-30 minutes to run the full algorithm on these tens of billions of data points and this approach was perfect for this use case. No overhead of Spark and all of its dependencies, just one program, a bunch of input files, and it's done when I get back from lunch.