Ah, I remember that. To paraphrase myself at the time, "never send a 747 to do a racecar's job." Yeah, as it turns out, it's not a good idea to use a piece of

Ah, I remember that. To paraphrase myself at the time, "never send a 747 to do a racecar's job." Yeah, as it turns out, it's not a good idea to use a piece of software optimized for workloads to large to even be handled by a single machine on large quantities of data, and maybe you should use tools designed for small workloads (Gigabytes, or even Terabytes, is pretty small, nowadays) for small workloads. This should be obvious. Besides, things like Manta exist, so you can scale up your small workload scripts to larger workloads in the future. Mind, I don't know if I trust Manta yet (I don't know the perf, and I haven't tried it personally), so maybe exercise a little bit of caution about going that route.