I agree, 1GB is small. I do similar processing on more 100GB-ish tasks. But I should say, this is a large enough dataset size that loading it all into memory

I agree, 1GB is small. I do similar processing on more 100GB-ish tasks. But I should say, this is a large enough dataset size that loading it all into memory is sometimes infeasible, at least in interactive interpreted environments like Python. That's an important boundary point.