I think command line tools is going to be fine if all you do is process one row at a time. Or if your data has a known order. But if you want to do some kind o

I think command line tools is going to be fine if all you do is process one row at a time. Or if your data has a known order. But if you want to do some kind of grouping or for example pivoting rows to columns, I think you will still benefit from a distributed tool like Spark or Trino. That can do the map/reduce job for you in a distributed way.