I don't disagree, but if you are fully invested in some cloud vendor, there are maybe easier and cheaper options than basically rolling your own mapreduce. In

I don't disagree, but if you are fully invested in some cloud vendor, there are maybe easier and cheaper options than basically rolling your own mapreduce. In AWS for instance, you are one API call away from running a Spark job in Glue or EMR, that effectively does all that for you. Plus you get logging, monitoring, concurrency limiting, retries. Or, even easier, you could just dump the big json docs in S3 and use Athena to directly query them with SQL.