> Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected. I've done something similar, you can just add a Pip

> Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected. I've done something similar, you can just add a Pipeline (Pipeline Task? not sure of the right terminology) which posts off the data somewhere else. You can also then store the full item logs in S3 so you can go back occasionally and check nothing has been missed. Works an absolute treat.