For large datasets, consider avoiding pagination complexity by having your api return a (asynch) job ID, and dumping the whole resultset to a file which can be

For large datasets, consider avoiding pagination complexity by having your api return a (asynch) job ID, and dumping the whole resultset to a file which can be fetched (over https) via the jobID. I have done this with both Postgres/S3 and Redshift/S3 backends and presigned URLs and looks like I could do it with Snowflake too.