2 ms·
You do not even need to use AWS. use Minio as S3 compatible system and Nifi to convert files to parquet once they land in Minio... No dependency on AWS.
by ekianjo 6y ago
You do not even need to use AWS. use Minio as S3 compatible system and Nifi to convert files to parquet once they land in Minio... No dependency on AWS.
- antman 6y agoMinio and nifi, require lots of resources for themselves. Better off using pure python and if one wants something lighweight and visually pleasing Mara [0] or Dagster with Dagit [1] will do the job [0] https://github.com/mara/mara-pipelines https://github.com/mara/mara-pipelines [1] https://docs.dagster.io/tutorial/execute https://docs.dagster.io/tutorial/execute
- ekianjo 6y agoYes, Minio and Nifi are not lightweight but if you look for robustness they have been used in large environments and proven to scale.