We also shared an article about how we run these indexing jobs at scale at deepset with kubernetes, SQS, s3 and KEDA. TL;DR: Queue upload events via SQS, uploa

We also shared an article about how we run these indexing jobs at scale at deepset with kubernetes, SQS, s3 and KEDA. TL;DR: Queue upload events via SQS, upload files to s3, scale consumers based on queue length with keda and use haystack to turn files into embeddings. This also works for arbitrary pipelines with your models, custom nodes (python code snippeds) and is pretty efficient. Part1 (application&architecture): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipelines-with-keda-and-haystack-part-1-the-application-45f54fcabd5d https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Part2 (scaling): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipelines-with-keda-and-haystack-part-2-deployment-8750dc96171f https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Example code: https://github.com/ArzelaAscoIi/haystack-keda-indexing https://github.com/ArzelaAscoIi/haystack-keda-indexing We actually also stared with celery, but moved to SQS to improve the stability.