To easily distribute jobs over a cluster. Running right now: I have a database of 100,000,000 molecules, and I need to compute the force field on all of them.

To easily distribute jobs over a cluster. Running right now: I have a database of 100,000,000 molecules, and I need to compute the force field on all of them. So I put all of the IDs in the celery queue, and then start a worker across 6 nodes, The job pulls the row from the database, computes the force field, and then stores the result. Lots of jobs like this: - I used to use it to tune hyperparameters of shallow models - Used it to batch convert a bunch of compressed files from one format to another - Use it with celery beat to schedule scraping of URLs