Something like Crawlab (https://github.com/crawlab-team/crawlab https://github.com/crawlab-team/crawlab) is what I was imagining. May fork this to make it work

Something like Crawlab (https://github.com/crawlab-team/crawlab https://github.com/crawlab-team/crawlab) is what I was imagining. May fork this to make it work if some of the other suggestions do not pan out.