What I've seen is that thousands of jobs for dozens of team are run on company's Hadoop cluster. Sure, each team could provision their own custom infra and run

What I've seen is that thousands of jobs for dozens of team are run on company's Hadoop cluster. Sure, each team could provision their own custom infra and run the job there, but having a centralized way to do it, with all the extra niceties (scalable capacity, good monitoring, logging, HA), can provide some company-wide efficiencies. Plus, some of the jobs can in fact be huge and you may need dozens of nodes to process them (we have such jobs in a bank, where we don't really have big data) - doing it without a cluster would be problematic.