at a previous employer we had a similar issue as you. from the employer side, we had a managed api platform that normalized all our messy data stores into a sin

at a previous employer we had a similar issue as you. from the employer side, we had a managed api platform that normalized all our messy data stores into a single api front end. our websites basically all called this api to populate the data in the page requests. except we didn't write this platform, we paid for it. and it is licensed per 100 queries. so when. a single account starts doing 30k queries every hour, 24/7. the pocketbook was directly hit. and yes, web scraping was in the tos that every account agreed to but never read