What is the exact point that a crawler needs to be decoupled (at the corporate level) from it's front end? Is it measured by MAUs? Pages scraped? Or is it going

What is the exact point that a crawler needs to be decoupled (at the corporate level) from it's front end? Is it measured by MAUs? Pages scraped? Or is it going by "it can be decoupled, so it should". The extreme version of your argument would have companies own the IP to a particular for loop's iterator variable name.