I don't think it makes economic sense to crawl everything with headless browsers, given it's several orders of magnitude more expensive, and in most cases doesn

I don't think it makes economic sense to crawl everything with headless browsers, given it's several orders of magnitude more expensive, and in most cases doesn't actually give you different results. (Many websites that seem like they'd not be crawlable with a dumb crawler, e.g. Medium, actually crawl just fine as long as you don't masquerade the U-A to look like Chrome) Makes a lot more sense to sample, identify and specifically crawl the sites where a classic retrieval doesn't work with headless browser, and do classic crawling elsewhere.