> Running their own crawler at this scale is probably not reasonable. You can interpret your comment in two ways, so maybe I misunderstood you, but they are ru

> Running their own crawler at this scale is probably not reasonable. You can interpret your comment in two ways, so maybe I misunderstood you, but they are running their own crawler, and they’re providing API access to their own web and news indices. These are geared towards niche content, and maybe that’s what you meant: they’re probably not running their crawler at a large scale.