There's more to it than that. What if instead of crawling the php generation of database rows with a bunch of cruft, the administrator published some kind of s

There's more to it than that. What if instead of crawling the php generation of database rows with a bunch of cruft, the administrator published some kind of schema with scraping and querying rules and you could alternatively make a single call to capture all of the data in a sematic schema. You can still do all the stuff you're talking about but it could make search more coherent. An entry for that humans and an entry for the computers. You can't trust everybody like this sure, but say imdb, discogs, wikipedia, all of which provide database dumps anyways (eg: https://datasets.imdbws.com/ https://datasets.imdbws.com/). That's what I'm advocating for revisiting. Lots of legit sites such as universities, newspapers, public records offices... You could even have a search toggle "screened sources" or whatever for the ones that make the cut