3 ms·
This is way less cool than the title suggests. They are doing a bunch of crawling and inserting the raw html content into their big centralized database, where
by anonytrary 8y ago
This is way less cool than the title suggests. They are doing a bunch of crawling and inserting the raw html content into their big centralized database, where you can run queries on the text inside:
select
url,
string_between(content, '<title>', '</title>') as title
from
resources
where
content_type like 'text/html%'
- sam0x17 8y agoI would say this is more exciting than it looks, though. I used to do a lot of crawling in the early 2000s, and almost all of it was expressed in terms of string_between calls. XPath is more convenient, but I'd say that 85% of the time you can collapse an XPath query into a string_between-style query. It can be awkward and even inconsistent, but in practice it often works well.
- sgc 8y agoHaving done a fair amount of work with xslt and using regex to strip out bad data, I agree. But 85% is terrible if you are creating a database. Any regex style query on xml or pseudo-xml requires bespoke treatment and a high amount of human hours to check the results before you can be sure an edge case didn't completely destroy your model.