3 ms·
Further thoughts on article extraction: one idea that comes to mind is including extraction rules in the source selection metadata. I'm using something along
by dredmorbius 1y ago
Further thoughts on article extraction: one idea that comes to mind is including extraction rules in the source selection metadata.
I'm using something along these lines right now to process sections within a given source, where I define the section-distinguishing-element from a headline URL, as well as the plaintext, position (within my generated page), lines of context, and maximum age (days) I'm interested in.
That could be extended or paired with a per-source rule that identifies the htmlq specifiers which pull out title, dateline, and byline elements from the source.
A further challenge is that such specifiers have a tendency to change as the publisher's back-end CMS varies, and finding ways to identify those is ... difficult.
But grist for the mill, at any rate.