2 ms·
Also: the Internet Archive's crawler (Heritrix) implements link extractors for HTML, XML, CSS, JS, PDF, Word document, and even Shockwave Flash. Heritrix extra
by mgdlbp 4y ago
Also: the Internet Archive's crawler (Heritrix) implements link extractors for HTML, XML, CSS, JS, PDF, Word document, and even Shockwave Flash.
Heritrix extractor Javadocs: https://www.javadoc.io/doc/org.archive.heritrix/heritrix-modules/latest/org/archive/modules/extractor/package-summary.html https://www.javadoc.io/doc/org.archive.heritrix/heritrix-mod...
Source of the JavaScript extractor: https://github.com/internetarchive/heritrix3/blob/master/modules/src/main/java/org/archive/modules/extractor/ExtractorJS.java https://github.com/internetarchive/heritrix3/blob/master/mod...
A script is included for running the extractors from the command line: https://github.com/internetarchive/heritrix3/blob/master/dist/src/main/bin/extractor https://github.com/internetarchive/heritrix3/blob/master/dis...