6 ms·
While this is how web scrapers work, this post is targeted at url unfurlers, which have different goals and almost never follow redirects. Facebook, slack, etc,
by hyperhopper 5y ago
While this is how web scrapers work, this post is targeted at url unfurlers, which have different goals and almost never follow redirects. Facebook, slack, etc, have documentation publicly available on how their unfurlers work: most effectively just curl the html then parse meta tags
- junon 5y agoThey follow redirects but only authoritative ones from the server. Not meta tag redirects. Just to be clear.
- hyperhopper 5y agoHence the "almost"