3 ms·
There are thousands of pieces of software that hit a URL and attempt to do something useful with the contents. Just try linking to a page on your own site on Tw
by simonw 12y ago
There are thousands of pieces of software that hit a URL and attempt to do something useful with the contents. Just try linking to a page on your own site on Twitter and tail your logfiles to see all the interesting bots that show up.
A couple of examples off the top of my head:
If you link to a web page on Reddit, they run a simple crawler to try and extract an image they can use as a thumbnail representation: https://github.com/reddit/reddit/blob/af09fa8dee69bef4f65a1662d9ad91c2329946e1/r2/r2/lib/media.py#L470-L523 https://github.com/reddit/reddit/blob/af09fa8dee69bef4f65a16...
If you link to a URL in Slack, they do something similar: https://api.slack.com/docs/unfurling https://api.slack.com/docs/unfurling