4 ms·

> Why can't it use the URL for the url? Because you can have multiple URLs for the same content and you need a way to say "by the way if you want to link to th

> Why can't it use the URL for the url? Because you can have multiple URLs for the same content and you need a way to say "by the way if you want to link to this page here's the proper URL" For example, if you have example.com/blog/foo, it's possible that m.example.com/blog/foo, example.com/blog/foo/, example.com/index.php?path=/blog/foo, example.com/index.php?p=1234, example.com/blog/foo?textonly=1 etc etc all point to the same blog post and in many cases you simply can't set up a redirect because the URL is valid.

runarberg 5y ago
What I don’t understand is if we omit e.g. the og:title meta tag, why can’t scrapers fall back to <title>? Or if there is no <meta name="og:description"> why not fall back to <meta name="description">? Most of the pages I make are unique and supplying these values twice is simply redundant. My only conclusion is that scrapers are stupid and developers are too laze to fix it (which ironically casts the work on other developers as we now need to supply these redundant values).
- kijin 5y ago
  The stupid scrapers we're talking about are major social networks like Facebook and Twitter. At this point I think it's more about unequal power than it has anything to do with developer laziness. Facebook and Twitter can tell others to add whatever stupid tags they demand, and people will comply. They can even call it a standard, and people will readily acknowledge it as such. Why change the status quo when you have the upper hand?
- collinmanderson 5y ago
  > What I don’t understand is if we omit e.g. the og:title meta tag, why can’t scrapers fall back to <title>? Or if there is no <meta name="og:description"> why not fall back to <meta name="description">? They do fall back to <title> and <meta name="description">. At least Facebook does. You can test it here (requires facebook login). It works though they complain about the missing tags. https://developers.facebook.com/tools/debug/ https://developers.facebook.com/tools/debug/

runarberg 5y ago

What I don’t understand is if we omit e.g. the og:title meta tag, why can’t scrapers fall back to <title>? Or if there is no <meta name="og:description"> why not fall back to <meta name="description">? Most of the pages I make are unique and supplying these values twice is simply redundant. My only conclusion is that scrapers are stupid and developers are too laze to fix it (which ironically casts the work on other developers as we now need to supply these redundant values).

kijin 5y ago
The stupid scrapers we're talking about are major social networks like Facebook and Twitter. At this point I think it's more about unequal power than it has anything to do with developer laziness. Facebook and Twitter can tell others to add whatever stupid tags they demand, and people will comply. They can even call it a standard, and people will readily acknowledge it as such. Why change the status quo when you have the upper hand?
collinmanderson 5y ago
> What I don’t understand is if we omit e.g. the og:title meta tag, why can’t scrapers fall back to <title>? Or if there is no <meta name="og:description"> why not fall back to <meta name="description">? They do fall back to <title> and <meta name="description">. At least Facebook does. You can test it here (requires facebook login). It works though they complain about the missing tags. https://developers.facebook.com/tools/debug/ https://developers.facebook.com/tools/debug/