29 ms·
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum,
by rgrieselhuber 4y ago
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
- Eduard 4y agoWhat's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages? I don't see this as a barrier unique to startups.
- rgrieselhuber 4y agoIt's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
- DocTomoe 4y agoDeep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation. We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
- rrwo 4y ago> The structured data on sites, unfortunately, still requires a crawler to index that content How is that any different from requiring a crawler to index XML sitemaps? > At a minimum, adding some metadata content to XML sitemaps The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date. Google has some extensions for identifying images and videos. But that adds more work for site maintainers, who have to duplicate work.