6 ms·

There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations. I'd rather that web robo

by rrwo 4y ago

There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations. I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine. Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.

rgrieselhuber 4y ago
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
- Eduard 4y ago
  What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages? I don't see this as a barrier unique to startups.
  - rgrieselhuber 4y ago
    It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
- DocTomoe 4y ago
  Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation. We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
- rrwo 4y ago
  > The structured data on sites, unfortunately, still requires a crawler to index that content How is that any different from requiring a crawler to index XML sitemaps? > At a minimum, adding some metadata content to XML sitemaps The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date. Google has some extensions for identifying images and videos. But that adds more work for site maintainers, who have to duplicate work.

by rrwo 4y ago

rgrieselhuber 4y ago

The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).

Eduard 4y ago
What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages? I don't see this as a barrier unique to startups.
- rgrieselhuber 4y ago
  It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
DocTomoe 4y ago
Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation. We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
rrwo 4y ago
> The structured data on sites, unfortunately, still requires a crawler to index that content How is that any different from requiring a crawler to index XML sitemaps? > At a minimum, adding some metadata content to XML sitemaps The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date. Google has some extensions for identifying images and videos. But that adds more work for site maintainers, who have to duplicate work.