3 ms·

I try to stay away from negative takes here, so I’ll keep this as constructive as I can: It’s surprising to see the author frame what seems like a basic conseq

I try to stay away from negative takes here, so I’ll keep this as constructive as I can: It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight. And then they posted it here for attention.

zem 1y ago
it's mostly that they didn't think of the page preview fetcher as a "crawler", and did not intend for their robots.txt to block it. it may not be profound but it's at the least not a completely trivial realisation. and heck, an actual human-written blog post can okay improve the average quality of the web.
- archievillain 1y ago
  The bots are called "crawlers" and "spiders", which to me evokes the image of tiny little things moving rapidly and mechanically from one place to another, leaving no niche unexplored. Spiders exploring a vast web. Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
  - JimDabell 1y ago
    > Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It’s definitely not crawling as robots.txt defines the term. : > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
- EPendragon 1y ago
  That's exactly it. It was one of those unintended consequences of blocking everything that led me down the road of figuring it out. Like other commenters have indicated, I will likely need to go back and allow some other social media to access the OPG data for previews to render properly. But since I mostly post on LinkedIn and HN, I don't feel the need to go and allow all of the other media at the moment. That might change in the future.
spookie 1y ago
They posted it here because they wouldn't appear on Google otherwise (:
coolgoose 1y ago
I agree, and I am also confused on how this got on the frontpage of all things. It's like reading a news article of 'water is wet'. You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
- EPendragon 1y ago
  Honestly, I am also surprised how this got on the frontpage. This was supposed to be a small post of what I have learnt in the process of fixing my LinkedIn previews. I don't know how we got here.
EPendragon 1y ago
I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works. I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.

zem 1y ago

it's mostly that they didn't think of the page preview fetcher as a "crawler", and did not intend for their robots.txt to block it. it may not be profound but it's at the least not a completely trivial realisation. and heck, an actual human-written blog post can okay improve the average quality of the web.

archievillain 1y ago
The bots are called "crawlers" and "spiders", which to me evokes the image of tiny little things moving rapidly and mechanically from one place to another, leaving no niche unexplored. Spiders exploring a vast web. Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
- JimDabell 1y ago
  > Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It’s definitely not crawling as robots.txt defines the term. : > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
EPendragon 1y ago
That's exactly it. It was one of those unintended consequences of blocking everything that led me down the road of figuring it out. Like other commenters have indicated, I will likely need to go back and allow some other social media to access the OPG data for previews to render properly. But since I mostly post on LinkedIn and HN, I don't feel the need to go and allow all of the other media at the moment. That might change in the future.

spookie 1y ago

They posted it here because they wouldn't appear on Google otherwise (:

coolgoose 1y ago

I agree, and I am also confused on how this got on the frontpage of all things. It's like reading a news article of 'water is wet'. You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.

EPendragon 1y ago
Honestly, I am also surprised how this got on the frontpage. This was supposed to be a small post of what I have learnt in the process of fixing my LinkedIn previews. I don't know how we got here.

EPendragon 1y ago

I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works. I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.