2 ms·

Hi there, Thanks, thats a good point! I will modify the crawler to respect those items, and also listed them as nofollow (similar to blocked by robots).