4 ms·
Yes, parse out HTML comments which is also kind of trivial if you've ever done any sort of parsing, listen for "<!--", whenever you come across it, ignore every
by embedding-shape 1y ago
Yes, parse out HTML comments which is also kind of trivial if you've ever done any sort of parsing, listen for "<!--", whenever you come across it, ignore everything until the next "-->". But then again, these people are using AI to build scrapers, so I wouldn't put too much pressure on them to produce high-quality software.
- stevage 1y agoLots of other ways to include URLs in an HTML document that wouldn't be visible to a real user, though.
- jcheng 1y agoIt's not quite as trivial as that; one could start the page with a <script> tag that contains "<!--" without matching "-->", and that would hide all the content from your scraper but not from real browsers. But I think it's moot, parsing HTML is not very expensive if you don't have to actually render it.