3 ms·
How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info o
by jmpe 13y ago
How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info on such a query, wouldn't it look the same?
Not looking for work atm, so my advice: this is not your problem.
- ChuckMcM 13y agoA typical crawler trap is a link to a page which adds some sort of "info" the url, so lets say you have a URL like http://not-a-real-host.com/some_page/ And that page has a link for comments which takes you to http://not-a-real-host.com/some_page/?show_comments=1 And the designer didn't neuter the show comments link and if you follow it, it takes you to: http://not-a-real-host.com/some_page/?show_comments=1&show_comments=1 Ad infiniteum. That is a bug but the same sort loop where the cgi arg is going from ?page=1, ?page=2 is valid. Basically folks who love working on web crawlers are people who like figuring out puzzles like this. The number of interesting things web sites can do that make crawling them not work is unlimited I believe. We have lots of regular checks (things like how many unique documents have I pulled from a host, or how many pages have the same MD5 hash, etc.) Designing defenses is part of the fun.
- ugexe 13y agoHe is asking how you plan to differentiate when it is ok for the query to be in the url (perhaps it is checked and validated with a regex on the server so it cant be abused) or when its actually capable of doing something malicious the maintainer is not aware of.