5 ms·

Any opinions on the best way to deal with this?

• ajross 18y ago
  There aren't any. The User-Agent header has always lied, at least a little bit. What AVG wants from your site is the same thing that the users do: the content (in this case, so that they can scan it). If you insist on giving them a different page (like an error page) than a user would see, they're going to take steps to prevent you from doing that. Remember that the desired outcome here is to see fewer AVG hits, not to "screen them out" just because they're not a human being (although they are a necessary condition for a human hit coming later). Try reducing the unique URL count on your site (basically: make it more RESTful and eliminate all the query strings on broadly identical content) so they have fewer URLs to crawl. Set your caching headers properly so that they don't have an excuse to recrawl you too often. That won't force them to hit you less, but it will "help them to help you" by reducing the total bandwidth. Remember they don't want to pull any more than they have to either.
  • BrandonM 18y ago
    What AVG wants from your site is the same thing that the users do Then shouldn't AVG report the same User-Agent that the user's browser is reporting?
    • aristus 18y ago
      If AVG didn't have their head up their asses, yes. Though that would make the problem for webmasters even harder. What really chews my liver is that they didn't even do it right.
• avalanche 18y ago
  I think the solution should be fairly simple. Since AVG is not a real browser, despite telling a web server that it is, it should be easy to craft a piece of Javascript and cookie on a webpage that would return a proper result in a real browser like IE, Firefox, Safari, or Opera. If a proper result is not returned, then the web server can just do a 302 meta redirect to AVG's website as mentioned earlier. If implemented widely on web servers (say as an apache module or PHP patch) this, in effect, would cause AVG to become it's very own DDoS attacker. I don't think they would want this, but it'll probably happen if they don't re-think the way they do things.
  • mleonhard 18y ago
    How would that work? The AVG robot probably only fetches the page's URL. By the time you serve up a JavaScript, you've already lost the ability to redirect the robot. Also, once enough people start redirecting the AVG robot to the AVG website, they'll just update the robot to ignore those redirects. It's antivirus software so it gets updated very often.
    • avalanche 18y ago
      I guess it could be done using some combination of first placing into every HTTP header: "Refresh: 2;URL=htpp://www.grisoft.com/", which would then allow a piece of Javascript on the web page up to 2 seconds to figure out if the visitor is a real browser (maybe even including some DOM checks too), and then acting accordingly (ie: sending the user to a page without the redirect header). If AVG's http fetcher doesn't execute the Javascript or fails to pass the browser test, then it gets redirected back to AVG's site because of the HTTP header. Convoluted? Yes. Will it work? It should, unless AVG decides to implement a full Javascript engine and DOM stack in their http fetcher. Also, the initial re-direct doesn't have to go to Grisoft, it could go each time to a randomly selected site from a list of their competitors <evil grin>. I think AVG is really playing with fire, it's really only a matter of time before things like this start popping up as defense mechanisms.
    • pmjordan 18y ago
      Presumably, AVG loads extra resources like images, CSS, iframes, <object>s etc. from your page. (if it didn't, it wouldn't be very good at finding malware, and be really easy to detect)
      • whatusername 18y ago
        actually - from skimming the article it appears that it doesn't.