6 ms·

AVG disguises fake traffic as IE6

33 points by nickb 18y ago

gm 18y ago
Any opinions on the best way to deal with this?
- ajross 18y ago
  There aren't any. The User-Agent header has always lied, at least a little bit. What AVG wants from your site is the same thing that the users do: the content (in this case, so that they can scan it). If you insist on giving them a different page (like an error page) than a user would see, they're going to take steps to prevent you from doing that. Remember that the desired outcome here is to see fewer AVG hits, not to "screen them out" just because they're not a human being (although they are a necessary condition for a human hit coming later). Try reducing the unique URL count on your site (basically: make it more RESTful and eliminate all the query strings on broadly identical content) so they have fewer URLs to crawl. Set your caching headers properly so that they don't have an excuse to recrawl you too often. That won't force them to hit you less, but it will "help them to help you" by reducing the total bandwidth. Remember they don't want to pull any more than they have to either.
  - BrandonM 18y ago
    What AVG wants from your site is the same thing that the users do Then shouldn't AVG report the same User-Agent that the user's browser is reporting?
    - aristus 18y ago
      If AVG didn't have their head up their asses, yes. Though that would make the problem for webmasters even harder. What really chews my liver is that they didn't even do it right.
- avalanche 18y ago
  I think the solution should be fairly simple. Since AVG is not a real browser, despite telling a web server that it is, it should be easy to craft a piece of Javascript and cookie on a webpage that would return a proper result in a real browser like IE, Firefox, Safari, or Opera. If a proper result is not returned, then the web server can just do a 302 meta redirect to AVG's website as mentioned earlier. If implemented widely on web servers (say as an apache module or PHP patch) this, in effect, would cause AVG to become it's very own DDoS attacker. I don't think they would want this, but it'll probably happen if they don't re-think the way they do things.
  - mleonhard 18y ago
    How would that work? The AVG robot probably only fetches the page's URL. By the time you serve up a JavaScript, you've already lost the ability to redirect the robot. Also, once enough people start redirecting the AVG robot to the AVG website, they'll just update the robot to ignore those redirects. It's antivirus software so it gets updated very often.
    - avalanche 18y ago
      I guess it could be done using some combination of first placing into every HTTP header: "Refresh: 2;URL=htpp://www.grisoft.com/", which would then allow a piece of Javascript on the web page up to 2 seconds to figure out if the visitor is a real browser (maybe even including some DOM checks too), and then acting accordingly (ie: sending the user to a page without the redirect header). If AVG's http fetcher doesn't execute the Javascript or fails to pass the browser test, then it gets redirected back to AVG's site because of the HTTP header. Convoluted? Yes. Will it work? It should, unless AVG decides to implement a full Javascript engine and DOM stack in their http fetcher. Also, the initial re-direct doesn't have to go to Grisoft, it could go each time to a randomly selected site from a list of their competitors <evil grin>. I think AVG is really playing with fire, it's really only a matter of time before things like this start popping up as defense mechanisms.
    - pmjordan 18y ago
      Presumably, AVG loads extra resources like images, CSS, iframes, <object>s etc. from your page. (if it didn't, it wouldn't be very good at finding malware, and be really easy to detect)
revicon 18y ago
Google / Yahoo do this as well to check up on sites to make sure whats served to their bots is the same as whats served to real users. Google has 3 or 4 different fake user agent strings they use for the googlebot
- jonknee 18y ago
  Google doing it isn't nearly the same as 10 million users doing it. In fact, this type of activity should be limited to search engines, not client-side apps.
Hexstream 18y ago
User-Agent information is inherently unreliable. I don't see why people are upset when a misguided practice that already provides false results with an unknown margin of error gets even worse. If they insist on using such a metric then it's their responsibility to assume the risks that it incurs.
petercooper 18y ago
One wonders if there's an "advantage" you could gain from this.. If your site is getting requested by AVG but NOT a valid browser, you can assume someone did a search that you appeared on but they did NOT click on you. This metric could be useful for improving your CTR in regular search results, something totally impossible to measure before.
- spydez 18y ago
  The problem being you've no idea why they did not click on you - only that they did. Thus, you can know something's wrong, but not why. And how are you going to improve if you don't know what's wrong?
  - petercooper 18y ago
    In SEO that's a pretty common problem.
Hoff 18y ago
The bad guys know how to bypass this check. It took seconds to figure out a way that could cloak malware against this AVG check. Malware is a problem. This is not a solution. This is another botnet.
Xichekolas 18y ago
Sounds like there is an easy solution to this problem... just block all IE6 users... kills two birds with one stone! </tongue_in_cheek>
- Stabback 18y ago
  I'm actually building a web app with a small start up, and we just do not have the time to make our service ie6 compatible. We are going to let users know that and redirect them to a very plane jane site where you can still get the information but not very efficiently. However our market is mainly tech savvy people so it should work out.
  - mdemare 18y ago
    Fair enough, but sometimes tech savvy people use computers of tech unsavvy people (friends and relatives, internet cafés, corporations with reactionary IT departments...)
  - Xichekolas 18y ago
    Just use the js widget on http://www.savethedevelopers.org/ http://www.savethedevelopers.org/ ... no sense writing your own. I'm also ignoring IE6, crappy old corporate computers be damned. It's just not worth my sanity.
    - Stabback 18y ago
      Sorry, that widget just gives that little dropdown redirecting to their site? I meant that I was going to just provide an info page where the user can still get the information on my site, just not in the same flashy page as everyone else.
ideamonk 18y ago
AVG is an antivirus. When users install it, they give it full privileges. A program like AVG Antivirus runs on many computers. Ofcourse this gives AVG enormous power to affect ad market, not only will CTRs go down, but also if AVG is corrupt, they can make auto clickers or have rival's ad income down by a great extent, or eat up adsense impressions of rivals. I wonder why didn't anyone think before implementing this.

33 points by nickb 18y ago

gm 18y ago

Any opinions on the best way to deal with this?

ajross 18y ago
There aren't any. The User-Agent header has always lied, at least a little bit. What AVG wants from your site is the same thing that the users do: the content (in this case, so that they can scan it). If you insist on giving them a different page (like an error page) than a user would see, they're going to take steps to prevent you from doing that. Remember that the desired outcome here is to see fewer AVG hits, not to "screen them out" just because they're not a human being (although they are a necessary condition for a human hit coming later). Try reducing the unique URL count on your site (basically: make it more RESTful and eliminate all the query strings on broadly identical content) so they have fewer URLs to crawl. Set your caching headers properly so that they don't have an excuse to recrawl you too often. That won't force them to hit you less, but it will "help them to help you" by reducing the total bandwidth. Remember they don't want to pull any more than they have to either.
- BrandonM 18y ago
  What AVG wants from your site is the same thing that the users do Then shouldn't AVG report the same User-Agent that the user's browser is reporting?
  - aristus 18y ago
    If AVG didn't have their head up their asses, yes. Though that would make the problem for webmasters even harder. What really chews my liver is that they didn't even do it right.
avalanche 18y ago
I think the solution should be fairly simple. Since AVG is not a real browser, despite telling a web server that it is, it should be easy to craft a piece of Javascript and cookie on a webpage that would return a proper result in a real browser like IE, Firefox, Safari, or Opera. If a proper result is not returned, then the web server can just do a 302 meta redirect to AVG's website as mentioned earlier. If implemented widely on web servers (say as an apache module or PHP patch) this, in effect, would cause AVG to become it's very own DDoS attacker. I don't think they would want this, but it'll probably happen if they don't re-think the way they do things.
- mleonhard 18y ago
  How would that work? The AVG robot probably only fetches the page's URL. By the time you serve up a JavaScript, you've already lost the ability to redirect the robot. Also, once enough people start redirecting the AVG robot to the AVG website, they'll just update the robot to ignore those redirects. It's antivirus software so it gets updated very often.
  - avalanche 18y ago
    I guess it could be done using some combination of first placing into every HTTP header: "Refresh: 2;URL=htpp://www.grisoft.com/", which would then allow a piece of Javascript on the web page up to 2 seconds to figure out if the visitor is a real browser (maybe even including some DOM checks too), and then acting accordingly (ie: sending the user to a page without the redirect header). If AVG's http fetcher doesn't execute the Javascript or fails to pass the browser test, then it gets redirected back to AVG's site because of the HTTP header. Convoluted? Yes. Will it work? It should, unless AVG decides to implement a full Javascript engine and DOM stack in their http fetcher. Also, the initial re-direct doesn't have to go to Grisoft, it could go each time to a randomly selected site from a list of their competitors <evil grin>. I think AVG is really playing with fire, it's really only a matter of time before things like this start popping up as defense mechanisms.
  - pmjordan 18y ago
    Presumably, AVG loads extra resources like images, CSS, iframes, <object>s etc. from your page. (if it didn't, it wouldn't be very good at finding malware, and be really easy to detect)

revicon 18y ago

Google / Yahoo do this as well to check up on sites to make sure whats served to their bots is the same as whats served to real users. Google has 3 or 4 different fake user agent strings they use for the googlebot

jonknee 18y ago
Google doing it isn't nearly the same as 10 million users doing it. In fact, this type of activity should be limited to search engines, not client-side apps.

Hexstream 18y ago

User-Agent information is inherently unreliable. I don't see why people are upset when a misguided practice that already provides false results with an unknown margin of error gets even worse. If they insist on using such a metric then it's their responsibility to assume the risks that it incurs.

petercooper 18y ago

One wonders if there's an "advantage" you could gain from this.. If your site is getting requested by AVG but NOT a valid browser, you can assume someone did a search that you appeared on but they did NOT click on you. This metric could be useful for improving your CTR in regular search results, something totally impossible to measure before.

spydez 18y ago
The problem being you've no idea why they did not click on you - only that they did. Thus, you can know something's wrong, but not why. And how are you going to improve if you don't know what's wrong?
- petercooper 18y ago
  In SEO that's a pretty common problem.

Hoff 18y ago

The bad guys know how to bypass this check. It took seconds to figure out a way that could cloak malware against this AVG check. Malware is a problem. This is not a solution. This is another botnet.

Xichekolas 18y ago

Sounds like there is an easy solution to this problem... just block all IE6 users... kills two birds with one stone! </tongue_in_cheek>

Stabback 18y ago
I'm actually building a web app with a small start up, and we just do not have the time to make our service ie6 compatible. We are going to let users know that and redirect them to a very plane jane site where you can still get the information but not very efficiently. However our market is mainly tech savvy people so it should work out.
- mdemare 18y ago
  Fair enough, but sometimes tech savvy people use computers of tech unsavvy people (friends and relatives, internet cafés, corporations with reactionary IT departments...)
- Xichekolas 18y ago
  Just use the js widget on http://www.savethedevelopers.org/ http://www.savethedevelopers.org/ ... no sense writing your own. I'm also ignoring IE6, crappy old corporate computers be damned. It's just not worth my sanity.
  - Stabback 18y ago
    Sorry, that widget just gives that little dropdown redirecting to their site? I meant that I was going to just provide an info page where the user can still get the information on my site, just not in the same flashy page as everyone else.

ideamonk 18y ago

AVG is an antivirus. When users install it, they give it full privileges. A program like AVG Antivirus runs on many computers. Ofcourse this gives AVG enormous power to affect ad market, not only will CTRs go down, but also if AVG is corrupt, they can make auto clickers or have rival's ad income down by a great extent, or eat up adsense impressions of rivals. I wonder why didn't anyone think before implementing this.