loving the dot com

Since late April 2008 a great many websites (this one included) have started noticing changes in their daily traffic – large sites especially. For some sites this traffic increase is huge, which can not only drastically increase bandwidth costs, but can also massively skew web analytics numbers.

As a result of this, it has become very difficult to report accurate numbers to advertisers of the actual eyeballs hitting the page. Many web-robots (GoogleBot/MSNBot et al) clearly identify themselves by the ‘user agent’ field they send to servers. This allows analytics software to simply ignore them. The problem with AVG’s LinkScanner software is that it does notonline casino netrent a car bulgaria clearly identify itself, instead the the user agent appears as follows:


Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)


Unlike many other kinds of shady web-spiders, the AVG software does not have any malious intent. Instead, it aims to protect it’s users from clicking on links (which may lead to exploits, virus or trojan infection etc) by visiting other links on a page before you do – so while you’re reading down a search results page trying to decide what to click on, the AVG software has already gone there and seen it.

Unfortunately, this means your PC will be visiting a lot of pages that you probably won’t even visit. Not only does this artifically inflate the traffic for the websites it visits, but it also means that your PC is potentally transferring a lot more data than you want it to. For users on limited-bandwidth contracts, this could be a problem: “What do you mean I’m over my traffic limit? I’ve hardly visited any websites!” – well, you haven’t, but your PC might have.

Back to a webmaster’s point of view – it would totally defeat the point of the software if AVG released ways to identify their bots, so don’t expect them to. The best way to figure out what to look for is to use Google, and search for what other webmasters have noticed. For the time being the software appears to use the user agent shown above, but that’s not to say it won’t sporadically change to more commonly used ones.

Luckily, the bot doesn’t appear to execute JavaScript, which means stats packages like Omniture that rely on events being fired off shouldn’t be affected. However, packages like DoubleClick’s DART which monitor traffic as it is fetched from their remote servers will.

If you’ve been suffering from this problem and want a quick idea of how to possibly fix it, here’s some PHP to drop into the top of a page which should stop AVG traffic from hitting it (for now!):

<?php

if (preg_match('/MSIE 6.0; Windows NT 5.1;1813/',
                     $_SERVER['HTTP_USER_AGENT']))
    return 0;

?>

If anyone else has been affected by this problem or has come up with some alternative solutions to deal with it, please comment and let me know.

Perhaps if enough people kick up a fuss, AVG will be inclined to start a discussion on how to better identify and deal with the traffic they are now generating.