loving the dot com

Since late April 2008 a great many websites (this one included) have started noticing changes in their daily traffic - large sites especially. For some sites this traffic increase is huge, which can not only drastically increase bandwidth costs, but can also massively skew web analytics numbers.

As a result of this, it has become very difficult to report accurate numbers to advertisers of the actual eyeballs hitting the page. Many web-robots (GoogleBot/MSNBot et al) clearly identify themselves by the ‘user agent’ field they send to servers. This allows analytics software to simply ignore them. The problem with AVG’s LinkScanner software is that it does notrent a car bulgaria clearly identify itself, instead the the user agent appears as follows:


Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)


Unlike many other kinds of shady web-spiders, the AVG software does not have any malious intent. Instead, it aims to protect it’s users from clicking on links (which may lead to exploits, virus or trojan infection etc) by visiting other links on a page before you do - so while you’re reading down a search results page trying to decide what to click on, the AVG software has already gone there and seen it.

Unfortunately, this means your PC will be visiting a lot of pages that you probably won’t even visit. Not only does this artifically inflate the traffic for the websites it visits, but it also means that your PC is potentally transferring a lot more data than you want it to. For users on limited-bandwidth contracts, this could be a problem: “What do you mean I’m over my traffic limit? I’ve hardly visited any websites!” - well, you haven’t, but your PC might have.

Back to a webmaster’s point of view - it would totally defeat the point of the software if AVG released ways to identify their bots, so don’t expect them to. The best way to figure out what to look for is to use Google, and search for what other webmasters have noticed. For the time being the software appears to use the user agent shown above, but that’s not to say it won’t sporadically change to more commonly used ones.

Luckily, the bot doesn’t appear to execute JavaScript, which means stats packages like Omniture that rely on events being fired off shouldn’t be affected. However, packages like DoubleClick’s DART which monitor traffic as it is fetched from their remote servers will.

If you’ve been suffering from this problem and want a quick idea of how to possibly fix it, here’s some PHP to drop into the top of a page which should stop AVG traffic from hitting it (for now!):

<?php

if (preg_match('/MSIE 6.0; Windows NT 5.1;1813/',
                     $_SERVER['HTTP_USER_AGENT']))
    return 0;

?>

If anyone else has been affected by this problem or has come up with some alternative solutions to deal with it, please comment and let me know.

Perhaps if enough people kick up a fuss, AVG will be inclined to start a discussion on how to better identify and deal with the traffic they are now generating.

This new technique has some real promise for mobile web content, and it looks like fun to play with!

Having been rediculously busy with various things, it has been rather a while since my last post, so here’s one that’s been sitting as a draft for ages, hopefully some of you will find it useful.

Wordpress Plugins - what’s the point?

Wordpress is a great publishing platform from the outset, but Matt also built in a great architecture for coders to expand the system to do all sorts of other neat stuff. I know of some blogs which use tens of plugins to make their site work, but at the moment I only use 4 main ones. here’s a quick rundown of what they do, and why I use them.

Akismet - stopping comment spam

If you haven’t already got some sort of anti-spam plugin installed on your blog, it’s more than likely you are getting lots of comments submitted which seem like utter junk, or just blatant spam. This is where Akismet comes in. This plugin actually comes preinstalled with Wordpress, but you need to signup to Wordpress.com and get an API Key to use it.

Since I started making more regular posts, the amount of comment spam I received has just been rediculous, and Akismet has successfully sorted all that out for me. Make sure you have this turned on, seriously.

Feedburner Feed Replacement

As I mentioned in my redirecting your Wordpress feed to Feedburner post, this plugin provides you an easy way of redirecting all subscribers to your feed through the Feedburner service, allowing you to accurately keep track of reader statistics.

Name Dropper - let your visitors publicise your posts

The name dropper plugin gives bloggers a simple way to allow their visitors to submit their blog posts to a number of social bookmarking sites, including StumbleUpon, Digg, Del.icio.us and many others. It’s well worth installing it - you can see it in use at the bottom of every post here.

Optimal Title - fix your titles for SEO

Optimal Title replaces Wordpress’ wp_title() function and just moves the seperator to after the title, rather than before it. This means the title of your post will be picked up by search engines, BEFORE the name of your blog. That way visitors will more easily find their way to your posts directly.

Google Sitemaps - helping Google spider your site

Having an accurate sitemap of your site can often appear to be more trouble to setup than it’s worth, but thanks to this plugin, the task no longer needs to be a tough one. Just install the plugin and hit Activate, all the rest is done for you. See the sitemap it generates for this site here.

Twitter Tools - simple integration to Twitter

My last post was all about the Twitter service. This plugin allows you to simply integrate with their API to view and submit your latest tweets. Seen here in the sidebar on the right.

These days there are a billion and one scripts and companies which can tell you all sorts of statistics about your website traffic, but there’s a new kind of statistic in town - the heat map

Instead of the conventional way of collecting statistics by parsing through log files or logging page hits in real time, heat mapping scripts actually respond to where users click on your pages. By using this data it’s possible to draw some really pretty pictures which can clearly illustrate how users physically move about the site - which elements are more popular than others? How far are people willing to scroll down? Is this Ad actually working? A heat map is the answer.

Commercial Heatmaps

The main names on the commercial side seem to be clickdensity, crazyegg and ClickTale - although ClickTale are still in beta testing and have yet to officially launch their service.

Here’s an example of the offering from clickdensity:

Once you sign up for the service, all you have to do to get their service working is paste a little bit of javascript directly before your closing </body> tag in your page and they immediately start tracking your visitors. As you can see from the picture you also get a lot of filter options on how to narrow down the results shown to you - whether it is in a heat map, or more traditional form. One of the particularly neat features splits the page up into it’s individual elements and gives you individual statistics as you hover over each one.

After trialing clickdensity’s service on a couple of websites at work it was immediately clear that there are a huge number of clicks which just appear to go absolutely nowhere - whether this is because of users clicking on their browser window to bring it into focus, or just randomly clicking it isn’t clear. It is very clear where the large majority of clicks are going though, and I think their map is very nicely rendered.

If you’re looking to test out heat mapping for your website, I would very much recommend you take a serious look at clickdensity as they have more detailed statistics than other providers I have seen. One thing to be wary of though, statistics are (currently) restricted to the exact domain names associated with the accounts - so subdomains count as seperate domains. This means if you have a large site which sprawls over a large number of subdomains, you’ll have to add a tracking account for each one.

Non-Commercial Heatmaps

Wherever there’s a good commercial project, there’s often a non-commercial open-source one too. This one isn’t as good as it’s paid-for counterparts, but then it is being built in people’s spare time and they have no financial motivation to make it as pretty.

The non-commercial offering comes in the form of LabsMedia’s ClickHeat. This system is written in JavaScript (of course) with a PHP/GD backend. Here’s what theirs looks like:

As you can see the interface can’t really compete with that of clickdensity, but most the functionality is there. I have briefly tested this system too and while it does work, I would be more inclined to use a paid-for service at the moment simply because the reporting side isn’t entirely up to scratch.

That said - having had a flick through the code which makes it work, it becomes clear that the system isn’t half as complicated as you might think. All the JavaScript has to do is tell the server where on the page the user has clicked, the majority of the code is dedicated to drawing the heatmap image. This is also fairly trivial in the grand scheme of things.

It’s often said that simple ideas are the best, and I’m surprised that more people haven’t implemented a similar sort of system to this. As it is so simple it does make you wonder how long clickdensity, clickegg and ClickTale can stay ahead of the game.

More and more websites these days are adding what’s called a “Favicon” (or ‘Favourites Icon’) so you can identify them better. For those of you thinking “wtf is a favicon”, it’s the little picture which might be shown next to the URL in the address bar, next to the website title or as part of the tab title - depending on your web browser. For example, if you’re really lucky, and look really carefully you might just see a very small picture of me!

Why are Favicons useful?

The main reason Favicons are useful these days is to promote brand familiarity - it’s the ideal place to put your company logo or some other recognisable little picture there. The problem is actually creating one - or at least, that’s what most people think. It’s actually really easy.

Favicons are stored as a .ico (Icon) file, which is really just a very small BMP file. There are a few programs which let you edit them easily but recently I came across this little gem:

http://www.rw-designer.com/online_icon_maker.php

By using this tool you can easily draw your own little logo, or upload a picture and have it automatically translate it for you. You can then easily save the icon and upload it to your website.

How to get the Favicon to display

Some web browsers support automatic discovery of favicons, that means all you have to do is upload your .ico file as ‘favicon.ico’ to the root directory of your site and it’ll do the rest - BUT - the best way to make sure is to also add the following tag into the header section of your website template, between the <head&ht; and </head> tags:


<link rel="shortcut icon" href="/favicon.ico" />

Adding a tag like this means the browser will be able to find your favicon wherever the user has navigated to (or entered) in your site.

Creating Animated Favicons

Some browsers, such as Firefox, have the ability to display animated favicons - these are just regular animated .gif files so they are even simpler to create. You can see an example of an animated favicon here.

To make your life simpler still, there’s an online generator for animated icons here! For maximum compatibility it’s always best if you’re going to include an animated favicon to also include a static one too because otherwise some browsers won’t show one at all.

To include your animated favicon along side your regular one, put the following code in your page header (as described above):


<link rel="icon" type="image/gif" href="/icon.gif" />

Just replace icon.gif with the filename of whatever you uploaded your animated favicon as, and there you have it.

Job done.

The popularity of RSS is snowballing as more and more people are finding ways to read content faster and more conveniently, especially with the new compatibility now inside Internet Explorer 7

For those not in the know, RSS is a data format used to “publish frequently updated content” - basically, it’s a way of laying out what you’ve written so machines can pick it up without scraping it directly from your website. This can then be passed onto your readers in whatever format they choose. The advantage is that they can read your content in the most convenient way, without necessarily having to open their web browser (thanks to RSS Readers).

By default, Wordpress doesn’t have a way of tracking how many readers you have to your RSS feed - this is where providers like FeedBurner come in. FeedBurner has a whole load of tools for you to optimize your feed and see who’s actually reading it - plus they have some neat ways to let you publish ads in your feed.

The question is, how do you get your Wordpress installation to redirect your RSS feeds to FeedBurner? Well, there are a few ways you can do it but just to make your life much simpler OrderedList has created a great FeedBurner Plugin for Wordpress, which lets you easily configure your RSS redirects from your normal Wordpress options. All you need to do is upload the plugin to your plugins directory, set the URL you want it to redirect to and bingo, job done.

Check out the lovingthe.com RSS Feed redirect.

So you’ve got a blog, and you’ve got something written down. Make sure the almighty blogosphere know it exists, and make sure people can easily find it again.

Recognition - LOOK! I’M OVER HERE! (aka XML-RPC Pings)

Most blogging software has the ability to send what’s called a ‘ping’ to blog indexing services (basically big lists of who’s written what and when), Wordpress is no different. This ping tells the indexing service that you’ve written something new and they should come and take a look at it. You should find that as soon as you hit “Publish!” you’ll get a load of hits from various web spiders as they digest your latest offering.

You can see the list of place your Wordpress blog pings by going to Options -> Writing in the control panel and looking at the “Update Services” box at the bottom of the page. By adding a list of servers in this box you can get Wordpress to automatically send ‘pings’ to all the servers each time you post a new article.

Here’s the list of servers I currently send a ping to whenever I publish a new post:

http://rpc.blogrolling.com/pinger/
http://www.blogdigger.com/RPC2
http://www.bitacoles.net/ping.php
http://rpc.wpkeys.com
http://rpc.technorati.com/rpc/ping
http://rpc.britblog.com
http://pingoat.com/goat/RPC2
http://bulkfeeds.net/rpc
http://rpc.weblogs.com/RPC2
http://api.my.yahoo.com/RPC2
http://services.newsgator.com/ngws/xmlrpcping.aspx
http://api.moreover.com/ping
http://api.moreover.com/RPC2
http://www.blogpeople.net/servlet/weblogUpdates
http://ping.fakapster.com/rpc
http://www.blogoon.net/ping/
http://ping.bloggers.jp/rpc/
http://bblog.com/ping.php
http://rpc.tailrank.com/feedburner/RPC2
http://ping.bitacoras.com
http://ping.feedburner.com
http://ping.myblog.jp
http://ping.syndic8.com/xmlrpc.php
http://ping.weblogalot.com/rpc.php
http://pinger.blogflux.com/rpc
http://blogsearch.google.com/ping/RPC2
http://blog.goo.ne.jp/XMLRPC
http://rpc.icerocket.com:10080
http://rpc.pingomatic.com
http://api.feedster.com/ping.php

There will always be more servers you can add to this list, so it’s well worth keeping an eye on it from time to time.

Syndication - Getting acknowledgement, and getting people to return

If you look in the toolbar on the right (well, it was on the right when I wrote this), you’ll see a load of little buttons under the “Syndication” heading. Each of these buttons allows someone to easily link your blog into their blog service of choice. That way they don’t have to bookmark your URL or remember the address, it’s automatically added right where they want it. This is a good thing. I know this because as soon as I added these buttons the number of return visitors to the site started adding up (thanks to Google Analytics for those statistics)

That’s it for now, but I’ll be adding more very soon, with a round-up at the end of the week.

Amazon’s latest project, ‘Mechanical Turk‘, provides a web services API for computers to integrate ‘artificial artificial intelligence’ into their software, by asking humans to complete certain tasks rather than relying on automatic processing.

The name itself is a little odd, and refers to a machine built in the 18th century which played chess against real people and frequently beat them - but almost 100 years later it was revealed that there was a man inside the machine playing the games.

Before you think “this sounds like another useless waste of time and TCP/IP packets”, take a moment to think what it can do for you…

The pseudo-code example Amazon use in their FAQ is a particularly good one:

read (photo);
photoContainsHuman = callMechanicalTurk(photo);
if (photoContainsHuman == TRUE) {
	acceptPhoto;
}
else {
	rejectPhoto;
}

What this basically means is instead of writing a program to read in an image and writing rediculously complex algorithms to try and determine whether a particular photo contains a human, you can submit the request to Amazon’s Mechanical Turk. From there someone else can choose to take on the task (for a fee) and quickly (and more reliably) give you an answer to your question.

This service isn’t just restricted to tasks like simply identifying photos. It could be used to do anything, translate and sort RSS Feeds, click on advertising - whatever you can think of.

Remember - if you will pay for it, someone will do it.

Recently it has come to my attention that web monkeys all over the Internet are getting all excited about Microformats and how they are “changing the web” - even one of the developers for Mozilla has started blogging about how exciting it is.

Don’t get your hopes up just yet though, it’s not as exciting as you might think.

The first line from Alex Faaborg’s blog puts it quite well into perspective:

Have you been over hearing people talk about microformats and thought to yourself ‘what are those?

To be honest no, no I haven’t. When I first heard about it I immediately assumed it was the latest semantic craze for web developers, and it seems I was right.

Continue reading…