loving the dot com

AVG LinkScanner and it’s fake traffic

alz — Wed, 25 Jun 2008 15:09:59 +0000

Since late April 2008 a great many websites (this one included) have started noticing changes in their daily traffic - large sites especially. For some sites this traffic increase is huge, which can not only drastically increase bandwidth costs, but can also massively skew web analytics numbers.

As a result of this, it has become very difficult to report accurate numbers to advertisers of the actual eyeballs hitting the page. Many web-robots (GoogleBot/MSNBot et al) clearly identify themselves by the ‘user agent’ field they send to servers. This allows analytics software to simply ignore them. The problem with AVG’s LinkScanner software is that it does notrent a car bulgaria clearly identify itself, instead the the user agent appears as follows:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

Unlike many other kinds of shady web-spiders, the AVG software does not have any malious intent. Instead, it aims to protect it’s users from clicking on links (which may lead to exploits, virus or trojan infection etc) by visiting other links on a page before you do - so while you’re reading down a search results page trying to decide what to click on, the AVG software has already gone there and seen it.

Unfortunately, this means your PC will be visiting a lot of pages that you probably won’t even visit. Not only does this artifically inflate the traffic for the websites it visits, but it also means that your PC is potentally transferring a lot more data than you want it to. For users on limited-bandwidth contracts, this could be a problem: “What do you mean I’m over my traffic limit? I’ve hardly visited any websites!” - well, you haven’t, but your PC might have.

Back to a webmaster’s point of view - it would totally defeat the point of the software if AVG released ways to identify their bots, so don’t expect them to. The best way to figure out what to look for is to use Google, and search for what other webmasters have noticed. For the time being the software appears to use the user agent shown above, but that’s not to say it won’t sporadically change to more commonly used ones.

Luckily, the bot doesn’t appear to execute JavaScript, which means stats packages like Omniture that rely on events being fired off shouldn’t be affected. However, packages like DoubleClick’s DART which monitor traffic as it is fetched from their remote servers will.

If you’ve been suffering from this problem and want a quick idea of how to possibly fix it, here’s some PHP to drop into the top of a page which should stop AVG traffic from hitting it (for now!):

If anyone else has been affected by this problem or has come up with some alternative solutions to deal with it, please comment and let me know.

Perhaps if enough people kick up a fuss, AVG will be inclined to start a discussion on how to better identify and deal with the traffic they are now generating.

Reading Festival, Web Scalability and Amazon coloured clouds

alz — Tue, 01 Apr 2008 00:19:32 +0000

While trying to book tickets to this year’s Reading Festival, I was once again frustrated by the main website (readingfestival.com) and it’s lack of ability to handle the increase in traffic as tickets are released for purchase and the lineup is announced. This frustration reminded me of a few sites I came across recently, some related research into designing sites for High Scalability, and systems for temporary load management. Here’s some stuff developers might find useful….

The first of these is the site HighScalability.com - as the title suggests, if offers tips and articles on exactly how to build a scalable site, and has some very handy interviews with tech staff from some very high traffic sites, such as YouTube and Flickr.

Looking at YouTube, there’s a great video from the Seattle Conference on Scalability from one of the YouTube staff about how they have successfully managed to grow their system on commodity hardware and open source software, such as MySQLкомпютри втора употреба. Watch the video here.

If you’re interested in working with MySQL, there is a PDF presentation available on Scribd which talks through a lot of the key issues developers face when scaling to heavy traffic loads. This one is well worth a read - see it here.

Lastly is a system which will allow you to quickly scale your application at low cost if you’re in absolute dire need. Enter Amazon’s Elastic Compute Cloud (EC2). There is too much behind the system to explain here, but it basically allows you to manage exactly how and what you want to scale by booting your own reusable images inside the cloud. Fans of Facebook will have no doubt heard of the iLike application, there’s a good explanation of how they used the cloud to scale their application here.

Oh - and don’t forget to get your Reading Festival tickets! No doubt the main site is still crippled under the load, so you can read the line-up and get ticket information from readinglineup.com.

Google Earth’s Flight Simulator Easter Egg

alz — Sun, 02 Sep 2007 23:09:06 +0000

As reported on Mashable, the Google Earth techies have coded quite a cool little easter egg into the new version of Google Earth - a flight simulator.

To enter flight sim mode, hit Ctrl+Alt+A on windows, or Command+Option+A on a Mac. This should pop up a window like the one above and allow you to start your flight. The help button is pretty comprehensive and gives you a fulll list of controls.

I did wonder a while ago how long it would take for real-world scenery to be properly integrated into one of the commercial flight sims. Maybe Microsoft might put some more thought into Flight Simulator 11 now.

TV.com Facebook app

alz — Sun, 02 Sep 2007 15:34:58 +0000

Check out this latest facebook app from TV.com. You can rate/review all your favourite tv shows, check out what’s currently hot, see what you should be watching. It’s also got a cool little quiz.

View the application here.

Image Resizing by Seam Carving

alz — Wed, 22 Aug 2007 20:21:20 +0000

This new technique has some real promise for mobile web content, and it looks like fun to play with!

IRC admins claim TimeWarner hijacks DNS to clean botnets

alz — Wed, 25 Jul 2007 14:35:35 +0000

Administrators of Internet Relay Chat (IRC) servers say they have recently become aware of dropping user counts from certain ISPs, especially TimeWarner owned Cox.net.

It is well known that botnets consisting of large numbers of compromised PCs are becoming more of a problem and are harder to find, harder to fight and harder to destroy. Until recently, the only people who seemed to be doing the fighting against this threat were the producers of anti-virus software. It is then up to the computer administrator, or home user to install this software in an attempt to remove any malicious programs from their system. As more and more viruses and trojans are released every day, this is not as simple as it sounds.

According to a number of IRC administrators, the Internet Service Provider Cox.net seems to have taken this fight away from it’s customers and into its own hands.

Many botnets use the IRC protocol for communication and control, so it makes sense that this is where Cox decided to strike. After administrators of the Ablenet IRC network noticed it’s users disappearing they started investigating the cause and found that a DNS server owned by Cox was returning falsified data and were hijacking DNS entries which caused any legitmate IRC connection to be redirected to a server owned by Cox.

After being redirected to Cox’s server a number of commands are sent to the user. To most users this is uncomprehensible data, but the server is infact issuing commands which attempt to uninstall one particular type of malicious trojan.

By hijacking a user’s connection in this manner, IRC networks are receiving a bad reputation because most connecting users are not aware that their connection has been hijacked. Users are left guessing why they cannot chat with their friends like normal, and aim their frustration toward the administrators of the IRC network. For most home users, avoiding this connection hijacking is not a trivial task.

More recently, administrators from EFnet, the oldest and 4th largest IRC network in the world, which currently holds around 60,000 concurrent users, have also suffered from their DNS entries being hijacked by Cox.

At present there are no laws in the United States to stop Cox, or other ISPs, hijacking and falsifying DNS entries, however the borderline on actually cleaning trojans automatically is much more vague. In the United Kingdom it is a definite breach of the Computer Misuse Act for any system or user to perform ‘Unauthorised Modification’ of another computer system. By issuing commands to a user’s PC without their prior consent, this is exactly what Cox appears to be currently doing.

One thing is for certain, with all the current emphasis on Net Neutrality laws in the United States, Cox may have very effectively demonstrated why so many people are campaigning to see it become a reality.

How many PS3s do you have?

alz — Wed, 02 May 2007 22:11:49 +0000

p.s. this isn’t me

spampress.net - catchalls and keywords

alz — Mon, 09 Apr 2007 01:20:13 +0000

Recently I setup another blog at spampress.net. This new blog (as suggeted by the name), will publish nothing but the spam it receives via email.

What’s the point?

Spampress.net has been setup as both a demonstration and experiment. Primarily, it will show exactly how much spam is received to a domain which is completely unpublicised. The second reason is an experiment to demonstrate how spammers are becoming more savvy with including specific keywords in their emails.

Why use keywords in email?

The term ‘keywords’ usually brings to mind SEO and other types of web targeting, but recently spammers have started adding more and more ‘random’ keywords to their email as a way of fooling spam filters. If you view your email in HTML format you might find a lot of spam is made up of one single image, by using this tactic the spammers can overcome a lot of the spam filters because the text isn’t directly readable, and therefore cannot be analysed. By viewing these same emails in text-only format often the email contains nothing but random keyword text. These are used to throw off the spam filter further.

What will this accomplish?

After the blog has been running a while I will be publishing the results collected by Google Analytics and the webserver’s local copy of Awstats.

Stay tuned.

A pick of the best Wordpress plugins

alz — Sat, 07 Apr 2007 23:03:46 +0000

Having been rediculously busy with various things, it has been rather a while since my last post, so here’s one that’s been sitting as a draft for ages, hopefully some of you will find it useful.

Wordpress Plugins - what’s the point?

Wordpress is a great publishing platform from the outset, but Matt also built in a great architecture for coders to expand the system to do all sorts of other neat stuff. I know of some blogs which use tens of plugins to make their site work, but at the moment I only use 4 main ones. here’s a quick rundown of what they do, and why I use them.

Akismet - stopping comment spam

If you haven’t already got some sort of anti-spam plugin installed on your blog, it’s more than likely you are getting lots of comments submitted which seem like utter junk, or just blatant spam. This is where Akismet comes in. This plugin actually comes preinstalled with Wordpress, but you need to signup to Wordpress.com and get an API Key to use it.

Since I started making more regular posts, the amount of comment spam I received has just been rediculous, and Akismet has successfully sorted all that out for me. Make sure you have this turned on, seriously.

Feedburner Feed Replacement

As I mentioned in my redirecting your Wordpress feed to Feedburner post, this plugin provides you an easy way of redirecting all subscribers to your feed through the Feedburner service, allowing you to accurately keep track of reader statistics.

Name Dropper - let your visitors publicise your posts

The name dropper plugin gives bloggers a simple way to allow their visitors to submit their blog posts to a number of social bookmarking sites, including StumbleUpon, Digg, Del.icio.us and many others. It’s well worth installing it - you can see it in use at the bottom of every post here.

Optimal Title - fix your titles for SEO

Optimal Title replaces Wordpress’ wp_title() function and just moves the seperator to after the title, rather than before it. This means the title of your post will be picked up by search engines, BEFORE the name of your blog. That way visitors will more easily find their way to your posts directly.

Google Sitemaps - helping Google spider your site

Having an accurate sitemap of your site can often appear to be more trouble to setup than it’s worth, but thanks to this plugin, the task no longer needs to be a tough one. Just install the plugin and hit Activate, all the rest is done for you. See the sitemap it generates for this site here.

Twitter Tools - simple integration to Twitter

My last post was all about the Twitter service. This plugin allows you to simply integrate with their API to view and submit your latest tweets. Seen here in the sidebar on the right.

So, what is this Twitter thing?

alz — Tue, 27 Mar 2007 01:40:18 +0000

Recently I’ve noticed more and more posts and articles relating to this “new” Web 2.0 service called Twitter. It’s been around for almost a year now (it was born on March 2006 I think), but it mainly took off in the US and hasn’t really appealed to the UK market much yet. What is it anyway?

Put simply - Twitter is a social networking site which allows it’s members to post updates of what they’re currently doing to each other - in 140 characters or less (basically like an SMS). At first glance, most people are going to think “What the hell is the point in a service which just lets me tell my friends what I’m currently doing?.” Well, it’s a good argument. A fair amount of people will disregard the service and not care - there are a few people who are familar with something similar already though. If you’ve got an account on Facebook, chances are you’ve updated your status at least once. This is what Twitter does best, except streamlined and linked to your IM client and mobile phone.

The advantage of the service being linked to your mobile phone instantly makes it easy to post your own little messages, co-ordinate when you’re going out, or even submit micro blog posts, wherever you are - especially for those moments when you don’t have a piece of paper to hand, or just want to quickly rant about something.

As Twitter seems to be steadily growing still, I thought I’d give it a look. If you’ve been reading this page recently you might have noticed the “Twitter feed” column in the right sidebar. This little feed comes directly from my Twitter account and is updated every 30 minutes - thanks to Alex King’s Twitter Tools plugin for Wordpress. As I’ve only been playing with it briefly there isn’t much going on in my feed. Maybe I’ll start updating it more frequently once I get suitably addicted to it.

Of course, being a Web 2.0 service, Twitter have published a handy API enabling people to plug all sorts of applications into their service (like the Twitter Tools plugin for Wordpress). There are some quirky amusing ones, such as Twittervision, which gives you a real-time feed of the latest Twitter “tweets” from the public feed. It’s quite amusing for the first few minutes, but you can get bored of it fairly quickly.

Some of the other people to plug into the service include BBC News, who send headline updates to the site - great if you like to keep track of breaking news. They also have a load of other Twitter feeds you can keep track of.

I have to say though, the most useful thing I’ve found on the service so far is the Twitter Tube Tracker by Tom Morris - it lets me get an SMS update of any delays on the London Underground so I don’t have the joyous pleasure of arriving at the station and finding the line is suspended.

There are a whole load of other ‘Twitterized’ services to play with at the Twitter Fan Wiki. If you’ve got a Twitter account, do check out the fan wiki - you might find something you like. Why not add me to your friends too.