Google Adsense Sucks!

Have you seen their new scrolling ad format? Thanks Google! Now I can spend more time cycling through adverts I didn’t want in the first place, just because you can’t do your job and display relevant adverts to begin with.

And in other news today, ITV have decided they will give users the chance to watch more adverts at the click of a red button, rather than watch the film they sat down to watch.

Cloaking! The definitive guide (Part 2 - IP delivery)

User-agent cloaking sucks! Take the code from cloaking part one, dissect it, understand it then hit that delete button. User-agent cloaking is waaaaaay too easily spoofed by anybody who wants to spoof it.

IP cloaking

Real men use IP based cloaking. That means we are going to need a list again. This time we have 3 choices; IPLists.com, Fantomaster’s SpiderSpy Database, or you can roll your own list. So which should you choose?

The IP Lists

  • IPLists.com: Only recommended if you are just experimenting a little with cloaking. Expect to get banned.

Advantages
It’s free
It’s easily accessible and parsable

Disadvantages
It’s rarely updated these days
It isn’t as comprehensive as it needs to be
You’re going to get banned lots if you base your system around this list

  • SpiderSpy: My recommendation. Ralph has done an amazing job at putting together the best damn bot list out there. Updated every 6 hours. If you get banned whilst using this list, it’s probably a manual review.

Advantages
It’s the most comprehensive list around
You can download it in a variety of formats for easy parsing
You can get support from Ralph

Disadvantages
It costs money (though the value for money as far as I’m concerned is great)

  • Roll your own: If you are skilled enough to roll your own, you probably aren’t going to learn much from this series of posts.

Advantages
It’s free
You get full control over how you identify robots

Disadvantages
You need to be a shit-hot coder
It will take time to build up a list of bots
Even after all that effort, I doubt your list would be as good as the SpiderSpy list

So now you’ve picked your method of creating a spider list, used a few instances of explode() to parse it and then put it into your database (MySQL of course). Lets assume it looks something like this:

ip botname
123.45.67.89 Google
98.76.543.21 Yahoo

You only really need the first row for cloaking, but it can be handy to have know which IP is attributed to which search engine for stats. My philosophy when it comes to data collection is a rather greedy one. I collect all the data I can, then decide what I’m going to do with it at a later date. This approach has allowed me to learn a lot about bot behaviour just by looking at my raw stats in phpMyAdmin. I’m sure if I started graphing the data and applying rules to it I could learn even more.

IP cloaking with PHP

Let’s write our first IP based cloaking script.

< ?
// Lets connect to MySQL
mysql_connect('localhost', 'username', 'password');
mysql_select_db('database');
// First we grab the visitors IP
$visitorIP = $_SERVER['REMOTE_ADDR'];
// Then we search the database for the IP
$result = mysql_query("SELECT * FROM table WHERE ip = '$visitorIP'");
$dbip = mysql_fetch_array($result);
$theip = $dbip['0'];
// We check if there is an instance
if(strlen($theip) &gt; 0) {
echo 'uber SE content';
}else{
echo 'standard page'
}
?>

The problem of potential spoofing has now gone, you can’t spoof an IP (well, to my knowledge you can’t). So only bots can see my content now, right? Wrong!

When bots crawl your page they store a copy of the page that they see in the cache. Which means somebody could just come along, click that lovely cache button and see your spammy highly optimised content. We don’t want to make it easy for these snoopers, so lets stop them seeing our content. Really you’ve got 2 options when it comes down to it.

Cache Busting?

  1. Tell Googlebot where to shove it’s cached page: Basically include <meta name=”ROBOTS” content=”NOARCHIVE”> in the <head></head> section of your page and the cache is no more! The problem with this approach is that it is a potential red flag to the search engines and sometimes they ignore the tag anyway.
  2. Pass a naughty little bit of Javascript. Let the user visit the page then pass some javascript that you stealthly stuffed into your page when the bot visited last. Just stick in this code:
url="http://www.yourdomain.com/target.html";
if (top.location != url) {top.location = url}

(You’ll have to visit Fantomaster’s page to get the actual code as this has no javascript tags. Wordpress was having none of it.)
That beautiful piece of code was stolen borrowed from Fantomaster and will redirect anybody trying to look at the cache to your actual site. So now we’ve locked out the snoopers. ..Unless they disable javascript when they view your cache. Now, I’m not one to be beaten so easily by the self appointed internet police who have nothing better to do than run around reporting sites to Google without being paid for their time. So take the source code from your uncloaked page, find the most cluttered part of your source code, then put in your optimised content. Now put in a div and add this to the div class:

text-indent: -12000px;

This will shift all the text 12,000 pixels to the left. Preferably hide that in the most cluttered part of your css and try and confuse any snoopers a bit. In part 3 I will tell you how to use cloaking for…

…Super Targeting

Cloaking! The definitive guide (Part 1 - An introduction)

I love cloaking! I think it’s brilliant. I love the fact I can create the perfect user experience whilst also presenting a perfectly optimised page to the search engines. That’s right, cloaking isn’t evil or immoral (at least it isn’t when done correctly) it can really help the user to find what they want.

Now, there is a grey area when defining cloaking. For instance; is “display:none” cloaking? To me it isn’t. Cloaking to me is delivering a page to a search engine where the source code is different from the page you deliver to the user.

A simple cloak

First we are going to start with a really simple form of cloaking - user-agent cloaking. So, we are going to need user-agents for the 3 major engines. So here they are:

  • Google: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Yahoo: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  • MSN/Live: msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)

Those are the main useragents for the three engines. They do use other variations, but I’m not going to go into detail with these. Lets knock up a script to do some user-agent cloaking.

< ?php
// First we need to collect the visitors user-agent
$visitorUserAgent = $_SERVER['HTTP_USER_AGENT'];
// Next we need to check the user-agent for occurrences
// of the words google or yahoo or msn
if(stristr($visitorUserAgent, 'google')
|| stristr($visitorUserAgent, 'yahoo')
|| stristr($visitorUserAgent, 'msn'))
{
// Now show the bot the page need to see
echo 'super, dooper, ultra optimised page';
}else{
// And show the user the bog standard web page
echo 'standard user page';
}
?>

So I can makez a pages rankz for ‘bowling balls’ but shows the user Viagrazzz?!!1!!1one!1!!!?

Sure if you really want to. But why oh why would you want to do that? That is an example of crap cloaking. If a user is looking for bowling balls that is what he wants; he doesn’t want some of your blue pills. I will go into more detail with regards to targeting later in the guide.

So that is the end of part one. It may be a bit thin on content for many, but it was only suppose to be an introduction to the world of cloaking. There are much more exciting things coming like how to trick them there pesky manual review monkeys.

In part two we will explore how to make…

…A stronger cloak

What Can Internet Marketers Learn From The X Factor?

After reading ShoeMoney’s ‘Things internet marketers can learn from…’ series of posts I decided to do my own.

After watching last night’s X Factor final, I realised how powerful the marketing of that show is. For anybody who doesn’t know what X Factor is all about, it’s a talent show where the public votes for their favourite talent, thus whittling down the contestants to just one talent who is promptly taken to market with a £1 million record deal.

So with that sorted, lets start the learning :)

Multiple streams of income

The makers of The X Factor could have taken the easy route and assumed as it’s a television show, the best way to monetise it is by selling the advertising spots in the breaks. Whilst they did capitalise on the advertising spots and made sure they gave the viewer a good chance to see the adverts, they also saw the bigger picture and took every opportunity to monetise the situation. Along with the TV ads, they also charged for voting via phone lines, sold advertising space on their (highly involving) website, charged for tickets to the live show, sold merchandise and I would be surprised if they don’t get a cut of the Christmas song soon to be released by the winner (Leon Jackson).

Empower and involve your audience

Ever trawled gaming forums? If you have, I’m sure you will have seen what are known as ‘fanboys’ (and ‘fangirls’ of course). These people will do anything to defend and promote their cause, and will of course jump at any chance when their vote will influence the outcome of a competition/debate involving their cause. The X Factor used this belief of the audience in their favourite talent as a way of monetisation. This idea of dedicated fans can also be applied to sites themselves, not just monetisation and interactivity within the site. If you provide a great service that people believe in, you can potentially build your own fan club without intending to. This will make it much harder for a competitor to move into your niche.

Celebrities don’t have to be expensive

In the final of The X Factor, Kylie Minogue made a guest appearance as the duet partner for Leon. You could argue that her sister, Dannii Minogue persuaded her to do it to help Leon to victory (Leon was in her category), but the fact that Kylie then went on and so kindly gave the audience a taste of her new song suggests that she was in it more for the publicity. There was exposure for Kylie’s new song in exchange for Kylie helping to raise the profile of The X Factor. The fact is celebrities need exposure or their fame fades away. If there is a particular celebrity or well known person within your niche, you may be surprised what you can offer them in exchange for the exposure they need.

Got customers?

I can almost guarantee that the UK Christmas number one will be Leon Jackson - When You Believe. Why? Is it because it’s a great song? Well partly, it is a very good song, but even if it were a mediocre song it would do well in the charts. The reason for that is The X Factor has a targeted and motivated customer base before they have even released the song. This can be compared to having a good mailing list or a lot of rss subscribers. Remember, it’s easier to keep hold of existing customers than find new ones.

Keeping an open mind

Sometimes a judge may not instantly like a contestant at the audition phase, but that contestant may go on to be extremely strong in the competition and vice versa, sometimes the judges may unanimously agree in the audition phase that a contestant is right for the competition and that contestant goes on to be a disappointment. The message in this is not to pin all your hopes on one strategy or website. Businesses rarely take the exact path they originally planned; be prepared to adapt to what your customers want and don’t get too attached to one strategy. This ties in with the multiple streams of income point.

YouTube overlay ads because of partner pressure

Could YouTube’s latest decision to show overlay ads have been forced by partners such as Universal?

I thought I would check out Universals numbers. Universal have been uploading music videos to youtube for 8 months. In that time they have uploaded 4211 videos and have achieved almost 330 million views.

First off, just so you know what I am talking about, you need to look at a partner page. So check out this shizzle with the D.O double gizzle. Instead of the standard layout, they get a Google ad along with one of their own ads and a “More Videos From This Channel” box.

Now Lets do the maths for these Google Ads. I’ll be generous and assume YouTube has a CTR of 1% on their partners pages, which equates to almost 3.3 million clicks over 8 months. Now say we assumed that each click was $0.10. That brings the total revenue from Universal’s partner page to almost $330,000 over the 8 months. Now not forgetting that the revenue has to be split between Universal and YouTube. A 65/35 split YouTube’s way is likely, which puts Universal’s revenue from YouTube at $115,500, or less than $15,000 per month.

That is just my estimate, other sources suggest that the situation might be much worse. On the Plentyoffish blog, Markus says that just over a year ago, he was paying just 30 cents cpm for ads on Youtube. Using that estimate Universal could expect to see just $34,000, or $4,300 per month.

$1 per video per month!!

I doubt they were the sorts of numbers Universal had in their head 8 months ago after deciding to partner with Google instead of sueing them.

So there is basically 2 ways for google to hold on to Universal and their other partners.

  1. Share more revenue with the partners and keep less for themselves.
  2. Maintain the same share percentage and increase the total revenue.

I think we all know which one makes more sense.

Whilst Google have probably always had this overlay idea in their mind and had always planned to implement it, if my guestimations are anywhere near accurate, then Google’s partners probably are having a big influence on Google’s decision.

If my maths is way off, then please let me know with a comment and tell me what an idiot I am.

SQL file too big for phpMyAdmin? No problem!

The other day I was confronted with a problem, a 109MB SQL file. The problem was (and still is) phpMyAdmin won’t accept anything bigger than 51,200KB and my file was over twice that.

I was about to go hunting for the shell command to import my file, but I stopped and thought, “Somebody must have created something in PHP to do this”.

And there was my answer, Big Dump!

Yeah the name conjured up some strange images in my head too…

poo-28215.jpg

…But anyway.

It’s just one PHP file that took me 2 seconds to upload. Just point your browser at the file, big dump reads all the files in the directory and determines whether it is a file that can can be imported into MySQL. You just click import and you can enjoy the import process with a progress bar (something that phpMyAdmin could benefit from).

The tool was created by Alexey Ozerov and you can download it at http://www.ozerov.de/bigdump.php