March 21, 2014
Share this Article:
Today’s post is a light hearted story about two common playgrounds for SEO tomfoolery; Source Code and Robots.txt. We’re going to look at some of the most creative uses of both and how, in some cases, they’ve led to some successful link acquisition!
Okay, so let’s first cover off exactly what we mean by robots.txt and not just assume everyone knows what it is. Google’s explanation of a robots.txt file is pretty detailed.
“A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)”
The robots.txt file is essentially the polite gatekeeper of your website who deals with bots requesting access. Unfortunately, polite gatekeepers can be bypassed more easily and this is quite often the case when it comes to different tools and crawlers sidestepping this particular minder.
One of the best uses of robots.txt we’ve seen for some time was by the MailOnline. Robots.txt is something often observed by SEOs and the bods at the MailOnline had obviously decided that they wanted to hire someone who had real SEO chops. Naturally, they thought the best way to find such a person was to advertise the job opportunity within their robots.txt file.
The Daily Mail take a different approach to recruitment.
Not only did the MailOnline sort the wheat from the chaff in terms of SEO applicants, they also bagged themselves a few links on the way.
|Number of Links||Linking Root Domains|
Obviously a few decent domains from the Tech and SEO industries;
All in all, a good effort!
Our second example is from Last FM, which instructed its robots.txt file to carry out some pretty essential orders. http://www.lastfm.com/robots.txt.
Last FM making sure their robots follow orders!
Unfortunately, these directories don’t seem to exist on the domain, and though it’s definitely good advice for any robot to heed, this effort didn’t receive much in the way of link love.
YouTube has joined the robot revolution with some jovial attempts of its own. Deserved or not, robots seem to have a bad reputation and there’s a slight Terminator feel to this one. https://www.youtube.com/robots.txt
YouTube Terminator Robots.txt
Links wise we’re looking at the following;
|Number of Links||Linking Root Domains|
It’s not just robots.txt that gets used as a developer/SEO playground; there are plenty of opportunities to use source code to show some creativity – and again, generate some quality links. The following is a selection of domains that have done just that.
Wren Living decided to take their branding to the next by adding their beloved Wren to their source code.
The Mighty Wren
The World’s most important man also recognises the potential link opportunities from this HTML trick. You’ll also notice the not-so-subtle use of only three letters in this particular example. http://www.barackobama.com
The President’s Source Code
Have you seen any other brands or personalities dabbling in their robots.txt or source code? If so let us know in the comments below – can anyone beat the MailOnline’s haul of links for their job advertisement?
From powerful brand propositions, customer-centric experiences or creative search campaigns, our proven approach overcomes obstacles and delivers commercial success.
We work fast and break norms by applying innovation, intelligence and creativity to everything we do. We look for people who share our passion and ambition.
Interested in finding out more about how we deliver outstanding results? Speak to our experts today.