Menu

Today’s post is a light hearted story about two common playgrounds for SEO tomfoolery; Source Code and Robots.txt. We’re going to look at some of the most creative uses of both and how, in some cases, they’ve led to some successful link acquisition!

Robots dot what?

Okay, so let’s first cover off exactly what we mean by robots.txt and not just assume everyone knows what it is. Google’s explanation of a robots.txt file is pretty detailed.

“A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)”

The robots.txt file is essentially the polite gatekeeper of your website who deals with bots requesting access. Unfortunately, polite gatekeepers can be bypassed more easily and this is quite often the case when it comes to different tools and crawlers sidestepping this particular minder.

Sorting the Wheat from the Chaff

One of the best uses of robots.txt we’ve seen for some time was by the MailOnline. Robots.txt is something often observed by SEOs and the bods at the MailOnline had obviously decided that they wanted to hire someone who had real SEO chops. Naturally, they thought the best way to find such a person was to advertise the job opportunity within their robots.txt file.

Daily Mail Robots.txt Job Advert

The Daily Mail take a different approach to recruitment.

Not only did the MailOnline sort the wheat from the chaff in terms of SEO applicants, they also bagged themselves a few links on the way.

Number of Links Linking Root Domains
142 52

Obviously a few decent domains from the Tech and SEO industries;

  • http://searchengineland.com
  • http://www.malcolmcoles.co.uk
  • http://www.switched.com

All in all, a good effort!

 

Good Advice

Our second example is from Last FM, which instructed its robots.txt file to carry out some pretty essential orders. http://www.lastfm.com/robots.txt.

Last FM Robots.txt Snippet

Last FM making sure their robots follow orders!

Unfortunately, these directories don’t seem to exist on the domain, and though it’s definitely good advice for any robot to heed, this effort didn’t receive much in the way of link love.

Robot Revolution

YouTube has joined the robot revolution with some jovial attempts of its own. Deserved or not, robots seem to have a bad reputation and there’s a slight Terminator feel to this one. https://www.youtube.com/robots.txt

YouTube Robots.txt

YouTube Terminator Robots.txt

Links wise we’re looking at the following;

Number of Links Linking Root Domains
26 26

 

Source Code: HTML Playground

It’s not just robots.txt that gets used as a developer/SEO playground; there are plenty of opportunities to use source code to show some creativity – and again, generate some quality links. The following is a selection of domains that have done just that.

Spot the Wren

Wren Living decided to take their branding to the next by adding their beloved Wren to their source code.

Wren Source Code

The Mighty Wren

 

Even the US President is doing it…

The World’s most important man also recognises the potential link opportunities from this HTML trick. You’ll also notice the not-so-subtle use of only three letters in this particular example. http://www.barackobama.com

Barack Obama Logo Source Code

The President’s Source Code

 

Over to You…

Have you seen any other brands or personalities dabbling in their robots.txt or source code? If so let us know in the comments below – can anyone beat the MailOnline’s haul of links for their job advertisement?

Comments

Loading Facebook Comments ...

Newcastle Upon Tyne

Mediaworks online marketing ltd.
Princesway North
Team Valley Trading Estate
Tyne & Wear, NE11 0NF
United Kingdom
+44 (0)191 404 0100
6309397

London

Mediaworks online marketing ltd.
64 Great Eastern Street
Shoreditch
London, EC2A 3QR
United Kingdom

Newsletter Sign Up