March 21 2014

Robots in Disguise: 5 Creative Uses of Robots.txt and Source Code

Today’s post is a light-hearted story about two common playgrounds for SEO tomfoolery; Source Code and Robots.txt. We’re going to look at some of the most creative uses of both and how, in some cases, they’ve led to some successful link acquisition!

Robots dot what?

Okay, so let’s first cover off exactly what we mean by robots.txt and not just assume everyone knows what it is. Google’s explanation of a robots.txt file is pretty detailed.

“A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password-protecting confidential information.)”

The robots.txt file is essentially the polite gatekeeper of your website that deals with bots requesting access. Unfortunately, polite gatekeepers can be bypassed more easily and this is quite often the case when it comes to different tools and crawlers sidestepping this particular minder.

Sorting the Wheat from the Chaff

One of the best uses of robots.txt we’ve seen for some time was by the MailOnline. Robots.txt is something often observed by SEOs and the bods at the MailOnline had obviously decided that they wanted to hire someone who had real SEO chops. Naturally, they thought the best way to find such a person was to advertise the job opportunity within their robots.txt file.

The Daily Mail take a different approach to recruitment.

Not only did the MailOnline sort the wheat from the chaff in terms of SEO applicants, they also bagged themselves a few links on the way.

Number of Links: 142

Linking Root Domains: 52

Obviously a few decent domains from the Tech and SEO industries;

All in all, a good effort!


Good Advice

Our second example is from Last FM, which instructed its robots.txt file to carry out some pretty essential orders.

Last FM making sure their robots follow orders!

Unfortunately, these directories don’t seem to exist on the domain, and though it’s definitely good advice for any robot to heed, this effort didn’t receive much in the way of link love.

Robot Revolution

YouTube has joined the robot revolution with some jovial attempts of its own. Deserved or not, robots seem to have a bad reputation and there’s a slight Terminator feel to this one.

YouTube Terminator Robots.txt

Links-wise we’re looking at the following;

Number of Links: 26

Linking Root Domains: 26

Source Code: HTML Playground

It’s not just robots.txt that gets used as a developer/SEO playground; there are plenty of opportunities to use source code to show some creativity – and again, generate some quality links. The following is a selection of domains that have done just that.

Spot the Wren

Wren Living decided to take their branding to the next by adding their beloved Wren to their source code.

The Mighty Wren


Even the US President is doing it…

The World’s most important man also recognises the potential link opportunities from this HTML trick. You’ll also notice the not-so-subtle use of only three letters in this particular example.

The President’s Source Code


Over to You…

Have you seen any other brands or personalities dabbling in their robots.txt or source code? If so let us know in the comments below – can anyone beat the MailOnline’s haul of links for their job advertisement?

You may also like …

July 22 2024

Smart Ways to Integrate SEO And PPC Marketing Strategies

As a leading digital marketing agency, Mediaworks has provided a blog on smart ways to integrate SEO and PPC marketing strategies. Read more here.
July 15 2024

Boosting Impact: Utilising Demand Gen and Performance Max Across the Funnel

Find out more about boosting the impact of your Performance Max campaigns in this blog from Mediaworks. Read more.