March 21 2014
Robots in Disguise: 5 Creative Uses of Robots.txt and Source Code
Today’s post is a light-hearted story about two common playgrounds for SEO tomfoolery; Source Code and Robots.txt. We’re going to look at some of the most creative uses of both and how, in some cases, they’ve led to some successful link acquisition!
Robots dot what?
Okay, so let’s first cover off exactly what we mean by robots.txt and not just assume everyone knows what it is. Google’s explanation of a robots.txt file is pretty detailed.
“A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password-protecting confidential information.)”
The robots.txt file is essentially the polite gatekeeper of your website that deals with bots requesting access. Unfortunately, polite gatekeepers can be bypassed more easily and this is quite often the case when it comes to different tools and crawlers sidestepping this particular minder.
Sorting the Wheat from the Chaff
One of the best uses of robots.txt we’ve seen for some time was by the MailOnline. Robots.txt is something often observed by SEOs and the bods at the MailOnline had obviously decided that they wanted to hire someone who had real SEO chops. Naturally, they thought the best way to find such a person was to advertise the job opportunity within their robots.txt file.
The Daily Mail take a different approach to recruitment.
Not only did the MailOnline sort the wheat from the chaff in terms of SEO applicants, they also bagged themselves a few links on the way.
Number of Links: 142
Linking Root Domains: 52
Obviously a few decent domains from the Tech and SEO industries;
All in all, a good effort!
Good Advice
Our second example is from Last FM, which instructed its robots.txt file to carry out some pretty essential orders. http://www.lastfm.com/robots.txt.
Last FM making sure their robots follow orders!
Unfortunately, these directories don’t seem to exist on the domain, and though it’s definitely good advice for any robot to heed, this effort didn’t receive much in the way of link love.
Robot Revolution
YouTube has joined the robot revolution with some jovial attempts of its own. Deserved or not, robots seem to have a bad reputation and there’s a slight Terminator feel to this one. https://www.youtube.com/robots.txt
YouTube Terminator Robots.txt
Links-wise we’re looking at the following;
Number of Links: 26
Linking Root Domains: 26
Source Code: HTML Playground
It’s not just robots.txt that gets used as a developer/SEO playground; there are plenty of opportunities to use source code to show some creativity – and again, generate some quality links. The following is a selection of domains that have done just that.
Spot the Wren
Wren Living decided to take their branding to the next by adding their beloved Wren to their source code.
The Mighty Wren
Even the US President is doing it…
The World’s most important man also recognises the potential link opportunities from this HTML trick. You’ll also notice the not-so-subtle use of only three letters in this particular example. http://www.barackobama.com
The President’s Source Code
Over to You…
Have you seen any other brands or personalities dabbling in their robots.txt or source code? If so let us know in the comments below – can anyone beat the MailOnline’s haul of links for their job advertisement?
