Search Engine Optimisation (SEO) Specialists

How robots.txt Can Affect Search Engine Optimisation Strategy

The search engine optimisation of a site involves a lot of different things. Some of these are very technical aspects, while others of them are substantially more creative. The site’s architecture, code and content all have a very important role to play in where the search engines place its link in the search engine results pages.

In the rush to use all available resources of a site, some things can be forgotten. It’s not always necessary or even desirable to show the search engines everything you’ve got. Sometimes, it’s a good idea to keep some files clear of the search engine spiders. The files you designate in your site’s robots.txt are a part of this. The whole subject of robots.txt often gets overlooked in SEO explanations because it sounds too technical for the average person to understand. It is a technical subject, but robots.txt is something it’s important for every site owner to have some knowledge of.

To begin with, robots.txt file can be accessed by typing in domainname.co.uk/robots.txt, substituting the domain of the site in the first half of the address. This will list the directories of the site that the webmaster wants search engines to disallow. This means that the search engine spiders will skip over the information contained in these files.

Robots.txt files can come in very handy beyond the search engines. It is possible to use robots.txt files to protect your site from malevolent web crawlers, which is useful to say the least. However, robots.txt files need to be handled carefully, as it is easy to make mistakes within these tiny little text files. Just because robots.txt files can be used to make the search engines overlook information, it doesn’t mean that the information is invisible. Many site owners make the mistake of including confidential information within a robots.txt file, which can lead to disaster if it is sampled by a malbot. Robots.txt files are also easily visible to the site’s users. Site owners also need to be careful when editing robots.txt files. The disallow feature within a robots.txt file is easy to write, and just as easy to muck up. The difference between the search engines only crawling desired files in your site and failing to crawl the site completely can be a simple ‘/’.

It’s important to note that robots.txt should not be used when swapping pages on your site. Some companies make the mistake of listing old pages in their robots.txt file when the changeover occurs. This can result in any old indexing power being lost. A simple 301 redirect should be used when you want to pass the search power of your old pages onto new ones.

Your robots.txt files can be useful in terms of SEO, preventing duplicate content and directing search engine attention to the important pages of your site. The understanding of proper use of robots.txt is one of the many reasons why it is usually worthwhile consulting an expert when approaching your site’s SEO. You can talk to our experts at SEO Consult for further advice.

Related posts:

  1. Avoid messing up your robots.txt
  2. Tell the Search Engines what you want them to see
  3. What makes a robot undesirable?
  4. SEO For On-Page Objects
  5. Optimising Your Code For Quick Page Load Times

Tags: , , , ,

Link to us

If you want to link to this blog, copy and paste the following HTML code to your website.

Leave a Reply

Search Blogs

Highest Rated Blogs

Tag Cloud

Recent Blogs

Blog Categories

About SEO
ASK SEO
Bing and Search Engine Optimisation
Black and White Hat SEO
Browser Software and Search Engine Optimisation
Cheap SEO
Clicktelligence & SEO
CMS and Admin Systems and SEO
eCommerce and Search Engine Optimisation
Email Marketing
Ethical Search Engine Optimisation
Flash and SEO
General SEO News
Geo Targeting Search Engine Optimisation
Google Analytics
Google and Search Engine Optimisation
Google PageRank
Google PR Update
Google Webmaster Tools
Increase Search Traffic with SEO
International SEO
Internet Marketing
Keyword Research and Search Engine Optimisation
Latest News
Link Campaigns and Search Engine Optimisation
Meta Tags and SEO
Mobile SEO
Offpage Optimisation
Online Advertising
Onpage Optimisation
Page Load and SEO
Page Rank Update
Pay Per Click & SEO
Press Releases
Professional SEO
Real Time Search
Search Engine Indexing and Crawling
Search Engine Optimisation
Search Engine Optimisation Advice
Search Engine Optimisation Analysis
Search Engine Optimization
Search Engine Optimization Keywords
Search Engine Optimization Marketing
SEO Analysis for Results
SEO and Business
SEO and Search Engines
SEO Blogging
SEO Companies
SEO Consult
seo consultancy
SEO Copywriting
SEO Costs
SEO Domain Names
SEO Experiment
SEO Links
SEO Lists
SEO Management
SEO Penalties
SEO Rankings
SEO Sitemaps
SEO Strategy
SEO Submissions
SEO Techniques
SEO Tips
SEO Tools
seo video
SEO vs PPC
Site Analysis and Search Engine Optimisation
Site Architecture & SEO
Social Media Optimisation
Social Networking and Search Engine Optimization
Target Marketing with SEO
Top SEO Tips
Traffic & SEO
Twitter and Search Engine Optimisation
User Driven SEO
W3C and Search Engine Optimisation
Web Design
Website Development
Website Optimization
Wordpress For SEO
Yahoo! and Search Engine Optimisation
View All Categories

Archives

Authors

W3C XHTML Valid W3C CSS Valid W3C WAI  Internet Marketing RSS Feed for SEO Consult