The key to a successful and focused SEO campaign is undoubtedly making the search engines aware of what you don’t want them to crawl on your website. Whether it be entire directories or external links there are ways and means of keeping the power circulating, this is done by robots.txt files, htaccess files, nofollow links and noindex functions.
A robots.txt file is a simple notepad file on the root of your domain, this is also known as an exclusion file and is the place to locate any directories that do not require spidering. Admin, image, pdf and various other folders can be huge in terms of size, and therefore will waste valuable crawling time. This will ensure the pages with potential are noticed and indexed more thoroughly.
The htaccess file is a file that directs one page to another page, a good example of this is when you have a homepage named www.seoconsult.co.uk/ and another called www.seoconsult.co.uk/index.htm, by using the htaccess effectively you can direct one to the other, thus creating one page of combined power.
Nofollow and noindex are functions that can be placed within the a tag. The nofollow will state that any page authority that the link is situated on will not be passed on, improving the strength of the other links that have not been nofollowed. This is useful when linking to external links, keeping the strength within your own site. Noindex is essentially an onpage exclusion function, stating to the search engines which links have pages that are not meant for indexing.
Related posts:
- How robots.txt Can Affect Search Engine Optimisation Strategy
- Avoid messing up your robots.txt
- Duplicate content
- URL Rewrites and Redirects
- What makes a robot undesirable?
Link to us
If you want to link to this blog, copy and paste the following HTML code to your website.
[...] put it plainly yet confusing the hell out of – search engines use ’spiders’ or ‘Robots‘ (actually programs) that ‘crawl’ their way through the websites of the world [...]