Robots Seo Basics Part 5

Robots SEO back to part 4Robots SEO

If you have landed on this robots seo page be sure to not miss the start of the SEO basics tutorials Here.

Robots txt SEO

By robots, I mean Robots SEO file, not a robot made from text on the right, I just want to make that clear.

This is a file that is placed in the root directory of your website, it tells search engine spiders where and where not to go to index your pages.

You can also use it to tell a search engine where your site map file is.

A “robots.txt” file tells search engines whether they can access and therefore crawl parts of your site.

You really do not want search engines indexing areas of your site that are not useful to your users. And believe it or not also using your servers resources.

Restrict crawling where it’s not needed with robots.txt. You can target specific search engines with examples below of User-agent:

Not all search engine spiders obey your robots.txt file so it is not a total solution but it is a must to put in place for all the main search engines.

I have tried out this robot file generator and it works fine, there are lots more available.

SEO Basics Part 6 Coming Soon…

Posts You Might Also Like...

Leave a Reply

Your email address will not be published. Required fields are marked *