If you have landed on this robots seo page be sure to not miss the start of the SEO basics tutorials Here.
By robots, I mean Robots SEO file, not a robot made from text on the right, I just want to make that clear.
This is a file that is placed in the root directory of your website, it tells search engine spiders where and where not to go to index your pages.
You can also use it to tell a search engine where your site map file is.
A “robots.txt” file tells search engines whether they can access and therefore crawl parts of your site.
You really do not want search engines indexing areas of your site that are not useful to your users.Â And believe it or not also using your servers resources.
Restrict crawling where it’s not needed with robots.txt. You can target specific search engines with examples below of User-agent:
User-agent: * //This is general and standard for all engines, * is a wildcard
User-agent: Googlebot-image //Targets just Googles Image engine
User-agent: Googlebot //Targets Google
User-agent: googlebot-image //Very important if you want your products in Google Shopping
Sitemap: http://www.your-site/site-map/sitemap.xml //Tells the engines where your sitemap is.Links
Not all search engine spiders obey your robots.txt file so it is not a total solution but it is a must to put in place for all the main search engines.
I have tried out this robot file generator and it works fine, there are lots more available.
SEO Basics Part 6 Coming Soon…