Robots SEO
If you have landed on this robots seo page be sure to not miss the start of the SEO basics tutorials Here.
By robots, I mean Robots SEO file, not a robot made from text on the right, I just want to make that clear.
This is a file that is placed in the root directory of your website, it tells search engine spiders where and where not to go to index your pages.
You can also use it to tell a search engine where your site map file is.
A “robots.txt” file tells search engines whether they can access and therefore crawl parts of your site.
You really do not want search engines indexing areas of your site that are not useful to your users. And believe it or not also using your servers resources.
Restrict crawling where it’s not needed with robots.txt. You can target specific search engines with examples below of User-agent:
|
1 2 3 4 5 6 7 8 9 10 |
User-agent: * //This is general and standard for all engines, * is a wildcard User-agent: Googlebot-image //Targets just Googles Image engine User-agent: Googlebot //Targets Google User-agent: googlebot-image //Very important if you want your products in Google Shopping Disallow: User-agent: googlebot Disallow: Sitemap: http://www.your-site/site-map/sitemap.xml //Tells the engines where your sitemap is.Links |
Not all search engine spiders obey your robots.txt file so it is not a total solution but it is a must to put in place for all the main search engines.
I have tried out this robot file generator and it works fine, there are lots more available.
SEO Basics Part 6 Coming Soon…
Posts You Might Also Like...
