|What is Robots.txt file and how to optimize for WordPress blog : eAskme|
WordPress robots fileUsually people get confuse about how to use Robots.txt, so we will answer it here.
Robots.txt is one of the hundreds of factors of SEO. By adding this small file you can seriously optimize your Website. usually blogger don`t try to edit it, but its an easy process. You just need to knwo some basic and you can easily edit Robots file, if you are new to this concept then read this post throughly.
Learn how to add Robots.txt file and how to optimize Robots.txt file.
What is WordPress Robots.txt and why to use it?Robots.txt file always help search engines to know which part of site they should crawl and which part they should avoid. When Search engine spider of bot comes to your blog to index your blog, first of all they follow Robots.txt file.
How to create robots.txt file?Robots.txt file is a text file. You can simply create this file on notepad and then upload to root directory for your site. Now let`s discuss records of Robots.txt:
These above lines shows that Google is allowed here to crawl and index site but But it is not allowing cgi-bin folder for indexing. This means Goggle will never index /cgi-bin directory.
Disallow option help you to stop any search engine to indexing party of your site. Usually sites use Disallow for archive folders and searches.
Where Can You Get names of Search bot?You can easily find names of search bots on your site directory, But I recommend that you should allow every search bot. for this purpose you should write User-agent: *. Let`s see an example:
This allow all search engines to index your site.
What You Should not do on Robots.txt file?Never use comments in Robots.txt file.
Never make ordinary space in file and never keep space in the beginning of any line. Now see some good Robots.txt and bad Robots.txt practices. Example:
- Never change rules of command.
- Don`t write names of directories together to disallow:
- Use small and capital letters properly.
- To index everything of your site:
- No to index anything of from your site:
Robots.Txt for WordPress:On wordpress you can use Robots meta or you can edit Robots.txt file by logging in FTP account. You should also add your sitemap URL in robots.txt file. It will help search engines to find your sitemap and index faster.
See below example:
How to check no content is affected by new Robots.txt file?After making changes in Robots.txt file now you need to check the impact by updating robots.txt file. Use Google Webmaster tool "Fetch as bot tool" to check if you can access your content with Robots.txt or not.
Also check for the crawl errors under Crawl error. Select Restricted by Robots.txt to see what are the links denied by Robots.txt on your blog.