Taking full control of search engine crawlers is the ultimate nirvana of any SEO expert. Yes, this can be achieved with the help of Robots Meta tag which tells search engines what to do and what not to do. This is similar to an instruction which is sincerely followed by search engine crawlers. Using Robots Meta tag, you can tell search engines that you want it to index only the first page of the website, or you want it to index the entire website except a particular page etc. Robots meta tag is mainly used when a publisher is not allowed to use robots.txt file. A number of options are provided to have full control on crawlers. This is the best way to prevent crawlers from crawling a few pages of your website. The syntax of Robots Meta tag is given as:
<meta name=”robots” content=”selection”>
As any other meta tag, this one is also placed inside the head section of HTML codes. As it is a robot tag, the name is “robots” and the valid values of “content” attribute are: “noindex” ,”nofollow”, “follow”, “index”. The basic meaning of attributes is:
- Index/noindex: determines whether a crawler should index this page or not.
- Follow/nofollow: determines whether a crawler should follow links on that page or not.
The default combination of robots meta tag is content=”index, follow” which tells search engines to index and crawl the links and all the pages of website. The different combinations and their explanations are given as below:
- <meta name=”robots” content=”noindex,nofollow” /> : Disallows indexing and following of current page and links in that page.
- <meta name=”robots” content=”noindex,follow” /> : Disallows indexing but allows crawlers to follow the links.
- <meta name=”robots” content=”index,nofollow” />: Allows crawlers to index the current pages but blocks it from following the links.
- <meta name=”robots” content=”none”>: Short hand method of writing the first one.
- <meta name=”robots” content=”all”>: Short hand method of writing <meta name=”robots” content=”index,follow”/>
Proper utilization of robots meta tags can boost up your rankings in search engines. Utilising them in an optimum way is very important. You would not want to block some of the important pages. Therefore, be selective in your process and choose correctly.
Professional SEO content Writing is not an easy task. Various factors have to be considered in this case. We have made the entire process easy for you by providing an easy and resourceful SEO Content Writing Guide For Professionals.