There are different methods to prevent a page from being indexed by Google or other search engines. From the configuration of the disallow command in the robots.txt file to the use of the nofollow tag in links, to the use of the HTML Noindex meta tag.
With the noindex meta tag it is possible to avoid the indexation of the page itself by adding an HTML code to it, not being necessary to be using other external methods to avoid its indexation.
What is the noindex meta tag?
The noindex meta tag is a tag that is inserted in the HTML code of a page to indicate to Google and other search engines that it should not index the page.
The main feature of this meta tag is that it does not block access to it from search engines (as does the nofollow tag or the robots.txt disallow command), but it does prevent it from being indexed in their website rankings.
The use of the follow and nofollow guidelines enhances the use of the meta noindex, allowing the indexing and crawling block to be adjusted according to the needs of each page.
¿Por qué es importante para el SEO?
The noindex meta tag is a very useful tool for SEO because it allows controlling some of the aspects related to content indexing such as:
- Do not index pages with low-quality content that you do not intend to remove or improve (thin content).
- Avoid indexing category pages with little value for the user
- Do not index pages with low-quality content that you do not intend to remove or improve (thin content).
- Avoid duplicate content by not indexing the page with the lowest ranking strength.
- Avoid indexing internal search results.
- Avoid indexing internal search results.
- Avoid indexing internal search results.
- Allows Googlebots to crawl all links on a page, without the need for indexing.
When performing an SEO audit it is common to find important pages of a website that are not even indexed in Google, most of the time for not having a correct configuration of the noindex meta tag or other tools such as the robots.txt file.
There are tools that allow you to analyze a website to see if its URLs have this meta tag activated and are blocking its indexation, as is the case of Screaming SEO Frog Spider. Using this application you can crawl a website as if it were a search engine bot, subsequently accessing a lot of important information.
In the directives tab of the application (directives), you can filter the results to see which pages are blocked from being indexed with the noindex and nofollow meta.
How to put a page in noindex?
Let’s see how the noindex tag is implemented on a page to avoid indexing by search engines.
1. Setting noindex in HTML
This is an HTML meta tag that must be included in the section or header of the code. This code is very simple, unlike other HTML tags and parameters.
It must be included in the head of the HTML:
<meta name="robots" content="noindex"> This line prevents any bot, whether Googlebots or other search engine crawlers, from indexing, that page in their site databases.
This code can be customized to refer to a specific bot. For example, if we want Google-image not to index the images of a URL we should include:
<meta name="googlebot" content="noindex"> Another interesting feature of this meta tag is that several meta tags can be inserted on the same page to perform custom indexing constraints.
2. Setting noindex in WordPress
As is often the case, in WordPress it is even easier to work with the noindex meta tag. With the use of plugins such as Yoast SEO or Rank Math SEO, just go to the page or blog post you want to avoid indexing, and check the box called No Index. This way, the plugin automatically adds the noindex meta tag to the HTML of the page.
Follow and nofollow directives accompanying noindex
This meta tag allows an even greater degree of customization, being used with the follow and nofollow attributes depending on the objective that has been set.
noindex, follow
By using this combination you are telling search engines that they should not index the page in their database, but they should crawl all the links included in it.
This is one of the most common SEO uses of this meta tag. The way to add it in HTML is the following:
<meta name="robots" content="noindex,follow"> A very simple process that only requires adding to noindex, the follow attribute separated by a comma.
noindex, nofollow
This combination of noindex and nofollow prevents search engines from indexing the website and also prevents them from crawling the links it contains. To implement it, the same simple and easy process is followed as in the previous case of combining noindex with follow.
<meta name="robots" content="noindex,nofollow"> Common errors using the noindex tag
When using this type of meta tag, many mistakes are made, many of them marked by Google Search Console itself when it detects them.
Among the main mistakes when implementing noindex tags on a page we have:
Sitemaps with URLs containing the noindex tag
It is common that when the sitemaps file is built with the list of URLs of the site to be indexed, those pages that have the noindex meta tag are not taken into account. This nonsense situation (a URL is sent through the sitemap that is marked with a tag not to be indexed) is detected by Google Search Console that marks it as an error.
The solution to this problem is quite simple and consists of either removing the URL from the sitemap or removing the noindex tag itself from that page if you really want it to be indexed.
Forget about the noindex tag
Although it may seem a somewhat absurd error, it happens on a regular basis. For a certain reason this meta tag is used to prevent content from being indexed in Google, but when the problem or reason is solved, it is forgotten that the tag is still active preventing indexing, so the page will still not appear in Google search results.
To solve this type of error the best thing to do is to always have a good organization, noting those pages that are not indexed with this tag (as well as those that are included in the robots.txt or use the nofollow tag in the links).
The noindex meta tag is a very interesting alternative to control which content to index and which to leave out of the Google database and other search engines.
It is an interesting alternative because it is able to block indexing, but it allows the bots to crawl the links contained in the page, something that is not possible with other alternatives such as the nofollow links tag, for example.
With the combined use of noindex metatag, the robots.txt file configuration, and the use of the HTML tag for nofollow and dofollow links, you can control the indexing and crawl of the different URLs that make up a website. They are ideal SEO tools to optimize the time available to the bots to crawl a site (crawl budget), allowing the most important content to be crawled, indexed and positioned correctly.
