Blogger SEO

Custom Robots Header Tags for Blogger, X-Robots tags

The robots header tags allow you to control how your website should display in search results. You can apply the robots tags using meta tags and X-robots tags in the response header. Blogger allows you to set both types of robots tag. In this article, we will understand the best practice of implementing Robots Header tags to boost the SEO of the Blogger website.

enable-custom-robots-header-tags-blogger

Recently, Blogger made many changes in its interface to improve the user experience. From the SEO point of view, the major update was on new themes. Blogger now supports HTML5 themes that drastically boost the SEO of the webpage. But along with the theme and content, you need to configure the X-Robots-Tag setting properly.

Please do not confuse Robots Header tags with custom robots.txt of Blogger. The robots.txt file serves as a site-wide guide for web spiders, residing in the root directory of a website. It instructs user agents on which sections of the site to crawl and which to avoid. This file applies globally to the entire website.

In contrast, the Robots Header tags are page-specific directives placed within the HTML head section of individual pages. They communicate to web spiders how to index or noindex a particular page, influencing the page’s visibility in search engine results. Unlike robots.txt, which governs the entire site, Robots Header tags operate page-by-page, providing targeted instructions for each webpage.

What is X-Robots-Tag or Robots Header tag and Robots Meta Tag?

X-Robots-Tags or Robots Header Tags: X-Robots-Tags are part of the HTTP header response sent by the server when a web page is accessed. These tags convey page-level settings and directives to search engine bots. Webmasters can control how search engines index their content by including X-Robots tags in the HTTP header. For example, an X-Robots-Tag might specify that a certain page should not be indexed or should have its links followed, influencing the search engine’s behavior for that specific page.

Robots-Tags Meta Tags: On the other hand, they are typically meta tags embedded within the HTML code of individual pages. These meta tags serve a similar purpose to X-Robots-Tags by providing page-level instructions to search engine bots. Webmasters can include meta tags like “noindex” or “nofollow” to specify whether a particular page should be excluded from search engine indexes or if links on that page should not be followed.

DirectiveMeaning
allThere are no restrictions for indexing or serving. Note: this directive is the default value and does not affect if explicitly listed.
noindexDo not show this page in search results.
nofollowDo not follow the links on this page.
noneEquivalent to noindex, nofollow.
noarchiveDo not show a “Cached” link in search results. You have to specify this directive. Otherwise, Google may generate a cached page, and users may access it through the search results.
nosnippetDo not show a text snippet or video preview snippet in the search results for this page.
noodpDo not use metadata from the Open Directory Project for titles or snippets shown for this page.
This tag is no longer supported.
notranslateDo not offer translation of this page in search results.
noimageindexImages on the page will not index.
unavailable_after:
[RFC-850 date/time]
Do not show this page in search results after the specified date/time. The date/time must be specified in the format including, but not limited to, RFC 822, RFC 850, and ISO 8601.
By default, there is no expiration date for content. This tag should be used for content that shouldn’t be visible after a certain date or time.

NOODP Robots Header tag is no longer supported.

We talked about the custom robots header tags available with Blogger. Many websites and YouTube channel guides regarding the robots header tags for the Blogger blog suggest enabling robots header tags noodp for the pages and posts. But that is an entirely wrong practice, then what is right?

noodp robots directive is not used by Google.

Search engines like Google no longer support the noodp tag. So don’t follow the wrong practice. It may spoil the On-Page SEO. I hope that Blogger will soon remove this option from its settings.

Who should enable custom robots header tags in the Blogger blog?

Before making any changes in the Blogger blog settings, we’ve to understand who should enable this custom robots header tag setting and who should not.

Who shouldn’t enable this tag option?

  • If you’re running a simple blog, then no need to enable this feature: This feature allows you to alter the default index status of Blogger posts or pages. The other ways to improve SEO of your blog are –

Who should enable and use these tags?

  • You are an advanced user who understands which pages to index and which not.
  • You are dealing with time-bound content that needs to be removed from search results after a specific time.
  • You have such complex written content that is readable in one language only. If translated, their meaning may be completely different. So you can stop translation on some pages.
  • You don’t want to highlight snippets.

How we enable custom robots header tags?

Log in to the Blogger dashboard to enable these X-robots header tags.

Enable custom robots header tags in Blogger
  1. Open the settings of the blog,
  2. Scroll to crawlers and indexing option,
  3. Switch on the “Enable custom robots header tags” option,
all X-robots settings available with Blogger
  1. Home page tags: Click on this, and the option shown in the above image will appear. You’ve to click on all, and save the settings,
  2. Archive and search page tags: from the options provided above, you’ve to select noindex. 
  3. Posts and Pages type: To index all the pages, you’ve to click on enable “all”. And then save the settings.

As we already discussed, enabling the noodp tag is unnecessary.

Enabling custom robots header tags will allow you to set custom header tags for every page and post. So you can decide on the suitable robots header tags.

x-robots setting inside blog post

The above image displays the settings of the Blogger blog post. As you can see, now there is an option available for custom robots tags available in the post’s sidebar.

If you enabled custom robots header settings for the simple blog, leave all these settings.

You can also check the HTTP response of the webpage to check all the settings.

Final Words

In this article, we understood the best practice of custom robots header tags for the Blogger blog. What should we choose, and what should we ignore? The best approach, in this case, is don’t touch this setting and make changes in the Robots meta tag of the blog.

I hope you liked this article. In case of any doubt, query, or feedback, comment in the comment section provided below. Thank You!

Ashok Sihmar

Ashok Kumar working in the Search Engine Optimization field since 2015. And worked on many successful projects since then. He shares the real-life experience of best SEO practices with his followers on seoneurons.com. You also can learn Advance level SEO for WordPress, Blogger, or any other blogging platform. Stay tuned.

16 Comments

  1. thankyou so much for this information i really appreciate your efforts to educate us thankyou once again , i will be visiting often .

  2. Hi @Ashok, I’m very much thankful to you. I’m also a continuous reader of seoneurons, especially for the blogger SEO.

  3. Very Helpful I was getting rejection when I request for indexing but after reading and following it is going through i hope my website appears for google search after the request

  4. Very Helpful I was getting rejection when I request for indexing but after reading and following it is going through i hope my website appears for google search after the request

  5. I have your this post-Custom Robots Header Tags for Blogger, Best SEO
    Practice 2022, really you have shared a piece of great and excellent information about the custom robots header.

  6. Hello sir, I am facing a page fetch error on every page of my site. So, no post is indexed
    Nepal Pharmacy Council model Question

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button