Do you know the default robots.txt file blocks many pages to crawl in a Blogger blog or website? This tutorial article will guide you on how to use robots.txt and robots meta tag combined for the best SEO results of Blogger blog—letting all the crawlers crawl and index the whole website or blog without duplication of content error.

Perfect Combination of Blogger robots.txt and meta robots tag

Blogger now gives you more control over its features than earlier. It supports advanced XML themes that enable us to enhance the On-Page SEO of the Blogger.

blogger-seo-best-practice

Blogger SEO index noindex tool. A smart way

You can add robot meta tags to control the index behavior at the page level. And allow all the search and archive sections to crawl with noindex value in the meta tags.

How does Google Indexing work?

When the search engine’s bot visits your blog, it first checks the robots.txt file rules. The Robots.txt file contains the rules that a search engine should follow. Blogger’s default robots.txt file blocks only blog’s search and label pages to appear in Google’s search results.

You can access your website’s robots.txt file at http://www.yourblog.com/robots.txt or https://example.blogspot.com/robots.txt.

As we already discussed, this file contains rules for bot crawl. Try visiting your robots.txt file.

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.xyz.blogspot.com/sitemap.xml

What is the mean of each line? Understand it here

  • The following line declared User-agent: * means bots or spiders of all search engines.
  • Disallow: /search means that all pages similar in structure to https://xyz.blogspot.com/search are not allowed to crawl.
  • Allow: / it states that all other pages are allowed.
  • And the last line declares the location of the sitemap: https://www.xyz.blogspot.com/sitemap.xml.

Blogger Robots.txt File settings will be.

Now I hope you understand all robots.txt file functions. So, some changes need to be made for better search engine crawling and indexing your web pages.

  • Visit the Blogger dashboard.
  • Dashboard > setting > Search Preference.
  • Find Custom robots.txt and enable it.
  • Paste the code given below. Note: Change the website name with yours.
Blogger robots.txt settings 
User-agent: *

Allow: /

Sitemap: https://www.xyz.com/sitemap.xml
Sitemap: https://www.xyz.com/sitemap-pages.xml

Here you submitted both sitemaps for pages and posts. And also, Google Spider or a bot can crawl your whole website. But for SEO, remember to submit only posts and pages for the Search Engine Index. Understand how noindex is important for SEO.

Blogger Robots Meta Tags settings.

Using robots meta tags, we’ll avoid junk/duplicate content to index in the search engine. Here we’re using the latest robots meta tag suggested by Google.

  1. Go to Theme
  2. Now click on Edit HTML
  3. Find <b:include data='blog' name='all-head-content'/>

Now, below this line <b:include data='blog' name='all-head-content'/>, past the code given below.

<b:if cond='!data:view.isHomepage and !data:view.isSingleItem'>
 <meta content='noindex,follow' name='robots'/>
<else/>
 <meta content='max-snippet:-1, max-image-preview:large, max-video-preview:-1' name='robots'/>
</b:if>

What happens with this code? Final SEO Results.

  • Boost SEO as only the content will index to google or other search engines.
  • Resolve the Google search console error “blocked by robots.txt” for all pages.
  • The website will interlink and, as a result, boost on-page SEO.
  • The bot or spider can visit all pages of your website/ webpage. That results in more pages to index.

You can download our Blogger themes, Best SEO, Speed, and User Friendly.

Step by Step robots.txt Blogger settings

I hope you like this article on Blogger SEO. If any questions regarding any issue, feel free to ask in the comment below. Thank you.

JOIN THE CONVERSATION

Comments

Have a question or something to add? Comments are moderated and appear here only after approval.

23 published comments, including 7 replies. Threads shown oldest first.

  1. emrah baldanReader #

    Couldn’t find this code in theme

    1. Ashok Sihmar (Ashok Kumar)Author #

      In reply to emrah baldan

      these code are not included in theme. You’ve to change these codes in the Blogger blog settings.

      1. huseyınReader #

        In reply to Ashok Sihmar

        bu kod ayarlarda yok

  2. onakpa danladiReader #

    thank you, i hope to see result with these settimgs you’re showimg me today

    1. Ashok Sihmar (Ashok Kumar)Author #

      In reply to onakpa danladi

      Along with these you can try tezzBlog Blogger theme to give a professional design to your blog.. and try to write enough length(at least 600 words) and error free(spell and grammar) content..

      1. matteo desimoneReader #

        In reply to Ashok Sihmar

        in this case, you should also explain what to do for those who don’t have this code in their template. Because that way you are not helping

        1. Ashok Sihmar (Ashok Kumar)Author #

          In reply to matteo desimone

          If these codes are not there, you can copy from here and paste.

  3. nowrozReader #

    Sir, are you sure this going work my blogspot/blogger ?

  4. iLinkReader #

    Thank you! you save me. I am now happy after this change. You are awesome master in this technique.

  5. iLinkReader #

    Can you help me more. I have backup my blog data the restore it again. There are all posts copy as duplicated. I have remove all posts the re upload blog data backup. The all posts are now new html link and there is Error 404 for the old links. Can you help me to resolve it. Please save me.

  6. iLinkReader #

    The service worker navigation preload request was cancelled before ‘preloadResponse’ settled. If you intend to use ‘preloadResponse’, use waitUntil() or respondWith() to wait for the promise to settle. i need help to fix this error please reply

  7. iLinkReader #

    The resource https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js was preloaded using link preload but not used within a few seconds from the window’s load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally. can you resolve it chrome error by inspecting url

  8. NabidReader #

    https://phonoming.blogspot.com/2022/07/html-bangla-tutorials-basic.html Sir can you please see this page and give a solution why my Bangla headings is showing less gap between lines?

  9. jdgaming.infoReader #

    Hi, I have an issue where on submitting my sitemap to Google Search Console for the website jdgaming.info this results in 0 pages being accessible, are you able to help? This was all previously working fine, it is as if the sitemap file for my website does not work.

    1. Biswajoyee BarmanReader #

      In reply to jdgaming.info

      try url your sitemap then submit

  10. Ecommerce TrendsReader #

    I appreciate your efforts to share this useful information. Your content is really beneficial to all the readers. well done!

  11. Nwali chineduReader #

    Bro. I so much love your articles, very easy to understand .. thank you so much

  12. OscarReader #

    Hello, friend, great blog =), it has helped me a lot. But in this case I can’t find this line of code in my blog: 1. Neither in a custom template nor in the default templates of Blogger. So I don’t know where to place the code of the article. What could I do in that case?

    1. OscarReader #

      In reply to Oscar

      This line

  13. Touhidul IslamReader #

    আমি ত Meta taq পাচ্ছি না।সমাধান কি?

  14. abdullahReader #

    this article is very informative for more information google bot

  15. mdijunReader #

    please 🙏 thanks for you.

  16. jdsezerReader #

    Eski bir Blogger kullanıcısı olarak bu bilgilerin ne kadar değerli olduğunu biliyorum. Makalede verilen robots.txt kodunu 2025’te de aynı şekilde mi kullanmalıyız, yoksa daha farklı bir yaklaşım izlememiz gerekir mi? Örneğin, yeni ortaya çıkan bot türleri için özel kurallar tanımlamalı mıyız?

JavaScript is required to submit a comment. Published comments above remain available to read.