{"id":2854,"date":"2020-12-17T16:37:00","date_gmt":"2020-12-17T11:07:00","guid":{"rendered":"https:\/\/seoneurons.com\/blog\/?p=2854"},"modified":"2023-05-27T14:44:03","modified_gmt":"2023-05-27T09:14:03","slug":"robots-txt-file","status":"publish","type":"technical-seo","link":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/","title":{"rendered":"Robots.txt file, what is it? And how to use it?"},"content":{"rendered":"\n<p>You created a website, and that website is crawlable for the search engine. But what if you don&#8217;t want the search engine to crawl some parts of the website? How can you block that parts of the website for the search engine? The robots.txt file.<\/p><div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">In this guide, we\u2019ll walk through:<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #494949;color:#494949\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #494949;color:#494949\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#understand-the-robotstxt-file-of-a-website-what-is-it-and-what-kind-of-functions-it-performs\" >Understand the robots.txt file of a website. What is it, and what kind of functions it performs?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#search-engine-optimization-and-robotstxt\" >Search Engine Optimization and Robots.txt<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#robotstxt-syntax-used\" >Robots.txt syntax used<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#user-agent\" >User-agent:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#disallow\" >Disallow:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#allow\" >Allow:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#sitemap\" >Sitemap:<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n\n<p>So you like to understand the robots.txt file? But if I say it is a crawling bot path inspector. That allows or disallows bots on some roads, and let&#8217;s block on others. This file allows or disallows search engine bots to crawl some sections of your webpage and disallow others.<\/p>\n\n\n\n\t\t<div class=\"stream-item stream-item-in-post stream-item-in-post-1\">\n\t\t<\/div>\n\t\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-understand-the-robots-txt-file-of-a-website-what-is-it-and-what-kind-of-functions-it-performs\"><span class=\"ez-toc-section\" id=\"understand-the-robotstxt-file-of-a-website-what-is-it-and-what-kind-of-functions-it-performs\"><\/span>Understand the robots.txt file of a website. What is it, and what kind of functions it performs?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt-1024x576.png\" alt=\"robots-txt file\" class=\"wp-image-3050\" srcset=\"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt-1024x576.png 1024w, https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt-300x169.png 300w, https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt-768x432.png 768w, https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt-390x220.png 390w, https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png 1280w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>robots.txt file<\/figcaption><\/figure><\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-search-engine-optimization-and-robots-txt\"><span class=\"ez-toc-section\" id=\"search-engine-optimization-and-robotstxt\"><\/span>Search Engine Optimization and Robots.txt<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><strong>What are the Search Engine Bots?: <\/strong>These are the bots that read data from a website or webpage and transfer that to their database like Google, Bing, Yandex, or any other. For example, suppose you create a website and continuously writing articles on that.<\/p>\n\n\n\n<p><strong>Internal Backlink and robots.txt: <\/strong>If you&#8217;re writing an article, and within that, you provide some internal link that may be of category or label or anything else that is blocked by robots.txt. Now that link you provided within the article is to follow as it is internal, and at the same time, you&#8217;re disallowing it by the permission set in the robots.txt file. So best practice for internal pages that shouldn&#8217;t be index but crawl is to noindex those pages, not to disallow using robots.txt file.<\/p>\n\n\n\n<p><strong>What pages should be blocked by the robots.txt file:<\/strong> It should block the sensitive pages. That may be the admin section of your website or blog. All other pages that cause junk or double content should be noindex using proper meta tags or x-robots tags.<\/p>\n\n\n\n\t\t<div class=\"stream-item stream-item-in-post stream-item-in-post-2\">\n\t\t<\/div>\n\t\n\n\n\n<p><strong>Links from external resources will block:<\/strong> Suppose someone provided a backlink of your website of category section, then in such case crawl engine will try to crawl your website but robots.txt block it from crawling and a hard-earned backlink will waste.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-robots-txt-syntax-used\"><span class=\"ez-toc-section\" id=\"robotstxt-syntax-used\"><\/span>Robots.txt syntax used<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-user-agent\"><span class=\"ez-toc-section\" id=\"user-agent\"><\/span><strong>User-agent:<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>This declared the bots or web crawler to which we&#8217;re giving instruction or controlling them for the various sections using allow and disallow function.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-disallow\"><span class=\"ez-toc-section\" id=\"disallow\"><\/span><strong>Disallow<\/strong>:<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>You can declare the page sections that shouldn&#8217;t be crawled by search engines. Using this method, we can save the crawling quota of the search engine dedicated to our website.<\/p>\n\n\n\n\t\t<div class=\"stream-item stream-item-in-post stream-item-in-post-3\">\n\t\t<\/div>\n\t\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-allow\"><span class=\"ez-toc-section\" id=\"allow\"><\/span><strong>Allow<\/strong>:<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>It is usually used for Googlebot to allow sections for the crawl of a website. For example, it may allow a subfolder that&#8217;s parent folder is disallowed to crawl.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-sitemap\"><span class=\"ez-toc-section\" id=\"sitemap\"><\/span><strong>Sitemap<\/strong>:<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>It declares the <a href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/xml-sitemap-for-seo\/\">XML sitemap&#8217;s<\/a> location\u00a0or blog\u2014search engines like Google, Bing, Yandex support this command.<\/p>\n\n\n\n<p><strong>robots.txt file example:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: Mediapartners-Google\nDisallow:\nUser-agent: *\nDisallow: \/*\/junk\/*\nDisallow: \/search\nAllow: \/\n\nSitemap: https:\/\/example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n<p>In the above robots.txt example<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><em><strong><span style=\"text-decoration: underline;\">User-agent: Mediapartners-Google<\/span><\/strong><\/em> declares instruction for Google AdSense, instructions are followed as <em>Disallow:<\/em> to nothing. That means AdSense can crawl your whole website and display ads.<\/li><li>The next command link is <em><span style=\"text-decoration: underline;\"><strong>User-agent: *, which<\/strong><\/span><\/em>&nbsp;means instruction for all other bots or crawlers other than Google AdSense.<\/li><li><span style=\"text-decoration: underline;\"><strong><em>Disallow: \/*\/junk\/*<\/em> <\/strong>and <strong><em>Disallow: \/search<\/em><\/strong><\/span> that disallows subfolder &#8220;junk&#8221; to any parent folder. And also parent folder &#8220;search&#8221;. Allow command allows the whole website to crawl. You can include private sections of the website that should not index in the search results.<\/li><li><strong><span style=\"text-decoration: underline;\">Sitemap: https:\/\/example.com\/sitemap.xml<\/span><\/strong> is the location of the sitemap added to the domain.<\/li><\/ul>\n\n\n\n\t\t<div class=\"stream-item stream-item-in-post stream-item-in-post-4\">\n\t\t<\/div>\n\t\n\n\n\n<p>Check how you can set it for WordPress and Blogger.<\/p>\n\n\n\n<div class=\"schema-faq wp-block-yoast-faq-block\"><div class=\"schema-faq-section\" id=\"faq-question-1620645949458\"><strong class=\"schema-faq-question\">Can I place the robots.txt file anywhere on my website?<\/strong> <p class=\"schema-faq-answer\">No, the robots.txt file is always located under the root directory with the name robots.txt. And, you can&#8217;t even change its name.<\/p> <\/div> <div class=\"schema-faq-section\" id=\"faq-question-1620646023221\"><strong class=\"schema-faq-question\">Can I provide more than one sitemap in the robots.txt file?<\/strong> <p class=\"schema-faq-answer\">Yes, You can provide multiple sitemaps in the robots.txt file.<\/p> <\/div> <div class=\"schema-faq-section\" id=\"faq-question-1620646062142\"><strong class=\"schema-faq-question\">Can I use the robots.txt file to remove a page from the search engine?<\/strong> <p class=\"schema-faq-answer\">No, for that, you&#8217;ve to use meta robots tag or X-Robots-Tag in the header response.<\/p> <\/div> <\/div>\n","protected":false},"excerpt":{"rendered":"<p>You created a website, and that website is crawlable for the search engine. But what if you don&#8217;t want the search engine to crawl some parts of the website? How can you block that parts of the website for the search engine? The robots.txt file. So you like to understand the robots.txt file? But if &hellip;<\/p>\n","protected":false},"author":2,"featured_media":3050,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"parentpage":[951],"class_list":["post-2854","technical-seo","type-technical-seo","status-publish","format-standard","has-post-thumbnail","hentry","parentpage-technical-seo"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.4 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Robots.txt file, what is it? How to use it for Best SEO Practice 2021<\/title>\n<meta name=\"description\" content=\"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt file, what is it? And how to use it?\" \/>\n<meta property=\"og:description\" content=\"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Neurons\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SEONeuronsAgency\/\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-27T09:14:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@SEONeurons\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":[\"WebPage\",\"FAQPage\"],\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/\",\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/\",\"name\":\"Robots.txt file, what is it? How to use it for Best SEO Practice 2021\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/robots-txt.png\",\"datePublished\":\"2020-12-17T11:07:00+00:00\",\"dateModified\":\"2023-05-27T09:14:03+00:00\",\"description\":\"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#breadcrumb\"},\"mainEntity\":[{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620645949458\"},{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646023221\"},{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646062142\"}],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#primaryimage\",\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/robots-txt.png\",\"contentUrl\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/robots-txt.png\",\"width\":1280,\"height\":720,\"caption\":\"robots-txt file\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/seoneurons.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Robots.txt file, what is it? And how to use it?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/\",\"name\":\"SEO Neurons\",\"description\":\"Create, Optimize Website and Drive More Organic Traffic\",\"publisher\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#organization\"},\"alternateName\":\"SEO Neurons\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#organization\",\"name\":\"SEO Neurons\",\"alternateName\":\"SEO Neurons\",\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/seoneurons-logo.svg\",\"contentUrl\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/seoneurons-logo.svg\",\"width\":1,\"height\":1,\"caption\":\"SEO Neurons\"},\"image\":{\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/SEONeuronsAgency\\\/\",\"https:\\\/\\\/x.com\\\/SEONeurons\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/seoneurons\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/SEONeurons\",\"https:\\\/\\\/www.instagram.com\\\/seo_neurons\\\/\",\"https:\\\/\\\/t.me\\\/SEONeurons\"],\"legalName\":\"SEO Neurons\",\"foundingDate\":\"2021-01-21\",\"numberOfEmployees\":{\"@type\":\"QuantitativeValue\",\"minValue\":\"1\",\"maxValue\":\"10\"}},{\"@type\":\"Question\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620645949458\",\"position\":1,\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620645949458\",\"name\":\"Can I place the robots.txt file anywhere on my website?\",\"answerCount\":1,\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"No, the robots.txt file is always located under the root directory with the name robots.txt. And, you can't even change its name.\",\"inLanguage\":\"en-US\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Question\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646023221\",\"position\":2,\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646023221\",\"name\":\"Can I provide more than one sitemap in the robots.txt file?\",\"answerCount\":1,\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Yes, You can provide multiple sitemaps in the robots.txt file.\",\"inLanguage\":\"en-US\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Question\",\"@id\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646062142\",\"position\":3,\"url\":\"https:\\\/\\\/seoneurons.com\\\/blog\\\/technical-seo\\\/robots-txt-file\\\/#faq-question-1620646062142\",\"name\":\"Can I use the robots.txt file to remove a page from the search engine?\",\"answerCount\":1,\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"No, for that, you've to use meta robots tag or X-Robots-Tag in the header response.\",\"inLanguage\":\"en-US\"},\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Robots.txt file, what is it? How to use it for Best SEO Practice 2021","description":"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt file, what is it? And how to use it?","og_description":"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.","og_url":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/","og_site_name":"SEO Neurons","article_publisher":"https:\/\/www.facebook.com\/SEONeuronsAgency\/","article_modified_time":"2023-05-27T09:14:03+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@SEONeurons","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["WebPage","FAQPage"],"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/","url":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/","name":"Robots.txt file, what is it? How to use it for Best SEO Practice 2021","isPartOf":{"@id":"https:\/\/seoneurons.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#primaryimage"},"image":{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#primaryimage"},"thumbnailUrl":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png","datePublished":"2020-12-17T11:07:00+00:00","dateModified":"2023-05-27T09:14:03+00:00","description":"Robots.txt file allows or disallows search engine bots to crawl some sections of your webpage and disallow others. How it is beneficial in SEO.","breadcrumb":{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#breadcrumb"},"mainEntity":[{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620645949458"},{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646023221"},{"@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646062142"}],"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#primaryimage","url":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png","contentUrl":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2020\/12\/robots-txt.png","width":1280,"height":720,"caption":"robots-txt file"},{"@type":"BreadcrumbList","@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/seoneurons.com\/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https:\/\/seoneurons.com\/blog\/"},{"@type":"ListItem","position":3,"name":"Robots.txt file, what is it? And how to use it?"}]},{"@type":"WebSite","@id":"https:\/\/seoneurons.com\/blog\/#website","url":"https:\/\/seoneurons.com\/blog\/","name":"SEO Neurons","description":"Create, Optimize Website and Drive More Organic Traffic","publisher":{"@id":"https:\/\/seoneurons.com\/blog\/#organization"},"alternateName":"SEO Neurons","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/seoneurons.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/seoneurons.com\/blog\/#organization","name":"SEO Neurons","alternateName":"SEO Neurons","url":"https:\/\/seoneurons.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/seoneurons.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2023\/06\/seoneurons-logo.svg","contentUrl":"https:\/\/seoneurons.com\/blog\/wp-content\/uploads\/2023\/06\/seoneurons-logo.svg","width":1,"height":1,"caption":"SEO Neurons"},"image":{"@id":"https:\/\/seoneurons.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SEONeuronsAgency\/","https:\/\/x.com\/SEONeurons","https:\/\/www.linkedin.com\/company\/seoneurons","https:\/\/www.youtube.com\/c\/SEONeurons","https:\/\/www.instagram.com\/seo_neurons\/","https:\/\/t.me\/SEONeurons"],"legalName":"SEO Neurons","foundingDate":"2021-01-21","numberOfEmployees":{"@type":"QuantitativeValue","minValue":"1","maxValue":"10"}},{"@type":"Question","@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620645949458","position":1,"url":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620645949458","name":"Can I place the robots.txt file anywhere on my website?","answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"No, the robots.txt file is always located under the root directory with the name robots.txt. And, you can't even change its name.","inLanguage":"en-US"},"inLanguage":"en-US"},{"@type":"Question","@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646023221","position":2,"url":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646023221","name":"Can I provide more than one sitemap in the robots.txt file?","answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"Yes, You can provide multiple sitemaps in the robots.txt file.","inLanguage":"en-US"},"inLanguage":"en-US"},{"@type":"Question","@id":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646062142","position":3,"url":"https:\/\/seoneurons.com\/blog\/technical-seo\/robots-txt-file\/#faq-question-1620646062142","name":"Can I use the robots.txt file to remove a page from the search engine?","answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"No, for that, you've to use meta robots tag or X-Robots-Tag in the header response.","inLanguage":"en-US"},"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/technical-seo\/2854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/technical-seo"}],"about":[{"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/types\/technical-seo"}],"author":[{"embeddable":true,"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/comments?post=2854"}],"version-history":[{"count":0,"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/technical-seo\/2854\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/media\/3050"}],"wp:attachment":[{"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/media?parent=2854"}],"wp:term":[{"taxonomy":"parentpage","embeddable":true,"href":"https:\/\/seoneurons.com\/blog\/wp-json\/wp\/v2\/parentpage?post=2854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}