Robots meta tag vs robots txt what are the main differences - SISTRIX Login Free trialSISTRIX BlogFree ToolsAsk SISTRIXTutorialsWorkshopsAcademy Home / Ask SISTRIX / Crawling and indexing / Robots meta tag vs. robots.txt
Robots meta tag vs robots txt what are the main differences
Steve Paine 22.03.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs robots txt what are the main differences What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overviewIf you want to influence the content on your website that search engines crawl and index, you have several options. Two of these options involve using robots meta tags and robots.txt. Whilst they may sound similar, they differ in important ways.ContentsContentsWhat are robots meta tags What is robots txt Robots meta tags vs. robots.txt - when you should use what Of course, you can simply do nothing and leave the crawling and indexing of your website entirely to Google. However, this has potential disadvantages, especially with larger sites. The crawling may take longer than necessary, and content might appear in the search results that should not be displayed there. Fortunately, both of these things can be prevented. What is important is that you take the appropriate action. This brings us to the topic of choosing either robots meta tags or robots.txt. What are robots meta tags
Robots meta tags are snippets that you place in the head section of a page. They look like this: You mark which search engine you want to address with the name attribute, and the content attribute indicates the desired action. In this example, the tag prevents the content from being indexed by all search engines. What is robots txt
A robots.txt file (Robots Exclusion Standard Protocol) is a text file that tells search engine crawlers which files or pages they can crawl. To do this, you must upload it to the website’s root directory. The search engine or its crawler is identified in the robots.txt file with user-agent. The disallow and allow commands can be used to specify which directories should and should not be crawled. You can also refer to the location of a sitemap in the robots.txt file. The result looks like this, for example:# Group 1 User-agent: Googlebot Disallow: /nogooglebot/ # Group 2 User-agent: * Allow: / Sitemap: http://www.example.com/sitemap.xml Robots meta tags vs robots txt – when you should use what
The key difference between robots meta tags and robots.txt is as follows. The robots.txt file is not suitable for safely excluding content from indexation. Incoming links may still cause content to be indexed under certain circumstances. Google therefore advises that you use the robots.txt file to manage crawling traffic and prevent image, video and audio files from appearing in search results. By using robots meta tags with the noindex instruction, you reliably prevent pages from appearing in search results. However, you cannot use them to exclude individual image, audio or video files from indexation. Tip: Make sure that the two measures do not interfere with each other. If a robots.txt file prohibits the crawling of a page, for example, the crawler will not be able to read the robots meta tags. This can, under certain circumstances, lead to the page being indexed – which is something that you of course want to avoid. 22.03.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs robots txt what are the main differences What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overview German English Spanish Italian French