What is the X-Robots-Tag - SISTRIX Login Free trialSISTRIX BlogFree ToolsAsk SISTRIXTutorialsWorkshopsAcademy Home / Ask SISTRIX / Crawling and indexing / x-robots tag

What is the X-Robots-Tag

From: SISTRIX Team Steve Paine 17.08.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overviewLike robots meta tags, X-Robots-Tags are used to manage the crawling and indexing of pages. But they have some key differences: X-Robots-Tags are provided in the http protocol response headers (not the html) and are therefore not limited to HTML files. Do you want some of your webpages to stop appearing in search results? You can achieve this through the use of robots meta tags. But what do you do if you want a page to be indexed, but not the image on it? In this case, and in a few other cases, the X-Robots-Tag is ideal.

How to use the X-Robots-Tag

The X-Robots-Tag is an HTTP protocol header. For example, a response with an X-Robots-Tag might look like this: HTTP/1.1 200 OK Date: Wed, 13 May 2020 19:30:25 GMT (...) X-Robots tag: noindex (...) This can be seen in Chrome developer tools when viewing the http response header. Here’s an example with the x-robots-tag at the bottom: As with robots meta tags, you have the option of combining several headers with one another. According to Google, the values are the same in both cases. However, the exact configuration depends on the web server and what is not to be indexed. With the most popular type of server, Apache, it is best to use httpd.conf and .htaccess files to set X-Robots-Tags. For example, if you want to keep all PNG and GIF files on a page from being indexed, enter the following code in the .htaccess or httpd.conf root file of the website in Apache: Header set X-Robots tag "noindex" The FIles and FilesMatch options allow regular expressions to be used to match large numbers of files in a single line of code.

When X-Robots Tags are the right choice

X-Robots-Tags should only be used by advanced users, as mistakes can have serious consequences for your web pages. That’s why we also recommend that you make a backup beforehand, just to stay on the safe side. Why should you use X-Robots-Tags at all? They are often the right choice despite their complicated application, especially in the following situations: You want to provide specific instructions for non-HTML files like images or PDFsYou want to deindex a large number of pages with certain parameters or a complete subdomain In the first case, X-Robots-Tags are the only option; in the second, they save time and effort compared to the laborious implementation of individual Robots meta tags. From: SISTRIX Team Steve Paine 17.08.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overview German English Spanish Italian French