How can I prevent the Google-Bot from crawling my website - SISTRIX Login Free trialSISTRIX BlogFree ToolsAsk SISTRIXTutorialsWorkshopsAcademy Home / Ask SISTRIX / Crawling and indexing / How can I prevent the Google-Bot from crawling my website

How can I prevent the Google-Bot from crawling my website

From: SISTRIX Team 16.05.2022 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overviewIf you want to prevent Google from crawling all or parts of your domain, you can do so within the robots.txt file.ContentsContentsBlocking the Google-Bot using the robots txtThe contents of the robots txtSome URLs may still be indexedVideo Lily Ray talks about blocking crawlers from a website using robots txtBlock crawlers using WordPressMore information on Googlebot and crawler control Note: If you are looking to block specific URLs, you can use the robots meta-tag. Also be aware that there are some cases where URLs will still be indexed, even if they are blocked using the robots.txt file.

Blocking the Google-Bot using the robots txt

The robots.txt is a simple text file with the name “robots”. It has to be placed in the root-directory of a website in order for search engines to follow the directives. If a website has a robots.txt, it can be accessed through the following path: http://www.my-domain.com/robots.txt

The contents of the robots txt

By using the following instructions, we exclusively forbid the Google-Bot access to our entire website: You have to add the following to your robots.txt to tell Google-Bot to stay away from the entire domain:User-Agent: Googlebot Disallow: / If you only want to restrict access to some directories or files instead of the entire website, the robots.txt has to contain the following: The following only tells Google-Bot that it is forbidden from accessing the directory “a-directory” as well as the file “one-file.pdf”:User-Agent: Googlebot Disallow: /a-directory/ Disallow: /one-file.pdf

Some URLs may still be indexed

The code examples shown here are only meant for Google-Bot. Crawlers from other search engines, such as Bing, will not be blocked.Restricting access for a specific crawler does not guarantee that the website or individual URLs will not (maybe) show up in the search results (SERPs). You can find additional information on this in our article “Why does a URL that is blocked through robots.txt show up in the search results?“

Video Lily Ray talks about blocking crawlers from a website using robots txt

Block crawlers using WordPress

WordPress has a built-in feature that will set a robots meta-tag to noindex in the header of every page. 1Assuming you have the Administrator rights in the WordPress site, go to the Settings -> Reading page and select “Discourage search engines from indexing this site” 1 as shown above.

More information on Googlebot and crawler control

What is the difference between robots.txt and the robots meta-tag? https://support.google.com/webmasters/answer/6062608?hl=en From: SISTRIX Team 16.05.2022 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget: What does this mean? These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overview German English Spanish Italian French