Crawl Budget What does this mean - SISTRIX Login Free trialSISTRIX BlogFree ToolsAsk SISTRIXTutorialsWorkshopsAcademy Home / Ask SISTRIX / Crawling and indexing / crawl budget

Crawl Budget What does this mean

Steve Paine 22.03.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget What does this mean These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overviewThe crawl budget is the amount of resources that Google spends on crawling websites. Since it is not unlimited, there is theoretically a risk that Google will have too little ‘budget’ for all of your site’s URLs. But how big is the budget, the risk, and what can you do about it?ContentsContentsCrawl budget - a definitionHow you can influence the crawl budgetHow important is the crawl budget   Regular and complete crawling of web pages is crucial for content to show up in Google searches. However, not even Google has unlimited resources at its disposal. Therefore, the crawl budget per website is limited.

Crawl budget – a definition

The crawl budget can be described as the maximum number of pages that the Googlebot can crawl. It is made up of two elements:Crawl rate: The crawl rate limit depends primarily on the crawl health, i.e. how quickly a website responds. As a website operator, you can also specify a limit in the Google Search Console. Crawl demand or crawl requirement: How high the crawl requirement is depends on the popularity of URLs. Obsolete and outdated content has a negative effect. Certain changes, such as a domain change, can increase the crawl requirement. Crawl rate and crawl demand are taken together to calculate the crawl budget. Note: In addition to the crawl budget, there is also the index budget. This determines how many pages can be indexed. The difference becomes clear when a page has many inaccessible subpages with the 404 error code. Crawling these pages is a burden on the crawl budget, but not on the index budget. According to a Google blog post, URLs with little added value have a negative effect on crawling and indexing: Soft ErrorsHacked pagesDuplicate contentSpam and low-quality contentFaceted navigation and session IDsInfinite Spaces In these cases, you should expect Google to reduce its crawling activities.

How you can influence the crawl budget

It naturally follows from the points made in the previous paragraph that, in order to increase the crawl budget, it is necessary to create high-quality content. There are a number of other ways to influence crawling and indexing, however:Optimise the internal linking so that the crawler can easily find all of the important contentOpt for a flat page architecture with few layersRemove duplicate content or mark it with canonical tagsRepair broken linksUse the robots.txt file to prevent the Googlebot from scanning unimportant pagesUpdate content regularly

How important is the crawl budget  

Google itself is relaxed on this question. Accordingly, owners of small or medium-sized websites with up to a few thousand URLs need not fear that the crawl budget will not be sufficient. Prioritisation only makes sense for large sites and those that automatically create content based on URL parameters. This does not mean that crawling and indexing is not an issue for smaller sites. After all, there is also a positive effect for smaller sites if the Googlebot indexes important pages without any problems and ignores inferior content. 22.03.2021 Google-Index, Google-Bot and the Crawling Process What is the Google Everflux? Robots meta tag vs. robots.txt: what are the main differences? What is an HTTP referrer? Our web site is no longer in the index - have we lost our rankings? What is a User-Agent? What is Google Search Console and How To Get Started Web Crawlers: How do They Work? Changing Google Search through Entities What is the X-Robots-Tag? What is the Mobile First Index? Rich Snippets: What are the advantages? Can the Google-Bot fill out and crawl forms? Crawl Budget What does this mean These are the CTR's For Various Types of Google Search Result Crawling and Indexing for extensive websites Google SERP Features: Result Types in the Search Results Why does the amount of indexed pages fluctuate so much? How can I quickly get a new page into Google's index? Why does a blocked, noindex URL show up in the search results? Is a website with and without the www harmful? Shelf space optimisation on Google Find out how many pages of a domain are indexed by Google The consequences of negative user-signals on Google's rankings Why am I getting different values for indexed pages in the Google search, the GSC and SISTRIX? How can I remove a URL on my website from the Google Index? Back to overview German English Spanish Italian French