Book a call

Daniel & Friends / Glossary / Crawl budget

SEO, GEO and AEO

Crawl budget

Crawl budget is the number of pages a search engine bot will fetch on your site in a given period. It depends on server performance and on how valuable Google considers the content. It matters mainly for large sites and online shops.

Daniel Votruba ·


In practice

Save it by keeping bots out of filters, duplicates and endless parameter combinations. Track it in the Crawl stats report in Search Console.

Related terms

  • robots.txt: The robots.txt file sits at the root of a domain and tells bots which parts of the site they may crawl.
  • Log file analysis: Log file analysis is the review of server records showing which pages Google, Bing and AI crawlers actually visit and how often.
  • Indexing: Indexing is the storing of a page in a search engine’s database so that it can appear in results.
  • XML sitemap: An XML sitemap is a file, usually sitemap.xml, that gives a search engine a list of a site’s URLs and the date of the last change.

More terms in the group SEO, GEO and AEO

E-E-A-T · Core Web Vitals · Entity SEO · Core update · Featured snippet · Content decay · GEO (generative engine optimization) · Canonical tag · Google AI Mode · Brand SERP · Google AI Overviews · Brand mentions · Google Business Profile · Backlink · Google Discover · Anchor text · Google Search Console · AEO (answer engine optimization) · Helpful content · 404 error · Hreflang · 301 and 302 redirects · YMYL (Your Money or Your Life) · Internal linking

Full glossary (590 terms)

Terms in practice

Need help putting them to work?

Terms are one thing, putting them to work is another. We go through which of them pay off for you and say so plainly. You get a first opinion for free and we reply within 24 hours.

Book a call

Enjoyed the article? Share it with friends.