Daniel & Friends / Glossary / robots.txt
SEO, GEO and AEOrobots.txt
The robots.txt file sits at the root of a domain and tells bots which parts of the site they may crawl. It controls crawling, so a URL blocked in it can still appear in the index without content if other pages link to it.
Daniel Votruba ·
In practice
To remove a page from results, use noindex. The file is also the place for the sitemap reference and for rules aimed at AI crawlers.
Related terms
- Noindex: Noindex is a directive that tells search engines to keep a page out of the index.
- Crawl budget: Crawl budget is the number of pages a search engine bot will fetch on your site in a given period.
- XML sitemap: An XML sitemap is a file, usually sitemap.xml, that gives a search engine a list of a site’s URLs and the date of the last change.
- AI crawlers (GPTBot and others): AI crawlers are bots run by AI companies that visit websites to train models or to fetch current sources for answers.
More terms in the group SEO, GEO and AEO
Schema markup and rich results · On-page SEO · Search intent · SEO · NAP and citations · SEO audit · Mobile-first indexing · SERP · Long-tail keywords · Structured data · Log file analysis · Topical authority · Local SEO · Topic cluster and pillar page · llms.txt · Link building · YMYL (Your Money or Your Life) · Knowledge Graph · 301 and 302 redirects · Keyword research · 404 error · Keyword difficulty · AEO (answer engine optimization) · Keyword cannibalization
Terms in practice
Need help putting them to work?
Terms are one thing, putting them to work is another. We go through which of them pay off for you and say so plainly. You get a first opinion for free and we reply within 24 hours.
Book a call