Book a call

Daniel & Friends / Glossary / AI crawlers (GPTBot and others)

AI and automation

AI crawlers (GPTBot and others)

AI crawlers are bots run by AI companies that visit websites to train models or to fetch current sources for answers. Well-known examples are GPTBot, ClaudeBot and PerplexityBot. Each identifies itself with its own name in the user-agent string.

Daniel Votruba ·


In practice

Decide in robots.txt which bots you admit, and remember that one company often runs several bots with different purposes. Verify actual visits in your server log.

Read more: How to get into ChatGPT's answers: a practical GEO guide

Related terms

  • llms.txt: llms.txt is a proposed file in the root of a site that offers AI tools an overview of the important pages.
  • GEO (generative engine optimization): GEO is the adjustment of a site and its content so that generative search engines and AI assistants, such as ChatGPT, Perplexity or Google AI Overviews, find, understand and cite them.
  • AI search (answer engines): AI search is a way of searching where the system composes one coherent answer from several sources and cites them.
  • SEO: SEO is the optimisation of a website for organic search results.
  • XML sitemap: An XML sitemap is a file, usually sitemap.xml, that gives a search engine a list of a site’s URLs and the date of the last change.

More terms in the group AI and automation

AI governance · AI costs and limits · AI personalization · AI automation · AI safety and alignment · AI agent · AI scoring · Agentic AI · AI SDR · Voice cloning · Vibe coding · AI slop · Vector database · AI workflow and orchestration · Token · Automation platforms (Zapier, Make, n8n) · Text-to-video · C2PA and content provenance (AI watermarks) · Text-to-speech (TTS) · Chain of thought · Text-to-image · ChatGPT Ads · Temperature · Computer vision

Full glossary (590 terms)

Terms in practice

Need help putting them to work?

Terms are one thing, putting them to work is another. We go through which of them pay off for you and say so plainly. You get a first opinion for free and we reply within 24 hours.

Book a call

Enjoyed the article? Share it with friends.