GPTBot
Stated to honor robots.txtGPTBot is the crawler OpenAI uses to collect web content for training and improving its models, including the GPT family that underpins ChatGPT. It is not used for ChatGPT's live search or browsing features — those use a separate, distinct crawler called OAI-SearchBot, and treating the two as interchangeable is the single most common mistake site owners make when deciding what to allow.
- User-agent
- GPTBot
- Operator
- OpenAI
Allow in robots.txt
User-agent: GPTBot
Allow: /Block in robots.txt
User-agent: GPTBot
Disallow: /Blocking GPTBot opts your content out of future OpenAI model training without necessarily affecting whether ChatGPT can find and cite you in live search answers, which is governed by OAI-SearchBot and ChatGPT-User instead. Decide the two separately rather than assuming one rule covers OpenAI's crawlers as a group.
GPTBot vs OpenAI's other crawlers
OpenAI operates at least three distinct crawlers with genuinely different jobs. GPTBot gathers training data that shapes future model versions — its effect shows up over the long term, in what the model eventually learns. OAI-SearchBot fetches current pages to power ChatGPT's live, search-grounded answers, with an effect that can show up within days. ChatGPT-User fetches a single page on demand when a user explicitly asks ChatGPT to open or summarise a specific URL. Blocking GPTBot alone does not stop the other two from citing you in a live answer, and site owners who assume it does often end up more invisible than they intended, or less protected than they assumed.
Should you allow or block GPTBot?
For most brands that want any chance of being referenced by ChatGPT, allowing GPTBot alongside OpenAI's other crawlers is the sensible default — training data inclusion is one more pathway to eventual visibility, and blocking it has no upside for a business trying to be found. The main legitimate reason to block it is a considered decision to opt your content out of AI training data collection specifically, separate from any concern about live citation.