Scoutern

Googlebot

Honors robots.txt

Googlebot is Google's core Search crawler, in service for decades — now also the primary feed for AI Overviews and AI Mode, since both are extensions of the same underlying Search index rather than a separate system.

User-agent
Googlebot
Operator
Google

Allow in robots.txt

User-agent: Googlebot
Allow: /

Block in robots.txt

User-agent: Googlebot
Disallow: /

Blocking Googlebot removes you from Google Search entirely, including AI Overviews and AI Mode — almost never advisable for a public marketing site, and worth triple-checking before any change here.

Frequently asked questions

It's Google's classic search crawler, but since AI Overviews and AI Mode are built on top of the same Search index, Googlebot access is a prerequisite for appearing in them too.

More AI crawlers

OpenAI

GPTBot

OpenAI

OAI-SearchBot

OpenAI

ChatGPT-User

Verify your robots.txt against every AI crawler

Scoutern's free AI bot checker tests exactly which bots can reach your site.

Check my robots.txtSee pricing