robots.txt: what it controls and what it can’t

robots.txt: what it controls, and what it can’t

A robots.txt file tells search engine crawlers which URLs on your site they may access; Google says it is used mainly to avoid overloading a site with requests, and that it is not a mechanism for keeping a page out of Google. To keep a page out of search results, Google points to a noindex rule or password protection instead.

General information from Google’s own documentation, not advice for your particular site. Google’s Search Central documentation is the place to check before you change anything, and it is updated over time.

Think of it as the line where the Park Lands meet the street grid: it says where respectable visitors may walk, but it isn’t a locked gate, and it doesn’t stop anyone pointing at what lies beyond it.

Fence or curtain: choosing the right tool

Google’s own warning is about using it to hide pages. The difference is between crawling (fetching a page) and indexing (keeping it in Google’s results).

What you want, and the tool Google’s documentation points to
You want to	Use	Why
Stop crawlers overloading your server, or skip unimportant or similar pages	robots.txt	That is its main job: managing crawler traffic.
Keep a web page out of Google’s results	`noindex`, or a password	A disallowed URL can still be indexed if other pages link to it, and may appear without a description.
Keep a PDF, image or other non-HTML file out of results	`X-Robots-Tag: noindex` HTTP header	The header works for non-HTML resources such as PDFs, video and images.
Keep images, video or audio out of Google Search	robots.txt	For media files, Google says robots.txt does prevent them appearing in results, though it can’t stop others linking to them.
Keep private information away from all crawlers	Password protection on the server	Robots rules can’t enforce anything: respectable crawlers obey them, others might not.

The trap: blocking a page you have marked noindex

For noindex to work, the page must not be blocked by robots.txt. If it is, Google can’t crawl the page, never sees the rule, and the URL can still appear in results when other pages link to it. Google also doesn’t support putting noindex inside robots.txt itself. A page that is both blocked and marked noindex has a fence in front of the curtain, and the curtain is never seen.

Where the file lives

It must be called robots.txt and sit at the root of the host it applies to, such as https://www.example.com/robots.txt. A site can have only one, and a copy in a subfolder isn’t valid.
It applies only to the protocol, host and port where it is posted. Rules at https://example.com/robots.txt don’t cover https://m.example.com/ or http://example.com/.
It must be UTF-8 plain text. Google suggests a plain text editor, not a word processor, which can add unexpected characters such as curly quotes.
Google enforces a size limit of 500 kibibytes and ignores anything after it.

A file, annotated

# Invented example for www.example.com
User-agent: *
Disallow: /search/
Disallow: /*?sessionid=
Allow: /search/help

User-agent: Googlebot
Disallow: /staging/

Sitemap: https://www.example.com/sitemap.xml

a A group begins with a User-agent line; * matches all crawlers except Google’s AdsBot crawlers, which must be named explicitly.
b Paths are case-sensitive and start with /. * means any run of characters and $ marks the end of the URL.
c Allow overrides a disallow for a page or folder inside it.
d A crawler follows only the most specific group that matches it. Googlebot here obeys its own group and ignores the * group, so it is not kept out of /search/.
e The Sitemap line isn’t tied to any user agent and must be a full URL.

Note d is easy to miss. Specific and global groups are not combined, so if you add a Googlebot group, repeat in it every rule you want Googlebot to follow. The order of groups in the file doesn’t matter.

When rules disagree

Within a group, Google uses the most specific rule, measured by the length of the rule’s path. When rules conflict equally, it uses the least restrictive one. These cases come from Google’s own table:

Which rule wins, per Google’s robots.txt specification
URL	Rules	Outcome
`/page`	`allow: /p` `disallow: /`	Allowed: `/p` is more specific.
`/folder/page`	`allow: /folder` `disallow: /folder`	Allowed: equal conflict, so the least restrictive wins.
`/page.htm`	`allow: /page` `disallow: /*.htm`	Disallowed: the longer path matches more characters.
`/`	`allow: /$` `disallow: /`	Allowed: `/$` is more specific.
`/page.htm`	`allow: /$` `disallow: /`	Disallowed: `/$` only matches the root.

What Google does when the file can’t be read

The server’s answer to a request for robots.txt changes how Google treats the whole site.

robots.txt status codes and Googlebot
Response	Google’s behaviour
2xx	Processes the file as served.
3xx	Follows at least five redirect hops, then treats it as a 404.
4xx (except 429)	Behaves as if there were no robots.txt: no crawl restrictions. Don’t use 401 or 403 to slow crawling.
5xx, or a network error	For the first 12 hours, stops crawling the site while retrying. Then for 30 days uses the last good copy. After that, if the site is otherwise available, behaves as if there were no file.

Google generally caches robots.txt for up to 24 hours, so a change isn’t instant. Only four fields are supported: user-agent, allow, disallow and sitemap. Others, such as crawl-delay, are ignored.

Don’t block what the page needs

You can block resource files such as unimportant scripts or styles, but Google warns not to if their absence makes a page harder to understand. The same applies on mobile: if resources there use different URLs, make sure they aren’t disallowed (see mobile-first indexing).

Testing

Open the file in a private browser window to check it is public, then use the robots.txt report in Search Console, or Google’s open-source robots.txt library if you are a developer. You don’t need to submit the file: Google’s crawlers find it.

After a site move, check that rules added to block a staging copy have been removed; Google lists forgotten blocks among the common migration mistakes.