Robots.txt is the name of a text file file that tells search engines which URLs or directories in a site should not be crawled. Robots.txt files are not inherited by subdomains or parent domains, and a given page can be affected by only one robots.txt file. To determine the URL, cut off everything after the host (and optional port) in the URL of a file and add "/robots.txt". A robots.txt file is located at the root of a protocol and domain. Per RFC 9309, the robots.txt file must be at the root of each protocol and host combination of your site. If your website is hosted on a website hosting service, it might not be easy to edit your robots.txt file. To request a recrawl, select the more settings icon next to a file in the robots file list and click Request a recrawl.

You can cycle through the errors and warnings using the arrow keys. If the robots.txt file has any errors or warnings, they will be highlighted in the displayed file contents. adrian casino This report is available only for properties at the domain level. Use noindex if you want to prevent content from appearing in search results. A robots.txt file is used to prevent search engines from crawling your site. Other techniques are used to prevent a page or image from appearing in search results.

You can see the last fetched version of a robots.txt file by clicking it in the files list in the report. To see fetch requests for a given robots.txt file in the last 30 days, click the file in the files list in the report, then click Versions. In a Domain property, the report includes robots.txt files from the top 20 hosts in that property. Do not use robots.txt to prevent a page from appearing in search results, only to prevent it from being crawled. To open a robots.txt file listed in this report, click the file in the list of robots.txt files. If this is your concern, search your hosting service for information about blocking pages from search engines. Note that most users are concerned with preventing files from appearing in Google Search, rather than crawled by Google. In that case, see your site host's documentation about how to block specific pages from being crawled or indexed by Google.

See the last fetched version

A request is included in the history only if the retrieved file or fetch result is different from the previous file fetch request. The report also enables you to request a recrawl of a robots.txt file for emergency situations. Post to the help community Get answers from community members