Where does a robots.txt file live on a server? +
It must live at the exact top-level root folder of your web site. If your website is `example.com`, the file should load at `https://example.com/robots.txt`. If placed inside subdirectories, search engine crawlers will not discover it.
What happens if my site does not have a robots.txt file? +
If no robots.txt file exists, search engines will assume there are no crawl restrictions. They will scan, index, and list every publicly accessible page, image, and asset on your site.
Does Robots.txt password protect page directories? +
No. Robots.txt is not a security tool. It relies on voluntary compliance by crawlers. Anyone can access the text, view what you disallowed, and attempt to fetch the restricted links directly. Use server passwords or noindex tags for actual security.
How does a Crawl Delay directive work? +
Crawl Delay tells bots to wait a specified number of seconds between requests. This helps prevent servers from being overloaded by aggressive crawling patterns, but note that Googlebot ignores the crawl delay directive (use Search Console instead).