Robots.txt
robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers which pages or files they are allowed or disallowed to request. It's the first line of defense in controlling how bots interact with your site infrastructure and helps optimize crawl budget.
Directing Bots to Your Best Content
Google allocates a limited "crawl budget" to your site—the number of pages its bots will crawl per day. If bots waste time crawling admin panels, duplicate printer-friendly pages, or cart/checkout URLs, they might miss your valuable translated product pages. robots.txt tells bots "Don't waste time on /admin/, focus on /en/, /fr/, /de/ instead." For international sites, you should disallow crawling of language auto-detection redirect pages, API endpoints, and any technical URLs that don't need to be indexed. However, NEVER accidentally block your language directories—that's a catastrophic mistake that kills all international SEO.
Allowing vs. Disallowing Crawl Access
Real-World Impact
Site has no robots.txt, bots crawl 10,000 cart URLs
Crawl budget wasted, product pages crawled slowly
New products take weeks to appear in search
Add robots.txt: Disallow /cart/, /checkout/, /api/
Bots focus 100% on product and language pages
New products indexed within 24 hours