4.4.6 – Crawl by the Rules
Bug fixes Fixed: The robots.txt checker could report a URL as ALLOWED while Google Search Console showed it blocked by robots.txt. The parser now follows how Google actually reads robots.txt (RFC 9309): consecutive User-agent lines grouped with only ignored directives between them (like Crawl-delay) stay in one group, so a trailing Disallow applies to the […]