Google hopes to standardize robots.txt by going open source

Google wants to make the Robots Exclusion Protocol an Internet standard.

Article Link: https://www.zdnet.com/article/google-hopes-to-standardize-robots-txt-by-going-open-source/#ftag=RSSbaffb68