Robots

https://www.instapaper.com/read/1957116947

Robots.txt and meta robots directives guide crawlers, controlling what gets indexed to protect sensitive pages and prioritize key content.