# robot exclusion for site
User-agent: *
Disallow: /dev
User-agent: *
Crawl-Delay:   60