d85f07c00d
* Fix legacy sitemap logic * Add aggregate failures * Fix formatting * Adjust robots.txt spec * Fix index minimum to >=
14 lines
No EOL
328 B
Text
14 lines
No EOL
328 B
Text
User-agent: *
|
|
Disallow: /users/auth/twitter*
|
|
Disallow: /users/auth/github*
|
|
Disallow: /report-abuse?url=*
|
|
Disallow: /connect/@*
|
|
Disallow: /search?q=*
|
|
Disallow: /search/?q=*
|
|
Disallow: /search/feed_content?*
|
|
Disallow: /listings*?q=*
|
|
Disallow: /mod/*
|
|
Disallow: /mod?*
|
|
Disallow: /admin/*
|
|
|
|
Sitemap: <%= URL.url("sitemap-index.xml") %> |