maverick/docbrown
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add Disallow URLs to robots.txt for uselessly crawled endpoints (#16501)

Browse source
This commit is contained in:
Ben Halpern 2022-02-09 12:19:28 -05:00 • committed by GitHub
parent 42b39da1a8
commit a43d25f6fe
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
app/views/pages/robots.text.erb
View file

@ -11,5 +11,7 @@ Disallow: /listings*?q=*
Disallow: /mod/*
Disallow: /mod?*
Disallow: /admin/*
Disallow: /reactions?*
Disallow: /async_info/base_data
Sitemap: <%= URL.url("sitemap-index.xml") %>