maverick/docbrown
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Disallow null user agent and obsolete browser (#15817)

Browse source 
We were previously disallowing empty user agents, but we weren't
blocking requests where the User-Agent header wasn't set at all. This
commit blocks those requests.

This commit also blocks Chrome 74. It's a 2.5-year-old version of an
evergreen browser that releases every 6 weeks, so this is clearly a bot
spoofing this header.
This commit is contained in:
Jamie Gaskins 2021-12-17 15:02:30 -05:00 • committed by GitHub
parent 325a9f9e3f
commit 329f33448f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
config/fastly/snippets/bad_bot_detection.vcl
View file

@ -1,7 +1,8 @@
sub vcl_recv {
# Disallow bot traffic based on user-agent
if (
req.http.user-agent ~ "^$"
req.http.user-agent ~ "^$" # Disallow empty User-Agent
|| req.http.user-agent !~ "." # Disallow null User-Agent
|| req.http.user-agent ~ "^Java"
|| req.http.user-agent ~ "^Jakarta"
|| req.http.user-agent ~ "IDBot"
@ -39,6 +40,9 @@ sub vcl_recv {
# Spoofed user agent, Firefox 62 was released in Sep 2018, this line added
# in Dec 2021.
|| (req.http.user-agent ~ "Firefox/62.0" && req.http.user-agent ~ "Win64")
# Spoofed user agent, Chrome 74 was released in April 2019, this line added
# in Dec 2021.
|| req.http.user-agent ~ "Chrome/74"
# DEV gets an order of magnitude more traffic from AhrefsBot than any other
# search crawler.
|| req.http.user-agent ~ "AhrefsBot"