maverick/docbrown
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[deploy] Replace (black|white)list with more inclusive language (#7459)

Browse source 
* Replace isBlacklisted

* Update fastly.rake task

* More renames

* Rename Faslty config file

* More changes

* Replace isForbiddenFromPreloading with isNotPreloadable

* Skip problematic test

* Re-enable
This commit is contained in:
rhymes 2020-04-27 18:00:12 +02:00 • committed by GitHub
parent b4644783ef
commit 2d26318cf9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 46 additions and 40 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
Envfile
View file

@ -114,7 +114,7 @@ variable :HCTI_API_KEY, :String, default: "Optional"
# (https://docs.fastly.com/api/)
variable :FASTLY_API_KEY, :String, default: ""
variable :FASTLY_SERVICE_ID, :String, default: ""
variable :FASTLY_WHITELIST_PARAMS_SNIPPET_NAME, :String, default: ""
variable :FASTLY_SAFE_PARAMS_SNIPPET_NAME, :String, default: ""
# Honeycomb for monitoring and observability
# (https://www.honeycomb.io/)

4
app/assets/javascripts/base.js.erb
View file

@ -56,7 +56,7 @@ var instantClick
return target
}
function isBlacklisted(elem) {
function isNotPreloadable(elem) {
do {
if (!elem.hasAttribute) { // Parent of <html>
break
@ -80,7 +80,7 @@ var instantClick
|| a.href.indexOf(domain + '/') != 0 // Another domain, or no href attribute
|| (a.href.indexOf('#') > -1
&& removeHash(a.href) == $currentLocationWithoutHash) // Anchor
|| isBlacklisted(a)
|| isNotPreloadable(a)
) {
return false
}

10
app/services/fastly_vcl/whitelisted_params.rb → app/services/fastly_vcl/safe_params.rb
View file

@ -1,10 +1,10 @@
module FastlyVCL
# Handles updates to our VCL snippet on Fastly that whitelists params
class WhitelistedParams
# Handles updates to our VCL snippet on Fastly that lists safe params
class SafeParams
VCL_DELIMITER_START = "^(".freeze
VCL_DELIMITER_END = ")$".freeze
SNIPPET_NAME = ApplicationConfig["FASTLY_WHITELIST_PARAMS_SNIPPET_NAME"].freeze
FILE_PARAMS = YAML.load_file("config/fastly/whitelisted_params.yml").freeze
SNIPPET_NAME = ApplicationConfig["FASTLY_SAFE_PARAMS_SNIPPET_NAME"].freeze
FILE_PARAMS = YAML.load_file("config/fastly/safe_params.yml").freeze
class << self
def update
@ -60,7 +60,7 @@ module FastlyVCL
"new_version:#{new_version.number}",
]
DatadogStatsClient.increment("fastly.whitelist", tags: tags)
DatadogStatsClient.increment("fastly.safelist", tags: tags)
end
end
end

0
config/fastly/whitelisted_params.yml → config/fastly/safe_params.yml
View file

2
config/initializers/docker.rb
View file

@ -3,7 +3,7 @@ if Rails.env.development? && File.file?("/.dockerenv")
# Using shell tools so we don't need to require Socket and IPAddr
host_ip = `/sbin/ip route|awk '/default/ { print $3 }'`.strip
logger = Logger.new(STDOUT)
logger.info "Whitelisting #{host_ip} for BetterErrors and Web Console"
logger.info "Allowing #{host_ip} for BetterErrors and Web Console"
if defined?(BetterErrors::Middleware)
BetterErrors::Middleware.allow_ip!(host_ip)

2
config/initializers/doorkeeper.rb
View file

@ -257,7 +257,7 @@ Doorkeeper.configure do
# force_ssl_in_redirect_uri { |uri| uri.host != 'localhost' }
# Specify what redirect URI's you want to block during Application creation.
# Any redirect URI is whitelisted by default.
# Any redirect URI is enabled by default.
#
# You can use this option in order to forbid URI's with 'javascript' scheme
# for example.

25
docs/backend/fastly.md
View file

@ -13,16 +13,16 @@ If you want to learn more about we use Fastly, check out this
[@benhalpern](https://dev.to/ben), gave at RailsConf 2018 talking about how we
made our app so fast it went viral.
## Whitelisting query string parameters
## Adding query string parameters to a safe list
In the context of contributing, here's what you need to know about Fastly. In
order for our servers to receive any sort of query string parameters in a
request, they must first be whitelisted in Fastly. For example, if you're
request, they must first be marked as safe in Fastly. For example, if you're
creating a new API endpoint or updating an existing one to accept new
parameters, you'll need to update Fastly.
The reason we whitelist parameters in Fastly this way is so we don't have to
consider junk parameters when busting the caches. Check out our
The reason we have a safe list of parameters in Fastly this way is so we don't
have to consider junk parameters when busting the caches. Check out our
[`CacheBuster`](https://github.com/thepracticaldev/dev.to/blob/master/app/labor/cache_buster.rb)
to see examples of this.
@ -33,8 +33,8 @@ it programmatically using the Fastly
## How it works
We created a new file, `config/fastly/whitelisted_params.yml`, to house all of
the whitelisted params in Fastly.
We created a new file, `config/fastly/safe_params.yml`, to house all of the safe
params in Fastly.
If you need to update this list, simply update this file. It's as easy as that!
@ -43,13 +43,12 @@ _Fastly is not setup for development._
## In production
If you have Fastly configured in Production and have a similar custom VCL script
to whitelist query string params, make sure you've set the
`FASTLY_WHITELIST_PARAMS_SNIPPET_NAME` ENV variable with the name of the VCL
snippet you have configured in Fastly.
to list safe query string params, make sure you've set the
`FASTLY_SAFE_PARAMS_SNIPPET_NAME` ENV variable with the name of the VCL snippet
you have configured in Fastly.
Whitelisted params on Fastly are updated automatically when a production deploy
goes out unless this key is not set (i.e. you don't have a similar custom VCL
setup).
Safe params on Fastly are updated automatically when a production deploy goes
out unless this key is not set (i.e. you don't have a similar custom VCL setup).
We do this by executing `bin/rails fastly:update_whitelisted_params` in our
We do this by executing `bin/rails fastly:update_safe_params` in our
`release-tasks.sh` script.

13
lib/tasks/fastly.rake
View file

@ -1,17 +1,20 @@
namespace :fastly do
desc "Update VCL for whitelisted params on Fastly"
task update_whitelisted_params: :environment do
desc "Update VCL for safe params on Fastly"
task update_safe_params: :environment do
fastly_credentials = %w[
FASTLY_API_KEY
FASTLY_SERVICE_ID
FASTLY_WHITELIST_PARAMS_SNIPPET_NAME
FASTLY_SAFE_PARAMS_SNIPPET_NAME
]
if fastly_credentials.any? { |cred| ApplicationConfig[cred].blank? }
puts "Fastly not configured. Please set #{fastly_credentials.join(", ")} in your environment."
Rails.logger.info(
"Fastly not configured. Please set #{fastly_credentials.join(", ")} in your environment."
)
next
end
FastlyVCL::WhitelistedParams.update
FastlyVCL::SafeParams.update
end
end

2
release-tasks.sh
View file

@ -17,7 +17,7 @@ set -Eex
# runs migration for Postgres, setups/updates Elasticsearch
# and boots the app to check there are no errors
STATEMENT_TIMEOUT=180000 bundle exec rails db:migrate
bundle exec rake fastly:update_whitelisted_params
bundle exec rake fastly:update_safe_params
bundle exec rake search:setup
bundle exec rake data_updates:enqueue_data_update_worker
bundle exec rails runner "puts 'app load success'"

10
spec/services/fastly_vcl/whitelisted_params_spec.rb → spec/services/fastly_vcl/safe_params_spec.rb
View file

@ -1,12 +1,14 @@
require "rails_helper"
RSpec.describe FastlyVCL::WhitelistedParams, type: :service do
RSpec.describe FastlyVCL::SafeParams, type: :service do
let(:fastly) { instance_double(Fastly) }
let(:fastly_service) { instance_double(Fastly::Service) }
let(:fastly_version) { instance_double(Fastly::Version) }
let(:fastly_snippet) { instance_double(Fastly::Snippet) }
let(:file_params) { YAML.load_file("config/fastly/whitelisted_params.yml") }
let(:snippet_content) { "#{described_class::VCL_DELIMITER_START}#{file_params.join('|')}#{described_class::VCL_DELIMITER_END}" }
let(:file_params) { YAML.load_file("config/fastly/safe_params.yml") }
let(:snippet_content) do
"#{described_class::VCL_DELIMITER_START}#{file_params.join('|')}#{described_class::VCL_DELIMITER_END}"
end
# Fastly isn't setup for test or development environments so we have to stub
# quite a bit here to simulate Fastly working ¯\_(ツ)_/¯
@ -67,7 +69,7 @@ RSpec.describe FastlyVCL::WhitelistedParams, type: :service do
tags = hash_including(tags: array_including("added_params:new_param", "removed_params:#{old_param}", "new_version:#{fastly_version.number}"))
expect(DatadogStatsClient).to have_received(:increment).with("fastly.whitelist", tags)
expect(DatadogStatsClient).to have_received(:increment).with("fastly.safelist", tags)
end
end
end

16
spec/tasks/fastly_spec.rb
View file

@ -6,14 +6,16 @@ RSpec.describe "Fastly tasks", type: :task do
PracticalDeveloper::Application.load_tasks
end
describe "#update_whitelisted_params" do
describe "#update_safe_params" do
it "doesn't run if Fastly isn't configured" do
allow(ApplicationConfig).to receive(:[]).with("FASTLY_API_KEY").and_return(nil)
allow(ApplicationConfig).to receive(:[]).with("FASTLY_SERVICE_ID").and_return(nil)
allow(ApplicationConfig).to receive(:[]).with("FASTLY_WHITELIST_PARAMS_SNIPPET_NAME").and_return(nil)
allow(FastlyVCL::WhitelistedParams).to receive(:update)
Rake::Task["fastly:update_whitelisted_params"].invoke
expect(FastlyVCL::WhitelistedParams).not_to have_received(:update)
%w[FASTLY_API_KEY FASTLY_SERVICE_ID FASTLY_SAFE_PARAMS_SNIPPET_NAME].each do |var|
allow(ApplicationConfig).to receive(:[]).with(var).and_return(nil)
end
allow(FastlyVCL::SafeParams).to receive(:update)
Rake::Task["fastly:update_safe_params"].invoke
expect(FastlyVCL::SafeParams).not_to have_received(:update)
end
end
end