docbrown/spec/rails_helper.rb
Jeremy Friesen 816e92e549
Configurable weighted feed strategy (#15240)
* AB Test for Feed Strategy

Adding the plumbing for performing AB Tests on the feed strategy.

You can add the environment variable `AB_TEST_SERVICE_FEED_STRATEGY` to
specify the field test to run.

For example, from the forem application home directory you can run the
following to force a strategy for all users:

```console
AB_TEST_SERVICE_FEED_STRATEGY=original bin/start
```

Forcing the strategy can significantly help with quality assurance testing.

* Adding configurable weighted feed strategy

I wrote about this exploration on [DEV.to][1].  This particular
implementation provides significant configuration options for A/B
testing.

At it's core this `Articles::Feeds::WeightedQueryStrategy` provides a
means to query articles that are relevant to the user.  The
configuration helps us test and refine those values.

Further more this implementation introduces the idea of some common
interfaces for feed strategies.  I'm not happy with the naming
convention but we can work to change that.

This even hints at a mechanism for administrators to configure.

Related to forem/rfcs#314, as I work towards understanding the feed
algorithm. And #15180

[1]:https://dev.to/jeremyf/practicing-postgresql-and-postulating-improvements-5m

* Renaming AbTestService to AbExperiment

I'm loath to end any production class's filename with "test".  I don't
want minitest or rspec coming along and attempting to "test" that file.
In addition, I find that Experiment better describes the concept.

Also, per conversations with Michael K, we favor nouns objects to go
in app/models and verb objects to go in app/services.  That's a great
separation.

* Factoring AbExperiment into a Controller concern

I wish this could be a stand-alone module but the implementation of
field test has cases that require controller context.

* Sanitizing configurable weighted strategy cases

* Minor refactoring of coding structure

* Renaming parameter for clarity

* Renaming variable to avoid confusion

Prior to the commit, `env` is a controller variable.  The injected `env`
variable didn't collide, but did create just a bit of confusion.

With this change, I'm trying to disentangle from that variable.

* Factoring AbExperiment away from mixin concern

Per discussion with Michael K, we both agree that we do not like
controller concern mixins.  They can quickly complicate the legibility
of any given codebase.

This refactor provides a common and simplified interface for a Plain Old
Ruby Object (PORO) that accounts for the implementation considerations
of the underlying "field_test" controller method.

* Adding Datadog.tracer.trace around feed actions

* Adding featured_article_factor

Prior to this commit, there was no consideration for how moderators
engage with an article.  This is a first nudge towards that effort.

* Removing spec

Add a spec that says yes or no we're feature testing something seems
fragile.  Namely, why are we wanting to assert that as a truism.

* Helping ensure we're "tracing" the same things

From the code comment:

> Hey, why the to_a you say?  Because the
> LargeForemExperimental has already done this.  But the
> weighted strategy has not.  I also don't want to alter the
> weighted query implementation as it returns a lovely
> ActiveRecord::Relation.  So this is a concession.

* Addressing parameters for tracing function

* For weight feed query only count positive tags

A user assigns explicit points to the tags that they do or do not want
to see.  This change treats the negative explicit_points as an indicator
that they don't want to see the tag.

* Adding weighted factor for privileged user reactions

* Updating comments to better clarify

* Adding additional constraint for published_at

* Updating documentation and adding logger

* Adding documentation concerning method call

* Upping logging level for experiments

See conversations in https://github.com/forem/forem/pull/15240

* Adding group by clauses to appease postgres configs

* Injecting relevant scoring method's joins

Prior to this commit, we included all of the joins regardless of the
scoring methods chosen.  With this commit, we only inject joins that are
used with the commit.

What this would allow us to do is to then have some pre-cooked scoring
methods but not turn them on.  Or to turn off the more expensive scoring
methods.

* Adding feature flag for feed strategy experiment
2021-11-29 10:46:56 -05:00

230 lines
8.1 KiB
Ruby

ENV["RAILS_ENV"] = "test"
require "knapsack_pro"
KnapsackPro::Adapters::RSpecAdapter.bind
require "spec_helper"
require File.expand_path("../config/environment", __dir__)
require "rspec/rails"
abort("The Rails environment is running in production mode!") if Rails.env.production?
# Add additional requires below this line. Rails is not loaded until this point!
require "fakeredis/rspec"
require "pundit/matchers"
require "pundit/rspec"
require "sidekiq/testing"
require "test_prof/factory_prof/nate_heckler"
require "validate_url/rspec_matcher"
require "webmock/rspec"
# Requires supporting ruby files with custom matchers and macros, etc, in
# spec/support/ and its subdirectories. Files matching `spec/**/*_spec.rb` are
# run as spec files by default. This means that files in spec/support that end
# in _spec.rb will both be required and run as specs, causing the specs to be
# run twice. It is recommended that you do not name files matching this glob to
# end with _spec.rb. You can configure this pattern with the --pattern
# option on the command line or in ~/.rspec, .rspec or `.rspec-local`.
#
# The following line is provided for convenience purposes. It has the downside
# of increasing the boot-up time by auto-requiring all files in the support
# directory. Alternatively, in the individual `*_spec.rb` files, manually
# require only the support files necessary.
Dir[Rails.root.join("spec/support/**/*.rb")].each { |f| require f }
Dir[Rails.root.join("spec/system/shared_examples/**/*.rb")].each { |f| require f }
Dir[Rails.root.join("spec/models/shared_examples/**/*.rb")].each { |f| require f }
Dir[Rails.root.join("spec/workers/shared_examples/**/*.rb")].each { |f| require f }
Dir[Rails.root.join("spec/initializers/shared_examples/**/*.rb")].each { |f| require f }
Dir[Rails.root.join("spec/mailers/shared_examples/**/*.rb")].each { |f| require f }
# Checks for pending migrations before tests are run.
# If you are not using ActiveRecord, you can remove this line.
ActiveRecord::Migration.maintain_test_schema!
# Disable internet connection with Webmock
# allow browser websites, so that "webdrivers" can access their binaries
# see <https://github.com/titusfortner/webdrivers/wiki/Using-with-VCR-or-WebMock>
allowed_sites = [
"chromedriver.storage.googleapis.com",
"github.com/mozilla/geckodriver/releases",
"selenium-release.storage.googleapis.com",
"developer.microsoft.com/en-us/microsoft-edge/tools/webdriver",
"api.knapsackpro.com",
]
WebMock.disable_net_connect!(allow_localhost: true, allow: allowed_sites)
RSpec::Matchers.define_negated_matcher :not_change, :change
Rack::Attack.enabled = false
# `browser`, a dependency of `field_test`, starting from version 3.0
# considers the empty user agent a bot, which will fail tests as we
# explicitly configure field tests to exclude bots
# see https://github.com/fnando/browser/blob/master/CHANGELOG.md#300
Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)
RSpec.configure do |config|
config.use_transactional_fixtures = true
config.fixture_path = "#{::Rails.root}/spec/fixtures"
config.include ActionMailer::TestHelper
config.include ApplicationHelper
config.include Devise::Test::ControllerHelpers, type: :view
config.include Devise::Test::IntegrationHelpers, type: :request
config.include Devise::Test::IntegrationHelpers, type: :system
config.include FactoryBot::Syntax::Methods
config.include OmniauthHelpers
config.include RpushHelpers
config.include SidekiqTestHelpers
config.extend WithModel
config.after(:each, type: :system) do
Warden::Manager._on_request.clear
end
config.after(:each, type: :request) do
Warden::Manager._on_request.clear
end
config.around do |example|
case example.metadata[:sidekiq]
when :inline
Sidekiq::Testing.inline! { example.run }
when :fake
Sidekiq::Testing.fake! { example.run }
when :disable
Sidekiq::Testing.disable! { example.run }
else
example.run
end
end
config.before(:suite) do
# Set the TZ ENV variable with the current random timezone from zonebie
# which we can then use to properly set the browser time for Capybara specs
ENV["TZ"] = Time.zone.tzinfo.name
end
config.before do
# Worker jobs shouldn't linger around between tests
Sidekiq::Worker.clear_all
# Disable SSRF protection for CarrierWave specs
# See: https://github.com/carrierwaveuploader/carrierwave/issues/2531
# rubocop:disable RSpec/AnyInstance
allow_any_instance_of(CarrierWave::Downloader::Base)
.to receive(:skip_ssrf_protection?).and_return(true)
# rubocop:enable RSpec/AnyInstance
# Enable the Connect feature flag for tests
# Doing this via a stub gets rid of the following error:
# "Please stub a default value first if message might be received with other args as well."
allow(FeatureFlag).to receive(:enabled?).and_call_original
allow(FeatureFlag).to receive(:enabled?).with(:connect).and_return(true)
end
config.around(:each, :flaky) do |ex|
ex.run_with_retry retry: 3
end
config.around(:each, throttle: true) do |example|
Rack::Attack.enabled = true
example.run
Rack::Attack.enabled = false
end
config.after do
Settings::General.clear_cache
end
# Only turn on VCR if :vcr is included metadata keys
config.around do |ex|
if ex.metadata.key?(:vcr)
ex.run
else
VCR.turned_off { ex.run }
end
end
config.before do
stub_request(:any, /res.cloudinary.com/).to_rack("dsdsdsds")
stub_request(:any, /emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/).to_rack("stubbed-emoji")
stub_request(:post, /api.fastly.com/)
.to_return(status: 200, body: "".to_json, headers: {})
stub_request(:any, /localhost:9090/)
.to_return(status: 200, body: "OK".to_json, headers: {})
# for twitter image cdn
stub_request(:get, /twimg.com/)
.to_return(status: 200, body: "", headers: {})
stub_request(:any, /api.mailchimp.com/)
.to_return(status: 200, body: "", headers: {})
stub_request(:any, /dummyimage.com/)
.to_return(status: 200, body: "", headers: {})
stub_request(:post, "http://www.google-analytics.com/collect")
.to_return(status: 200, body: "", headers: {})
stub_request(:any, /robohash.org/)
.with(headers:
{
"Accept" => "*/*",
"Accept-Encoding" => "gzip;q=1.0,deflate;q=0.6,identity;q=0.3",
"User-Agent" => "Ruby"
}).to_return(status: 200, body: "", headers: {})
allow(Settings::Community).to receive(:community_description).and_return("Some description")
allow(Settings::UserExperience).to receive(:public).and_return(true)
allow(Settings::General).to receive(:waiting_on_first_user).and_return(false)
# Default to have field a field test available.
config = { "experiments" =>
{ "wut" =>
{ "variants" => %w[base var_1],
"weights" => [50, 50],
"goals" => %w[user_creates_comment
user_creates_comment_four_days_in_week
user_views_article_four_days_in_week
user_views_article_four_hours_in_day
user_views_article_nine_days_in_two_week
user_views_article_twelve_hours_in_five_days] } },
"exclude" => { "bots" => true },
"cache" => true,
"cookies" => false }
begin
# Add the field tests that are currently configured (if any).
field_tests = Psych.load_file("config/field_test.yml")
config["experiments"].merge!(field_tests.fetch("experiments", {}))
rescue StandardError
# Accept that we may not have experiments.
end
allow(FieldTest).to receive(:config).and_return(config)
end
config.after do
Timecop.return
end
config.after(:suite) do
WebMock.disable_net_connect!(
allow_localhost: true,
allow: allowed_sites,
)
end
OmniAuth.config.test_mode = true
OmniAuth.config.logger = Rails.logger
config.infer_spec_type_from_file_location!
# Filter lines from Rails gems in backtraces.
config.filter_rails_from_backtrace!
# arbitrary gems may also be filtered via:
# config.filter_gems_from_backtrace("gem name")
end