docbrown/docs/technical-overview/ab_testing.md

A/B testing

We use the Field Test gem for conducting simple A/B tests.

If you want to propose an A/B test of a feature, you may make a pull request which defines the hypotheses and what admins should look for to declare a winner. As A/B tests are going to have results that may differ from Forem to Forem, the process is relatively immature. In the future we may have more studies that can return anonymous ecosystem-wide results.

A/B tests are inherently the most useful in large Forems, where qualitative feedback may be more useful on small Forems. As such, DEV is our largest Forem and therefore can provide us with the most feedback for our existing A/B tests. However, we must keep in mind that DEV results may not apply well to future large Forems. We should seek to re-run useful experiments within the ecosystem after time has passed.

Creating a new A/B test

Follow the guidelines of the field test gem and add the test info to config/field_test.yml.

Then where you want to trigger the variant, you'll add some code like this:

  test_variant = field_test(:follow_implicit_points, participant: user)
  case test_variant
  when "no_implicit_score"
    0
  when "half_weight_after_log"
    Math.log(occurrences + bonus + 1) * 0.5
  when "double_weight_after_log"
    Math.log(occurrences + bonus + 1) * 2.0
  when "double_bonus_before_log"
    Math.log(occurrences + (bonus * 2) + 1)
  when "without_weighting_bonus"
    Math.log(occurrences + 1)
  else # base - Our current "default" implementation
    Math.log(occurrences + bonus + 1) # + 1 in all cases is to avoid log(0) => -infinity
  end

Which would find or create the test variant for that user in particular. If this code is not called in the controller or view, you'll need to first include the gem helpers at the top of the file...

include FieldTest::Helpers

To record a successful field test outcome, you should call something like this

    Users::RecordFieldTestEventWorker
      .perform_async(user_id, :follow_implicit_points, "user_creates_reaction")

And modify that class as needed to determine whether to record the successful trial.