In the general sense we have an existing `order by` that follows the
following format: `order(x) = f(x) * rand() ^ ( 1 / g(x))`
For the present [20220603-variant-b][1], `g(x)` is the article’s score.
And `f(x)` is 1.
This pull request introduces a few named variations for the above
`order(x)` formula.
The three named formats each use a different `g(x)` function; namely
`greatest(0.1, ln(1 + greatest(0, public_reactions_count)))`. The more
public_reactions the article has the closer to zero the resulting
positive number will be. And raising a random number, with range
`(0..1)`, to the inverse of that number will tend to mean that artciles
with many reactions will tend to group towards the higher end of the
random range, whereas articles with few reactions will have a more even
distribution across the range.
Further the `ln` function helps dampen the weight we give to reactions.
The variation within these three order levers is on the `f(x)` portion:
1. Using `published_at`
2. Using `last_commented_at`
3. Using a mix of `published_at` and `last_commented_at`
In all cases, those dates are converted to an integer. A date 2 weeks
ago will result in a number less than a date 1 week ago which will be
less than a today.
The idea for the mix of `published_at` and `last_commented_at` is to see
a mix of things where some “older” articles (from the underlying result
set) will get a chance to show up earlier in the feed.
My rational for this order structure is because the `RANDOM() ^ (1 /
score)` is showing possible success in a current experiment. And in
past trials long past, it rose to the top as a likely and viable
contender.
These new `order_by` lever patterns introduce three things:
1. A variable based on a date in time
2. A dampening of what can be a large number; (we have scores in the
publication range that are between -200 and 750 so I think it is
important to apply a dampening. For example `(0.1 ^ (1/100)) =
0.977` and `(0.01 ^ (1/100)) = 0.955`; in other words these large
scores aggressively pushed their corresponding posts to the top of
the relevance; even as we weren't explicitly querying for those
scores.
3. With the score being quasi-magical, it doesn’t make sense to sort on
that value but to instead sort on a property that represents what
we’re setting as our experiment goals (e.g. publishing something or
commenting on something and/or reacting to something)
Note: it would be feasible to extract these into a singular order lever
that we would configure. But that's a future concern.
Also note, none of these are yet part of any experiment, but are instead
being "queued up" to add to our available corpus.
Closesforem/forem#17834
Related to:
- forem/forem#17827
- forem/forem#17826
- forem/forem#17833
- forem/forem#16128 :: for past order queries to seek as inspiration
[1]:https://github.com/forem/forem/blob/main/config/feed-variants/20220603-variant-b.json
The commit includes three changes that work to allow us to "seed" the
randomization. When using the same seed, and randomizer should/must
return the same sequence of numbers.
First, I added the "seed" parameter to the variant query. If you want
the same sequence of random numbers from query to query, pass the same
seed. Otherwise, we'll use a Ruby generated random seed.
Second, I added a virtual column to the virtual `article_relevancies`
table. If you provide a seed, and call the query twice, the first row
in each of `article_relevancies` will have the same `randomized_value`,
likewise the second row, etc.
Third, I amended the existing `order_by_lever` that had a `RANDOM()`
postgresql function call. I replaced that with the `randomized_value`
which is computed based on the provided seed.
Fundamentally the idea is to allow for randomness but introduce possible
repeatability; a hard thing considering that some of the inner selection
criteria is not fixed (e.g. number of reactions can change from moment
to moment).
I wrote a [complimentary blog post][1] to further explain what's
happening.
Related to forem/forem#17826
[1]:https://takeonrules.com/2022/06/03/adding-reproducible-randomization-to-sql-queries/
* Adding new relevancy lever for privileged reactions
This PR includes four changes:
1. Renaming the variant to remove Jennie's name
2. Starting a new experiment
3. Adding a new relevancy lever
4. Adding a new variant that uses the relevancy lever
1 and 2 is somewhat straight-forward.
For 3, we're looking at having a total of 5 "slots" for privileged user
reactions. Hence the four boundary values.
For 4, we're configuring the granular levers with the following:
- `[-∞..-5)` should have a 0.05
- `[-5..0)` should have a 0.5
- `[0..5)` should have a 0.9
- `[5..10)` should have a 0.98
- `[10..∞)` should have a 1.0
The 20220509-variant.json is a copy of 20220422-variant.json but
replaces the `privileged_user_reaction` lever with the
`privileged_user_reaction_granular` lever.
Below is the diff to highlight the similarities and differences between
the two variants.
```shell
❯ diff config/feed-variants/20220509-variant.json config/feed-variants/20220422-variant.json
90c90
< "privileged_user_reaction_granular": {
---
> "privileged_user_reaction": {
92,96c92,93
< [-2, 0.05],
< [-1, 0.5],
< [0, 0.9],
< [1, 0.98],
< [2, 1]
---
> [-1, 0.2],
> [1, 1]
98,102c95,97
< "fallback": 0.9,
< "very_negative_reaction_threshold": -5,
< "negative_reaction_threshold": 0,
< "positive_reaction_threshold": 5,
< "very_positive_reaction_threshold": 10
---
> "fallback": 0.95,
> "negative_reaction_threshold": -10,
> "positive_reaction_threshold": 10
```
Closesforem/forem#17584
* Update app/models/articles/feeds.rb
Co-authored-by: Josh Puetz <josh@dev.to>
* Adjusting testing logic to reflect latest experiment
* Adjusting negative threshold
Co-authored-by: Josh Puetz <josh@dev.to>
Prior to this commit, the SQL fragments included variables that were set
configured at a global level.
With this commit, we're now saying that each lever "knows" what variable
it needs; and providing the means at lever declaration time to "say"
what those variable names are. (e.g. `Articles::Feeds::LEVER_CATALOG`).
Then as part of the variant configuration (in the
`./config/feed-variants/*.json` files) we now include the expected value
of those parameters; which by convention (and coercion) are integers.
This relates to forem/forem#17584 because we want to move from a
privileged user reaction that has two values (`negative` and `positive`)
into four values (`very_negative`, `negative`, `positive`, and
`very_positive`). To do that, we'll create a new lever; but that's for
another pull request.
You had a good ride, but your successor, the
`Articles::Feeds::VariantQuery`, is doing well in production. And has
been for quite awhile.
You helped us get to a better spot in regards to the feed algorithm, but
your time has come to rest as a memory in our git history.
* Allowing VariantQuery for Feed Generation
Apologies for the breadth of this pull request, I had considered many
small commits, but felt that would've been more effort for the value
provided.
This commit includes the following:
- Documentation updates to the feed variant (though not the final pass)
- Renaming and adding RelevancyLevers that help differentiate
- Adding RelevancyLever#range to provide documentation
- Reducing redundent controller logic by making a
`Articles::Feeds.feed_for` method.
- Adding some configuration validation for RelevancyLevers
- Adding constants for better clarification
- Testing unhappy paths for feed configuration
- Adjusting the module namespace of some objects
- Exposing top-level configurations for variants (along with their
defaults)
- Creating the VariantyQuery that at present inherits from the
`Articles::Feeds::WeightedQueryStrategy`
As implemented, we can deploy this code to production without using the
new VariantQuery. Once we toggle on the
`:feed_uses_variant_query_feature` FeatureFlag, it will switch to using
the VariantQuery. The VariantQuery's two variants and the
internal configuration of `Articles::Feeds::WeightedQueryStrategy`
produce the same query.
The goal of this factor is to allow for a quick on and off toggle of the
feed query; to ensure that what we introduce remains performant.
- Closesforem/forem#17272
- Closesforem/forem#17276
- Closesforem/forem#17216
In addition, I will be recording a code-walkthrough and linking that
recording to the pull request.
* Apply suggestions from code review
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
**tl;dr** This PR is looking to provide a means to programmatically
assemble, from system configurations, what is currently the
[Articles::Feeds::WeightedQueryStrategy::SCORING_METHOD_CONFIGURATIONS][1].
Once we merge this PR, instead of having that constant, we'll initialize
the Articles::Feeds::WeightedQueryStrategy with the variant
configuration that we've assembled.
**Introduction**
This pull request has quite a bit going on, but as of now the production
code does not use any of it.
*Note: None of this code is bleeding into production code paths.*
Put your Ruby hat on and let's go for a ride. And apologizes for not
making this a smaller pull request. As I built this, I made many small
commits, but this became the smallest commit that provided the most
context without integrating into production code.
Before we get started, you may want to familiarize yourself with where
we're going. The `./app/models/articles/feeds/README.md` provides
further guidance; but fair warning reader, I have not reconciled the
README's language with what emerged as I wrote this pull request.
Those of you who are part of Forem Core team, you can read the
[Refinements to Our Present Feed Configuration][2]
to provide some insight into what's happening.
**Why not reconcile?** *Because I want to have all of the language and
thoughts available for you to help consider how best to name and model
this.*
**On to the Review**
The purpose of this pull request is to provide a mechanism for
engineering to provide a series of variant query levers. And to allow
us to easily configure those available query levers into a variant
query.
*Why the mix of Ruby configuration and JSON?*
The Ruby levers are meant to indicate that this is code we don't want to
expose in text based configuration because it might create SQL inject
points. (More on that when we integrate the code of this PR into the
production implementation)
The JSON levers are meant to indicate that "anyone" can pick the
appropriate/available levers and configure how they are set. In other
words, these values do not create SQL injection issues.
**What I Need from You**
I am putting this forward as a draft so you can read this code ahead of
time. I'm then going to schedule a synchronous code review where we
record our collective walk through of the implementation.
Closes:
- forem/forem#17268
- forem/forem#17269
- forem/forem#17270
- forem/forem#17271
- forem/forem#17273
Relates to:
- forem/forem#17308
- forem/forem#17245
[1]:6818ef3ed0/app/services/articles/feeds/weighted_query_strategy.rb (L84-L231)
[2]:https://forem.team/jeremy/refinements-to-our-present-feed-configuration-1p0c
