* Bypassing validiation for data script
Prior to this commit, we attempted to run a script against all articles
using validation. In the case of DEV, we have lots of articles that
would no longer validate. This change now updates the column value
without running validation.
Relates to #16075
* Bump for travis
* Add boolean attribute main_image_from_frontmatter to indicate if cover image was set via frontmatter
* use article model spec for main_image_from_frontmatter test cases
* Add data migration script and spec for main_image_from_frontmatter for articles
* Update app/models/article.rb
Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com>
Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com>
Co-authored-by: Michael Kohl <me@citizen428.net>
* move home link to a customisable navlink
* add trailing slash for path
* update specs
* replace positions, set home link to -1
* add update script for adding home navigation link
* update data update script
On the DEV.to database, I ran the following SQL:
```sql
SELECT name, short_summary FROM tags WHERE short_summary LIKE '%<%';
```
There were three tags with a `<` in them:
- changelog
- cnc2018
- javascript
This PR will tidy up our short summary as part of saving a tag. It will
also tidy up the existing tags.
* Generator output
Add your logic here!
* replicate the logic from the forem connect migration
* Add data update script
This will generate the expected broadcast message to fix#16059
* Add apple connect to seeds file
* feat/WIP: first version of the svg logo to png logo DUS
* feat: logoSVG Uploader
* refactor: remove the if original_filename
* feat: add a test for the logo_svg_uploader
* feat: update the test and dus
* add soem error handling
* updae the rails spec helper
* feat: provide a content type
* feat: add content type
* Try https://travis-ci.community/t/build-times-out-with-no-apparent-reason/5083/4
* chore: try this suggestion https://stackoverflow.com/questions/41138404/how-to-install-newer-imagemagick-with-webp-support-in-travis-ci-container
* chore: remove libweb adn fix = in the travis.yml
* chore: undo the changes to attempt to update the version for image magick
* feat: install gsfonts for convert
* chore: add a comment for svg
* feat: use the good practices for tempfile, update error handling and update the tests
* chore: update the timestamp on the DUS
* feat: convert to png with a transparent background using Image Magick
* feat: substitute some css variables with real css colors
* chore: revert the path
* fix: set content type
* Update lib/data_update_scripts/20220105112823_migrate_logo_svg_data.rb
Co-authored-by: Jamie Gaskins <jamie@forem.com>
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Tidying up and documenting Tag model
Prior to this commit we had a custom `where(alias_for: [nil, ""])`
call. That call highlighted that we lacked a term for a Tag that was
not an alias. As part of this commit, I named that a "concrete" tag.
Further, I added scopes to assist in helping "name" those concepts.
This commit also adds a data migration and utilization of
StringAttributeCleaner to hopefully get away from `alias_for == ""`
situations.
As of writing this commit <2022-01-04 Tue 17:22 UTC>, in DEV.to we had 5
tags with `alias_for == ""`:
- actionshackathon21
- regex
- atlashackathon
- hotwire
- foremfest
In https://dev.to/admin/blazer I ran the following:
```sql
SELECT name FROM tags WHERE alias_for = ''
```
* Renaming concrete to direct
* Remove reactions to a user when deleting the user
Fixes#15245
---
I have no idea why `create(:vomit_reaction, reactable: user)` gave a
validation error about category being invalid, but `build().save`
worked fine.
* Move relation details to reaction scope `for_user`
Add a user method to access this by passing self to reaction scope
* Create a moderator when flagging user in factory
This might be a missing requirement in the reactions factory - but
this suppresses an error about invalid category I was getting when
setting a privileged reaction on the user via
create(:vomit_reaction, reactable: user)
Everywhere else it looks like we do this with `user: moderator` (where
there's a trusted user in the surrounding context).
* Move trusted user requirement into vomit_reaction factory
* Add a data update script to remove reactions to deleted users
This cleans up the remaining invalid references and should make the
change in #15249 obsolete (i.e. we don't need to limit the view to
exclude items that no longer exist).
The data update script "nullify empty tag colors" failed to update
because of the way it was written - in two passes, updating the
background color for tags where the background color was empty, then
updating the foreground color for tags where the foreground color was
empty.
Because both attributes have a validation regex, neither pass updated
any tags (the problematic cases observed had empty string values in both
columns).
Perform in a single loop, over the union of the two sets (expected to
include only the same rows as before), an update for both columns at
once.
Additionally, use update_columns which bypasses _other_ validation
issues (the only one I could suspect is the alias_for check).
Nullify empty strings in alias_for (simplifies a few things, now it's
either valid or null).
When a tag is detected with a "danlging alias" repair it by setting
alias for nil.
* Removing the stackbit integration
The feature didn't quite work and Stackbit no longer supports this
integration.
Yes there are a few places where the webhooks has a string of "stackbit"
but I'm hesitant to remove that, as they are the part of the Webhooks
tests.
Closes#15700
* Update lib/data_update_scripts/20211206222716_remove_stackbit_page.rb
Co-authored-by: Michael Kohl <citizen428@forem.com>
Co-authored-by: Michael Kohl <citizen428@forem.com>
* AB Test for Feed Strategy
Adding the plumbing for performing AB Tests on the feed strategy.
You can add the environment variable `AB_TEST_SERVICE_FEED_STRATEGY` to
specify the field test to run.
For example, from the forem application home directory you can run the
following to force a strategy for all users:
```console
AB_TEST_SERVICE_FEED_STRATEGY=original bin/start
```
Forcing the strategy can significantly help with quality assurance testing.
* Adding configurable weighted feed strategy
I wrote about this exploration on [DEV.to][1]. This particular
implementation provides significant configuration options for A/B
testing.
At it's core this `Articles::Feeds::WeightedQueryStrategy` provides a
means to query articles that are relevant to the user. The
configuration helps us test and refine those values.
Further more this implementation introduces the idea of some common
interfaces for feed strategies. I'm not happy with the naming
convention but we can work to change that.
This even hints at a mechanism for administrators to configure.
Related to forem/rfcs#314, as I work towards understanding the feed
algorithm. And #15180
[1]:https://dev.to/jeremyf/practicing-postgresql-and-postulating-improvements-5m
* Renaming AbTestService to AbExperiment
I'm loath to end any production class's filename with "test". I don't
want minitest or rspec coming along and attempting to "test" that file.
In addition, I find that Experiment better describes the concept.
Also, per conversations with Michael K, we favor nouns objects to go
in app/models and verb objects to go in app/services. That's a great
separation.
* Factoring AbExperiment into a Controller concern
I wish this could be a stand-alone module but the implementation of
field test has cases that require controller context.
* Sanitizing configurable weighted strategy cases
* Minor refactoring of coding structure
* Renaming parameter for clarity
* Renaming variable to avoid confusion
Prior to the commit, `env` is a controller variable. The injected `env`
variable didn't collide, but did create just a bit of confusion.
With this change, I'm trying to disentangle from that variable.
* Factoring AbExperiment away from mixin concern
Per discussion with Michael K, we both agree that we do not like
controller concern mixins. They can quickly complicate the legibility
of any given codebase.
This refactor provides a common and simplified interface for a Plain Old
Ruby Object (PORO) that accounts for the implementation considerations
of the underlying "field_test" controller method.
* Adding Datadog.tracer.trace around feed actions
* Adding featured_article_factor
Prior to this commit, there was no consideration for how moderators
engage with an article. This is a first nudge towards that effort.
* Removing spec
Add a spec that says yes or no we're feature testing something seems
fragile. Namely, why are we wanting to assert that as a truism.
* Helping ensure we're "tracing" the same things
From the code comment:
> Hey, why the to_a you say? Because the
> LargeForemExperimental has already done this. But the
> weighted strategy has not. I also don't want to alter the
> weighted query implementation as it returns a lovely
> ActiveRecord::Relation. So this is a concession.
* Addressing parameters for tracing function
* For weight feed query only count positive tags
A user assigns explicit points to the tags that they do or do not want
to see. This change treats the negative explicit_points as an indicator
that they don't want to see the tag.
* Adding weighted factor for privileged user reactions
* Updating comments to better clarify
* Adding additional constraint for published_at
* Updating documentation and adding logger
* Adding documentation concerning method call
* Upping logging level for experiments
See conversations in https://github.com/forem/forem/pull/15240
* Adding group by clauses to appease postgres configs
* Injecting relevant scoring method's joins
Prior to this commit, we included all of the joins regardless of the
scoring methods chosen. With this commit, we only inject joins that are
used with the commit.
What this would allow us to do is to then have some pre-cooked scoring
methods but not turn them on. Or to turn off the more expensive scoring
methods.
* Adding feature flag for feed strategy experiment
* Enable Forem (Passport) Auth
* Remove feature flag DUS
* Add prompt for Forem Passport in admin
* Add spec & fix broken one
* Link to docs instead of passport site
* Nullify invalid tag colors
Set bg_color_hex and text_color_hex to nil when they were an empty
string.
This is a cleanup of legacy data (from form submissions, when the
field was unset and another attribute was updated, the value could be
saved as '' rather than set nil). This is no longer possible due to
validations added in #10495 so only needs to be handled once.
Because of the way the the scripts to run works, there's always
already a script with this file name by the time this starts.
Remove the guard clause and let it run.
* Caching sum privileged reaction scores
This relates to work on improving the feed. Namely that in the current
proposal in PR #15240 I'm not accounting for how privileged users have
given feedback on an article.
With this change, I will now have a cached value on the articles table
that can be a proxy for how the privileged users have reacted.
In addition there's a small refactor that moves a constant to the
correct scoping object.
Dependent on #15283.
* Fixing method name to `exists?`
Prior to this commit, I was using `exist?` which doesn't work for
ActiveRecord::Relation objects.
* Add one reaction to each article when seeding the database
* Add reaction to article when created
and sync reactions count.
There's still a gap where the redis-cached reaction count could be
present but not cleared during seeder runs (redis is disconnected for
caching and rails cache is set to the null store). Since we are able
to enqueue sidekiq jobs redis is live - but it's a bad idea to use
sidekiq's connection to flush keys in redis.
* Make the reaction count update script a no-op
* Remove unused data update script and limit user id queries
Pull all user ids into an array before you start creating articles,
and sample from the array rather than repeatedly asking for the first
user from a random ordered db result.
* fail if we can't create reaction
thanks @jgaskins
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Revert "fail if we can't create reaction"
This reverts commit c35f6d47d2e8956a240133b9e6ad0d144f1b4722.
There is a uniqueness constraint on (user, reactable, category) that
could be triggered when seeding due to random selection. It's better
to skip creation when a validation error occurs than to break the seed
completely.
There are expected to be 0-9 reactions per article, we don't require a
set number on each article, and likely won't notice any specific
problem if the random number is one lower than it would have been
because some reactions were skipped due to uniqueness violations.
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Remove check_box type from profile fields
* Fix profile field spec
* Delete emphasized_comments_experiment.rb
This file was not supposed to be part of this PR
* Initial forem omniauth strategy setup work
* Finish basic raw proof of concept
* Some playing around
* use OAuth payloads + PASSPORT_OAUTH_URL for local dev
* Use FeatureFlag for Forem Passport Auth
* Working on tests
* Fix tests 🤞🏼 & some cleanup
* Use correct namespace within lib directory (match class namespace)
* Test to ensure Forem Passport auth is restricted by FeatureFlag
* Add broadcast + work on tests
* Update spec/lib/data_update_scripts/insert_forem_connect_broadcast_message_spec.rb
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Hash format
* Schema cleanup + inline comments
* Use temprorary Heroku domain
* More cleanup
* Missed one
* Back to passport.forem.com
* Require correct path in lib
* Apply suggestions from code review
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Use with_indifferent_access for symbol hash access in Forem strategy
Co-authored-by: benhalpern <bendhalpern@gmail.com>
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Add feature flag for connect
* Start using Connect feature flag
* Add feature flag check to plaintext mailers
* Minor fixes/formatting issues
* Fix typo
* Enable Connect feature flag for tests
* Fix feature flag in tests
* Update rails_helper
* Fix typo
* (╯°□°）╯︵ ┻━┻
* One more try
* connect announcement
* Use connect feature flag in admin and admin routes
* Add aria-live attribute to announcement
* Undo schema.rb changes
Co-authored-by: Paweł Ludwiczak <ludwiczakpawel@gmail.com>
* Removing :apple_auth feature flag and debugging
* Fix tests failing due to nil in Devise initializer
* Remove db/schema.rb changes
* Devise config tweak
* cleaning up
* Fix spec + delete debug logs
* Removes temporary beta_access_providers spec + remove feature flag DUS
* Add test to avoid connecting an existing user with SIWA
* Fix merge conflict mistake + more feature flag cleanups
* Add additional test cases to the DUS
And then handle them.
* rubocop changes
Keep let blocks together
prefer blank? to ! + present?
* Test copied script as well
Since the file was fixed, then copied - we want to test the copy that
will actually run (these _should_ be idempotent and running both back
to back will cause no harm).
* Validate website_url profile field is a url
related to issue #14300
May need a data update script to (fixup? remove?) invalid profiles
since not being able to save existing profiles will cause problems.
* Check that URI is a valid url
Require the scheme to be one of https or http, not something like
mailto:// or telnet:// (nobody would do that, but don't try and link
to it if they did).
* Rubocop fixup for validation rule
URI::regexp is obsolete and should not be used. Instead, use URI::DEFAULT_PARSER.make_regexp
Prefer %w[] literals for arrays of words
Prefer the new style validations `validates :column, format: value` to validates_format_of
* Permit empty website_url fields
The previous validation was rejecting nil, which was the default. This
caused a lot of user factory calls to fail (since users didn't need
website urls in the profiles during testing unless the test was about
the website url link).
* Use :url validation as suggested
* Update validation error message
The validate_url gem gives a more complete error message to the
user. Update the spec to expect this.
* Use url_field rather than text_field in profile form
https://apidock.com/rails/v6.1.3.1/ActionView/Helpers/FormHelper/url_field
* Add data update to either fixup or clear invalid website urls
Checking blazer there are about 2600 profiles with "invalid" website
urls, typically a hostname, sometimes a hostname + path component,
which should be fixed up.
Naively add https:// to the front of the url, check that a valid
scheme and host are present on the resulting url, and save.
If an invalid url is generated (specifically, if host or scheme are
nil), just set the website url (with the invalid link) to an empty
string.
It's possible we'd want to notify users affected by this, that was not
included in this pass.
* Skip validations for intentionally invalid test cases
* Remove unused profile fields
* Clean up CSV
* Account for user settings in Profiles::Update
* Move brand color validation to Users::Setting
* Fix specs
* schema file undelete description
* update with main
* update with origin
* Add "section" enum to navlink model
* create migration for section column
* create migration and data-update script
* update related rake tasks
* add scopes for default_links and other_links
* update E2E seeds
* ran migration; correct enum reference
* fix requests and model specs
* write DUS spec; fix factory and DUS
* yarn install
* address PR review comments
* fix migration; add null: false
* Remove yarn.lock from commit
* remove yarn.lock changes from commit
* sync yarn.lock with main
* newline
* schema file undelete description
* feat: v1 of the script
* Flesh out remaining enums under their categories
* complete UsersSettings data update script
* complete DUS for relevant attributes in users and profiles tables
* complete DUS for users_notification_settings
* alphabetize user_settings sql file
* safeguard against null values for "null: false" settings
* Set up actual UsersSettings DUS and specs files
* fix broken DUS script
* complete specs for UsersSetting DUS
* Address QA of specs
* complete specs for users_notification_settings DUS
* fix the typos (thanks Julianna!)
* begin implementation
* still building
* add missing attribute "email_membership_newsletter"
* complete sync code (except race condition for user profile)
* complete implementation, remains tests
* Address PR review and fix Travis fails
* remove superfluous Profile.new
* fix travis fails
* feat: update the users_notification_setting attributes from the user model
* feat: use the config fonts enums to display the fonts
* feat: loop through the keys
* fix profile = nil blowing up; add specs for notification_setting model
* remove unneeded spec
* remove feed validation until after sync code removed; fixes feed_import spec failures
* remove spec associated with feed_url validation in user_setting model
* fix failing spec 😅
* add TODO
* feat: set the user settings in the user controller and use it in the customization form
* feat: move some update logic to the users settings controller thats being used from customization
* feat: show the updated values form the users_settingd and not the user instance
* Generalize redirect back to current tab
* still trying to reflect changed theme upon refresh
* customizations take effect on refresh
* remove 'with_feed' scope from user model
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* start with takeover for fields previously in profiles table
* Takeover code for `publishing_from_rss` section in Settings (#13914)
* implement takeover code part 1
* implement takeover code
* fix feed fetch
* need rhymes help
* complete implementation; specs pending
* fix STUPID omission that caused so many headaches 😫
* implement profile fields pointing to users_settings 🎉
* run migrations
* implement inbox type & guidelines takeover code; specs pending (#13911)
* Point changes in notification settings to `users_notification_settings` table (#13910)
* implement takeover code; remains specs
* address PR feedback; remove related sync code
* address PR review feedback
* need help with routing and specs
* address pr review
* addressing pr review
* Treat implementation edge cases and omissions 😅
* fix uncommented comment
* fixing implementation cases
* address more PR review feedback
* fixing notifications use-cases
* refactor settings controller
* more pr review changes
* solving bugs
* fix broken onboarding
* handle eperience_level calls
* more fixes
* remove unneeded mappings
* add To-dos for quety updates
* remove done TODO
* purge done TODOs
* update notification_settings-related queries
* start fixing specs
* fixing specs
* fix notification and lrg_forem specs
* fixing broken specs
* still fixing
* fix line dif and remove reloads from user.rb
* run specs
* silence bullet and other fixes
* remove setting migration scripts and specs, fix more settings for specs
* handle missing user for article builder and fix notification specs
* fix some final controller specs and re-add incorrectly removed specs
* remove deprecated data update scripts and related workers, put travis back
* refactor admin tags mods controller, write/move specs for users notifications settings controller
* schema cleanup and other small refactors for consistency
* set field we can invalidate in spec via active record instead of at the db level
* remove I think an uneccessary hook call from subscribe_to_mailchimp_newsletter
* use bnefore_create to setup settings, please dont blow up the test suite
* mailchimp bot fix
* remove decorator in favor of single model method
Co-authored-by: Arit Amana <msarit@gmail.com>
Co-authored-by: Ridhwana <ridhwana.khan16@gmail.com>
Co-authored-by: Arit Amana <32520970+msarit@users.noreply.github.com>
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Removes :runtime_banner feature flag + repositioning
* Switch to https://udl.forem.com + fix spec
* Fix URL specs
* Adds data_update_script to remove feature flag
* Fix banner overlay on reactions for small screens + iPad max width
* Append -only to Runtime Filter CSS
* Add import to fix assets:precompile
* Implement design update
* Apply suggestions from code review
Co-authored-by: Lisa Sy <lisasyis@gmail.com>
* Add ;
* Trigger Travis
Co-authored-by: Lisa Sy <lisasyis@gmail.com>
* Replaced granular emails with the default one
* Set reply to header in emails
* Extract shared_examples from NotifyMailer
* Removed unused code
* Replaced SiteConfig with Settings::General
* Fixes related to replacing SiteConfig with Settings
* Removed duplicate line from specs
* Move static profile fields to profiles columns
In order to remove some ambiguity around the availability of certain
profile fields, we can designate certain fields as "static." These
fields are intended to exist on every Forem regardless of configuration;
they contain basic personal info that most Forem's will likely use.
Because these fields already exist on some Forems it is necessary to
migrate the data from existing profile's data column to their respective
columns.
This change should behave as expected irrespective of the existence of
the static fields having associated ProfileFields, however, the UI that
is rendered in a user's settings still depends on the presence of the
ProfileFields. We can address that in a future change when we are
prepared to delete those ProfileFields entirely. We should make sure the
migration occurs without issue before moving to that step, in my
opinion.
* Apply suggestions from code review
Co-authored-by: Michael Kohl <me@citizen428.net>
* Add clarifying comments
Co-authored-by: Michael Kohl <me@citizen428.net>
Co-authored-by: Michael Kohl <me@citizen428.net>
* Run user setting DUS per-user via Sidekiq
Doing it as a single query trips the query timeout beyond a certain
quantity of users.
* Run DataUpdateScripts::MigrateRelevantFieldsFromUsersToUsersNotificationSettings asynchronously
* schema file undelete description
* feat: v1 of the script
* Flesh out remaining enums under their categories
* complete UsersSettings data update script
* complete DUS for relevant attributes in users and profiles tables
* complete DUS for users_notification_settings
* alphabetize user_settings sql file
* safeguard against null values for "null: false" settings
* Set up actual UsersSettings DUS and specs files
* fix broken DUS script
* complete specs for UsersSetting DUS
* Address QA of specs
* complete specs for users_notification_settings DUS
* fix the typos (thanks Julianna!)
* add missing attribute "email_membership_newsletter"
Co-authored-by: Arit Amana <msarit@gmail.com>
* Rename SiteConfig
* More renaming
* Update spec
* Update mandatory settings mapping
* More renaming
* e2e test fixes
* You have a rename, and you have a rename
* Spec fix
* More changes
* Temporarily disable specs
* After-merge update
* Undo rename for migration
* undo rename of DUS
* Fix DUS
* Fix merge problem
* Remove redundant DUS
* Fix specs
* Remove unused code
* Change wrong class name
* More cleanup
* Re-add missing values to constant
* Fix constant
* Fix spec
* Remove obsolete fields
* Add accidentally removed field
* Update spec
* Move methods from Settings::General to ForemInstance
* Remove unneeded model
* Change mentions of 'site config'
