* Moved logging and deleting comments to Moderator::UnpublishAllArticlesWorker
* Added logging for admin action (unpublish_all_articles)
* Extracted code from UnpublishAllArticlesWorker to the service
* Make it possible to add a note while unpublishing all via admin action
* Added ability to add notes for unpublishing all from moderator action panel
* Fixed setting slug for the AuditLog on unpublishing + spec
* Remove context_notifications when unpublishing all posts
* Fixed typo
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Fixed typo
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Article query spec for scheduled articles
* Added scheduled article badge on the user dashboard
* Added published_at field to editor options
* Accept and validate published_at from editor
* Refactor published_at validation
* Allow 1-minute difference in published_at
* Notice on an unpublished article page
* Added specs for 'Click to edit' link on scheduled article preview page
* ContextNotification model
* Articles::Publish worker
* Added specs for articles publish worker
* Schedule publish articles worker
* Added tests to check for scheduled posts in feeds
* Don't allow managing scheduled articles
* Don't send notifications for scheduled articles
* Set published_at in Articles::Updater when publishing
* Published_at value in post options
* Pass timezone and set published_at accordingly
* Limit setting published_at to the future
* Readonly published_at for articles that were already published
* Chagning published_at format in editor v1 (start)
* Changed published_at format in frontmatter, specs
* Added specs for updating published_at from frontmatter
* Fixed accepting past published_at for articles published_from_feed
* Enabled published_at validation: don't allow updating published_at for already published articles
* Validate published_at on create
* Added a spec for updating published_at for exported articles
* Fixed specs related to creating articles with past published_at
* Fixed specs related to past published_at for articles
* Added a hack so that admins would be able to update published_at
* Switch button text schedule/publish when changin publishedAt
* Fixed saving published_at with timezone
* Added a feature flag for scheduling articles
* Default text in markdown editor depends on feature flag
* Enable article editor cache again
* Fixed the default value in the markdown editor
* Fix sitemaps spec
* Removed tooltip
* Fixed articles update specs
* Added missing locales
* Fixed article create specs
* Fixed spec
* Removed commented code
* Returned enabling extensions in the schema
* Returned accidentally deleted constraint
* Make articles query spec more stable
Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com>
* Removed commented code
* Removed unused code
* A clearer policy
Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com>
* Use StringInquirer for article current state
* Added a note and todo to articles factory past trait
* Remove duplicated PropType
Co-authored-by: Suzanne Aitchison <suzanne@forem.com>
* Refactor query in the Articles::PublishWorker
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Refactor articleForm.jsx
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Removed specs that are no longer relevant
* Removed useless onKeyUp on a hidden input
* Refactored articleForm
* Hide scheduling from post options when published_at is readonly
* Run sends notifications worker every 5 minutes instead of every minute
Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com>
Co-authored-by: Suzanne Aitchison <suzanne@forem.com>
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Moving Experiments into AbExperiment namespace
This commit entails two major concepts:
1. Extracting logic out of a worker.
2. Cleaning the conditions in which we convert experiments.
In addition, I revisited some spec names to tidy them up for
clarification and structure.
**Extracting Logic...**
For workers, my preference is that they be a simple router to another
object. A non-worker object, with it's initialize method, can both
better manage instance variables. Further, workers are somewhat "flat"
in hierarchy,
In this case, having the AbExperiment be knowledgeable both in setting
the experiment and handling conversions makes (to me) organizational
sense. For example, the AbExperiment is constructed as a wrapper to the
FieldTest gem. But the RecordFieldTestEventWorker had knowledge of
FieldTest.
With this refactor, it does not have that knowledge. This also provided
an opportunity to replace magic strings with constants.
**Cleaning the conditions...**
This relates directly to and closesforem/forem#17530. I was looking at
the experiments after 2 days, and realized that there shouldn't be
results for
`user_views_pages_on_at_least_nine_different_days_within_two_weeks`;
after all the experiment started 2 days ago, how can we have results
that look at 2 weeks.
By adding the "max" function calls, we ensure a clear boundary of
"before the experiment began" versus "while the experiment is running".
Closesforem/forem#17530
* Update spec/models/ab_experiment_spec.rb
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
* Apply suggestions from code review
Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
* feat: update all the paths as a first order of business
* feat: update all the folders to no longer appear under users
* feat: update specs
* feat: update specs
* feat: remove User namespace
* feat: remove route namespace
* feat: path change
* feat: update url
We avoid busting a user cache for 15 minutes but the tag cache is 5
hours. So we want to "nudge things along".
What this is trying to solve is the server side rendering of whether we
hide or show a button.
Related to forem/forem#17324
Related to forem/forem#17119
Prior to this commit, when we change attributes for a user we might
resave each article. The purpose of the resave is to update some of the
cached attributes of an article.
This removes some redundant code. The following removed code has
an `article.save` call. Which, the callbacks in the article (see below)
already clear the cache.
```ruby
def resave_articles
articles.find_each do |article|
if article.path
cache_bust = EdgeCache::Bust.new
cache_bust.call(article.path)
cache_bust.call("#{article.path}?i=i")
end
article.save
end
end
```
8e6981aac5/app/models/article.rb (L164)8e6981aac5/app/models/article.rb (L881-L888)
This was discovered in triaging forem/forem#17041
* Limit feeds import fanout to users with feeds
This filtering for feed settings occurs in
Feeds::Import#filter_users_from already, so the enqueued ForUser
jobs were mostly no-op, but the queue latency spiked as hundreds of
thousands of jobs were added to default.
Prescreen users to avoid enqueuing a noop job every hour for every single
user without a feed.
* Update tests to check only jobs for users with feeds are enqueued
* spec cleanup
Only include alice when we check no job is enqueued for her.
Assert no job is enqueued for alice.
Rename bob to user when alice isnt there to contrast.
* Update spec
Use update_columns to bypass setting validation (checking feed is
valid/reachable)
Remove unneeded timecop block when passing a set time
Remove sidekiq: fake and allow Import.call to be received
* Empty commit to trigger CI rebuild
* Fan out Feeds::ImportArticlesWorker
Doing all that work within a single Sidekiq job has begun taking over an
hour on DEV. Regardless of the reasons we did it that way originally, we
should be able to handle this concurrently. If we cannot, we need to
investigate why and handle it properly rather than consigning it to
sequential work.
* Fix specs
The specs make assumptions about how the code under test is implemented.
This commit does not change that, as much as I would like to. Instead,
it just aligns the assumptions with the new implementation.
The previous tests didn't actually represent reality though, since we
can't get a Time or ActiveSupport::TimeWithZone instance inside of the
`perform` method while running it through Sidekiq. Instead, the specs
seem to be relying on the fact that the time instance gets serialized to
ISO-8601/RFC3339 format and that that format is understood by Postgres.
Otherwise, I'm not sure how it would work in production as written.
The new specs reflect reality more closely. The `earlier_than` value
will be converted into an ISO8601/RFC3339 string when passed through
Sidekiq.
* Add parens to perform_bulk call
Turns out, we actually do this pretty consistently. I could've sworn I
saw a bunch of these calls without parens. ¯\_(ツ)_/¯
* Improve variable naming
This is not a list of ids, it's a list of lists of arguments for Sidekiq
jobs, the inner of which contains an id, but that's not the only thing
it contains.
This change makes it easier to resolveforem/forem#16487.
What do I mean by that? To address forem/forem#16487, I need to only
select user's who are authorized to create articles. In
forem/forem#16732 I added a method that will allow chaining of a User
scope. So with this current commit and forem/forem#16732, I'm
triangulating on an approach that will make that change easier.
I did not want to conflate those two, as mixing this PR's change and
what is necessary for the closing PR would create a more complicated
review. Not unduly complicated, but one that will require more tests
and a change in logic. And for someone reviewing the diff, those
concerns could easily be lost.
Yes, I have changed the method signature, but that method signature is
limited to one location:
```shell
❯ rg "Feeds::Import.call"
app/workers/feeds/import_articles_worker.rb
21: ::Feeds::Import.call(users_scope: users_scope, earlier_than: earlier_than)
```
So I look to the method signature a bit as an internal API, hence the change.
Related to forem/forem#16487
* Convert symbol hash keys to strings when calling .perform_async
Fixes a warning from Sidekiq 6.4.0+ about perform_async arguments
which are not equal when passed to perform (`JSON.parse(JSON.dump(arg))`
should equal arg).
This is a safety measure to prevent passing objects (like classes, or
model instances) rather than their representations (like a class name,
or a model's attributes hash).
This warning will be an error in sidekiq 7
* Turn warning into an error in non-production environments
* Use string keys for reaction notification and article fetched
Missed these two on the first pass
* Update example argument hashes for #enqueues_on_correct_queue
Since this calls perform_async under the hood we need to pass json
safe hashes in the test cases as well.
https://github.com/forem/forem/blob/main/spec/workers/shared_examples/enqueues_on_correct_queue.rb
* Convert keys from FollowData#to_h to string before perform_async
I'm not sure enough where else (outside of notification) to_h is being
called, so I'm converting here when building args, rather than in
FollowData#to_h, which might be my next step.
* Let FollowData#to_h return a hash with string keys
Update spec to use string keys as well.
* Make to_h return string keys for ReactionData
Like FollowData, the #to_h method is only used to call
notifications (this is used to enqueue sidekiq jobs).
* Remove a key that was in the hash
Since reaction_data calls to_h, it gets string and not symbol,
keys. Call Hash#except with a key that was actually there.
* Ensuring we don't track views of author or unpublished
Prior to this commit, I was surprised to learn that we:
1) Tracked an author's view of their article.
2) Tracked views of an unpublished article.
This came up from a Forem creator asking if they could reset the view
counter. Or trigger the reset on publication.
I think a general business logic policy of don't track views for the
author and don't track views for unpublished articles is a reasonable
default.
Were we to pursue the clear views on publication, we'd need to consider
something that went from unpublished -> published -> unpublished ->
published. Without a more explicit state machine, triggering a
busineiss logic behavior seems a bit unexpected.
In other words, I wrote an article. There are 20 views when I realize
that I need to unpublished it. I make the changes in the unpublished
state, and re-publish. I'd assume that those 20 views would still be
"recorded" and counted towards my article's view counts.
* Adjusting condition structure
Prior to this commit, the `if` clause was rather far to the right. This
helps make the if clause more pronounced.
* Restoring tracking of custom impressions without GA
In #15967 I (@jeremyf) introduced the "don't do tracking if Google
Analytics ID is blank". However, we still likely want to
trackCustomImpressions().
Further, I did some sleuthing and adding a couple of comments on whether
we should record page views for articles in draft status or any views by
the author.
* Ensuring we don't track views of author or unpublished
Prior to this commit, I was surprised to learn that we:
1) Tracked an author's view of their article.
2) Tracked views of an unpublished article.
This came up from a Forem creator asking if they could reset the view
counter. Or trigger the reset on publication.
I think a general business logic policy of don't track views for the
author and don't track views for unpublished articles is a reasonable
default.
Were we to pursue the clear views on publication, we'd need to consider
something that went from unpublished -> published -> unpublished ->
published. Without a more explicit state machine, triggering a
busineiss logic behavior seems a bit unexpected.
In other words, I wrote an article. There are 20 views when I realize
that I need to unpublished it. I make the changes in the unpublished
state, and re-publish. I'd assume that those 20 views would still be
"recorded" and counted towards my article's view counts.
* Revert "Ensuring we don't track views of author or unpublished"
This reverts commit 321ecbed0ac4552e175d39bf17655a2fc93b265c.
The download app broadcast (in the Generator) returns early if the
user was created less than 7 days ago. It seems counterproductive to
filter out _only_ users created in the last 7 days.
Add one day to the query range, so users created less than 8 days ago,
but more than 7 days ago, receive the final message.
* Name the created after timestamp and remove temporary variables
The logic used to calculate the live at time, the week ago time, then
compare them to find the more recent value.
Move all of this logic to a method (there's a guard to exit if
the setting is nil, so this should be safe).
* Mark method private
This is not part of the public api (for workers, this should be
`#perform` alone). Mark it private
* Proposing a new feed experiment
This commit involves tweaking a few subtle aspects of the challenger:
1) Remove the sort by published date on the relevance score. The
articles will now be listed in the order of perceived relevance.
2) Disable a few levers that did little. There really aren't any
`articles.featured = true` articles in the database.
3) Gently increase the weight of each comment in a linear manner.
4) Give a little more weight to posts that don't have followed tags.
In addition, it involves renaming the conversions to better convey their
implementation. This renaming helps create more descriptive labels for
the test results at `/admin/abtests`.
This commit also adds logic to repurpose the AbExperiment feed_strategy
logic; I envision adjusting the weighted feed strategy levers with some
frequency.
Per discussions, I'm also disabling the "not logged in" feed testing.
We'll use the LargeForemExperimental for this.
* Flipping attribute name to positive
Mentally "not disabled" is harder to parse than "enabled". This change
helps with setting an optimistic attribute.
* Bump for travis
* Bump for travis
* Removing file committed by mistake
* Extracting method
* Load article before creating a page view for an invalid one
This should resolve a validation error in PageView.create! when the
article does not exist (perhaps it was deleted, or the user suspended,
or it was invalid data sent from the client).
This had been happening dozens of times per day for the last 10
months.
* Use an intention revealing symbol instead of calculated id
Since we only require that find_by not find anything, pass a symbol
that's not going to be the id for any article under any conditions.
As an added benefit, this provides a clear indication of the purpose
of the symbol, without needing to mentally confirm that
```sql
SELECT * FROM articles
WHERE id IN (
SELECT (1 + MAX(id)) FROM articles
) LIMIT 1
```
actually never gives any articles back.
* Use the tag_list method, rather than the cached_tag_list attribute
This is a follow on to #15456 and #15637
Saving the article in #15637 was a work-around to force the article to
have a valid cached tag list. However, we can rely on the tag_list
method to find the associated tags.
Update the worker to use tag_list (rather than the decorator's
cached_tag_list.array) and flatten the tag list rather than flat
mapping String#split over the non-nil cached_tag_lists.
* Remove unused join on tags
The query logging showed each article's tags were loaded twice in a
row (select by id, immediately followed by select by id). Removing the
join removed this behavior. I suspect mapping `#tag_list` over
articles is ignoring the loaded tags.
Performance (in my local environment) improved about 30% when this join was removed.
* Guarding against missing experiments
This is a short-circuit to ensure we don't throw exceptions in the
worker. In #15240 we introduce an AbExperiment module that we could use
to insulate against FieldTest implementations.
* Update app/workers/users/record_field_test_event_worker.rb
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Favoring guard condition over coercion
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Remove Connect
* Remove more Connect specs
* Remove a lot more Connect code
* 🚮
* It all has to go
* Explicitly add httpclient
* Update application layout
* Remove messages association from User
* Start fixing specs
* reintroduce util function and refactor references
* Remove Connect Cypress test
* Fix more specs
* Remove Connect from listings
* Ignore contact_via_connect column on listings
* Remove contact_via_connect usages
* Ignore mod_chat_channel_id on tags
* Drop Connect tables
* Remove email_connect_messages from user notification settings
* Re-add httpclient 2.8.3
This was mistakenly removed as a merge conflict
* Don't need to exclude removed chat channel file
* Remove unneeded style for chat channels
* Remove unneeded channel list prop type
* Remove chat channels index/connect-link from getPageEntries
* Re-add comment from httpclient in Gemfile
* Remove connect references from mailers
Tag Moderators no longer have a chat channel
No longer will users be notified about new messages (there won't be
any)
No longer will users be notified about channel invites (you can't
invite anyone anymore)
* Don't configure Pusher and remove PUSHER_* from .env_sample
since it's removed from gemfile, the Pusher constant will not resolve, if this is
configured in the environment variables we'll fail to boot.
Co-authored-by: Suzanne Aitchison <suzanne@forem.com>
Co-authored-by: Dan Uber <dan@forem.com>
* Add PGHero for more insights into the DB
Andrew Kane's Ruby gems will continue being added until morale improves!
* Add linux x86_64 protobuf
* Fix discrepancy between Gemfile and Gemfile.lock
No idea how this happened, but it happened when merging `main` back into
this branch.
Co-authored-by: Dan Uber <dan@forem.com>
* Initial work to expand detection functionality
* Finish up functionality and tests
* Fix class name
* Fix tests
* Update spec/services/articles/enrich_image_attributes_spec.rb
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Update spec/services/articles/enrich_image_attributes_spec.rb
Co-authored-by: Michael Kohl <citizen428@forem.com>
Turns out this caused a *different* problem. It would work for the first
delete batch in `BulkSqlDelete`, but subsequent `DELETE` queries would
then get their `statement_timeout`s reset to the global setting. The
issue was that `BulkSqlDelete` uses methods that don't respect nested
transactions, and we were already resetting the value of the setting to
its previous value anyway, so there was really no reason to run it
inside a transaction.
The only failure mode I know of here is that the `ensure` block breaks
because `connection` is severed. In that case, the value is reset anyway
when the connection pool replaces the connection so the failure does not
propagate.
* Allow Emails::RemoveOldEmailsWorker to run longer
We set the SQL statement timeout based on the STATEMENT_TIMEOUT
environment variable (defaulting to 2.5 seconds in production), but not
all SQL queries are created equal. Since some may take longer out of
necessity, this PR introduces a `Model.with_statement_timeout` method
that allows you to change the SQL statement timeout only for a given
block.
* EmailMessage doesn't inherit ApplicationRecord
It inherits from `Ahoy::Message`, which inherits directly from
`ActiveRecord::Base`.
[3] pry(main)> EmailMessage.ancestors
=> [EmailMessage(id: integer, clicked_at: datetime, content: text, feedback_message_id: integer, mailer: string, sent_at: datetime, subject: text, to: text, token: string, user_id: integer, user_type: string, utm_campaign: string, utm_content: string, utm_medium: string, utm_source: string, utm_term: string),
EmailMessage::GeneratedAssociationMethods,
EmailMessage::GeneratedAttributeMethods,
Ahoy::Message(id: integer, clicked_at: datetime, content: text, feedback_message_id: integer, mailer: string, sent_at: datetime, subject: text, to: text, token: string, user_id: integer, user_type: string, utm_campaign: string, utm_content: string, utm_medium: string, utm_source: string, utm_term: string),
Kaminari::ConfigurationMethods,
Kaminari::ActiveRecordModelExtension,
Ahoy::Message::GeneratedAssociationMethods,
Ahoy::Message::GeneratedAttributeMethods,
Bullet::SaveWithBulletSupport,
ActiveRecord::Base,
...
* Use milliseconds to match STATEMENT_TIMEOUT units
* show top tags when search is empty
* add cypress tests
* nudge travis
* tweak to e2e test
* show a 'top tags' heading
* only suggest tags that are supported
* Add supported scope to Tag
Co-authored-by: Michael Kohl <citizen428@forem.com>
