* Limit feeds import fanout to users with feeds
This filtering for feed settings occurs in
Feeds::Import#filter_users_from already, so the enqueued ForUser
jobs were mostly no-op, but the queue latency spiked as hundreds of
thousands of jobs were added to default.
Prescreen users to avoid enqueuing a noop job every hour for every single
user without a feed.
* Update tests to check only jobs for users with feeds are enqueued
* spec cleanup
Only include alice when we check no job is enqueued for her.
Assert no job is enqueued for alice.
Rename bob to user when alice isnt there to contrast.
* Update spec
Use update_columns to bypass setting validation (checking feed is
valid/reachable)
Remove unneeded timecop block when passing a set time
Remove sidekiq: fake and allow Import.call to be received
* Empty commit to trigger CI rebuild
* Fan out Feeds::ImportArticlesWorker
Doing all that work within a single Sidekiq job has begun taking over an
hour on DEV. Regardless of the reasons we did it that way originally, we
should be able to handle this concurrently. If we cannot, we need to
investigate why and handle it properly rather than consigning it to
sequential work.
* Fix specs
The specs make assumptions about how the code under test is implemented.
This commit does not change that, as much as I would like to. Instead,
it just aligns the assumptions with the new implementation.
The previous tests didn't actually represent reality though, since we
can't get a Time or ActiveSupport::TimeWithZone instance inside of the
`perform` method while running it through Sidekiq. Instead, the specs
seem to be relying on the fact that the time instance gets serialized to
ISO-8601/RFC3339 format and that that format is understood by Postgres.
Otherwise, I'm not sure how it would work in production as written.
The new specs reflect reality more closely. The `earlier_than` value
will be converted into an ISO8601/RFC3339 string when passed through
Sidekiq.
* Add parens to perform_bulk call
Turns out, we actually do this pretty consistently. I could've sworn I
saw a bunch of these calls without parens. ¯\_(ツ)_/¯
* Improve variable naming
This is not a list of ids, it's a list of lists of arguments for Sidekiq
jobs, the inner of which contains an id, but that's not the only thing
it contains.
This change makes it easier to resolveforem/forem#16487.
What do I mean by that? To address forem/forem#16487, I need to only
select user's who are authorized to create articles. In
forem/forem#16732 I added a method that will allow chaining of a User
scope. So with this current commit and forem/forem#16732, I'm
triangulating on an approach that will make that change easier.
I did not want to conflate those two, as mixing this PR's change and
what is necessary for the closing PR would create a more complicated
review. Not unduly complicated, but one that will require more tests
and a change in logic. And for someone reviewing the diff, those
concerns could easily be lost.
Yes, I have changed the method signature, but that method signature is
limited to one location:
```shell
❯ rg "Feeds::Import.call"
app/workers/feeds/import_articles_worker.rb
21: ::Feeds::Import.call(users_scope: users_scope, earlier_than: earlier_than)
```
So I look to the method signature a bit as an internal API, hence the change.
Related to forem/forem#16487
* Add script to remove feature flag
* Remove :feeds_import feature flag and RssReader and related classes
* Change Feeds::ImportArticlesWorker signature to support Sidekiq Cron serialization
* Replace RssReader::Assembler with Feeds::AssembleArticleMarkdown
* Removing Assembler
* Add earlier_than to Feeds::Import
* Ignore feeds that have been imported in the last 4 hours
* Make earlier_than mandatory in Feeds::ImportArticlesWorker
* Add explanatory comment to how RssReaderFetchUserWorker works
* Add Feeds::ImportWorker worker
* Enable Feeds::Import for single user feed fetching behind a feature flag
* Add Feeds::ValidateUrl service
* Add feature flag to user's validate feed URL feature
* Remove todo notices
* Add feature flag to fetch_all_rss task
* Add feature flag to RssReaderWorker
* Bring back short-circuit for Articles::RssReaderWorker
