follow on to #15942 which enabled downloading existing DEV feeds, this
allows setting a feed url to DEV.
It's possible this user agent should be brought inline with the
tags choice of community name and url, but in the short term I'm
making this consistent with the existing feed importer.
* Only fetch RSS of users who can create articles
This commit leverages changes from forem/forem#16732 and
forem/forem#16763 to ensure that we're only fetching RSS feeds for
folks who have permission to create articles.
In addition, I chose to rename the variable `user_scope` to
`users_scope` to bring consistency in the variable names.
I also chose to refactor a private method so that it wasn't setting two
instance variables but was instead returning a value that could be used
to set an instance variable.
Closesforem/forem#16487
* Amending method name
This change makes it easier to resolveforem/forem#16487.
What do I mean by that? To address forem/forem#16487, I need to only
select user's who are authorized to create articles. In
forem/forem#16732 I added a method that will allow chaining of a User
scope. So with this current commit and forem/forem#16732, I'm
triangulating on an approach that will make that change easier.
I did not want to conflate those two, as mixing this PR's change and
what is necessary for the closing PR would create a more complicated
review. Not unduly complicated, but one that will require more tests
and a change in logic. And for someone reviewing the diff, those
concerns could easily be lost.
Yes, I have changed the method signature, but that method signature is
limited to one location:
```shell
❯ rg "Feeds::Import.call"
app/workers/feeds/import_articles_worker.rb
21: ::Feeds::Import.call(users_scope: users_scope, earlier_than: earlier_than)
```
So I look to the method signature a bit as an internal API, hence the change.
Related to forem/forem#16487
* Send custom user agent string when fetching feeds
dev.to blocks access to the feeds for user agent "Ruby"
Use "Forem Feeds Import" as alternate user agent when fetching feeds.
fixes https://github.com/forem/forem/issues/15939
* Update app/services/feeds/import.rb
Co-authored-by: Michael Kohl <citizen428@forem.com>
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Use the feed source url when the feed url (from links) is empty
This probably should just be using @feed_source_url always to make a
base path for absolute paths without a hostname, rather than expecting
this from the feed.
An example why we should not be using @feed.url would be
individual posts at domain/slug but the "index" page at
domain/blog (with a generic screen on /), and images hosted from
/assets/ without /blog/ anywhere in the asset path.
Partial fix for #15488
* Use item.url as base_url for relative links
It's possible the rss feed and the contained articles are served from
separate hosts/subdomains/paths. Don't assume the entries are related
to the feed or the feed_source_url, use the item's url for relative
links in that item.
* Update failing spec to add url to item double
* schema file undelete description
* feat: v1 of the script
* Flesh out remaining enums under their categories
* complete UsersSettings data update script
* complete DUS for relevant attributes in users and profiles tables
* complete DUS for users_notification_settings
* alphabetize user_settings sql file
* safeguard against null values for "null: false" settings
* Set up actual UsersSettings DUS and specs files
* fix broken DUS script
* complete specs for UsersSetting DUS
* Address QA of specs
* complete specs for users_notification_settings DUS
* fix the typos (thanks Julianna!)
* begin implementation
* still building
* add missing attribute "email_membership_newsletter"
* complete sync code (except race condition for user profile)
* complete implementation, remains tests
* Address PR review and fix Travis fails
* remove superfluous Profile.new
* fix travis fails
* feat: update the users_notification_setting attributes from the user model
* feat: use the config fonts enums to display the fonts
* feat: loop through the keys
* fix profile = nil blowing up; add specs for notification_setting model
* remove unneeded spec
* remove feed validation until after sync code removed; fixes feed_import spec failures
* remove spec associated with feed_url validation in user_setting model
* fix failing spec 😅
* add TODO
* feat: set the user settings in the user controller and use it in the customization form
* feat: move some update logic to the users settings controller thats being used from customization
* feat: show the updated values form the users_settingd and not the user instance
* Generalize redirect back to current tab
* still trying to reflect changed theme upon refresh
* customizations take effect on refresh
* remove 'with_feed' scope from user model
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* start with takeover for fields previously in profiles table
* Takeover code for `publishing_from_rss` section in Settings (#13914)
* implement takeover code part 1
* implement takeover code
* fix feed fetch
* need rhymes help
* complete implementation; specs pending
* fix STUPID omission that caused so many headaches 😫
* implement profile fields pointing to users_settings 🎉
* run migrations
* implement inbox type & guidelines takeover code; specs pending (#13911)
* Point changes in notification settings to `users_notification_settings` table (#13910)
* implement takeover code; remains specs
* address PR feedback; remove related sync code
* address PR review feedback
* need help with routing and specs
* address pr review
* addressing pr review
* Treat implementation edge cases and omissions 😅
* fix uncommented comment
* fixing implementation cases
* address more PR review feedback
* fixing notifications use-cases
* refactor settings controller
* more pr review changes
* solving bugs
* fix broken onboarding
* handle eperience_level calls
* more fixes
* remove unneeded mappings
* add To-dos for quety updates
* remove done TODO
* purge done TODOs
* update notification_settings-related queries
* start fixing specs
* fixing specs
* fix notification and lrg_forem specs
* fixing broken specs
* still fixing
* fix line dif and remove reloads from user.rb
* run specs
* silence bullet and other fixes
* remove setting migration scripts and specs, fix more settings for specs
* handle missing user for article builder and fix notification specs
* fix some final controller specs and re-add incorrectly removed specs
* remove deprecated data update scripts and related workers, put travis back
* refactor admin tags mods controller, write/move specs for users notifications settings controller
* schema cleanup and other small refactors for consistency
* set field we can invalidate in spec via active record instead of at the db level
* remove I think an uneccessary hook call from subscribe_to_mailchimp_newsletter
* use bnefore_create to setup settings, please dont blow up the test suite
* mailchimp bot fix
* remove decorator in favor of single model method
Co-authored-by: Arit Amana <msarit@gmail.com>
Co-authored-by: Ridhwana <ridhwana.khan16@gmail.com>
Co-authored-by: Arit Amana <32520970+msarit@users.noreply.github.com>
Co-authored-by: Jamie Gaskins <jamie@forem.com>
* Shorten long titles from feed items if needed
Fixes#13959
When the title is too long to create a valid article, put a shortened
title in the generated article markdown.
Added unit tests to cover the default case and the shortening case
* Fix Struct initialization
Rubocop suggested struct over openstruct because of performance, but
Struct.new creates a "class" with a positional initializer, while
Openstruct's new takes a hash like keyword set and builds an object
from those directly.
I mistakenly reused the openstruct syntax after the conversion. This
changes the feed and item to use Struct's initializer.
* Use String#truncate in place of custom trimming
`separator` enables a natural break (preventing space between last
character and ellipsis), and omission text "..." is the default behavior.
* Use verifying doubles in test, use method access to members in code
The instance_double for Feedjira::Parser::RSSEntry
necessitated changing hash key access to reader method access for the
data members (title, categories, summary, description).
* This change abstracts the DatadogStatsClient into a ForemStatsClient.
The purpose of this abstraction is to set the foundation for a subsequent PR that will allow one to use New Relic for recording Forem stats, instead of Datadog, if there is a New Relic configuration found.
This specific change creates an abstraction layer that can be built upon, without changing any actual default behavior. All specs still pass.
* Use delegate instead of explicit methods.
* Delegate instead of explicit methods.
* Fix the error.
* Refactor according to the suggestions in the comments.
* Ooops. Stats work better when all of the code is committed.
* Removing the alias of count to increment since that was done in error.
* Add script to remove feature flag
* Remove :feeds_import feature flag and RssReader and related classes
* Change Feeds::ImportArticlesWorker signature to support Sidekiq Cron serialization
* Replace RssReader::Assembler with Feeds::AssembleArticleMarkdown
* Removing Assembler
* Add earlier_than to Feeds::Import
* Ignore feeds that have been imported in the last 4 hours
* Make earlier_than mandatory in Feeds::ImportArticlesWorker
* Add explanatory comment to how RssReaderFetchUserWorker works
* Add Feeds::ImportWorker worker
* Enable Feeds::Import for single user feed fetching behind a feature flag
* Add Feeds::ValidateUrl service
* Add feature flag to user's validate feed URL feature
* Remove todo notices
* Add feature flag to fetch_all_rss task
* Add feature flag to RssReaderWorker
* Bring back short-circuit for Articles::RssReaderWorker
* Extract Feeds::CheckMediumReplyItem service
* Extract Feeds::CheckItemPreviouslyImported
* Clarified comment on Feeds::CheckItemMediumReply (thanks @maestromac)
* Refactor a bit to be more Rubeyeyey
* Add the parallel gem
* Add prototype version of Feeds::Import with parallel URL fetching and parsing
* Tune Feeds::Import and add rake task for local tests
* Add rough measurer tool
* Add specs
* Apply suggestions by @citizen428
* Replace silence with silent
* No need for eager loading
* Add temporary Articles::DevFeedsImportWorker
* Remove temporary rake task
* Add basic error handling, copied from RssReader
* Fix error handling
* Remove temporary measuring rake task
* Remove logging added for development purposes
* Add info and error logging on error level
