Commit graph

263 commits

Author SHA1 Message Date
Alex
393ba00221
Remove Elasticsearch ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ (#13606)
* Attempt number 1

* Fix rack_attack specs

* Fix users_searches_users spec

* Fix display_users_search_spec

* Fix comment typo

* Remove search:destroy task from cypress

* Remove port 9300 from gitpod

* Stub response in attack_spec
2021-05-03 11:09:45 -04:00
rhymes
c20c4abe3f
[Search 2.0] Use Article.cached_tagged_with* in queries (#13566)
* Add Symbol to Article.cached_tagged_with* and fix error bug

* Use Article.cached_tagged_with* in search

* Apply PR feedback suggestions
2021-04-29 12:00:49 +02:00
rhymes
2ff317687e
[Search 2.0] Articles (#13540)
* Rename the search scope to reflect its generic usage

* Add and use Search::Postgres::Article
2021-04-27 18:59:01 +02:00
Michael Kohl
610f6151e6
Split Settings::Community from SiteConfig (#13403)
* Add settings_community_contents model

* Add settings_communities model

* Update usage

* Add controller and update code

* Add e2e test

* Add data update script

* Update schema.rb

* Fix specs

* PR feedback

* Remove experience_* from Settings::Community

* Update spec

* Fix spec
2021-04-26 10:46:35 +02:00
Michael Kohl
f2f5e911cf
Add Settings::Mascot (#13451)
* Add Settings::Mascot

* Add DUS

* Update usage

* Fix e2e test and controller

* Fix specs

* Fix remaining spec
2021-04-26 11:39:19 +07:00
rhymes
05c220619f
[15 min fix] Enforce an upper limit for articles.body_markdown (#13418)
* Enforce a length limit for articles.body_markdown

* Fix spec assertion

* Add bytesize validation to body_markdown
2021-04-16 20:20:11 +02:00
Vaidehi Joshi
956caf69c7
Rename comment-mentioned-user class added by Html::Parser (#13263)
* Rename comment-mentioned-user class added by Html::Parser

* Add DataUpdateScripts for resaving articles, comments, and messages

* Swap order of messages and articles DUS
2021-04-13 14:33:22 -07:00
rhymes
b6e562f14c
[Search 2.0] Optimize search reading list (#13275)
* Use trigger and tsvector column to speed up reading list search

* Add organization destroy spec and todo note

* Fix failing data update script due to not null constraint

* Remove the leading anchor in the trigger regexp

* Fix reading list specs

* Address feedback
2021-04-12 16:40:40 +02:00
Vaidehi Joshi
97678456d6
[15 Min Fix]: Extract a post's followers out of service and onto Article model (#13229)
* Move authors followers out of Notifications::NotifiableAction::Send and into Article model

* Rename authors_followers to followers

* Update followers method in service, oops

* Update app/services/notifications/notifiable_action/send.rb

Co-authored-by: Michael Kohl <citizen428@dev.to>

Co-authored-by: Michael Kohl <citizen428@dev.to>
2021-04-07 06:46:32 -07:00
Jacob Herrington
a5b2d109d5
Rename banned and comment_banned roles (#12270)
* Rename banned and comment_banned roles

* Add data update script to rename roles containing 'ban'

* Add named error for Suspended users

* Update unidiomatic method names

* Rename misc banned to suspended

* Apply suggestions from code review

Co-authored-by: Michael Kohl <me@citizen428.net>

* Add unit tests for suspended methods

This commit also adds TODO comments for removing banned and
comment_banned from the codebase after data update scripts have
successfully run on all of our Forems.

Co-authored-by: Michael Kohl <me@citizen428.net>
2021-04-06 10:12:14 -05:00
Vaidehi Joshi
484c97fc8d
Limit mentions in posts (#13259)
* Limit mentions in articles

* Bump MAX_USER_MENTION_LIVE_AT to April 7th, 2021 UTC time

* Use stubbed constants in article and comment specs

* Add TODOs around extracting MAX_USER_MENTIONS into constant
2021-04-05 18:19:46 -07:00
rhymes
b6360fb80c
Upgrade Rails to 6.1.3.1 (#11333)
* Upgrade Rails to 6.1

* Switch to unreleased bullet

* Run rails app:update

* Add deprecation notices after reading changelogs

* Fix app:update error

* Move middleware in the correct place

* Temporarily disable ransack which does not support Rails 6.1

* Remove wrongly merged file

* Re-run spring binstub

* Fix double quotes

* Track ransack branch with Rails 6.1 support

* Fix deprecation

* Switch to Ransack 2.4 which supports Rails 6.1 rc1

* Fix missing default params for the duration substitution

* Fix new behavior of relation.pluck with contradictory queries

* Fix uploaders specs by tracking rspec-rails main repo

* Disable bullet temporarily

* Fix remaining fixture_file_upload usages

* Add default seconds for video article duration

* Trigger Travis CI

* Upgrade Rails to 6.1.0.rc2

* Remove file deleted on master

* Add Rails 6.1 gem to Gemfile

* Trigger Travis CI

* Revert "Disable bullet temporarily"

This reverts commit cee0c2ce61fb72cbc16d52c94b12ee681e873031.

* Fix bullet version

* Upgrade to acts-as-taggable 7 and fix conflict

* Update Gemfile and rspec-* gems

* Fix nokogiri in Gemfile.lock

* Switch to rspec-rails main branch

* Remove leftover vendored cached items

* Fix path for Rails 6.1

* Use latest release of erb_lint

* Re-run rails app:update to incorporate new changes

* Disable erb_lint's ErbSafety checks

* Fix Gemfile.lock and re-add platform specific gems of Nokogiri

* Add mini_portile2 as well

* Fix latest merge conflict by removing now unused faraday_middleware

* Upgrade to Rails 6.1.3.1

* Regenerate Gemfile.lock and vendor/cache

* Add x86_64 linux gem

* Mark spec as flaky

* Revert "Mark spec as flaky"

This reverts commit 3caba94b33645f9b59c84ba78ee8df042fc7aee0.

Co-authored-by: Mac Siri <krairit.siri@gmail.com>
2021-04-05 10:39:48 -04:00
Jamie Gaskins
9533c77d7f
User#followed_articles->Article.cached_tagged_with (#13174)
* User#followed_articles->Article.cached_tagged_with

This should improve performance on StoriesController#index

* Make cached_* work like their tagged_with cousins

* Swap to `cached_tagged_with_any`

This was using `tagged_with(tags, any: true)` before so this commit
swaps to using a scope that aligns with that method.

* Allow for searching by Tag models

* do the thing

* abracadabra run the build

* Add comment explaining Postgres regex anchors
2021-04-01 16:20:29 -04:00
Michael Kohl
1732396abc
Remove ignored Buffer columns (#13139) 2021-03-26 17:37:29 +01:00
Julianna Tetreault
af5a391429
Part 2: Shows Relevant Posts in Homepage Sidebar (#12942) [deploy]
* Adjust active_threads class method to display posts in homepage sidebar

* WIP: add specs around active_threads class method

* Uses newer AR query syntax in Article#active_threads

* WIP: Updates article_spec.rb to get tests passing

* WIP: Adjust test to return articles that dont fall into constraints

* Adjusts spec to update articles properly

* Adjust article.update_columns to match initial article columns

* Extracts .active_threads into a query object and removes class method
 - Removes class method from Article model
 - Removes class method tests from article_spec.rb
 - Adds Articles::ActiveThreadsQuery to app/queries/articles
 - Adds active_threads_query_spec.rb to spec/queries/articles
 - Replaces Article.active_threads with Articles::ActiveThreadsQuery
in _homepage_content.html.erb and _sidebar_activity.html.erb
 - General code clean up

* Rename number to count in Articles::ActiveThreadsQuery

* WIP new specs

* Update spec

* Repleaces let! with before block to appease rubocop

* Refactors Articles::ActiveThreadsQuery#call per PR review suggestion

* Calls Articles::ActiveThreadsQuery within articles_helper instead of view

Co-authored-by: Mac Siri <krairit.siri@gmail.com>
2021-03-25 08:03:55 -06:00
rhymes
83852038e4
[Search 2.0] Reading list (#13052)
* Step one in populating the reading list with PG

This first attempt tries to recycle the `Search::ArticleSerializer` which is only
used in input in ES, but we're using it in output in PG.
For this reason it's currently 15.55x times slower

* Serialize only what is requested by the frontend

`Search::ArticleSerializer` which is only used in ES in the indexing step aims
to add as much info as possible for broader purposes, in this case
(with serialization in output) we should aim to save only what's requested from
the frontend.

* Optimize selection of articles columns

* Select only needed columns for users

* Compute total of reading list items

* Attach the basic filtering based on PG on the search controller

* Restructure in methods

* Add tags support

* Use LIKE on articles.cached_tag_list

* Fix tags as nil

* Fix default pagination

* Add optional FTS for reading list

* Reworded the tags comment explaining why

* Add index to reactions.status

* Fix total counter in Preact readingList component

* Fix total count in reading list backend search

* Add GIN index to articles.cached_tag_list

* Add service tests

* Add search request specs

* Added missing early return

* Update spec/requests/search_spec.rb

Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>

* Extract MAX_PER_PAGE constant and add comments

Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
2021-03-24 15:40:00 +01:00
Michael Kohl
7c31a9cc98
Remove buffer gem and related code: step 1 (#12993)
* Remove buffer gem and related code

* Remove more Buffer-related code

* Remove one more Buffer spec

* Remove DUS

* Remove more Buffer code
2021-03-24 09:58:38 +07:00
Jamie Gaskins
68025ae472
Optimize Article.published scope to use an index (#13071)
* Optimize Article.published scope

The index on `articles.published` is not being invoked due to lack of
diversity in the values in that column (it only has 2 possible values
because it's a boolean).

The index on `articles.published_at` is extremely diverse and will be
invoked any time it can reduce the scope of a query by an order of
magnitude or more.

On DEV, this does not change the row count for the scope:

    irb(main):001:0> Article.where(published: true).count == Article.where(published: true).where(Arel.sql("published_at < now()")).count
    => true

The query plan for the `Articles::Feeds::LargeForemExperimental` service
invoked in the `Stories::FeedController#show` endpoint goes from this:

    Planning Time: 0.271 ms
    Execution Time: 329.258 ms

to this:

    Planning Time: 0.330 ms
    Execution Time: 0.468 ms

This is a reduction in query time of 99.7%

* Set published_at in articles factory

An article that is published should always have a `published_at`
timestamp

* Use Time.current for Zonebie

* Add clarifying comment to Article.published scope

* Generate timestamp in Ruby instead of SQL

I think using transactions for specs was interfering with comparing
timestamps generated in SQL, so this commit generates the timestamp in
Ruby.

* Move published_at outside of the transient block

This was causing articles to be marked as `published_at` right now even
if `published_at` was specified in the `FactoryBot.create` call.

* published_at: nil is no longer the factory default

* published_at is no longer nil by default

* published_at is no longer nil by default

* We didn't actually want to clear this published_at

This was intentionally left out to show that published_at does not get
cleared when we flip published true->false.
2021-03-23 13:07:27 -04:00
Daniel Uber
d6b4e83190
When updating article scores, use new score for blackbox (#13009)
Fixes #13008

Add a test case for blackbox stability

This failed as expected on master and passed on this branch, when a
reaction (which would change the article's calculated score during
update) is added, the hotness should be stable, and not modified
during the second call to `#update_score`.
2021-03-18 14:20:27 -05:00
rhymes
576aceab76
[Hotfix] Restore article content in RSS feed (#12935)
When https://github.com/forem/forem/pull/11268 we didn't test it thoroughly and
thus didn't detect that it wasn't working correctly and that it
resulted in an empty string in the `<description>` field.
2021-03-08 14:53:05 +01:00
Alex Timofeev
0ff8284a68
Exclude code highlight panel markup from RSS description (#11268)
* Introduce `plain_html` method with minified article body

* Add corresponding test case
2021-03-05 12:06:50 +01:00
Alex
d109b6824e
Refactor cache purging (#12811)
* Refactor caching

* Bring back old bust class

* Refactor with PR feedback

* Remove useless return values
2021-03-02 10:09:05 -05:00
Julianna Tetreault
27f27caff3
Part 1: Shows Relevant #help Posts in Homepage Sidebar (#12670) [deploy]
* Replace #active_help scope with #active_help class method

* WIP: Add tests around #active_help class method

* Refactor #active_help and update article_spec.rb tests to pass

* Replaces .active_help class method with .active_help scope
  - Reverts change from scope to class method, back to scope
  - Updates to use newer AR query syntax using endless ranges
  - Refactors scope to be more concise
2021-03-01 10:24:57 -07:00
Michael Kohl
544030ff2c
Drop language settings columns (#12699) 2021-02-16 23:29:44 +07:00
Michael Kohl
0c3ad5a1b5
Remove language columns (#12685) 2021-02-16 09:30:01 +07:00
Michael Kohl
e258e36c92
Remove language settings (#12502)
* Remove language settings

* Remove more language related code

* Remove leftover spec

* Remove language from ES mapping
2021-02-04 08:35:00 +07:00
Mac Siri
bdd05f7a9d
Reindex comments' Elasticsearch doc when commentable is updated (#12330) 2021-01-20 09:44:59 -05:00
Andy Zhao
8d462801ec
Add new cache key for article sidebar & bottom_content (#12185)
* Add new cache key for article sidebar & bottom_content

* Trigger build

* Undo automatic changes from 'merge conflict fixes' commit

* Revert to actual previous version, oops

* Trying again...

* Trying again...

* Trying again...

* Remove unnecessary variable

Co-authored-by: Mac Siri <krairit.siri@gmail.com>

Co-authored-by: Mac Siri <krairit.siri@gmail.com>
2021-01-19 14:03:15 -05:00
Alex
31689fd76a
Break MarkdownFixer into MarkdownProcessor::Fixer services (#12241)
* Create new MarkdownProcessor::Fixer services

* Remove old MarkdownFixer

* Code cleanup

* Capitalize Base in code comments

* Remove comments related to inheritance

* Add fix_methods method to hold METHDOS constant
2021-01-14 10:26:46 -05:00
Alex
a5e6f7942c
Rename MarkdownParser to MarkdownProcessor::Parser (#12248)
* Rename MarkdownParser to MarkdownProcessor::Parse

* Wake up, Travis

* Fix typo
2021-01-13 10:39:13 -05:00
Michael Kohl
bfb505d088
Move LanguageDetector to service (#12035) 2021-01-04 19:12:24 -05:00
Molly Struve
4db300e348
Bug Fix:Update Article Cached Org and Docs When Org is Deleted (#12100)
* Bug Fix:Update Article Cached Org and Docs When Org is Deleted

* update cached entity spec
2021-01-04 12:36:28 -06:00
Alex
f70af66258
Update bust to EdgeCache::Bust (#12052)
* Update bust to EdgeCache::Bust

* Update specs

* Fix more specs

* Fix more specs :)
2020-12-29 09:53:15 +01:00
Ben Halpern
673b1406be
Allow any admin to publish welcome thread (#11597)
* Allow any admin to publish welcome thread

* Update app/models/article.rb

Co-authored-by: rhymes <rhymes@hey.com>

* Fix specs by making user for welcome thread an admin

* revert spec changes and update code to include either admin, super_admin, or site mascot

* add specs for scope

* Modify tests and include mascot user

* Fix test

Co-authored-by: rhymes <rhymes@hey.com>
Co-authored-by: mstruve <mollylbs@gmail.com>
2020-12-22 16:18:32 -05:00
Michael Kohl
59e2f5a025
Remove app/labor/color_from_image.rb (#11825)
* Remove app/labor/color_from_image.rb

* Remove miro gem

* Delete TODO
2020-12-10 10:08:06 +07:00
Fernando Valverde
c2982ef0fc
Ensure Article's username method returns a downcased string (#11813)
* Adds a lower() query to the DB in Stories#show

* Restructure spec

* bypass article callbacks in order to test edge case

* Fixes spec

* Change the approach to unsure downcase path in Article model

* Move .downcase call to set_caches before_save callback

* Adds data update script for fixing offending Article paths

* Moves model tests to correct block

* Apply suggestions from code review

Co-authored-by: Vaidehi Joshi <vaidehi.sj@gmail.com>

Co-authored-by: Vaidehi Joshi <vaidehi.sj@gmail.com>
2020-12-09 12:56:40 -06:00
Michael Kohl
c5a29826e8
Move app/labor/error_message_cleaner.rb to service (#11679) 2020-12-02 11:05:36 +07:00
Arit Amana
845e00797b
[EOY 2020] Replace "ban" with "suspend" in Admin (#11581)
* Implement changes

* display "Suspended" if user role.name is banned

* Correct equality symbol

Co-authored-by: Michael Kohl <me@citizen428.net>

Co-authored-by: Michael Kohl <me@citizen428.net>
2020-11-25 09:15:07 -05:00
Akash Srivastava
2ee5ea69bd
Added published date as editable in admin articles list/show (#11124)
* Added published date as editable in admin articles list/show

* Added requests spec for articles published date update
2020-11-18 10:40:13 -06:00
Ben Halpern
98b600da5c
[deploy] Move spam check to after save (#10685) 2020-10-06 19:54:30 -04:00
Ben Halpern
6cb68f134e
[deploy] Allow pattern match for spam terms (#10684) 2020-10-06 19:18:29 -04:00
Ben Halpern
c77a13c0d0
[deploy] Auto-suspend multiple-spammers (#10679)
* Auto-ban multiple-spammers

* Fix xit
2020-10-06 18:07:48 -04:00
Ben Halpern
eefbe25bd3
[deploy] Add ability for admin to add anti-spam terms (#10615)
* Add ability for admin to add anti-spam terms

* Fix specs

* Fix tests

* Fix typo

* Fix test
2020-10-06 10:10:30 -04:00
Julianna Tetreault
c87974b7ac
[deploy] Add co_author_ids to Articles (#10339)
* Replaces second_user_id and third_user_id with co_author_ids on the Articles table

* Adjusts migration that adds co_author_ids to also add array: true

* Replaces second_user_id and third_user_id with co_author_ids in all applicable files
  - Use co_author_ids in _individual_article view
  - Use co_author_ids in articles and stories controllers
  - Use co_author_ids in article and podcast_episode models
  - Use co_author_ids in article_dashboard
  - Use co_author_ids in comments_helper
  - Use co_author_ids in articles_spec

* Replaces second_user_id and third_user_id with co_author_ids in article.rb model

* Remove unused #assign_co_authors method from the Stories::Controller

* Add #co_authors and #co_author_name_and_path to article_decorator.rb

* Adds a descriptive form field for co_author_ids for Admins in /admin/articles

* Adds a conditional in article/show.html.erb to render co_author names on an Article in a human-readable way

* Replaces get_co_author_name_and_path with proper method name, co_author_name_and_path in show.html.erb

* Adds latest schema to (hopefully) resolve Travis conflicts

* Adds the safe operator to show.html.erb

* Adjusts articles_spec.rb to use an array in the first test

* Replaces unless &.empty? with if ./present? in articles/show.html.erb

* Replaces second_user_id with co_author_ids in stories_show_spec.rb test

* Replaces elsif with else in articles/show.html.erb

* Cleans up show.html.erb by removing conditional in favor of decorator method

* Refactors splitting of params in Articles::Controller and optimizes query in #co_authors

* Reverts removal of second_user_id and third_user_id and migration file

* Adds a data_update script to update co_author_ids with existing second and third user_ids

* Adds validations to co_authors and flash_messages to indicate whether an update was successful or not
  - Adds to methods to article.rb to validate the IDs of co_authors and authors
  - Adds flash messages to the Admin::Articles::Controller and a redirect to the show page
  - Removes the JS highlighting upon submit when updating an Article in Admin
  - Refactors #update action in Admin::Articles::Controller

* Adds tests to article_spec.rb around co_author_id validations in article.rb

* Adjusts #validate_co_authors_must_not_be_the_same to use .include? instead

* Uses Field::Select.with_options in article_dashboard.rb to properly display co_author_ids

* Reverts removal of assigning co_author_ids in the Stories::Controller

* Adjusts error message and adjusts return logic in article.rb (thanks, Fernando!)

* Fixes failing article_spec.rb test that checks for co_author_ids uniqueness

* Adds a default array to co_author_ids and checks if they are .blank?

* Refactor data_update script to use a single SQL statement (thank you, Michael)

* Preserve array order of co_author_ids in article_decorator.rb

* Add db file for default: []

* Add validation to fix bug related to text inputs and invalid users when adding co_authors

* Adds tests to ensure that co_author_ids are both an integer and an integer > 0

* Updates admin/articles_spec.rb to default [] instead of nil

* Adjusts validations in article.rb to be DRY-er and more consistent

* Consolidates validations further

* Refactors validations in article.rb to use procs

* Refactors data_update script to remove nil values from array
2020-10-02 11:06:11 -04:00
rhymes
10efa25c0a
Add missing presence validators to models (#10495)
* Add missing presence validator for Tag

* Add missing presence validators to Follow

* Add missing presence validators to ProfileField

* Add missing presence validators to Comment

* Add missing presence validators to Profile

* Add missing presence validators to Organization

* Add missing presence validators to Badge

* Add missing presence validators to Webhook::Endpoint

* Add missing presence validators to NotificationSubscription

* Add missing presence validators to PodcastEpisode

* Add missing presence validators to Sponsorship

* Add missing presence validators to PollOption

* Add missing presence validators to Poll

* Add missing presence validators to Article

* Add missing presence validators to EmailAuthorization

* Add missing presence validators to AuditLog

* Add missing presence validators to DataUpdateScript

* Add missing presence validators to User

* Add missing presence validators to SiteConfig

* Fix specs
2020-10-01 16:15:32 +02:00
Molly Struve
e9cb7b2fd9
[deploy] Refactor:Remove Reaction Search Code (#10377) 2020-09-23 12:27:50 -05:00
rhymes
5ba8c44b05
[deploy] Add missing foreign keys to ahoy_messages and articles (#10321)
* Add FK between ahoy_messages and feedback_messages

* Add FKs between articles.second_user_id, articles.third_user_id and users

* Fix associations and specs

* Add more validations

* Remove FKs to articles.second_user_id and .third_user_id as they are about to be removed
2020-09-16 19:47:03 +02:00
Molly Struve
f2b59cc882
Optimization:Only Enqueue UpdateMainImageBackgroundHexWorker If Main Image Changes (#10325) 2020-09-14 11:37:38 -05:00
Michael Kohl
7b729bbf06
[deploy] Move video_duration_in_minutes_integer to serializer (#10268)
* Move video_duration_in_minutes_integer to serializer

* Add data update script to reindex video articles
2020-09-14 09:47:54 +07:00
Julianna Tetreault
751c910291
Give Admins the Ability to Add Co-Authors to Posts (#10252) [deploy]
* Adds second_user_id and third_user_id to Admin::Articles::Controller and Article model
  - Add second_user_id to #update
  - Add third_user_id to #update
  - Add second_user_id to #article_params
  - Add third_user_id to #article_params
  - Add second and third user_ids to :limited_columns_internal_select

* Adds a second_user_id and third_user_id form field to _individual_article.html.erb

* Adds tests around adding a co-author and multiple co-authors to an article in articles_spec.rb
2020-09-09 10:25:25 -06:00