Commit graph

103 commits

Author SHA1 Message Date
rhymes
9e4de49b79
[Hotfix] [Search 2.0] Optimize performance of listings and articles and fix bugs (#13577)
* Add tsvector index on listings

* Fix sorting order when fetching tag flares

* Add published_at as a sorting condition for Homepage::ArticlesQuery

* Re-added param needed by ES, this got lost somewhere down the line
2021-04-29 17:37:59 +02:00
rhymes
c20c4abe3f
[Search 2.0] Use Article.cached_tagged_with* in queries (#13566)
* Add Symbol to Article.cached_tagged_with* and fix error bug

* Use Article.cached_tagged_with* in search

* Apply PR feedback suggestions
2021-04-29 12:00:49 +02:00
Alex
11138c4f18
[Search 2.0] Change podcast_episode query (#13568)
* Change podcast_episode query

* Add the index of all indexes

* Update ATTRIBUTES
2021-04-29 08:49:47 +02:00
Alex
0211a7c341
[Search 2.0] Fix podcast values (#13550)
* Fix podcast values

* Also select slug from podcast_episodes
2021-04-28 08:58:45 -04:00
rhymes
8762815709
[Search 2.0] Match both word boundaries when filtering tags (#13554) 2021-04-28 11:24:45 +02:00
rhymes
f48cf141a6
Use regexp operator to avoid matching partial tags (#13547) 2021-04-27 20:27:11 +02:00
Alex
315dca98c9
[Search 2.0] Podcasts on search page (#13475)
* Working MVP

* Add specs

* Add partial index on podcasts.published

* Add tsvector indexes to podcast_episodes

* Filter for reachable PodcastEpisodes

* Fix indentation/formatting

* Add podcast attribute to serializer
2021-04-27 13:29:22 -04:00
rhymes
2ff317687e
[Search 2.0] Articles (#13540)
* Rename the search scope to reflect its generic usage

* Add and use Search::Postgres::Article
2021-04-27 18:59:01 +02:00
rhymes
996f6e980f
[Search 2.0] Users (#13453)
* Add Search::Postgres::User

* Add sorting params

* Use a better algorithm to filter out suspended users

* Add tsvector index on users.name

* Add specs

* Add UserRole spec

* Update spec/services/search/postgres/user_spec.rb

Co-authored-by: Michael Kohl <citizen428@dev.to>

* Fix search with quotes in users's names

Co-authored-by: Michael Kohl <citizen428@dev.to>
2021-04-24 19:06:06 +02:00
Alex
f233be805b
Preload users instead of querying separately when searching for comments (#13490)
* Preload users instead of querying separately

* Add conditional to comment.user
2021-04-23 16:30:06 -04:00
Alex
1b6f753f73
Performance improvements to comments search (#13489)
* Explicitly pluck user_id from comments table

* Change comment query

* Remove indexes and add partial indexes

* Update schema
2021-04-23 13:21:30 -04:00
dependabot[bot]
29dcea3750
Bump rubocop-performance from 1.10.2 to 1.11.0 (#13471)
* Bump rubocop-performance from 1.10.2 to 1.11.0

Bumps [rubocop-performance](https://github.com/rubocop/rubocop-performance) from 1.10.2 to 1.11.0.
- [Release notes](https://github.com/rubocop/rubocop-performance/releases)
- [Changelog](https://github.com/rubocop/rubocop-performance/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop/rubocop-performance/compare/v1.10.2...v1.11.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Enable new cops and fix violations

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: rhymes <rhymes@hey.com>
2021-04-22 09:31:25 -05:00
rhymes
1fd38aa172
[Search 2.0] Allow mention autocomplete to search users names (#13462)
* Add tsvector index on users.name

* [Search 2.0] Allow mention autocomplete to search users names
2021-04-21 18:47:15 +02:00
rhymes
14d98fc590
[Search 2.0] Homepage query (#13339)
* Added basic homepage query object

* Add serializers and initial service object

* Add flare tags, select only required columns

* Test default Homepage::FetchArticles behind a feature flag

* Add all needed params for the homepage feeds

* Add Homepage::ArticlesQuery specs

* Add Homepage::FetchTagFlares specs

* Add Homepage::FetchArticles specs

* Add approved filter to Homepage::ArticlesQuery

* Add specs for SearchController

* Addressed feedback

* Apply PR feedback to cleanup SearchController#feed_content

* Update app/services/homepage/fetch_tag_flares.rb

Co-authored-by: Jamie Gaskins <jamie@forem.com>

* Update spec/queries/homepage/articles_query_spec.rb

Co-authored-by: Jamie Gaskins <jamie@forem.com>

* Update spec/queries/homepage/articles_query_spec.rb

Co-authored-by: Jamie Gaskins <jamie@forem.com>

* Apply PR feedback

* Modify travis

* Revert "Modify travis"

This reverts commit b42548e866804af69542602556db47d37a31034e.

Co-authored-by: Jamie Gaskins <jamie@forem.com>
Co-authored-by: Mac Siri <krairit.siri@gmail.com>
2021-04-21 12:41:24 +02:00
Alex
cf0baf05f9
[Search 2.0] Comments on search page (#13401)
* Working MVP

* Add comment to COMMENTABLE_TYPES

* Add specs to search_spec

* Add service specs and fix ordering

* Fixed queries and specs

* Clarify COMMENTABLE_TYPES comment

* Move FORCED_EAGER_LOAD_QUERY to Comment

* Add highlighting to Comment search

* Add explicit .to_s on class_name params

* Remove Comment#forced_eager_load_serialized_data

* Revert CommentSerializer changes

* Hardcode class_name attribute

* Add PR feedback

* Use one liner syntax in serializer

* Update user comment

* Update path attribute logic

* Fix user_ids

* Remove PodcastEpisode skipped spec

* Add sort order to Search::Postgres::Comment

* Fix search_specs

* Move pagination TODO comment
2021-04-20 12:34:21 -04:00
rhymes
b6e562f14c
[Search 2.0] Optimize search reading list (#13275)
* Use trigger and tsvector column to speed up reading list search

* Add organization destroy spec and todo note

* Fix failing data update script due to not null constraint

* Remove the leading anchor in the trigger regexp

* Fix reading list specs

* Address feedback
2021-04-12 16:40:40 +02:00
Alex
3891c7d468
✂️ Remove search from Connect (#13235)
* Remove search from connect

* Remove alternate approach

* Alphabetize CHAT_CHANNEL_PARAMS

* Remove default value for user_ids

* Add spec for result ordering
2021-04-06 13:03:31 -04:00
Jacob Herrington
a5b2d109d5
Rename banned and comment_banned roles (#12270)
* Rename banned and comment_banned roles

* Add data update script to rename roles containing 'ban'

* Add named error for Suspended users

* Update unidiomatic method names

* Rename misc banned to suspended

* Apply suggestions from code review

Co-authored-by: Michael Kohl <me@citizen428.net>

* Add unit tests for suspended methods

This commit also adds TODO comments for removing banned and
comment_banned from the codebase after data update scripts have
successfully run on all of our Forems.

Co-authored-by: Michael Kohl <me@citizen428.net>
2021-04-06 10:12:14 -05:00
Alex
1597318db8
[Search 2.0] Listings (#13133)
* Add PgSearch to Listing model

* Working MVP - pre-optimizations

* Remove includes

* Update search_spec

* Add more specs

* Fix specs

* Refactor filter_by_category

* Update block parameter name

* Add published index to classified_listings table

* Preload user, organization, and listing_category

* Working MVP - pre-optimizations

* Refactor filter_by_category

* Update schema

* Fix schema for real

* Remove constant specs and make them private
2021-03-29 13:22:16 -04:00
rhymes
83852038e4
[Search 2.0] Reading list (#13052)
* Step one in populating the reading list with PG

This first attempt tries to recycle the `Search::ArticleSerializer` which is only
used in input in ES, but we're using it in output in PG.
For this reason it's currently 15.55x times slower

* Serialize only what is requested by the frontend

`Search::ArticleSerializer` which is only used in ES in the indexing step aims
to add as much info as possible for broader purposes, in this case
(with serialization in output) we should aim to save only what's requested from
the frontend.

* Optimize selection of articles columns

* Select only needed columns for users

* Compute total of reading list items

* Attach the basic filtering based on PG on the search controller

* Restructure in methods

* Add tags support

* Use LIKE on articles.cached_tag_list

* Fix tags as nil

* Fix default pagination

* Add optional FTS for reading list

* Reworded the tags comment explaining why

* Add index to reactions.status

* Fix total counter in Preact readingList component

* Fix total count in reading list backend search

* Add GIN index to articles.cached_tag_list

* Add service tests

* Add search request specs

* Added missing early return

* Update spec/requests/search_spec.rb

Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>

* Extract MAX_PER_PAGE constant and add comments

Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
2021-03-24 15:40:00 +01:00
Alex
eb741029b7
Use serializer over as_json (#13029) 2021-03-19 11:16:52 -04:00
rhymes
8b9f2d546c
Use faster serializers for Search::Postgres::Tag (#13022)
* Use faster serializers for Search::Postgres::Tag

* Fix spec
2021-03-18 09:30:56 +01:00
Alex
1e4d4db562
[Search 2.0] Add Search::Postgres::Username behind a feature flag (#12975)
* Add PostgreSQL FTS for usernames

* Change profile_image_90 logic

* Make search_users private

* Add tsvector index on usernames

* Limit the number of search results

* Update index name
2021-03-17 13:45:58 -04:00
rhymes
c0ea4e5878
[Search 2.0] Add Search::Postgres::Tag behind a feature flag (#12937)
* Add Search::Postgres::Tag to retrieve tags for autocompletion

* Add Search::Postgres::Tag in SearchController behind a feature flag

* Update spec/services/search/postgres/tag_spec.rb

Co-authored-by: Michael Kohl <citizen428@dev.to>

* Add index to tags.supported

* Fix hotness_score ordering

Co-authored-by: Michael Kohl <citizen428@dev.to>
2021-03-11 19:12:30 +01:00
Molly Struve
37f609e5c5
Return Additional Data Needed for Autocomplete (#12823)
* Return Additional Data Needed for Autocomplete

* return name instead of username twice
2021-02-25 10:55:46 -06:00
Kirk Haines
ed74f2f245
Abstract DatadogStatsClient to ForemStatsClient (#12304)
* This change abstracts the DatadogStatsClient into a ForemStatsClient.
The purpose of this abstraction is to set the foundation for a subsequent PR that will allow one to use New Relic for recording Forem stats, instead of Datadog, if there is a New Relic configuration found.
This specific change creates an abstraction layer that can be built upon, without changing any actual default behavior. All specs still pass.

* Use delegate instead of explicit methods.

* Delegate instead of explicit methods.

* Fix the error.

* Refactor according to the suggestions in the comments.

* Ooops.  Stats work better when all of the code is committed.

* Removing the alias of count to increment since that was done in error.
2021-01-27 11:25:44 -05:00
Molly Struve
fd8f83dc7d
Feature:Backend Username Search Endpoint for Mention Completion (#11479)
* Feature:Backend Username Search Endpoint for Mention Completion

* ensure user is authenticated to use usernames endpoint

* update spec with authaa
2020-11-18 16:43:59 -06:00
rhymes
fe75dcde1c
Routine rubocop fixes (#10478) 2020-09-29 10:14:26 -04:00
Molly Struve
e9cb7b2fd9
[deploy] Refactor:Remove Reaction Search Code (#10377) 2020-09-23 12:27:50 -05:00
Molly Struve
86cb66e858
[deploy] Feature: Serve Reading List from FeedContent Index (#10294)
* Feature:Serve Reading List from FeedContent Index
* Fix paging, order ID desc, improve comments
2020-09-17 17:20:14 -05:00
Molly Struve
313bfef28c
[deploy] Feature:Allow for Search by ID for FeedContent (#10289) 2020-09-11 14:45:23 -05:00
Molly Struve
9c7c4e7062
[deploy] Optimization:Increase refresh_interval For Possible High Indexing Indexes (#10215) 2020-09-09 17:07:15 -05:00
rhymes
2d881ef571
Routine rubocop fixes (#10237) 2020-09-08 12:06:47 -04:00
Molly Struve
d0d3c2a2ab
[deploy] Bug Fix:Sort Reactions by created_at (#10171) 2020-09-03 14:18:02 -05:00
rhymes
c9e72858b7
Enable pending Style/SingleArgumentDig (#9842) 2020-08-18 10:26:40 -04:00
Molly Struve
db98a8e224
[deploy] Bug Fix:Remove User Roles From Elasticsearch Returned Data (#9831) 2020-08-17 13:17:53 -05:00
rhymes
085c60992b
Rubocop: Enable and fix Style/OptionalBooleanParameter cop (#9711)
* Enable and fix Style/OptionalBooleanParameter cop

* Keep regular parameters for workers

* Keep regular parameters for workers

* Fix spec

* Trigger Travis correctly
2020-08-10 19:35:55 +02:00
rhymes
cc8bfcb5c0
Enable Rubocop 0.89 cops and fix Lint/ violations (#9709)
* Enable new Lint/ cops and run rubocop -a

* Fixing last remaining things with rubocop -a

* Enable and fix Style/ExplicitBlockArgument and Style/GlobalStdStream

* Forgot parenthesis
2020-08-10 16:57:12 +02:00
Molly Struve
d120fc7e54
[deploy] Bug Fix:Rename Listing Search Field in Elasticsearch to Fix Searching (#9609) 2020-08-03 10:18:17 -05:00
Molly Struve
6ac637b9ef
[deploy] Boost Scores For Adjacent Words (#9562)
* Feature:Boost Scores For Adjacent Words

* using equal for spec array to ensure order
2020-07-30 15:24:11 +02:00
luigi
7c22026d30
Use Enumerable#filter_map in more places (#9389)
because it's a little faster without lose readability
2020-07-20 18:54:38 +02:00
rhymes
5b62811c98
[deploy] Rubocop: fix violations of Layout/LineLength (#9197) 2020-07-08 08:36:36 -05:00
Molly Struve
85a10524d4
Change Highlight Tags to mark instead of em (#9098)
* Change Highlight Tags to mark instead of em

* Ensure User ID is never present and stub Sitconfig
2020-07-03 21:50:06 +02:00
Michael Kohl
7f75f99560
[deploy] Rename classified listings (#7910)
* Change models and related files

* Update controllers and specs

* More renaming

* Seek and destroy, I mean search and replace

* Round up the stragglers

* Ground control to Major Travis...

* More fixes

* PR feedback

* Various fixes

* Rename view

* Fix list query builder

* Unify request specs

* Fix some API spec errors

* Fix remaining API specs

* Make spec conform to API

* Fix leftover problems

* Fix JS tests

* Fix column name in select

* Fix API specs

* Fix search specs

* Paging Mr. Travis
2020-05-27 13:35:09 +00:00
Andy Zhao
c42fd3461e
[deploy] Rename positive_reactions_count to public_reactions_count in logic (#7926)
* Rename positive_reactions_count to public_reactions_count

* Add positive reactions count back in so we can remove it

* Use public_category method for reactions

* Add positive_reactions_count in case any old caches rely on it

* Add positive_rxn_count to account for API endpoints

* Remove unused method

* One more spot...

* Add method back in because of caches

* Update specs to match new functionality

* Fix typo

* Remove unused methods
2020-05-26 12:36:28 -04:00
Mac Siri
9c58ebc61e
[deploy] Apply HTML encoder to FeedContent's query (#7985)
* Use .textContent to apply Search Snippets

* Improve var name

* Apply HTML encoder to FeedContent's query

* Add spec

* Expand FeedContent specs

* Actually let's not use approval
2020-05-25 13:06:35 -04:00
Andy Zhao
e686d63d78
[deploy] Add public reactions count fields to Elasticsearch (#7956)
* Add public_reactions_count field to elasticsearch

* Add new data update scripts and update old ones

* Use BulkIndexWorker to batch indexing

* Use proper search class for handling feed content

* Remove redundant script oops

* Rename data update script and remove old specs

* Combine specs into one file
2020-05-20 13:06:48 -04:00
Molly Struve
dbb5c752ab
[deploy] Sample Noisy Honeycomb Spans (#7891) 2020-05-18 10:36:01 -05:00
rhymes
3130261384
Refactor GitHub::Client to extract Github::OauthClient for users (#7848)
* Introduce Github::OAuthClient and refactor shortcut Github::Client

* Better stubbing and fix specs

* Fix Badge rewarder

* Add dummy values for Travis

* Fix spec?

* Simplify

* Fix spec take 2?

* Is this the right one?

* Hopefully this is the last time, for realz

* Fix spec for the nth time?

* Which deity do I have to offer a sacrifice to?

* This is it

* Fix Honeycomb name
2020-05-18 14:23:55 +02:00
Sarthak Sharma
90d60bf73f
[deploy] 🚀 Feature: Ability to search discoverable channels and send request to join (#7385)
* Feature 🚀 : Ability to delete messages in chat channels

- Sending message ID to frontend
- Deleting Message
- Use pusher to delete message realtime

* Minor Bug 🐞: Show message action only for current user

- User can delete or edit their own messages

* Test cases added

* Bug 🐞: Update message id for receiver

Message id was not sent to receiver by pusher

* Refactoring🛠: Message controller refactoring

* Test Cases📝 : Specs for Delete message added

* Feature 🚀 : Ability to edit messages

* Test Cases📝 : Specs for Edit message added

* added new column discoverable for public channel and changes in elasticsearch

* fix corrections

* changes in serializer file, added channel_discoverable

* added checkbox with discoverable policy

* changes in code for discoverable channels

* accept and reject member invitation by mod

* add joining request to closed groups

* fixed merge error

* changes in joining member to closed groups

* join to closed group

* changes in message action of joining

* changes in chat upopened json

* 🚀Feature : Ability to search Global Channels

* touch updated_at in after commit update

* 🚀Feature : Ability to send request to discoverable channel

* 🚀Feature : Ability to send request to discoverable channel

* created new route for joining closed channel

* 🚀 Success handle on joining request sent

* removed redundant code

* join_channel improvement, removed authorization for join channel

* list of joining requests

* added joining_request status channels on search list

* changes in mailer and query builder optimized

* 🛠 Adding filter for new Query

* added rspec for add membership method in controller

* rspec for sending join channel request

* invite join request channel list rspec

* test case for query builder discoverable

* viewable and discoverable channel list

* refactored logic for search channels

* 🚀 Check if Request already sent

* 🛠 Optimizing code and making channelButton Component

* 🛠 Optimizing codefor SVG problem

* 🛠 Optimized action.js

* changes in search controller query

* hot fix

* removed unwanted code

* 🛠  Fix the Channel Name problem

* 🛠  Optimizing code further for CodeClimate

* added new column discoverable for public channel and changes in elasticsearch

* fix corrections

* changes in serializer file, added channel_discoverable

* added checkbox with discoverable policy

* changes in code for discoverable channels

* accept and reject member invitation by mod

* add joining request to closed groups

* fixed merge error

* changes in joining member to closed groups

* join to closed group

* changes in message action of joining

* changes in chat upopened json

* touch updated_at in after commit update

* 🚀Feature : Ability to search Global Channels

* 🚀Feature : Ability to send request to discoverable channel

* 🚀Feature : Ability to send request to discoverable channel

* created new route for joining closed channel

* 🚀 Success handle on joining request sent

* removed redundant code

* join_channel improvement, removed authorization for join channel

* list of joining requests

* added joining_request status channels on search list

* changes in mailer and query builder optimized

* added rspec for add membership method in controller

* rspec for sending join channel request

* invite join request channel list rspec

* 🛠 Adding filter for new Query

* test case for query builder discoverable

* viewable and discoverable channel list

* refactored logic for search channels

* 🚀 Check if Request already sent

* 🛠 Optimizing code and making channelButton Component

* 🛠 Optimizing codefor SVG problem

* 🛠 Optimized action.js

* changes in search controller query

* hot fix

* 🛠  Fix the Channel Name problem

* 🛠  Fixing merge problem

* 🛠  Optimizing code further for CodeClimate

* updated UI for membership edit

* fixed test casses and fixed UI for chat_channel_edit

* 🛠napshots added

* test cases fixed

* 🛠 Test cases added for new component

* channel settings accesible only by mod

* 🛠 More Test cases added

* 🛠 Svg code optimized

* 🛠 Svg code optimized in videocontent

* optimized code for search query

* fix test case for query builder

* changes in joining closed channel logic

* refactored join channel

* redirect to edit channel after joining request

* changes in test case for redirect

* optimized code

* 🛠 Eslint bugs fixed

* test case fixed

* optimization

* olving channel repetition problem

* optimization of code for query builder

* changes in query builder

* test cases fixed

* 🛠 Handling reduntant data on frontend

* 🛠  Don't show data if no filter query

* 🛠 Optimized code for fixing bugs and code coverage

* 🛠  Optimizing code further for CodeClimate

Co-authored-by: Parasgr-code <paras.gaur@skynox.tech>
2020-05-11 09:29:15 -04:00