Remove Elasticsearch ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ (#13606)

* Attempt number 1

* Fix rack_attack specs

* Fix users_searches_users spec

* Fix display_users_search_spec

* Fix comment typo

* Remove search:destroy task from cypress

* Remove port 9300 from gitpod

* Stub response in attack_spec

.env_sample

@@ -33,9 +33,6 @@ REDIS_SIDEKIQ_URL="redis://localhost:6379"
 # Redis Devices/Rpush storage
 REDIS_RPUSH_URL="redis://localhost:6379"
 
-# Elasticsearch
-ELASTICSEARCH_URL="http://localhost:9200"
-
 # OpenResty
 OPENRESTY_URL=""
 
@@ -164,4 +161,4 @@ SENDGRID_API_KEY="Optional, Production only"
 
 
 # Disable simplecov coverage testing when running rspec locally
-COVERAGE="false"
+COVERAGE="false"

.gitpod.dockerfile

@@ -18,12 +18,3 @@ RUN sudo apt-get update \
   && sudo apt-get install -y \
   redis-server \
   && sudo rm -rf /var/lib/apt/lists/*
-
-# Install Elasticsearch
-ARG ES_REPO=https://artifacts.elastic.co/downloads/elasticsearch
-ARG ES_ARCHIVE=elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
-RUN wget "${ES_REPO}/${ES_ARCHIVE}" \
-  && wget "${ES_REPO}/${ES_ARCHIVE}.sha512" \
-  && shasum -a 512 -c ${ES_ARCHIVE}.sha512 \
-  && tar -xzf ${ES_ARCHIVE} \
-  && rm ${ES_ARCHIVE} ${ES_ARCHIVE}.sha512

.gitpod.yml

@@ -10,16 +10,11 @@ ports:
     onOpen: ignore
   - port: 6379
     onOpen: ignore
-  - port: 9200
-    onOpen: ignore
-  - port: 9300
-    onOpen: ignore
 
 tasks:
   - name: Forem Server
     before: |
       redis-server &
-      /home/gitpod/elasticsearch-7.5.2/bin/elasticsearch &
     init: >
       cp .env_sample .env &&
       gem install solargraph &&

.travis.yml

@@ -39,20 +39,11 @@ env:
     - KNAPSACK_PRO_CI_NODE_TOTAL=3
     - COVERAGE_REPORTS_TOTAL=4
     - FOREM_OWNER_SECRET="secret" # test secret so e2e tests can run properly.
-    - ELASTICSEARCH_URL="http://localhost:9200"
 before_install:
   - gem install bundler:"<2.3"
 install:
   - date --rfc-3339=seconds
   - nvm install
-  - pkill -9 -f elasticsearch || true
-  - curl -s -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.2-amd64.deb
-  - sudo dpkg -i --force-confnew elasticsearch-7.5.2-amd64.deb
-  - sudo sed -i.old 's/-Xms1g/-Xms128m/' /etc/elasticsearch/jvm.options
-  - sudo sed -i.old 's/-Xmx1g/-Xmx128m/' /etc/elasticsearch/jvm.options
-  - echo -e '-XX:+DisableExplicitGC\n-Djdk.io.permissionsUseCanonicalPath=true\n-Dlog4j.skipJansi=true\n-server\n' | sudo tee -a /etc/elasticsearch/jvm.options
-  - sudo chown -R elasticsearch:elasticsearch /etc/default/elasticsearch
-  - sudo systemctl start elasticsearch
   - bundle config set path 'vendor/bundle'
   - cp .env_sample .env
   - bin/ci-bundle

Gemfile

@@ -28,7 +28,6 @@ gem "devise" # Flexible authentication solution for Rails
 gem "devise_invitable", "~> 2.0.5" # Allows invitations to be sent for joining
 gem "dogstatsd-ruby", "~> 4.8" # A client for DogStatsD, an extension of the StatsD metric server for Datadog
 gem "doorkeeper", "~> 5.5" # Oauth 2 provider
-gem "elasticsearch", "~> 7.12" # Powers DEVs core search functionality
 gem "email_validator", "~> 2.2" # Email validator for Rails and ActiveModel
 gem "emoji_regex", "~> 3.2" # A pair of Ruby regular expressions for matching Unicode Emoji symbols
 gem "fastly", "~> 3.0" # Client library for the Fastly acceleration system
@@ -65,7 +64,7 @@ gem "omniauth-github", "~> 2.0" # OmniAuth strategy for GitHub
 gem "omniauth-rails_csrf_protection", "~> 1.0" # Provides CSRF protection on OmniAuth request endpoint on Rails application.
 gem "omniauth-twitter", "~> 1.4" # OmniAuth strategy for Twitter
 gem "parallel", "~> 1.20" # Run any kind of code in parallel processes
-gem "patron", "~> 0.13.3" # HTTP client library based on libcurl, used with Elasticsearch to support http keep-alive connections
+gem "patron", "~> 0.13.3" # HTTP client library based on libcurl, used with GitHub OAuth client
 gem "pg", "~> 1.2" # Pg is the Ruby interface to the PostgreSQL RDBMS
 gem "pg_search", "~> 2.3.5" # PgSearch builds Active Record named scopes that take advantage of PostgreSQL's full text search
 gem "puma", "~> 5.2.2" # Puma is a simple, fast, threaded, and highly concurrent HTTP 1.1 server

Gemfile.lock

@@ -223,14 +223,6 @@ GEM
     dotenv-rails (2.7.6)
       dotenv (= 2.7.6)
       railties (>= 3.2)
-    elasticsearch (7.12.0)
-      elasticsearch-api (= 7.12.0)
-      elasticsearch-transport (= 7.12.0)
-    elasticsearch-api (7.12.0)
-      multi_json
-    elasticsearch-transport (7.12.0)
-      faraday (~> 1)
-      multi_json
     em-websocket (0.5.2)
       eventmachine (>= 0.12.9)
       http_parser.rb (~> 0.6.0)
@@ -880,7 +872,6 @@ DEPENDENCIES
   dogstatsd-ruby (~> 4.8)
   doorkeeper (~> 5.5)
   dotenv-rails (~> 2.7.6)
-  elasticsearch (~> 7.12)
   email_validator (~> 2.2)
   emoji_regex (~> 3.2)
   erb_lint (~> 0.0.37)

README.md

@@ -116,7 +116,6 @@ A more complete overview of our stack is available in
 - [ImageMagick](https://imagemagick.org/): please refer to ImageMagick's
   [installation instructions](https://imagemagick.org/script/download.php).
 - [Redis](https://redis.io/) 4 or higher.
-- [Elasticsearch](https://www.elastic.co) 7 or higher.
 
 #### Containers
 

app/controllers/api/v0/health_checks_controller.rb

@@ -7,14 +7,6 @@ module Api
         render json: { message: "App is up!" }, status: :ok
       end
 
-      def search
-        if Search::Client.ping
-          render json: { message: "Search ping succeeded!" }, status: :ok
-        else
-          render json: { message: "Search ping failed!" }, status: :internal_server_error
-        end
-      end
-
       def database
         if ActiveRecord::Base.connected?
           render json: { message: "Database connected" }, status: :ok

app/controllers/chat_channels_controller.rb

@@ -159,7 +159,6 @@ class ChatChannelsController < ApplicationController
     authorize chat_channel
     chat_channel.status = "blocked"
     chat_channel.save
-    chat_channel.chat_channel_memberships.map(&:index_to_elasticsearch)
     render json: { status: "success", message: "chat channel blocked" }, status: :ok
   end
 

app/controllers/search_controller.rb

@@ -93,10 +93,6 @@ class SearchController < ApplicationController
     render json: { result: result }
   end
 
-  def users
-    render json: { result: user_search }
-  end
-
   def usernames
     result = Search::Postgres::Username.search_documents(params[:username])
 
@@ -197,10 +193,6 @@ class SearchController < ApplicationController
     )
   end
 
-  def user_search
-    Search::User.search_documents(params: user_params.to_h)
-  end
-
   def chat_channel_params
     params.permit(CHAT_CHANNEL_PARAMS)
   end

app/lib/acts_as_taggable_on/tag.rb

@@ -1,11 +0,0 @@
-module ActsAsTaggableOn
-  class Tag
-    after_commit on: %i[create update] do
-      ::Tag.find(id).index_to_elasticsearch
-    end
-
-    after_commit on: :update do
-      DataSync::Elasticsearch::Tag.new(self).call
-    end
-  end
-end

app/lib/forem_stats_drivers/datadog_driver.rb

@@ -9,7 +9,6 @@ module ForemStatsDrivers
         c.tracer enabled: ENV["DD_API_KEY"].present?
         c.tracer partial_flush: true
         c.tracer priority_sampling: true
-        c.use :elasticsearch
         c.use :sidekiq
         c.use :redis, service_name: "redis", describes: { url: ENV["REDIS_URL"] }
         c.use :redis, service_name: "redis-sessions", describes: { url: ENV["REDIS_SESSIONS_URL"] }

app/models/article.rb

@@ -3,14 +3,9 @@ class Article < ApplicationRecord
   include ActionView::Helpers
   include Storext.model
   include Reactable
-  include Searchable
   include UserSubscriptionSourceable
   include PgSearch::Model
 
-  SEARCH_SERIALIZER = Search::ArticleSerializer
-  SEARCH_CLASS = Search::FeedContent
-  DATA_SYNC_CLASS = DataSync::Elasticsearch::Article
-
   acts_as_taggable_on :tags
   resourcify
 
@@ -112,9 +107,6 @@ class Article < ApplicationRecord
                                                  }
 
   after_commit :async_score_calc, :touch_collection, on: %i[create update]
-  after_commit :index_to_elasticsearch, on: %i[create update]
-  after_commit :sync_related_elasticsearch_docs, on: %i[update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
 
   # The trigger `update_reading_list_document` is used to keep the `articles.reading_list_document` column updated.
   #
@@ -361,7 +353,6 @@ class Article < ApplicationRecord
 
   def touch_by_reaction
     async_score_calc
-    index_to_elasticsearch
   end
 
   def comments_blob

app/models/chat_channel_membership.rb

@@ -1,10 +1,6 @@
 class ChatChannelMembership < ApplicationRecord
   attr_accessor :invitation_usernames
 
-  include Searchable
-  SEARCH_SERIALIZER = Search::ChatChannelMembershipSerializer
-  SEARCH_CLASS = Search::ChatChannelMembership
-
   ROLES = %w[member mod].freeze
   STATUSES = %w[active inactive pending rejected left_channel removed_from_channel joining_request].freeze
 
@@ -18,9 +14,6 @@ class ChatChannelMembership < ApplicationRecord
 
   validate  :permission
 
-  after_commit :index_to_elasticsearch, on: %i[create update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
-
   delegate :channel_type, to: :chat_channel
 
   scope :eager_load_serialized_data, -> { includes(:user, :channel) }

app/models/comment.rb

@@ -4,10 +4,6 @@ class Comment < ApplicationRecord
 
   include PgSearch::Model
   include Reactable
-  include Searchable
-
-  SEARCH_SERIALIZER = Search::CommentSerializer
-  SEARCH_CLASS = Search::FeedContent
 
   BODY_MARKDOWN_SIZE_RANGE = (1..25_000).freeze
 
@@ -63,18 +59,9 @@ class Comment < ApplicationRecord
   after_create_commit :send_to_moderator
 
   after_commit :calculate_score, on: %i[create update]
-  after_commit :index_to_elasticsearch, on: %i[create update]
 
   after_update_commit :update_notifications, if: proc { |comment| comment.saved_changes.include? "body_markdown" }
 
-  after_commit :remove_from_elasticsearch, on: [:destroy]
-
-  # [@atsmith813] this is adapted from the `search_field` property in
-  # `config/elasticsearch/mappings/feed_content.json` and
-  # `app/serializers/search/comment_serializer.rb`
-  #
-  # highlighter settings are taken from
-  # Search::QueryBuildersFeedContent#add_highlight_fields
   pg_search_scope :search_comments,
                   against: %i[body_markdown],
                   using: {

app/models/concerns/searchable.rb

@@ -1,32 +0,0 @@
-module Searchable
-  def search_id
-    id
-  end
-
-  def index_to_elasticsearch
-    Search::IndexWorker.perform_async(self.class.name, id)
-  end
-
-  def index_to_elasticsearch_inline
-    self.class::SEARCH_CLASS.index(search_id, serialized_search_hash)
-  end
-
-  def remove_from_elasticsearch
-    # Callbacks can cause index and removal jobs to be enqueued at the same time
-    # to avoid indexing a document after removing it we delay the removal job by 5 seconds to
-    # ensure it is run last
-    Search::RemoveFromIndexWorker.perform_in(5.seconds, self.class::SEARCH_CLASS.to_s, search_id)
-  end
-
-  def serialized_search_hash
-    self.class::SEARCH_SERIALIZER.new(self).serializable_hash.dig(:data, :attributes)
-  end
-
-  def elasticsearch_doc
-    self.class::SEARCH_CLASS.find_document(search_id)
-  end
-
-  def sync_related_elasticsearch_docs
-    self.class::DATA_SYNC_CLASS.new(self).call
-  end
-end

app/models/listing.rb

@@ -3,12 +3,8 @@ class Listing < ApplicationRecord
   # We standardized on the latter, but keeping the table name was easier.
   self.table_name = "classified_listings"
 
-  include Searchable
   include PgSearch::Model
 
-  SEARCH_SERIALIZER = Search::ListingSerializer
-  SEARCH_CLASS = Search::Listing
-
   attr_accessor :action
 
   # NOTE: categories were hardcoded at first and the model was only added later.
@@ -19,8 +15,6 @@ class Listing < ApplicationRecord
   before_validation :modify_inputs
   before_save :evaluate_markdown
   before_create :create_slug
-  after_commit :index_to_elasticsearch, on: %i[create update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
   acts_as_taggable_on :tags
   has_many :credits, as: :purchase, inverse_of: :purchase, dependent: :nullify
 
@@ -33,8 +27,6 @@ class Listing < ApplicationRecord
   validate :restrict_markdown_input
   validate :validate_tags
 
-  # [@atsmith813] this is adapted from the `listing_search` property in
-  # `config/elasticsearch/mappings/listings.json`
   pg_search_scope :search_listings,
                   against: %i[body_markdown cached_tag_list location slug title],
                   using: { tsearch: { prefix: true } }

app/models/message.rb

@@ -31,7 +31,6 @@ class Message < ApplicationRecord
 
   def update_chat_channel_last_message_at
     chat_channel.touch(:last_message_at)
-    ChatChannels::IndexesMembershipsWorker.perform_async(chat_channel.id)
   end
 
   def update_all_has_unopened_messages_statuses

app/models/organization.rb

@@ -30,7 +30,6 @@ class Organization < ApplicationRecord
   # This callback will eventually invoke Article.update_cached_user to update the organization.name
   # only when it has been changed, thus invoking the trigger on Article.reading_list_document
   after_update_commit :update_articles_cached_organization
-  after_update_commit :sync_related_elasticsearch_docs
   after_destroy_commit :bust_cache, :article_sync
 
   has_many :articles, dependent: :nullify
@@ -173,10 +172,6 @@ class Organization < ApplicationRecord
     errors.add(:slug, "is taken.") if slug_taken
   end
 
-  def sync_related_elasticsearch_docs
-    DataSync::Elasticsearch::Organization.new(self).call
-  end
-
   def article_sync
     # Syncs article cached organization and updates Elasticsearch docs
     Article.where(id: cached_article_ids).find_each(&:save)

app/models/podcast_episode.rb

@@ -4,10 +4,6 @@ class PodcastEpisode < ApplicationRecord
   ]
 
   include PgSearch::Model
-  include Searchable
-
-  SEARCH_SERIALIZER = Search::PodcastEpisodeSerializer
-  SEARCH_CLASS = Search::FeedContent
 
   acts_as_taggable
 
@@ -38,13 +34,6 @@ class PodcastEpisode < ApplicationRecord
   after_destroy :purge, :purge_all
   after_save :bust_cache
 
-  after_commit :index_to_elasticsearch, on: %i[update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
-
-  # [@atsmith813] this is adapted from the `search_field` property in
-  # `config/elasticsearch/mappings/feed_content.json` and
-  # `app/serializers/search/podcast_episode_serializer.rb` with a couple of
-  # extras
   pg_search_scope :search_podcast_episodes,
                   against: %i[body subtitle title],
                   using: { tsearch: { prefix: true } }

app/models/tag.rb

@@ -1,5 +1,3 @@
-require_relative "../lib/acts_as_taggable_on/tag"
-
 class Tag < ActsAsTaggableOn::Tag
   attr_accessor :points, :tag_moderator_id, :remove_moderator_id
 
@@ -8,8 +6,6 @@ class Tag < ActsAsTaggableOn::Tag
 
   # This model doesn't inherit from ApplicationRecord so this has to be included
   include Purgeable
-  include Searchable
-
   include PgSearch::Model
 
   ALLOWED_CATEGORIES = %w[uncategorized language library tool site_mechanic location subcommunity].freeze
@@ -39,9 +35,6 @@ class Tag < ActsAsTaggableOn::Tag
   before_save :mark_as_updated
 
   after_commit :bust_cache
-  after_commit :index_to_elasticsearch, on: %i[create update]
-  after_commit :sync_related_elasticsearch_docs, on: [:update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
 
   pg_search_scope :search_by_name,
                   against: :name,
@@ -49,10 +42,6 @@ class Tag < ActsAsTaggableOn::Tag
 
   scope :eager_load_serialized_data, -> {}
 
-  SEARCH_SERIALIZER = Search::TagSerializer
-  SEARCH_CLASS = Search::Tag
-  DATA_SYNC_CLASS = DataSync::Elasticsearch::Tag
-
   # possible social previews templates for articles with a particular tag
   def self.social_preview_templates
     Rails.root.join("app/views/social_previews/articles").children.map { |ch| File.basename(ch, ".html.erb") }

app/models/user.rb

@@ -1,8 +1,8 @@
 class User < ApplicationRecord
   resourcify
+  rolify
 
   include CloudinaryHelper
-  include Searchable
   include Storext.model
 
   # @citizen428 Preparing to drop profile columns from the users table
@@ -100,12 +100,6 @@ class User < ApplicationRecord
                 :add_credits, :remove_credits, :add_org_credits, :remove_org_credits, :ip_address,
                 :current_password
 
-  rolify after_add: :index_roles, after_remove: :index_roles
-
-  SEARCH_SERIALIZER = Search::UserSerializer
-  SEARCH_CLASS = Search::User
-  DATA_SYNC_CLASS = DataSync::Elasticsearch::User
-
   acts_as_followable
   acts_as_follower
 
@@ -310,9 +304,6 @@ class User < ApplicationRecord
 
   after_create_commit :send_welcome_notification
   after_commit :bust_cache
-  after_commit :index_to_elasticsearch, on: %i[create update]
-  after_commit :sync_related_elasticsearch_docs, on: %i[create update]
-  after_commit :remove_from_elasticsearch, on: [:destroy]
 
   def self.dev_account
     find_by(id: Settings::Community.staff_user_id)
@@ -676,14 +667,6 @@ class User < ApplicationRecord
     follows.destroy_all
   end
 
-  def index_roles(_role)
-    return unless persisted?
-
-    index_to_elasticsearch_inline
-  rescue StandardError => e
-    Honeybadger.notify(e, context: { user_id: id })
-  end
-
   def can_send_confirmation_email
     return if changes[:email].blank? || id.blank?
 

app/services/data_sync/elasticsearch/article.rb

@@ -1,16 +0,0 @@
-module DataSync
-  module Elasticsearch
-    class Article < Base
-      RELATED_DOCS = %i[
-        comments
-      ].freeze
-
-      SHARED_FIELDS = %i[
-        published
-        title
-      ].freeze
-
-      delegate :comments, to: :@updated_record
-    end
-  end
-end

app/services/data_sync/elasticsearch/base.rb

@@ -1,33 +0,0 @@
-module DataSync
-  module Elasticsearch
-    class Base
-      attr_reader :updated_record
-
-      def initialize(updated_record)
-        @updated_record = updated_record
-      end
-
-      def call
-        return unless sync_needed?
-
-        sync_related_documents
-      end
-
-      private
-
-      def sync_needed?
-        updated_fields.slice(*self.class::SHARED_FIELDS).any?
-      end
-
-      def sync_related_documents
-        self.class::RELATED_DOCS.each do |relation_name|
-          __send__(relation_name).find_each(&:index_to_elasticsearch)
-        end
-      end
-
-      def updated_fields
-        updated_record.saved_changes
-      end
-    end
-  end
-end

app/services/data_sync/elasticsearch/organization.rb

@@ -1,17 +0,0 @@
-module DataSync
-  module Elasticsearch
-    class Organization < Base
-      RELATED_DOCS = %i[
-        articles
-      ].freeze
-
-      SHARED_FIELDS = %i[
-        slug
-        name
-        profile_image
-      ].freeze
-
-      delegate :articles, to: :@updated_record
-    end
-  end
-end

app/services/data_sync/elasticsearch/tag.rb

@@ -1,30 +0,0 @@
-module DataSync
-  module Elasticsearch
-    class Tag < Base
-      RELATED_DOCS = %i[
-        articles
-        podcast_episodes
-      ].freeze
-
-      SHARED_FIELDS = %i[
-        keywords_for_search
-      ].freeze
-
-      private
-
-      def articles
-        ::Article.cached_tagged_with(updated_record.name)
-      end
-
-      def reactions
-        ::Reaction.readinglist.where(reactable: articles)
-      end
-
-      def podcast_episodes
-        ::PodcastEpisode.where(
-          id: updated_record.taggings.where(taggable_type: "PodcastEpisode").select(:taggable_id),
-        )
-      end
-    end
-  end
-end

app/services/data_sync/elasticsearch/user.rb

@@ -1,30 +0,0 @@
-module DataSync
-  module Elasticsearch
-    class User < Base
-      RELATED_DOCS = %i[
-        articles
-        podcast_episodes
-        chat_channel_memberships
-        comments
-      ].freeze
-
-      SHARED_FIELDS = %i[
-        username
-        name
-        profile_image_url
-      ].freeze
-
-      delegate :articles, :chat_channel_memberships, :comments, to: :@updated_record
-
-      private
-
-      def reactions
-        updated_record.reactions.readinglist
-      end
-
-      def podcast_episodes
-        PodcastEpisode.for_user(updated_record)
-      end
-    end
-  end
-end

app/services/edge_cache/bust_commentable.rb

@@ -7,7 +7,6 @@ module EdgeCache
 
       cache_bust = EdgeCache::Bust.new
       cache_bust.call("#{commentable.path}/comments")
-      commentable.index_to_elasticsearch_inline
     end
   end
 end

app/services/podcasts/create_episode.rb

@@ -69,7 +69,6 @@ module Podcasts
 
     def finalize(episode)
       episode.purge_all
-      episode.index_to_elasticsearch
       episode.save if episode.processed_html.blank?
     end
   end

app/services/search/base.rb

@@ -1,143 +0,0 @@
-module Search
-  class Base
-    class << self
-      def index(doc_id, serialized_data)
-        Search::Client.index(
-          id: doc_id,
-          index: self::INDEX_ALIAS,
-          body: serialized_data.merge(last_indexed_at: Time.current),
-        )
-      end
-
-      def bulk_index(data_hashes)
-        indexing_hashes = data_hashes.map do |data_hash|
-          model_hash = data_hash.with_indifferent_access
-          model_hash[:last_indexed_at] = Time.current
-          {
-            index: {
-              _index: self::INDEX_ALIAS,
-              _id: model_hash[:id],
-              data: model_hash
-            }
-          }
-        end
-
-        process_hashes(indexing_hashes)
-      end
-
-      def find_document(doc_id)
-        Search::Client.get(id: doc_id, index: self::INDEX_ALIAS)
-      end
-
-      def delete_document(doc_id)
-        Search::Client.delete(id: doc_id, index: self::INDEX_ALIAS)
-      end
-
-      def index_exists?(index_name: self::INDEX_NAME)
-        Search::Client.indices.exists(index: index_name)
-      end
-
-      def create_index(index_name: self::INDEX_NAME)
-        Search::Client.indices.create(index: index_name, body: settings)
-      end
-
-      def update_index(index_name: self::INDEX_NAME)
-        return unless dynamic_index_settings.any?
-
-        Search::Client.indices.put_settings(index: index_name, body: dynamic_settings)
-      end
-
-      def delete_index(index_name: self::INDEX_NAME)
-        Search::Client.indices.delete(index: index_name)
-      end
-
-      def refresh_index(index_name: self::INDEX_ALIAS)
-        Search::Client.indices.refresh(index: index_name)
-      end
-
-      def add_alias(index_name: self::INDEX_NAME, index_alias: self::INDEX_ALIAS)
-        Search::Client.indices.put_alias(index: index_name, name: index_alias)
-      end
-
-      def update_mappings(index_alias: self::INDEX_ALIAS)
-        Search::Client.indices.put_mapping(index: index_alias, body: self::MAPPINGS)
-      end
-
-      def document_count
-        Search::Client.count(index: self::INDEX_ALIAS)["count"]
-      end
-
-      def search_documents(params:)
-        set_query_size(params)
-        query_hash = "Search::QueryBuilders::#{name.demodulize}".safe_constantize.new(params: params).as_hash
-
-        results = search(body: query_hash)
-        hits = results.dig("hits", "hits").map { |hit| prepare_doc(hit) }
-        paginate_hits(hits, params)
-      end
-
-      private
-
-      def prepare_doc(hit)
-        hit["_source"]
-      end
-
-      def search(body:)
-        Search::Client.search(index: self::INDEX_ALIAS, body: body)
-      end
-
-      def set_query_size(params)
-        params[:page] ||= self::DEFAULT_PAGE
-        params[:per_page] ||= self::DEFAULT_PER_PAGE
-
-        # pages start at 0
-        params[:size] = params[:per_page].to_i * (params[:page].to_i + 1)
-      end
-
-      def paginate_hits(hits, params)
-        start = params[:per_page] * params[:page]
-        hits[start, params[:per_page]] || []
-      end
-
-      def settings
-        { settings: { index: index_settings } }
-      end
-
-      def index_settings
-        raise "Search classes must implement their own index settings"
-      end
-
-      def dynamic_settings
-        { settings: { index: dynamic_index_settings } }
-      end
-
-      def dynamic_index_settings
-        {}
-      end
-
-      def process_hashes(indexing_hashes)
-        indexing_hashes.in_groups_of(1000, false).flat_map do |hashes|
-          indexing_chunks(hashes).filter_map { |chunk| Search::Client.bulk(body: chunk) }
-        end
-      end
-
-      def indexing_chunks(hashes)
-        return [] unless hashes.any?
-        return to_enum(__method__, hashes) unless block_given?
-
-        size = 0
-        chunk = []
-        hashes.each do |hash|
-          chunk << hash
-          size += hash.to_s.bytesize
-          next unless size > 5.megabytes
-
-          yield chunk
-          size = 0
-          chunk = []
-        end
-        yield chunk
-      end
-    end
-  end
-end

app/services/search/chat_channel_membership.rb

@@ -1,28 +0,0 @@
-module Search
-  class ChatChannelMembership < Base
-    INDEX_NAME = "chat_channel_memberships_#{Rails.env}".freeze
-    INDEX_ALIAS = "chat_channel_memberships_#{Rails.env}_alias".freeze
-    MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/chat_channel_memberships.json"),
-                          symbolize_names: true).freeze
-    DEFAULT_PAGE = 0
-    DEFAULT_PER_PAGE = 30
-
-    class << self
-      private
-
-      def index_settings
-        if Rails.env.production?
-          {
-            number_of_shards: 3,
-            number_of_replicas: 1
-          }
-        else
-          {
-            number_of_shards: 1,
-            number_of_replicas: 0
-          }
-        end
-      end
-    end
-  end
-end

app/services/search/client.rb

@@ -1,65 +0,0 @@
-module Search
-  # Search client (uses Elasticsearch as a backend)
-  class Client
-    class << self
-      # adapted from https://api.rubyonrails.org/classes/Module.html#method-i-delegate_missing_to
-      def method_missing(method, *args, &block)
-        return super unless target.respond_to?(method, false)
-
-        # define for re-use
-        self.class.define_method(method) do |*new_args, &new_block|
-          request do
-            target.public_send(method, *new_args, &new_block)
-          end
-        end
-
-        # call the original method, this will only be called the first time
-        # as in subsequent calls, the newly defined method will prevail
-        request do
-          target.public_send(method, *args, &block)
-        end
-      end
-
-      # adapted from https://api.rubyonrails.org/classes/Module.html#method-i-delegate_missing_to
-      def respond_to_missing?(method, _include_all = false)
-        target.respond_to?(method, false) || super
-      end
-
-      private
-
-      TRANSPORT_EXCEPTIONS = [
-        Elasticsearch::Transport::Transport::Errors::BadRequest,
-        Elasticsearch::Transport::Transport::Errors::NotFound,
-      ].freeze
-
-      def request
-        yield
-      rescue *TRANSPORT_EXCEPTIONS => e
-        class_name = e.class.name.demodulize
-        record_error(e.message, class_name)
-
-        # raise specific error if known, generic one if unknown
-        error_class = "::Search::Errors::Transport::#{class_name}".safe_constantize
-        raise error_class, e.message if error_class
-
-        raise ::Search::Errors::TransportError, e.message
-      end
-
-      def record_error(error_message, class_name)
-        Honeycomb.add_field("elasticsearch.result", "error")
-        Honeycomb.add_field("elasticsearch.error", class_name)
-        ForemStatsClient.increment("elasticsearch.errors", tags: ["error:#{class_name}", "message:#{error_message}"])
-      end
-
-      def target
-        @target ||= Elasticsearch::Client.new(
-          url: ApplicationConfig["ELASTICSEARCH_URL"],
-          retry_on_failure: 5,
-          request_timeout: 30,
-          adapter: :patron,
-          log: Rails.env.development?,
-        )
-      end
-    end
-  end
-end

app/services/search/cluster.rb

@@ -1,72 +0,0 @@
-module Search
-  class Cluster
-    SEARCH_CLASSES = [
-      Search::ChatChannelMembership,
-      Search::Listing,
-      Search::FeedContent,
-      Search::Tag,
-      Search::User,
-    ].freeze
-
-    class << self
-      def recreate_indexes
-        delete_indexes
-        setup_indexes
-      end
-
-      def setup_indexes
-        update_settings
-        create_indexes
-        update_indexes
-        add_aliases
-        update_mappings
-      end
-
-      def update_settings
-        Search::Client.cluster.put_settings(body: default_settings)
-      end
-
-      def create_indexes
-        SEARCH_CLASSES.each do |search_class|
-          next if search_class.index_exists?
-
-          search_class.create_index
-        end
-      end
-
-      def update_indexes
-        SEARCH_CLASSES.each(&:update_index)
-      end
-
-      def add_aliases
-        SEARCH_CLASSES.each(&:add_alias)
-      end
-
-      def update_mappings
-        SEARCH_CLASSES.each(&:update_mappings)
-      end
-
-      def delete_indexes
-        return if Rails.env.production?
-
-        SEARCH_CLASSES.each do |search_class|
-          next unless Search::Client.indices.exists(index: search_class::INDEX_NAME)
-
-          search_class.delete_index
-        end
-      end
-
-      private
-
-      def default_settings
-        {
-          persistent: {
-            action: {
-              auto_create_index: false
-            }
-          }
-        }
-      end
-    end
-  end
-end

app/services/search/feed_content.rb

@@ -1,83 +0,0 @@
-module Search
-  class FeedContent < Base
-    INDEX_NAME = "feed_content_#{Rails.env}".freeze
-    INDEX_ALIAS = "feed_content_#{Rails.env}_alias".freeze
-    MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/feed_content.json"), symbolize_names: true).freeze
-    DEFAULT_PAGE = 0
-    DEFAULT_PER_PAGE = 60
-
-    INCLUDED_CLASS_NAMES = %w[Article Comment PodcastEpisode].freeze
-
-    class << self
-      INCLUDED_CLASS_NAMES.each do |class_name|
-        define_method("#{class_name.underscore.pluralize}_document_count") do
-          Search::Client.count(
-            index: self::INDEX_ALIAS, body: count_filter(class_name),
-          )["count"]
-        end
-      end
-
-      private
-
-      def prepare_doc(hit)
-        source = hit["_source"]
-        source["tag_list"] = hit["tags"]&.map { |t| t["name"] } || []
-        source["id"] = source["id"].split("_").last.to_i
-        source["tag_list"] = source["tags"]&.map { |t| t["name"] } || []
-        source["flare_tag"] = source["flare_tag_hash"]
-        source["user_id"] = source.dig("user", "id")
-        source["highlight"] = hit["highlight"]
-        source["readable_publish_date"] = source["readable_publish_date_string"]
-        source["podcast"] = {
-          "slug" => source["slug"],
-          "image_url" => source["main_image"],
-          "title" => source["title"]
-        }
-        source["_score"] = hit["_score"]
-
-        source.merge(timestamps_hash(hit))
-      end
-
-      def timestamps_hash(hit)
-        published_at = hit.dig("_source", "published_at")
-        published_at_timestamp = Time.zone.parse(published_at || "")
-        {
-          "published_at_int" => published_at_timestamp.to_i,
-          "published_timestamp" => published_at
-        }
-      end
-
-      def count_filter(class_name)
-        {
-          query: {
-            bool: {
-              filter: { term: { class_name: class_name } }
-            }
-          }
-        }
-      end
-
-      def index_settings
-        if Rails.env.production?
-          {
-            number_of_shards: 10,
-            number_of_replicas: 1
-          }
-        else
-          {
-            number_of_shards: 1,
-            number_of_replicas: 0
-          }
-        end
-      end
-
-      def dynamic_index_settings
-        if Rails.env.production?
-          { refresh_interval: "5s" }
-        else
-          { refresh_interval: "1s" }
-        end
-      end
-    end
-  end
-end

app/services/search/listing.rb

@@ -1,29 +0,0 @@
-module Search
-  class Listing < Base
-    # We used to use both "classified listing" and "listing" throughout the app.
-    # We standardized on the latter in most places, but kept the index name here.
-    INDEX_NAME = "classified_listings_#{Rails.env}".freeze
-    INDEX_ALIAS = "classified_listings_#{Rails.env}_alias".freeze
-    MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/listings.json"), symbolize_names: true).freeze
-    DEFAULT_PAGE = 0
-    DEFAULT_PER_PAGE = 75
-
-    class << self
-      private
-
-      def index_settings
-        if Rails.env.production?
-          {
-            number_of_shards: 2,
-            number_of_replicas: 1
-          }
-        else
-          {
-            number_of_shards: 1,
-            number_of_replicas: 0
-          }
-        end
-      end
-    end
-  end
-end

app/services/search/query_builders/chat_channel_membership.rb

@@ -1,57 +0,0 @@
-module Search
-  module QueryBuilders
-    class ChatChannelMembership < QueryBase
-      FILTER_KEYS = %i[
-        channel_status
-        channel_type
-        status
-        viewable_by
-      ].freeze
-
-      TERMS_FILTER_KEYS = %i[viewable_by status].freeze
-
-      QUERY_KEYS = %i[
-        channel_text
-      ].freeze
-
-      DEFAULT_PARAMS = {
-        sort_by: "channel_last_message_at",
-        sort_direction: "desc",
-        size: 0
-      }.freeze
-
-      def initialize(params:)
-        super()
-
-        @params = params.deep_symbolize_keys
-        @params[:viewable_by] = @params[:user_id]
-
-        # TODO: @mstruve: When we want to allow people like admins to
-        # search ALL memberships this will need to change
-        @params[:status] = %w[active joining_request]
-
-        build_body
-      end
-
-      private
-
-      def build_queries
-        @body[:query] = {}
-        @body[:query][:bool] = { filter: filter_conditions }
-        @body[:query][:bool][:must] = query_conditions if query_keys_present?
-      end
-
-      def filter_conditions
-        FILTER_KEYS.filter_map do |filter_key|
-          next if @params[filter_key].blank? || @params[filter_key] == "all"
-
-          if TERMS_FILTER_KEYS.include?(filter_key)
-            { terms: { filter_key => @params[filter_key] } }
-          else
-            { term: { filter_key => @params[filter_key] } }
-          end
-        end
-      end
-    end
-  end
-end

app/services/search/query_builders/feed_content.rb

@@ -1,202 +0,0 @@
-module Search
-  module QueryBuilders
-    class FeedContent < QueryBase
-      # In order for highlighting to work properly we have to search the fields we want to highlight
-      QUERY_KEYS = {
-        search_fields: [
-          "tags.keywords_for_search",
-          "tags.name^3",
-          "body_text^2",
-          "title^6",
-          "user.name",
-          "user.username",
-          "organization.name",
-        ]
-      }.freeze
-
-      # Search keys from our controllers may not match what we have stored in Elasticsearch so we map them here,
-      # this allows us to change our Elasticsearch docs without worrying about the frontend
-      TERM_KEYS = {
-        id: "id", # NOTE: FeedContent ids are formatted article_#, podcast_episode_#, comment_#
-        tag_names: "tags.name",
-        approved: "approved",
-        user_id: "user.id",
-        class_name: "class_name",
-        published: "published",
-        organization_id: "organization.id"
-      }.freeze
-
-      RANGE_KEYS = %i[
-        published_at
-      ].freeze
-
-      DEFAULT_PARAMS = {
-        sort: [
-          :_score,
-          { score: "desc" },
-          { hotness_score: "desc" },
-          { comments_count: "desc" },
-        ],
-        size: 0
-      }.freeze
-
-      HIGHLIGHT_FIELDS = %w[
-        body_text
-      ].freeze
-
-      SOURCE = %i[
-        id
-        class_name
-        cloudinary_video_url
-        comments_count
-        flare_tag_hash
-        main_image
-        path
-        public_reactions_count
-        published_at
-        readable_publish_date_string
-        reading_time
-        slug
-        tags
-        title
-        video_duration_in_minutes
-        video_duration_string
-        user
-        organization
-      ].freeze
-
-      attr_accessor :params, :body
-
-      def initialize(params:)
-        super()
-
-        @params = params.deep_symbolize_keys
-
-        # Default to only showing published articles to start
-        @params[:published] = true
-
-        build_body
-        add_function_scoring unless sort_params_present?
-      end
-
-      private
-
-      def add_sort
-        return @body[:sort] = DEFAULT_PARAMS[:sort] unless sort_params_present?
-
-        @body[:sort] = { @params[:sort_by] => @params[:sort_direction] }
-      end
-
-      def add_highlight_fields
-        highlight_fields = { encoder: "html", pre_tags: "<mark>", post_tags: "</mark>", fields: {} }
-        HIGHLIGHT_FIELDS.each do |field_name|
-          # This hash can be filled with options to further customize our highlighting
-          # https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-highlighting
-          highlight_fields[:fields][field_name] = { order: :score, number_of_fragments: 2, fragment_size: 75 }
-        end
-        @body[:highlight] = highlight_fields
-      end
-
-      def filter_source
-        @body[:_source] = SOURCE
-      end
-
-      def build_queries
-        @body[:query] = { bool: {} }
-        @body[:query][:bool][:filter] = filter_conditions if filter_keys_present?
-        return unless query_keys_present?
-
-        @body[:query][:bool][:must] = query_conditions
-        # Boost the score of queries that match these conditions but if they dont match any,
-        # minimum_should_match: 0, then that is OK
-        @body[:query][:bool][:should] = match_phrase_conditions
-        @body[:query][:bool][:minimum_should_match] = 0
-      end
-
-      def filter_conditions
-        filter_conditions = []
-
-        filter_conditions.concat(term_keys) if terms_keys_present?
-        filter_conditions.concat(range_keys) if range_keys_present?
-
-        filter_conditions
-      end
-
-      def filter_keys_present?
-        range_keys_present? || terms_keys_present?
-      end
-
-      def terms_keys_present?
-        TERM_KEYS.detect { |key, _| @params[key].present? }
-      end
-
-      def range_keys_present?
-        RANGE_KEYS.detect { |key| @params[key].present? }
-      end
-
-      def term_keys
-        TERM_KEYS.filter_map do |term_key, search_key|
-          next unless @params.key? term_key
-
-          values = Array.wrap(@params[term_key])
-
-          if params[:tag_boolean_mode] == "all" && term_key == :tag_names
-            values.map { |val| { term: { search_key => val } } }
-          else
-            { terms: { search_key => values } }
-          end
-        end.compact
-      end
-
-      def range_keys
-        RANGE_KEYS.filter_map do |range_key|
-          next unless @params.key? range_key
-
-          { range: { range_key => @params[range_key] } }
-        end
-      end
-
-      def query_hash(key, fields)
-        {
-          simple_query_string: {
-            query: key.downcase,
-            fields: fields,
-            lenient: true,
-            analyze_wildcard: true,
-            minimum_should_match: 2
-          }
-        }
-      end
-
-      def add_function_scoring
-        @body[:query] = { function_score: { query: @body[:query] } }
-        @body[:query][:function_score][:functions] = scoring_functions
-      end
-
-      def scoring_functions
-        [
-          scoring_filter(1000, 2.5),
-          scoring_filter(500, 2),
-          scoring_filter(100, 1.5),
-        ]
-      end
-
-      def scoring_filter(score, weight)
-        {
-          filter: {
-            range: {
-              score: {
-                gte: score
-              }
-            }
-          },
-          weight: weight
-        }
-      end
-
-      def sort_params_present?
-        @params[:sort_by] && @params[:sort_direction]
-      end
-    end
-  end
-end

app/services/search/query_builders/listing.rb

@@ -1,91 +0,0 @@
-module Search
-  module QueryBuilders
-    class Listing < QueryBase
-      TERM_KEYS = %i[
-        category
-        contact_via_connect
-        location
-        published
-        slug
-        tags
-        title
-        user_id
-      ].freeze
-
-      RANGE_KEYS = %i[
-        bumped_at
-        expires_at
-      ].freeze
-
-      QUERY_KEYS = %i[
-        listing_search
-      ].freeze
-
-      DEFAULT_PARAMS = {
-        published: true,
-        sort_by: "bumped_at",
-        sort_direction: "desc",
-        size: 0
-      }.freeze
-
-      def initialize(params:)
-        super()
-
-        @params = params.deep_symbolize_keys
-
-        # For now, we're not allowing searches for Listings that are
-        # not published. If we want to change this in the future we can do:
-        # @params[:published] = DEFAULT_PARAMS[:published] unless @params.key?(:published)
-        @params[:published] = DEFAULT_PARAMS[:published]
-
-        build_body
-      end
-
-      private
-
-      def build_queries
-        @body[:query] = { bool: {} }
-        @body[:query][:bool][:filter] = filter_conditions
-        @body[:query][:bool][:must] = query_conditions if query_keys_present?
-      end
-
-      def filter_conditions
-        filter_conditions = []
-
-        # term_keys are always present because we are setting a filter of
-        # published: true by default
-        filter_conditions.concat term_keys
-
-        filter_conditions.concat range_keys if range_keys_present?
-
-        filter_conditions
-      end
-
-      def term_keys
-        TERM_KEYS.flat_map do |term_key|
-          next unless @params.key? term_key
-
-          values = Array.wrap(@params[term_key])
-
-          if params[:tag_boolean_mode] == "all" && term_key == :tags
-            values.map { |tag| { terms: { term_key => Array.wrap(tag) } } }
-          else
-            { terms: { term_key => values } }
-          end
-        end.compact
-      end
-
-      def range_keys
-        RANGE_KEYS.filter_map do |range_key|
-          next unless @params.key? range_key
-
-          { range: { range_key => @params[range_key] } }
-        end
-      end
-
-      def range_keys_present?
-        RANGE_KEYS.detect { |key| @params[key].present? }
-      end
-    end
-  end
-end

app/services/search/query_builders/query_base.rb

@@ -1,80 +0,0 @@
-module Search
-  module QueryBuilders
-    class QueryBase
-      attr_accessor :params, :body
-
-      def as_hash
-        @body
-      end
-
-      private
-
-      def build_body
-        @body = ActiveSupport::HashWithIndifferentAccess.new
-        build_queries
-        add_sort
-        set_size
-        add_highlight_fields
-        filter_source
-      end
-
-      def add_highlight_fields; end
-
-      def filter_source; end
-
-      def add_sort
-        sort_key = @params[:sort_by] || self.class::DEFAULT_PARAMS[:sort_by]
-        sort_direction = @params[:sort_direction] || self.class::DEFAULT_PARAMS[:sort_direction]
-        @body[:sort] = {
-          sort_key => sort_direction
-        }
-      end
-
-      def set_size
-        # By default we will return 0 documents if size is not specified
-        @body[:size] = @params[:size] || self.class::DEFAULT_PARAMS[:size]
-      end
-
-      def query_keys_present?
-        self.class::QUERY_KEYS.detect { |key, _| @params[key].present? }
-      end
-
-      def query_conditions
-        self.class::QUERY_KEYS.filter_map do |query_key, query_fields|
-          next if @params[query_key].blank?
-
-          fields = query_fields.presence || [query_key]
-
-          query_hash(@params[query_key], fields)
-        end
-      end
-
-      def match_phrase_conditions
-        self.class::QUERY_KEYS.filter_map do |query_key, _fields|
-          next if @params[query_key].blank?
-
-          match_phrase(@params[query_key], query_key)
-        end
-      end
-
-      def match_phrase(phrase, query_key)
-        {
-          match_phrase: {
-            query_key => { query: phrase, slop: 0 }
-          }
-        }
-      end
-
-      def query_hash(key, fields)
-        {
-          simple_query_string: {
-            query: "#{key}*",
-            fields: fields,
-            lenient: true,
-            analyze_wildcard: true
-          }
-        }
-      end
-    end
-  end
-end

app/services/search/query_builders/user.rb

@@ -1,52 +0,0 @@
-module Search
-  module QueryBuilders
-    class User < QueryBase
-      QUERY_KEYS = %i[
-        search_fields
-      ].freeze
-
-      # In the event we want to search for documents that do NOT contain certain values
-      EXCLUDED_TERM_KEYS = {
-        exclude_roles: "roles"
-      }.freeze
-
-      DEFAULT_PARAMS = {
-        sort_by: "hotness_score",
-        sort_direction: "desc",
-        size: 0
-      }.freeze
-
-      def initialize(params:)
-        super()
-
-        @params = params.deep_symbolize_keys
-
-        # default to excluding users who are suspended
-        # TODO: [@jacobherrington] banned can be removed once the data scripts have succesfully run on all Forems
-        @params[:exclude_roles] = %w[suspended banned]
-
-        build_body
-      end
-
-      private
-
-      def build_queries
-        @body[:query] = { bool: {} }
-        @body[:query][:bool][:must] = query_conditions if query_keys_present?
-        @body[:query][:bool][:must_not] = excluded_term_keys if excluded_term_keys_present?
-      end
-
-      def excluded_term_keys_present?
-        self.class::EXCLUDED_TERM_KEYS.detect { |key, _| @params[key].present? }
-      end
-
-      def excluded_term_keys
-        EXCLUDED_TERM_KEYS.filter_map do |term_key, search_key|
-          next unless @params.key? term_key
-
-          { terms: { search_key => Array.wrap(@params[term_key]) } }
-        end
-      end
-    end
-  end
-end

app/services/search/reading_list.rb

@@ -1,78 +0,0 @@
-module Search
-  # This class does not inherit from Search::Base like our other search classes bc it
-  # is using the FeedContent index to search for ReadingList reaction articles rather than its
-  # own separate index. The primary function of this class is to help combine and parse reaction
-  # data with article Elasticsearch data to populate the Reading List view.
-  class ReadingList
-    DEFAULT_PAGE = 0
-    DEFAULT_PER_PAGE = 60
-    DEFAULT_STATUS = %w[valid confirmed].freeze
-
-    # Using a class method here to follow the pattern of the other Search classes
-    def self.search_documents(params:, user:)
-      new(params: params, user: user).reading_list_reactions
-    end
-
-    def initialize(params:, user:)
-      self.status = params.delete(:status) || DEFAULT_STATUS
-      self.view_page = params.delete(:page) || DEFAULT_PAGE
-      self.view_per_page = params.delete(:per_page) || DEFAULT_PER_PAGE
-      self.user = user
-      self.search_params = params
-    end
-
-    def reading_list_reactions
-      ordered_articles = parse_and_order_articles(article_docs)
-      { "reactions" => paginate_articles(ordered_articles), "total" => total }
-    end
-
-    private
-
-    attr_accessor :search_params, :user, :status, :view_page, :view_per_page, :total
-
-    def paginate_articles(ordered_articles)
-      start = view_per_page * view_page
-      ordered_articles[start, view_per_page] || []
-    end
-
-    def article_docs
-      return @article_docs if @article_docs
-
-      # Gather articles from Elasticsearch based on search criteria containing
-      # tags, text search, status, and the list of IDs of all articles in a user's
-      # reading list
-      docs = FeedContent.search_documents(
-        params: search_params.merge(
-          id: search_ids,
-          class_name: "Article",
-          page: 0,
-          per_page: reading_list_article_ids.count,
-        ),
-      )
-      self.total = docs.count
-      @article_docs = docs.index_by { |doc| doc["id"] }
-    end
-
-    def reading_list_article_ids
-      # Collect all reading list IDs and article IDs for a user
-      @reading_list_article_ids ||= user.reactions.readinglist.where(status: status).order(id: :desc).pluck(
-        :reactable_id, :id
-      ).to_h
-    end
-
-    def search_ids
-      reading_list_article_ids.keys.map { |id| "article_#{id}" }
-    end
-
-    def parse_and_order_articles(articles)
-      # Combines reaction and article data to create hashes that contain the fields
-      # the reading list view needs. Ensures articles are returned in order of reaction ID
-      reading_list_article_ids.filter_map do |article_id, reaction_id|
-        found_article_doc = articles[article_id]
-        next unless found_article_doc
-
-        { "id" => reaction_id, "user_id" => user.id, "reactable" => articles[article_id] }
-      end
-    end
-  end
-end

app/services/search/tag.rb

@@ -1,45 +0,0 @@
-module Search
-  class Tag < Base
-    INDEX_NAME = "tags_#{Rails.env}".freeze
-    INDEX_ALIAS = "tags_#{Rails.env}_alias".freeze
-    MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/tags.json"), symbolize_names: true).freeze
-
-    class << self
-      def search_documents(query_string)
-        results = search(body: query(query_string))
-        results.dig("hits", "hits").map { |tag_doc| tag_doc["_source"] }
-      end
-
-      private
-
-      def query(query_string)
-        {
-          query: {
-            query_string: {
-              query: query_string,
-              analyze_wildcard: true,
-              allow_leading_wildcard: false
-            }
-          },
-          sort: {
-            hotness_score: "desc"
-          }
-        }
-      end
-
-      def index_settings
-        if Rails.env.production?
-          {
-            number_of_shards: 1,
-            number_of_replicas: 1
-          }
-        else
-          {
-            number_of_shards: 1,
-            number_of_replicas: 0
-          }
-        end
-      end
-    end
-  end
-end

app/services/search/user.rb

@@ -1,80 +0,0 @@
-module Search
-  class User < Base
-    INDEX_NAME = "users_#{Rails.env}".freeze
-    INDEX_ALIAS = "users_#{Rails.env}_alias".freeze
-    MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/users.json"), symbolize_names: true).freeze
-    DEFAULT_PAGE = 0
-    DEFAULT_PER_PAGE = 20
-
-    class << self
-      def search_usernames(username)
-        results = search(body: username_query(username))
-        results.dig("hits", "hits").map do |doc|
-          source = doc["_source"]
-          {
-            "username" => source["username"],
-            "name" => source["name"],
-            "profile_image_90" => source["profile_image_90"]
-          }
-        end
-      end
-
-      private
-
-      def username_query(username)
-        {
-          query: {
-            query_string: {
-              query: "username:#{username}*",
-              analyze_wildcard: true,
-              allow_leading_wildcard: false
-            }
-          },
-          size: Search::Postgres::Username::MAX_RESULTS # Limit the number of results we return to the frontend
-        }
-      end
-
-      def prepare_doc(hit)
-        source = hit["_source"]
-        {
-          "user" => {
-            "username" => source["username"],
-            "name" => source["username"],
-            "profile_image_90" => source["profile_image_90"]
-          },
-          "title" => source["name"],
-          "path" => source["path"],
-          "id" => source["id"],
-          "class_name" => "User",
-          "public_reactions_count" => source["public_reactions_count"],
-          "comments_count" => source["comments_count"],
-          "badge_achievements_count" => source["badge_achievements_count"],
-          "last_comment_at" => source["last_comment_at"],
-          "user_id" => source["id"]
-        }
-      end
-
-      def index_settings
-        if Rails.env.production?
-          {
-            number_of_shards: 10,
-            number_of_replicas: 1
-          }
-        else
-          {
-            number_of_shards: 1,
-            number_of_replicas: 0
-          }
-        end
-      end
-
-      def dynamic_index_settings
-        if Rails.env.production?
-          { refresh_interval: "10s" }
-        else
-          { refresh_interval: "1s" }
-        end
-      end
-    end
-  end
-end

app/services/users/delete_activity.rb

@@ -2,8 +2,6 @@ module Users
   module DeleteActivity
     module_function
 
-    # If you're removing data that is in Elasticsearch, make sure to use
-    # .destroy_all to trigger the callback to remove the document(s)
     def call(user)
       delete_social_media(user)
       delete_profile_info(user)

app/services/users/delete_articles.rb

@@ -12,10 +12,8 @@ module Users
           comment.reactions.delete_all
           EdgeCache::BustComment.call(comment.commentable)
           EdgeCache::BustUser.call(comment.user)
-          comment.remove_from_elasticsearch
           comment.delete
         end
-        article.remove_from_elasticsearch
         article.delete
         article.purge
       end

app/services/users/delete_comments.rb

@@ -9,7 +9,6 @@ module Users
         comment.reactions.delete_all
         EdgeCache::BustComment.call(comment.commentable)
         comment.remove_notifications
-        comment.remove_from_elasticsearch
         comment.delete
       end
       EdgeCache::BustUser.call(user)

app/workers/articles/score_calc_worker.rb

@@ -9,7 +9,6 @@ module Articles
       return unless article
 
       article.update_score
-      article.index_to_elasticsearch_inline
     end
   end
 end

app/workers/chat_channels/indexes_memberships_worker.rb

@@ -1,12 +0,0 @@
-module ChatChannels
-  class IndexesMembershipsWorker
-    include Sidekiq::Worker
-
-    sidekiq_options queue: :high_priority, lock: :until_executing
-
-    def perform(chat_channel_id)
-      chat_channel = ChatChannel.find(chat_channel_id)
-      chat_channel.chat_channel_memberships.each(&:index_to_elasticsearch)
-    end
-  end
-end

app/workers/metrics/record_data_counts_worker.rb

@@ -14,16 +14,6 @@ module Metrics
 
         Rails.logger.info(message: "db_table_size", table_info: { table_name: model.table_name, table_size: db_count })
         ForemStatsClient.gauge("postgres.db_table_size", db_count, tags: ["table_name:#{model.table_name}"])
-
-        next unless model.const_defined?(:SEARCH_CLASS)
-
-        document_count = if model::SEARCH_CLASS.respond_to?("#{model.to_s.underscore.pluralize}_document_count")
-                           model::SEARCH_CLASS.public_send("#{model.to_s.underscore.pluralize}_document_count")
-                         else
-                           model::SEARCH_CLASS.document_count
-                         end
-        ForemStatsClient.gauge("elasticsearch.document_count", document_count,
-                               tags: ["table_name:#{model.table_name}"])
       end
     end
   end

app/workers/rating_votes/assign_rating_worker.rb

@@ -16,7 +16,6 @@ module RatingVotes
         experience_level_rating_distribution: ratings.max - ratings.min,
         last_experience_level_rating_at: Time.current,
       )
-      article.index_to_elasticsearch_inline
     end
   end
 end

app/workers/search/bulk_index_worker.rb

@@ -1,18 +0,0 @@
-module Search
-  class BulkIndexWorker
-    include Sidekiq::Worker
-
-    sidekiq_options queue: :high_priority, lock: :until_executing
-
-    def perform(object_class, ids)
-      data_hashes = object_class.constantize
-        .eager_load_serialized_data
-        .where(id: ids)
-        .find_each.map(&:serialized_search_hash)
-
-      search_class = object_class.constantize::SEARCH_CLASS
-
-      search_class.bulk_index(data_hashes)
-    end
-  end
-end

app/workers/search/index_worker.rb

@@ -1,12 +0,0 @@
-module Search
-  class IndexWorker
-    include Sidekiq::Worker
-
-    sidekiq_options queue: :high_priority, lock: :until_executing
-
-    def perform(object_class, id)
-      object = object_class.constantize.find_by(id: id)
-      object&.index_to_elasticsearch_inline
-    end
-  end
-end

app/workers/search/remove_from_index_worker.rb

@@ -1,17 +0,0 @@
-module Search
-  class RemoveFromIndexWorker
-    include Sidekiq::Worker
-
-    sidekiq_options queue: :medium_priority, lock: :until_executing
-
-    def perform(search_class, id)
-      search_class.safe_constantize.delete_document(id)
-    rescue Search::Errors::Transport::NotFound
-      # Often race conditions cause us never to index a document and
-      # we end up with an error when trying to remove it. Because
-      # we have count checks that run every hour to track the counts in
-      # Elasticsearch and db to ensure they match we can ignore this error
-      nil
-    end
-  end
-end

bin/setup

@@ -30,9 +30,6 @@ FileUtils.chdir APP_ROOT do
     FileUtils.cp "config/database.yml.sample", "config/database.yml"
   end
 
-  puts "\n== Preparing Test Elasticsearch =="
-  system! 'RAILS_ENV="test" bin/rails search:setup'
-
   puts "\n== Initializing the application =="
   system! "bin/rails app_initializer:setup forem:setup"
 

config/elasticsearch/mappings/chat_channel_memberships.json

@@ -1,61 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "status": {
-      "type": "keyword"
-    },
-    "viewable_by": {
-      "type": "keyword"
-    },
-    "chat_channel_id": {
-      "type": "keyword"
-    },
-    "last_opened_at": {
-      "type": "date"
-    },
-    "channel_text": {
-      "type": "text"
-    },
-    "channel_last_message_at": {
-      "type": "date"
-    },
-    "channel_status": {
-      "type": "keyword"
-    },
-    "channel_status": {
-      "type": "keyword"
-    },
-    "channel_type": {
-      "type": "keyword"
-    },
-    "channel_username": {
-      "type": "keyword"
-    },
-    "channel_name": {
-      "type": "text"
-    },
-    "channel_image": {
-      "type": "keyword"
-    },
-    "channel_modified_slug": {
-      "type": "text",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "channel_messages_count": {
-      "type": "integer"
-    },
-    "channel_discoverable": {
-      "type": "boolean"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    }
-  }
-}

config/elasticsearch/mappings/feed_content.json

@@ -1,186 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "approved": {
-      "type": "boolean"
-    },
-    "body_text": {
-      "type": "text",
-      "copy_to": "search_fields"
-    },
-    "class_name": {
-      "type": "keyword"
-    },
-    "cloudinary_video_url": {
-      "type": "keyword"
-    },
-    "comments_count": {
-      "type": "integer"
-    },
-    "experience_level_rating": {
-      "type": "integer"
-    },
-    "experience_level_rating_distribution": {
-      "type": "integer"
-    },
-    "featured": {
-      "type": "boolean"
-    },
-    "featured_number": {
-      "type": "integer"
-    },
-    "flare_tag_hash": {
-      "dynamic": "strict",
-      "properties": {
-        "name": {
-          "type": "keyword"
-        },
-        "bg_color_hex": {
-          "type": "keyword"
-        },
-        "text_color_hex": {
-          "type": "keyword"
-        }
-      }
-    },
-    "hotness_score": {
-      "type": "integer"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    },
-    "main_image": {
-      "type": "keyword"
-    },
-    "organization": {
-      "dynamic": "strict",
-      "properties": {
-        "slug": {
-          "type": "keyword"
-        },
-        "name": {
-          "type": "text",
-          "copy_to": "search_fields",
-          "fields": {
-            "raw": {
-              "type": "keyword"
-            }
-          }
-        },
-        "id": {
-          "type": "keyword"
-        },
-        "profile_image_90": {
-          "type": "keyword"
-        }
-      }
-    },
-    "path": {
-      "type": "keyword"
-    },
-    // deprecated in favor of public_reactions_count, to be removed
-    "positive_reactions_count": {
-      "type": "integer"
-    },
-    "public_reactions_count": {
-      "type": "integer"
-    },
-    "published": {
-      "type": "boolean"
-    },
-    "published_at": {
-      "type": "date"
-    },
-    "quote": {
-      "type": "text"
-    },
-    "reactions_count": {
-      "type": "integer"
-    },
-    "readable_publish_date": {
-      "type": "date"
-    },
-    "readable_publish_date_string": {
-      "type": "keyword"
-    },
-    "reading_time": {
-      "type": "integer"
-    },
-    "score": {
-      "type": "integer"
-    },
-    "search_fields": {
-      "type": "text"
-    },
-    "search_score": {
-      "type": "integer"
-    },
-    "slug": {
-      "type": "keyword"
-    },
-    "subtitle": {
-      "type": "text"
-    },
-    "summary": {
-      "type": "text"
-    },
-    "tags": {
-      "dynamic": "strict",
-      "properties": {
-        "name": {
-          "type": "keyword",
-          "copy_to": "search_fields"
-        },
-        "keywords_for_search": {
-          "type": "keyword",
-          "copy_to": "search_fields"
-        }
-      }
-    },
-    "title": {
-      "type": "text",
-      "copy_to": "search_fields",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "user": {
-      "dynamic": "strict",
-      "properties": {
-        "username": {
-          "type": "keyword",
-          "copy_to": "search_fields"
-        },
-        "name": {
-          "type": "text",
-          "copy_to": "search_fields",
-          "fields": {
-            "raw": {
-              "type": "keyword"
-            }
-          }
-        },
-        "id": {
-          "type": "keyword"
-        },
-        "profile_image_90": {
-          "type": "keyword"
-        }
-      }
-    },
-    "video_duration_in_minutes": {
-      "type": "integer"
-    },
-    "video_duration_string": {
-      "type": "keyword"
-    },
-    "website_url": {
-      "type": "keyword"
-    }
-  }
-}

config/elasticsearch/mappings/listings.json

@@ -1,87 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "author": {
-      "dynamic": "strict",
-      "properties": {
-        "username": {
-          "type": "keyword"
-        },
-        "name": {
-          "type": "keyword"
-        },
-        "profile_image_90": {
-          "type": "keyword"
-        }
-      }
-    },
-    "body_markdown": {
-      "type": "text",
-      "copy_to": "listing_search"
-    },
-    "originally_published_at": {
-      "type": "date"
-    },
-    "bumped_at": {
-      "type": "date"
-    },
-    "category": {
-      "type": "keyword"
-    },
-    "listing_search": {
-      "type": "text"
-    },
-    "contact_via_connect": {
-      "type": "boolean"
-    },
-    "expires_at": {
-      "type": "date"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    },
-    "location": {
-      "type": "text",
-      "copy_to": "listing_search",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "processed_html": {
-      "type": "keyword"
-    },
-    "published": {
-      "type": "boolean"
-    },
-    "slug": {
-      "type": "text",
-      "copy_to": "listing_search",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "tags": {
-      "type": "keyword",
-      "copy_to": "listing_search"
-    },
-    "title": {
-      "type": "text",
-      "copy_to": "listing_search",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "user_id": {
-      "type": "keyword"
-    }
-  }
-}

config/elasticsearch/mappings/reactions.json

@@ -1,96 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "category": {
-      "type": "keyword"
-    },
-    "created_at": {
-      "type": "date"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    },
-    "reactable": {
-      "dynamic": "strict",
-      "properties": {
-        "id": {
-          "type": "keyword"
-        },
-        "body_text": {
-          "type": "text",
-          "copy_to": "search_fields"
-        },
-        "class_name": {
-          "type": "keyword"
-        },
-        "path": {
-          "type": "keyword"
-        },
-        "published_date_string": {
-          "type": "keyword"
-        },
-        "reading_time": {
-          "type": "integer"
-        },
-        "tags": {
-          "dynamic": "strict",
-          "properties": {
-            "name": {
-              "type": "keyword",
-              "copy_to": "search_fields"
-            },
-            "keywords_for_search": {
-              "type": "keyword",
-              "copy_to": "search_fields"
-            }
-          }
-        },
-        "title": {
-          "type": "text",
-          "copy_to": "search_fields",
-          "fields": {
-            "raw": {
-              "type": "keyword"
-            }
-          }
-        },
-        "user": {
-          "dynamic": "strict",
-          "properties": {
-            "username": {
-              "type": "keyword",
-              "copy_to": "search_fields"
-            },
-            "name": {
-              "type": "text",
-              "copy_to": "search_fields",
-              "fields": {
-                "raw": {
-                  "type": "keyword"
-                }
-              }
-            },
-            "id": {
-              "type": "keyword"
-            },
-            "profile_image_90": {
-              "type": "keyword"
-            }
-          }
-        }
-      }
-    },
-    "search_fields": {
-      "type": "text"
-    },
-    "status": {
-      "type": "keyword"
-    },
-    "user_id": {
-      "type": "keyword"
-    }
-  }
-}

config/elasticsearch/mappings/tags.json

@@ -1,31 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    },
-    "name": {
-      "type": "text",
-      "fields": {
-        "raw": {
-          "type": "keyword"
-        }
-      }
-    },
-    "hotness_score": {
-      "type": "integer"
-    },
-    "supported": {
-      "type": "boolean"
-    },
-    "short_summary": {
-      "type": "text"
-    },
-    "rules_html": {
-      "type": "text"
-    }
-  }
-}

config/elasticsearch/mappings/users.json

@@ -1,87 +0,0 @@
-{
-  "dynamic": "strict",
-  "properties": {
-    "id": {
-      "type": "keyword"
-    },
-    "available_for": {
-      "type": "keyword",
-      "copy_to": "search_fields"
-    },
-    "badge_achievements_count": {
-      "type": "integer"
-    },
-    "comments_count": {
-      "type": "integer"
-    },
-    "employer_name": {
-      "type": "keyword",
-      "copy_to": "search_fields"
-    },
-    "hotness_score": {
-      "type": "integer"
-    },
-    "last_comment_at": {
-      "type": "date"
-    },
-    "last_indexed_at": {
-      "type": "date"
-    },
-    "mostly_work_with": {
-      "type": "keyword",
-      "copy_to": "search_fields"
-    },
-    "name": {
-      "type": "text",
-      "copy_to": "search_fields"
-    },
-    "path": {
-      "type": "keyword"
-    },
-    // deprecated in favor of public_reactions_count, to be removed
-    "positive_reactions_count": {
-      "type": "integer"
-    },
-    "public_reactions_count": {
-      "type": "integer"
-    },
-    "profile_image_90": {
-      "type": "keyword"
-    },
-    "reactions_count": {
-      "type": "integer"
-    },
-    "roles": {
-      "type": "keyword"
-    },
-    "search_fields": {
-      "type": "text"
-    },
-    "username": {
-      "type": "keyword",
-      "copy_to": "search_fields"
-    },
-    "profile_fields": {
-      "type": "nested",
-      "properties": {
-        "name": {
-          "type": "keyword"
-        },
-        "value": {
-          "type": "keyword"
-        }
-      }
-    },
-    "custom_profile_fields": {
-      "type": "nested",
-      "properties": {
-        "name": {
-          "type": "keyword"
-        },
-        "value": {
-          "type": "keyword"
-        }
-      }
-    }
-  }
-}

config/initializers/cypress_rails.rb

@@ -22,7 +22,6 @@ end
 CypressRails.hooks.before_server_stop do
   # Called once, at_exit
   puts "Cleaning up and stopping server for end to end tests."
-  Rake::Task["search:destroy"].invoke
   Rake::Task["db:truncate_all"].invoke
   puts "The end to end test server shutdown gracefully."
 end

config/routes.rb

@@ -89,7 +89,6 @@ Rails.application.routes.draw do
         resources :health_checks, only: [] do
           collection do
             get :app
-            get :search
             get :database
             get :cache
           end
@@ -212,7 +211,6 @@ Rails.application.routes.draw do
     get "/search/tags", to: "search#tags"
     get "/search/chat_channels", to: "search#chat_channels"
     get "/search/listings", to: "search#listings"
-    get "/search/users", to: "search#users"
     get "/search/usernames", to: "search#usernames"
     get "/search/feed_content", to: "search#feed_content"
     get "/search/reactions", to: "search#reactions"

container-compose.yml

@@ -8,13 +8,11 @@ services:
     depends_on:
       - bundle
       - db
-      - elasticsearch
       - redis
       - yarn
     environment:
       RAILS_ENV: development
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
@@ -24,7 +22,7 @@ services:
       APP_DOMAIN: rails
     volumes:
       - .:/opt/apps/forem:z
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s"]
     command: [ "bash", "-c", "./scripts/entrypoint.sh bootstrap && bundle exec rails server -b 0.0.0.0 -p 3000"]
 
   bundle:
@@ -36,7 +34,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:z
     command: ["./scripts/bundle.sh"]
@@ -50,7 +47,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:z
     command: [ "bash", "-c", "yarn install --dev"]
@@ -67,7 +63,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:z
     entrypoint: ["dockerize", "-wait", "file:///opt/apps/forem/node_modules/.bin/webpack-dev-server", "-timeout", "300s"]
@@ -80,17 +75,15 @@ services:
       - rails
       - redis
       - db
-      - elasticsearch
     environment:
       RAILS_ENV: development
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:z
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
     command: ["bundle", "exec", "rake","db:seed"]
 
   sidekiq:
@@ -100,17 +93,15 @@ services:
       - rails
       - redis
       - db
-      - elasticsearch
     environment:
       RAILS_ENV: development
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:z
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
     command: ["bundle", "exec", "sidekiq","-c","2"]
 
   db:
@@ -131,23 +122,5 @@ services:
     ports:
       - "6379:6379"
 
-  elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
-    container_name: forem_elasticsearch
-    environment:
-      - cluster.name=forem
-      - bootstrap.memory_lock=true
-      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
-      - "discovery.type=single-node"
-      - xpack.security.enabled=false
-      - xpack.monitoring.enabled=false
-      - xpack.graph.enabled=false
-      - xpack.watcher.enabled=false
-    ports:
-      - "9200:9200"
-    # volumes:
-    #   - es_data:/usr/share/elasticsearch/data:Z
-
 volumes:
   db_data:
-  # es_data:

docker-compose.yml

@@ -9,13 +9,11 @@ services:
     depends_on:
       - bundle
       - db
-      - elasticsearch
       - redis
       - yarn
     environment:
       RAILS_ENV: development
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
@@ -25,7 +23,7 @@ services:
       APP_DOMAIN: rails
     volumes:
       - .:/opt/apps/forem:delegated
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
     command: [ "bash", "-c", "./scripts/entrypoint.sh bootstrap && bundle exec rails server -b 0.0.0.0 -p 3000"]
 
   bundle:
@@ -38,7 +36,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:delegated
     command: ["./scripts/bundle.sh"]
@@ -53,7 +50,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:delegated
     command: yarn install --dev
@@ -72,7 +68,6 @@ services:
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:delegated
     entrypoint: ["dockerize", "-wait", "file:///opt/apps/forem/node_modules/.bin/webpack-dev-server", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
@@ -86,17 +81,15 @@ services:
       - rails
       - redis
       - db
-      - elasticsearch
     environment:
       RAILS_ENV: development
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:delegated
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
     command: ["bundle", "exec", "rake","db:seed"]
 
   sidekiq:
@@ -107,17 +100,15 @@ services:
       - rails
       - redis
       - db
-      - elasticsearch
     environment:
       RAILS_ENV: development
       REDIS_SESSIONS_URL: redis://redis:6379
       REDIS_SIDEKIQ_URL: redis://redis:6379
       REDIS_URL: redis://redis:6379
       DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
-      ELASTICSEARCH_URL: http://elasticsearch:9200
     volumes:
       - .:/opt/apps/forem:delegated
-    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
+    entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
     command: ["bundle", "exec", "sidekiq","-c","2"]
 
   db:
@@ -138,28 +129,5 @@ services:
     ports:
       - "6379:6379"
 
-  elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
-    container_name: forem_elasticsearch
-    environment:
-      - cluster.name=forem
-      - bootstrap.memory_lock=true
-      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
-      - "discovery.type=single-node"
-      - xpack.security.enabled=false
-      - xpack.monitoring.enabled=false
-      - xpack.graph.enabled=false
-      - xpack.watcher.enabled=false
-    volumes:
-      - es_data:/usr/share/elasticsearch/data:delegated
-    ports:
-      - "9200:9200"
-    ulimits:
-      memlock:
-        soft: -1
-        hard: -1
-
 volumes:
   db_data:
-  es_data:
-

docs/admin/admin-search.md

@@ -16,9 +16,9 @@ The view responsible for managing user reports can be found at
 action of the [`admin/feedback_messages_controller`][feedback_messages].
 
 For Forem, Ransack is being used exclusively in admin, for search problems in
-other parts of the app we use [Elasticsearch][elasticsearch].
+other parts of the app we use [PostgreSQL Full Text Search][postgres_fts].
 
 [feedback_messages]:
   https://github.com/forem/forem/blob/4e41e4a2ac893fa2a6c36990cfe475858ffb086a/app/controllers/admin/feedback_messages_controller.rb#L4
 [ransack]: https://github.com/activerecord-hackery/ransack
-[elasticsearch]: /backend/elasticsearch
+[postgres_fts]: https://www.postgresql.org/docs/11/textsearch.html

docs/backend/data-update-scripts.md

@@ -8,9 +8,7 @@ Data Update Scripts were introduced in
 [this PR](https://github.com/forem/forem/pull/6025) and allow us to run any data
 updates we might need. For example, if we added a column to the database and
 then wanted to backfill that column with data, rather than going and manually
-doing it in a console, we would use a DataUpdateScript. Another example might be
+doing it in a console, we would use a DataUpdateScript.
-adding a new attribute to Elasticsearch. We could then use a DataUpdateScript to
-reindex all of our models.
 
 ## How it works
 

docs/backend/elasticsearch.md

@@ -1,92 +0,0 @@
----
-title: Elasticsearch
----
-
-# What is [Elasticsearch](https://www.elastic.co/what-is/elasticsearch)?
-
-> Elasticsearch is a distributed, open-source search and analytics engine for
-> all types of data, including textual, numerical, geospatial, structured, and
-> unstructured. Elasticsearch is built on Apache Lucene and was first released
-> in 2010 by Elastic.
-
-# How Forem uses Elasticsearch
-
-### Searchable Models
-
-At Forem we use Elasticsearch for all of our user-facing searching needs. Models
-that are searched using Elasticsearch are:
-
-- Tags
-- Listings
-- Chat Channels
-- Users
-- Podcast Episodes
-- Comments
-- Articles
-
-### Index Setup
-
-The above models are organized in Elasticsearch in different
-[indexes](https://www.elastic.co/blog/what-is-an-elasticsearch-index) to make
-searching more accurate and performant. The index breakdown is:
-
-- Tags index -> Tags
-- Listings index -> Listings
-- Chat Channel Membership index -> Chat Channels
-- User's index -> Users
-- Feed Content index -> Podcast Episodes, Comments, and Articles
-
-You will notice that Podcast Episodes, Comments, and Articles all share the same
-index. This allows us to search and aggregate across all of these documents at
-the same time. These documents also share a lot of similar fields so putting
-them in the same index made sense. In the future, depending on our search needs
-this index structure might change.
-
-Each index that is listed above has a corresponding
-[mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html)
-that defines what fields can and cannot be indexed to it. These mappings are
-similar to a db schema. In our case, all of our index mappings are defined as
-[`strict`](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic)
-which means if we try to index a field that is not defined it will throw an
-error. Mappings for each index can be found in
-`app/config/elasticsearch/mappings`.
-
-### Indexing documents into Elasticsearch
-
-In order to index the data from Postgres into Elasticsearch, we rely on
-callbacks in our ActiveRecord models. Whenever a model is created or updated we
-use an `after_commit` callback to enqueue a `Search::IndexWorker` which handles
-indexing the document into Elasticsearch. In order to translate our ActiveRecord
-data to Elasticsearch we use serializers. Each model has its own serializer
-which can be found in `app/serializers/search`
-
-### Searching Elasticsearch
-
-Once the documents are in Elasticsearch then we can search for them. This is
-handled by code in our `app/services/search` directory. Here you will see a
-collection of classes used to help us index and search documents in
-Elasticsearch. The nested `query_builders` directory contains all the logic we
-need to help us build complex search queries to send to Elasticsearch.
-
-# Working With Elasticsearch
-
-## How to add a new field to Elasticsearch
-
-When adding a new field to Elasticsearch here are the steps you have to follow:
-
-1. Add field to mapping by editing the appropriate JSON file in
-   `app/config/elasticsearch/mappings`. If you plan to test locally you will
-   need to run `bin/setup` to update your local Elasticsearch mappings.
-2. Add field to serializer for model located in `app/serializers/search`.
-3. If the new field needs to be backfilled write a DataUpdateScript to reindex
-   all affected models so the field is available and populated in Elasticsearch
-4. DEPLOY - Before you can start using a new field you need to ensure that it is
-   added to Elasticsearch and populated properly. The worker that runs the
-   DataUpdateScript will run 5 min after a deploy completes. This means you have
-   to deploy the code for adding the field separate from the code that will use
-   it.
-5. Add code to search for your new field. You likely will need to update the
-   allowed params in the searches controller as well as the pertinent query
-   builder.
-6. DEPLOY new search code
-7. DONE!

docs/backend/readme.md

@@ -10,7 +10,6 @@ items:
   - authorization.md
   - configuration.md
   - data-update-scripts.md
-  - elasticsearch.md
   - emails.md
   - fastly.md
   - internationalization.md

docs/installation/containers.md

@@ -108,12 +108,6 @@ The script executes the following steps:
 
 ### Docker on Mac
 
-- Should you experience problems with the Elasticsearch container, try to
-  increase the memory and/or swap allocation for Docker. On macOS this can be
-  done via the GUI:
-
-  ![docker gui](https://user-images.githubusercontent.com/47985/74210448-b63b7c80-4c83-11ea-959b-02249b2a6952.png)
-
 - In case `rails server` starts with the following message:
 
   ```shell

docs/installation/linux.md

@@ -66,21 +66,6 @@ We recommend following Digital Ocean's extensive guides
 [How to Install and Secure Redis](https://www.digitalocean.com/community/tutorial_collections/how-to-install-and-secure-redis)
 to setup Redis.
 
-### Elasticsearch
-
-Forem requires a version of Elasticsearch between 7.1 and 7.5. Version 7.6 is
-not supported. We recommend version 7.5.2.
-
-We recommend following
-[Elasticsearch's guide for installing on Linux](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-linux).
-
-Elasticsearch is also available as as
-[Debian package](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/deb.html)
-or a
-[RPM package](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/rpm.html).
-
-NOTE: Make sure to download **the OSS version**, `elasticsearch-oss`.
-
 ## Installing Forem
 
 1. Fork Forem's repository, e.g. <https://github.com/forem/forem/fork>

docs/installation/mac.md

@@ -8,18 +8,25 @@ title: macOS
 
 ### Ruby
 
-1. **Note:** MacOS ships with a version of Ruby, needed for various operating systems. 
+1. **Note:** MacOS ships with a version of Ruby, needed for various operating
-   To avoid causing an issue with your operating system you should use a version manager for Ruby.
+   systems. To avoid causing an issue with your operating system you should use
+   a version manager for Ruby.
+
+   If you don't already have a Ruby version manager, we highly recommend
+   [rbenv](https://github.com/rbenv/rbenv). This will allow you to have
+   different versions running on a per project basis. The MacOS system version
+   of Ruby will stay intact while giving you the ability to use the version
+   needed for this Forem project. Please follow their
+   [installation guide](https://github.com/rbenv/rbenv#installation).
 
-   If you don't already have a Ruby version manager, we highly recommend [rbenv](https://github.com/rbenv/rbenv). This will allow you to have different versions running on a per project basis. The MacOS system version of Ruby will stay intact while giving you the ability to use the version needed for this Forem project.
-Please follow their [installation guide](https://github.com/rbenv/rbenv#installation).
 2. With the Ruby version manager, install the Ruby version listed on our badge.
    (i.e. with rbenv: `rbenv install $(cat .ruby-version)`)
-   
+
-   **Note:** The repository must be forked and cloned before running the `rbenv install $(cat .ruby-version)` command.
+   **Note:** The repository must be forked and cloned before running the
-    
+   `rbenv install $(cat .ruby-version)` command.
 
 ### Yarn
+
 Please refer to their [installation guide](https://yarnpkg.com/en/docs/install).
 
 ### PostgreSQL
@@ -63,146 +70,6 @@ You can test if it's up and running by issuing the following command:
 redis-cli ping
 ```
 
-### Elasticsearch
-
-Forem requires Elasticsearch 7.x to run. We recommend version 7.5.2.
-
-You have the option of installing Elasticsearch with Homebrew or through an
-archive. We **recommend** installing from archive on Mac.
-
-### Installing Elasticsearch from the archive
-
-We recommend that you **do not** install Elasticsearch in the app directory.
-Instead, we recommend installing it in your home directory (for example,
-`cd $HOME`). (This also ensures that we don't accidentally commit Elasticsearch
-code to the project's repository!)
-
-The following directions were
-[taken from the Elasticsearch docs themselves](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-macos),
-so check those out if you run into any issues or want further information. Make
-sure to download **the OSS version** of Elasticsearch, `elasticsearch-oss`.
-
-Please note that you will need `wget` in order to proceed with this installation
-(`brew install wget`).
-
-```shell
-wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz
-wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz.sha512
-shasum -a 512 -c elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz.sha512
-tar -xzf elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz
-```
-
-To start elasticsearch, make sure you are in the correct directory:
-
-```shell
-cd elasticsearch-7.5.2
-```
-
-You can then start it by running:
-
-```shell
-./bin/elasticsearch
-```
-
-To start elasticsearch as a daemonized process:
-
-```shell
-./bin/elasticsearch -d
-```
-
-### Installing Elasticsearch with Homebrew
-
-To install Elasticsearch with Homebrew we will use the following commands to:
-
-- tap the Elastic Homebrew repository
-- install the latest OSS distribution
-- pin the latest OSS distribution.
-
-```shell
-brew tap elastic/tap
-brew install elastic/tap/elasticsearch-oss
-brew pin elasticsearch-oss
-```
-
-After installation you can manually test if the Elasticsearch server starts by
-issuing the command `elasticsearch` in the shell. You can then start the server
-as a service with `brew services start elasticsearch-oss`.
-
-You can find further info on your local Elasticsearch installation by typing
-`brew info elastic/tap/elasticsearch-oss`.
-
-#### Troubleshooting startup issues
-
-Two possible startup issues you might encounter:
-
-- `java.nio.file.FileSystemLoopException`:
-
-```text
-Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.FileSystemLoopException: /usr/local/etc/elasticsearch/elasticsearch
-Likely root cause: java.nio.file.FileSystemLoopException: /usr/local/etc/elasticsearch/elasticsearch
-```
-
-This happens because the installation of Elasticsearch might have a recursive
-link in the configuration directory causing the infinite loop:
-
-```shell
-> ll /usr/local/etc/elasticsearch
-elasticsearch -> /usr/local/etc/elasticsearch
-```
-
-By manually removing the link with
-`rm -i /usr/local/etc/elasticsearch/elasticsearch` the issue should be fixed.
-
-- `java.lang.IllegalStateException`:
-
-```text
-java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [plugins]
-Likely root cause: java.nio.file.NoSuchFileException: /usr/local/Cellar/elasticsearch-oss/7.6.0/libexec/plugins/plugins/plugin-descriptor.properties
-```
-
-This happens for a similar reason as the previous error, the installation might
-create a recursive link in the plugins directory.
-
-```shell
-> ll /usr/local/var/elasticsearch/plugins
-plugins -> /usr/local/var/elasticsearch/plugins
-```
-
-By manually removing the link with
-`rm -i /usr/local/var/elasticsearch/plugins/plugins` the issue should be fixed.
-
-### Testing if Elasticsearch is running
-
-Once installed and started you can test if it's up and running correctly by
-issuing the following command:
-
-```shell
-curl http://localhost:9200
-```
-
-You should receive in response a JSON document containing some information about
-your local Elasticsearch installation, for example:
-
-```json
-{
-  "name": "hostname",
-  "cluster_name": "elasticsearch_...",
-  "cluster_uuid": "...",
-  "version": {
-    "number": "7.5.2",
-    "build_flavor": "oss",
-    "build_type": "tar",
-    "build_hash": "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
-    "build_date": "2020-01-15T12:11:52.313576Z",
-    "build_snapshot": false,
-    "lucene_version": "8.3.0",
-    "minimum_wire_compatibility_version": "6.8.0",
-    "minimum_index_compatibility_version": "6.0.0-beta1"
-  },
-  "tagline": "You Know, for Search"
-}
-```
-
 ## Installing Forem
 
 1. Fork Forem's repository, e.g. <https://github.com/forem/forem/fork>
@@ -240,8 +107,7 @@ your local Elasticsearch installation, for example:
 
 ### Possible error messages
 
-**Error:** 
+**Error:** `rbenv install hangs at ruby-build: using readline from homebrew`
-`rbenv install hangs at ruby-build: using readline from homebrew`
 
 **_Solution:_**
 [Stackoverflow answer](https://stackoverflow.com/questions/63599818/rbenv-install-hangs-at-ruby-build-using-readline-from-homebrew)

docs/installation/windows.md

@@ -176,60 +176,6 @@ We recommend to follow
 [this guide](https://redislabs.com/blog/redis-on-windows-10/) to run Redis under
 WSL.
 
-### Elasticsearch
-
-Forem requires a version of Elasticsearch between 7.1 and 7.5. Version 7.6 is
-not supported. We recommend version 7.5.2.
-
-We recommend that you **do not** install Elasticsearch in the app directory.
-Instead, we recommend installing it in your home directory (for example,
-`cd $HOME`). (This also ensures that we don't accidentally commit Elasticsearch
-code to the project's repository!)
-
-The following directions were taken from
-[the Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-linux),
-check them out to learn more about the installation process and troubleshooting
-issues. Make sure to refer to **the OSS version**, `elasticsearch-oss` while
-going through the Elasticsearch docs.
-
-To install Elasticsearch perform the following steps:
-
-1. Execute the following commands:
-
-   ```shell
-   cd
-   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
-   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-linux-x86_64.tar.gz.sha512
-   shasum -a 512 -c elasticsearch-oss-7.5.2-linux-x86_64.tar.gz.sha512
-   tar -xzf elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
-   ```
-
-2. Next, switch to the correct directory with:
-
-   ```shell
-   cd elasticsearch-7.5.2/
-   ```
-
-3. To start Elasticsearch, run the following command:
-
-   ```shell
-   ./bin/elasticsearch
-   ```
-
-   or, start it as a daemonized process with:
-
-   ```shell
-   ./bin/elasticsearch -d
-   ```
-
-4. Once Elasticsearch is running,
-   [verify Elasticsearch's installation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#_checking_that_elasticsearch_is_running)
-   by executing the `cURL` command as follows:
-
-   ```shell
-   curl -X GET "localhost:9200/?pretty"
-   ```
-
 ## Installing Forem
 
 1. Fork Forem's repository, eg. <https://github.com/forem/forem/fork>
@@ -277,8 +223,8 @@ To install Elasticsearch perform the following steps:
    - You do not need "real" keys for basic development. Some features require
      certain keys, so you may be able to add them as you go.
 
-1. After ensuring that Elasticsearch, the PostgreSQL server, and the Redis
+1. After ensuring that the PostgreSQL server and the Redis server are running,
-   server are running, run `bin/setup`.
+   run `bin/setup`.
 
    > The `bin/setup` script is responsible for installing a varienty of
    > dependencies. One can find it inside the `bin` folder by the name of
@@ -290,8 +236,8 @@ To install Elasticsearch perform the following steps:
    > - It then installs JavaScript dependencies using the script in `bin/yarn`
    >   file. These dependencies are located in `package.json` in the root of the
    >   repository.
-   > - Next, it uses various Rake files located inside the `lib` folder to setup
+   > - Next, it uses various Rake files located inside the `lib` folder for the
-   >   ElasticSearch environment, PostgreSQL database creation and updation.
+   >   PostgreSQL database creation and updation.
    > - Finally it cleans up all the log files and restarts the Puma server.
 
 ### Possible error messages

docs/maintainers/deploying.md

@@ -57,10 +57,10 @@ stage will do nothing. If the branch has been merged into main, then this stage
 will kick off a deploy. The deploy will run in its own job deploying our
 application to Heroku.
 
-Prior to deploying the code, Heroku will run database migrations, Elasticsearch
+Prior to deploying the code, Heroku will run database migrations and do some
-updates, and do some final checks (more information on that below) to make sure
+final checks (more information on that below) to make sure everything is working
-everything is working as expected. If these all succeed, then the deploy
+as expected. If these all succeed, then the deploy completes and our team is
-completes and our team is notified.
+notified.
 
 ## Deploying to Heroku
 
@@ -84,13 +84,8 @@ directory. During this release step we do a few checks.
    successfully before the code that uses it goes live.
 3. We run any data update scripts that need to be run. A data update script is
    one that allows us to update data in the background separate from a
-   migration. For example, if we add a new field to Elasticsearch and need to
+   migration.
-   reindex all of our documents we would use a data update script.
+4. Following updating all of our datastores we use the Rails runner to output a
-4. We update Elasticsearch. Elasticsearch contains indexes which have mappings.
-   Mappings are similar to database schema. The same way we run a migration to
-   update our database we have to run a setup task to update any Elasticsearch
-   mappings.
-5. Following updating all of our datastores we use the Rails runner to output a
    simple string. Executing a Rails runner command ensures that we can boot up
    the entire app successfully before it is deployed. We deploy asynchronously,
    so the website is running the new code a few minutes after deploy. A new

docs/technical-overview/architecture.md

@@ -72,10 +72,10 @@ Rails][fastly_rails] docs, but we bust specific URLs via `EdgeCache::Bust#call`.
 
 The home feed is based on a combination of recent collective posts that are
 cached and delivered the same to everyone in the HTML, and additional articles
-fetched from an Elasticsearch index after page load. To determine which posts a
+fetched from a PostgreSQL Full Text Search index after page load. To determine
-user sees, they are ranked based on the user's followed tags, followed users,
+which posts a user sees, they are ranked based on the user's followed tags,
-and relative weights for each tag. Additional fetched articles also follow this
+followed users, and relative weights for each tag. Additional fetched articles
-general pattern.
+also follow this general pattern.
 
 Currently, the top post on the home feed, which must have a cover image, is
 shared among all users.

docs/technical-overview/stack.md

@@ -7,7 +7,8 @@ title: Stack
 For the Forem tech stack we use:
 
 - [_Puma_](https://github.com/puma/puma) as the web server
-- [_PostgreSQL_](https://www.postgresql.org/) as the primary database
+- [_PostgreSQL_](https://www.postgresql.org/) as the primary database and for
+  Full Text Search
 - [_Redis_](https://redis.io/) to store cached data
 - [_Fastly_](https://www.fastly.com/) for
   [edge caching](https://dev.to/ben/making-devto-insanely-fast)
@@ -18,7 +19,6 @@ For the Forem tech stack we use:
 - [_Sidekiq_](https://github.com/mperham/sidekiq) and
   [_Active Job_](https://guides.rubyonrails.org/active_job_basics.html) for
   background workers
-- [_Elasticsearch_](https://www.elastic.co/elasticsearch/) for in app searching
 - [Ransack](https://github.com/activerecord-hackery/ransack) for internal search
 - [_Redcarpet_](https://github.com/vmg/redcarpet) and
   [_Rouge_](https://github.com/jneen/rouge) to parse Markdown

docs/tests/readme.md

@@ -69,8 +69,3 @@ retry the individual job that failed rather than the entire suite in order to
 save some time. When you retry the individual job, make sure to also retry the
 Deploy job. Even though you may not be deploying that job must complete for the
 entire build to pass.
-
-Please note that you will need to have Elasticsearch installed and running for
-certain tests in our test suite. You can find instructions on how to install and
-run Elasticsearch specific to your environment in the
-[Installation Guide](/installation).

docs/tests/test-flags.md

@@ -11,59 +11,15 @@ test example.
 Some flags that we use are:
 
 - `js: true`
-- `elasticsearch_reset: true`
-- `elasticsearch: <search class name>`
-- `stub_elasticsearch: true`
 - `throttle: true`
 - `type: <test type>`
 
-### Elasticsearch Flags
-
-Two Elasticsearch flags that can be used when dealing with specs that will be
-hitting Elasticsearch. When running a spec that is going to interact with
-Elasticsearch, you want Elasticsearch to be clean like your database. Since we
-don't have a "database cleaner" for Elasticsearch, we have to do it manually.
-There are two ways to do this:
-
-1. `elasticsearch: reset` - This will trigger a complete tear down and rebuild
-   of Elasticsearch via a block. This takes time so it's not something you want
-   to be doing unless you absolutely have to.
-
-```ruby
-  config.around(:each, elasticsearch_reset: true) do |example|
-    Search::Cluster.recreate_indexes
-    example.run
-    Search::Cluster.recreate_indexes
-  end
-```
-
-2. `elasticsearch: <search class name>` - This will clear the data for the given
-   search class index. For example, if you passed "User" as your search class
-   name it would clear out the user index data using code from this block.
-
-```ruby
-  config.around(:each, :elasticsearch) do |ex|
-    klasses = Array.wrap(ex.metadata[:elasticsearch]).map do |search_class|
-      Search.const_get(search_class)
-    end
-    klasses.each { |klass| clear_elasticsearch_data(klass) }
-    ex.run
-  end
-```
-
 ### `js: true` Flag
 
 `js: true` indicates that we want the JavaScript on the page to be executed when
 the page is rendered, and a headless chrome instance will be initialized to do
 so (instead of the default
-[rack_test](https://github.com/teamcapybara/capybara#racktest) driver). One side
+[rack_test](https://github.com/teamcapybara/capybara#racktest) driver).
-effect of running our JavaScript in our specs is that a lot of pages will hit
-Elasticsearch. Since we don't clean out Elasticsearch between every single spec
-(because it is very costly) this can lead to unexpected data being loaded for a
-spec. To prevent this from happening, we can use the `:stub_elasticsearch` flag.
-The `:stub_elasticsearch` flag will stub all index and search requests made to
-Elasticsearch and return an empty response. This will ensure that no unwanted
-data shows up on your spec's page.
 
 If you are debugging a `js: true` spec and want to see the browser, you can set
 `HEADLESS=false` before running a spec:

docs/troubleshooting.md

@@ -33,66 +33,6 @@ The first command creates the user **ec2-user** and the second one creates the
 database for this user because every user needs its database. Even if the first
 command fails, run the second command to create the missing database.
 
-## Elasticsearch
-
-### Index read-only error
-
-If you encounter an error similar to the following:
-
-```shell
-{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [tags_development] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"index [tags_development] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}
-```
-
-it means that Elasticsearch went into read only mode because its disk allocator
-noticed how the disk drive is nearly full:
-
-> Elasticsearch enforces a read-only index block
-> (`index.blocks.read_only_allow_delete`) on every index that has one or more
-> shards allocated on the node that has at least one disk exceeding the flood
-> stage.
-
-This is an indication that you might not have enough space for ES to work
-correctly.
-
-If you want to disable the threshold check on your local machine you can open a
-Rails console (with `rails console`) and issue the following command:
-
-```ruby
-Search::Client.cluster.put_settings(body: {
-  persistent: {
-    "cluster.routing.allocation.disk.threshold_enabled" => false,
-  }
-})
-```
-
-Alternatively you can use the following command in a terminal session:
-
-```shell
-curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
-```
-
-To disable the "read only" mode to allow operations on the Elasticsearch indexes
-you can issue the following command, similary in the Rails console:
-
-```ruby
-Search::Client.indices.get(index: "*").keys.each do |index_name|
-  Search::Client.indices.put_settings(
-    index: index_name,
-    body: { "index.blocks.read_only_allow_delete" => nil }
-  )
-end
-```
-
-Or alternatively:
-
-```shell
-curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
-```
-
-If instead you want to tune the Elasticsearch disk allocator's settings, please
-refer to
-[Disk-based shard allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html#disk-allocator).
-
 ## CORS
 
 If you are experiencing CORS issues locally or need to display more information

lib/data_update_scripts/20200218195023_index_chat_channel_memberships_to_elasticsearch.rb

@@ -1,7 +1,7 @@
 module DataUpdateScripts
   class IndexChatChannelMembershipsToElasticsearch
     def run
-      ChatChannelMembership.find_each(&:index_to_elasticsearch_inline)
+      # ChatChannelMembership.find_each(&:index_to_elasticsearch_inline)
     end
   end
 end

lib/data_update_scripts/20200410152018_resync_elasticsearch_documents.rb

@@ -2,13 +2,13 @@ module DataUpdateScripts
   class ResyncElasticsearchDocuments
     def run
       # Previous bug: Not getting removed properly
-      sync_docs(Article.ids, "Article")
+      # sync_docs(Article.ids, "Article")
-      sync_docs(Comment.ids, "Comment")
+      # sync_docs(Comment.ids, "Comment")
-      sync_docs(User.ids, "User")
+      # sync_docs(User.ids, "User")
 
       # Previous bug: Not getting indexed on creation properly
-      index_docs(PodcastEpisode.ids, "PodcastEpisode")
+      # index_docs(PodcastEpisode.ids, "PodcastEpisode")
-      index_docs(Tag.ids, "Tag")
+      # index_docs(Tag.ids, "Tag")
     end
 
     private

lib/data_update_scripts/20200519142908_re_index_feed_content_and_users_to_elasticsearch.rb

@@ -1,21 +1,21 @@
 module DataUpdateScripts
   class ReIndexFeedContentAndUsersToElasticsearch
     def run
-      Article.select(:id).in_batches(of: 100) do |batch|
+      # Article.select(:id).in_batches(of: 100) do |batch|
-        Search::BulkIndexWorker.set(queue: :default).perform_async(
+      #   Search::BulkIndexWorker.set(queue: :default).perform_async(
-          "Article", batch.ids
+      #     "Article", batch.ids
-        )
+      #   )
-      end
+      # end
-      Comment.select(:id).in_batches(of: 100) do |batch|
+      # Comment.select(:id).in_batches(of: 100) do |batch|
-        Search::BulkIndexWorker.set(queue: :default).perform_async(
+      #   Search::BulkIndexWorker.set(queue: :default).perform_async(
-          "Comment", batch.ids
+      #     "Comment", batch.ids
-        )
+      #   )
-      end
+      # end
-      PodcastEpisode.select(:id).in_batches(of: 100) do |batch|
+      # PodcastEpisode.select(:id).in_batches(of: 100) do |batch|
-        Search::BulkIndexWorker.set(queue: :default).perform_async(
+      #   Search::BulkIndexWorker.set(queue: :default).perform_async(
-          "PodcastEpisode", batch.ids
+      #     "PodcastEpisode", batch.ids
-        )
+      #   )
-      end
+      # end
 
       # See: https://github.com/forem/forem/pull/10313#discussion_r487646864
       # User.select(:id).in_batches(of: 200) do |batch|

lib/data_update_scripts/20200803142830_reindex_listing_search_column.rb

@@ -3,7 +3,7 @@ module DataUpdateScripts
     def run
       # Choose to do inline so development envs are ready immediately after
       # this is run
-      Listing.find_each(&:index_to_elasticsearch_inline)
+      # Listing.find_each(&:index_to_elasticsearch_inline)
     end
   end
 end

lib/data_update_scripts/20200904132553_remove_draft_articles_with_duplicate_feed_source_url.rb

@@ -5,76 +5,76 @@ module DataUpdateScripts
 
       # This statement deletes all draft articles in excess found to be duplicate over feed_source_url,
       # excluding those whose body_markdown is different from the other duplicate occurrences
-      result_same_body = ActiveRecord::Base.connection.execute(
+      # result_same_body = ActiveRecord::Base.connection.execute(
-        <<-SQL,
+      #   <<-SQL,
-          WITH duplicates_draft_articles AS
+      #     WITH duplicates_draft_articles AS
-              (SELECT id
+      #         (SELECT id
-               FROM
+      #          FROM
-                   (SELECT id,
+      #              (SELECT id,
-                           published,
+      #                      published,
-                           body_markdown,
+      #                      body_markdown,
-                           LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
+      #                      LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
-                                                      ORDER BY id ASC) AS previous_body_markdown,
+      #                                                 ORDER BY id ASC) AS previous_body_markdown,
-                           ROW_NUMBER() OVER(PARTITION BY feed_source_url
+      #                      ROW_NUMBER() OVER(PARTITION BY feed_source_url
-                                             ORDER BY id ASC) AS row_number
+      #                                        ORDER BY id ASC) AS row_number
-                    FROM articles
+      #               FROM articles
-                    WHERE feed_source_url IS NOT NULL ) duplicates
+      #               WHERE feed_source_url IS NOT NULL ) duplicates
-               WHERE duplicates.row_number > 1
+      #          WHERE duplicates.row_number > 1
-                   AND published = 'f' -- drafts
+      #              AND published = 'f' -- drafts
-                   AND body_markdown = previous_body_markdown -- with the same body
+      #              AND body_markdown = previous_body_markdown -- with the same body
-           )
+      #      )
-          DELETE
+      #     DELETE
-          FROM articles
+      #     FROM articles
-          WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
+      #     WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
-        SQL
+      #   SQL
-      )
+      # )
-
+      #
       # Now that all duplicates with the same body are gone, we need to deal with duplicate feed source URLs
       # with different bodies.
       # We thus select the oldest for removal preserving the most recent one
-      result_different_bodies = ActiveRecord::Base.connection.execute(
+      # result_different_bodies = ActiveRecord::Base.connection.execute(
-        <<-SQL,
+      #   <<-SQL,
-          WITH duplicates_draft_articles AS
+      #     WITH duplicates_draft_articles AS
-              (SELECT id
+      #         (SELECT id
-               FROM
+      #          FROM
-                   (SELECT id,
+      #              (SELECT id,
-                           published,
+      #                      published,
-                           body_markdown,
+      #                      body_markdown,
-                           LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
+      #                      LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
-                                                      ORDER BY created_at DESC) AS previous_body_markdown,
+      #                                                 ORDER BY created_at DESC) AS previous_body_markdown,
-                           ROW_NUMBER() OVER(PARTITION BY feed_source_url
+      #                      ROW_NUMBER() OVER(PARTITION BY feed_source_url
-                                             ORDER BY created_at DESC) AS row_number
+      #                                        ORDER BY created_at DESC) AS row_number
-                    FROM articles
+      #               FROM articles
-                    WHERE feed_source_url IS NOT NULL ) duplicates
+      #               WHERE feed_source_url IS NOT NULL ) duplicates
-               WHERE duplicates.row_number > 1
+      #          WHERE duplicates.row_number > 1
-                   AND published = 'f' -- drafts
+      #              AND published = 'f' -- drafts
-                   AND body_markdown != previous_body_markdown -- with different bodies
+      #              AND body_markdown != previous_body_markdown -- with different bodies
-           )
+      #      )
-          DELETE
+      #     DELETE
-          FROM articles
+      #     FROM articles
-          WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
+      #     WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
-        SQL
+      #   SQL
-      )
+      # )
-
+      #
-      result_same_body_ids = result_same_body.map { |row| row["id"] }
+      # result_same_body_ids = result_same_body.map { |row| row["id"] }
-      result_different_bodies_ids = result_different_bodies.map { |row| row["id"] }
+      # result_different_bodies_ids = result_different_bodies.map { |row| row["id"] }
 
       # Remove all articles from Elasticsearch
-      (result_same_body_ids + result_different_bodies_ids).each do |id|
+      # (result_same_body_ids + result_different_bodies_ids).each do |id|
-        Search::RemoveFromIndexWorker.perform_in(5.seconds, "Search::FeedContent", "article_#{id}")
+      #   Search::RemoveFromIndexWorker.perform_in(5.seconds, "Search::FeedContent", "article_#{id}")
-      end
+      # end
 
       # Store deleted IDs temporarily in Redis for safe keeping
-      Rails.cache.write(
+      # Rails.cache.write(
-        "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::SameBody",
+      #   "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::SameBody",
-        result_same_body_ids,
+      #   result_same_body_ids,
-        expires_in: 2.weeks,
+      #   expires_in: 2.weeks,
-      )
+      # )
-      Rails.cache.write(
+      # Rails.cache.write(
-        "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::DifferentBodies",
+      #   "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::DifferentBodies",
-        result_same_body_ids,
+      #   result_same_body_ids,
-        expires_in: 2.weeks,
+      #   expires_in: 2.weeks,
-      )
+      # )
     end
   end
 end

lib/data_update_scripts/20200910140109_cleanup_published_articles_with_duplicate_user_id_title_body_markdown.rb

@@ -34,7 +34,7 @@ module DataUpdateScripts
 
           # save all the magic
           article.save
-          article.index_to_elasticsearch_inline
+          # article.index_to_elasticsearch_inline
 
           articles_to_delete_ids += articles_to_graft.map { |a| a["id"] }
         end

lib/data_update_scripts/20200911045602_reindex_articles_with_videos.rb

@@ -1,8 +1,8 @@
 module DataUpdateScripts
   class ReindexArticlesWithVideos
     def run
-      articles = Article.where.not(video: nil).or(Article.where.not(video: ""))
+      # articles = Article.where.not(video: nil).or(Article.where.not(video: ""))
-      articles.find_each(&:index_to_elasticsearch_inline)
+      # articles.find_each(&:index_to_elasticsearch_inline)
     end
   end
 end

lib/data_update_scripts/20200924140813_remove_reaction_index_by_name.rb

@@ -1,10 +1,10 @@
 module DataUpdateScripts
   class RemoveReactionIndexByName
     def run
-      index_name = "reactions_#{Rails.env}"
+      # index_name = "reactions_#{Rails.env}"
-      return unless Search::Client.indices.exists(index: index_name)
+      # return unless Search::Client.indices.exists(index: index_name)
 
-      Search::Client.indices.delete(index: index_name)
+      # Search::Client.indices.delete(index: index_name)
     end
   end
 end

lib/data_update_scripts/20201030134117_reindex_users_for_username_search.rb

@@ -1,9 +1,9 @@
 module DataUpdateScripts
   class ReindexUsersForUsernameSearch
     def run
-      User.select(:id).in_batches(of: 200) do |batch|
+      # User.select(:id).in_batches(of: 200) do |batch|
-        Search::BulkIndexWorker.set(queue: :default).perform_async("User", batch.ids)
+      #   Search::BulkIndexWorker.set(queue: :default).perform_async("User", batch.ids)
-      end
+      # end
     end
   end
 end

lib/data_update_scripts/20210118194138_resync_unpublished_articles_comments_elasticsearch_document.rb

@@ -1,9 +1,9 @@
 module DataUpdateScripts
   class ResyncUnpublishedArticlesCommentsElasticsearchDocument
     def run
-      Article.unpublished.where.not(comments_count: 0).each do |article|
+      # Article.unpublished.where.not(comments_count: 0).each do |article|
-        article.comments.each(&:index_to_elasticsearch)
+      #   article.comments.each(&:index_to_elasticsearch)
-      end
+      # end
     end
   end
 end

lib/tasks/app_initializer.rake

@@ -1,9 +1,6 @@
 namespace :app_initializer do
   desc "Prepare Application on Boot Up"
   task setup: :environment do
-    puts "\n== Preparing Elasticsearch =="
-    Rake::Task["search:setup"].execute
-
     puts "\n== Preparing database =="
     system("bin/rails db:prepare") || exit!(1)
     Rake::Task["db:migrate"].execute # it'll re-alphabetize the columns in `schema.rb`

lib/tasks/search.rake

@@ -1,26 +0,0 @@
-namespace :search do
-  desc "set up Elasticsearch indexes"
-  task setup: :environment do
-    Search::Cluster.setup_indexes
-  end
-
-  desc "update Elasticsearch index mappings"
-  task update_mappings: :environment do
-    Search::Cluster.update_mappings
-  end
-
-  desc "tear down Elasticsearch indexes"
-  task destroy: :environment do
-    if Rails.env.production?
-      puts "Will NOT destroy indexes in production"
-      next
-    end
-
-    Search::Cluster.delete_indexes
-  end
-end
-
-if Rails.env.development?
-  Rake::Task["db:create"].enhance(["search:setup"])
-  Rake::Task["db:drop"].enhance(["search:destroy"])
-end

release-tasks.sh

@@ -14,8 +14,7 @@ set -Eex
 # abort release if deploy status equals "blocked"
 [[ $DEPLOY_STATUS = "blocked" ]] && echo "Deploy blocked" && exit 1
 
-# runs migration for Postgres, setups/updates Elasticsearch
+# runs migration for Postgres and boots the app to check there are no errors
-# and boots the app to check there are no errors
 STATEMENT_TIMEOUT=4500000 bundle exec rails app_initializer:setup
 bundle exec rake fastly:update_configs
 bundle exec rails runner "puts 'app load success'"

spec/factories/tags.rb

@@ -4,9 +4,5 @@ FactoryBot.define do
   factory :tag do
     name { generate :name }
     supported { true }
-
-    trait :search_indexed do
-      after(:create, &:index_to_elasticsearch_inline)
-    end
   end
 end

spec/initializers/rack/attack_spec.rb

@@ -11,12 +11,13 @@ describe Rack::Attack, type: :request, throttle: true do
   describe "search_throttle" do
     it "throttles /search endpoints based on IP" do
       Timecop.freeze do
-        allow(Search::User).to receive(:search_documents).and_return({})
+        allow(Search::Postgres::Username).to receive(:search_documents).and_return({})
+
         valid_responses = Array.new(5).map do
-          get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
+          get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
         end
-        throttled_response = get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
+        throttled_response = get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
-        new_ip_response = get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "1.1.1.1" }
+        new_ip_response = get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "1.1.1.1" }
 
         valid_responses.each { |r| expect(r).not_to eq(429) }
         expect(throttled_response).to eq(429)

spec/lib/acts_as_taggable_on/tag_spec.rb

@@ -1,31 +0,0 @@
-require "rails_helper"
-
-RSpec.describe ActsAsTaggableOn::Tag, type: :lib do
-  describe "#after_commit" do
-    it "on create indexes tag to elasticsearch" do
-      tag_name = "muffintag"
-      create(:article, body_markdown: "---\ntitle: Me#{rand(1000)}\ntags: #{tag_name}\n---\n\nMeMeMe")
-      sidekiq_perform_enqueued_jobs
-      tag = Tag.find_by(name: tag_name)
-      expect(tag.elasticsearch_doc).not_to be_nil
-    end
-
-    it "syncs related elasticsearch documents" do
-      article = create(:article)
-      podcast_episode = create(:podcast_episode)
-      tag = article.tags.first
-      podcast_episode.tags << tag
-      new_keywords = "keyword1, keyword2, keyword3"
-      sidekiq_perform_enqueued_jobs
-
-      tag.update(keywords_for_search: new_keywords)
-      sidekiq_perform_enqueued_jobs
-      expect(collect_keywords(article)).to include(new_keywords)
-      expect(collect_keywords(podcast_episode)).to include(new_keywords)
-    end
-  end
-
-  def collect_keywords(record)
-    record.elasticsearch_doc.dig("_source", "tags").flat_map { |t| t["keywords_for_search"] }
-  end
-end

spec/lib/data_update_scripts/index_chat_channel_memberships_to_elasticsearch_spec.rb

@@ -1,11 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20200218195023_index_chat_channel_memberships_to_elasticsearch.rb")
-
-describe DataUpdateScripts::IndexChatChannelMembershipsToElasticsearch, elasticsearch: "ChatChannelMembership" do
-  it "indexes chat channel memberships to Elasticsearch" do
-    chat_channel_membership = FactoryBot.create(:chat_channel_membership)
-    expect { chat_channel_membership.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-    described_class.new.run
-    expect(chat_channel_membership.elasticsearch_doc).not_to be_nil
-  end
-end

spec/lib/data_update_scripts/re_index_feed_content_and_users_to_elasticsearch_spec.rb

@@ -1,28 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20200519142908_re_index_feed_content_and_users_to_elasticsearch.rb")
-
-describe DataUpdateScripts::ReIndexFeedContentAndUsersToElasticsearch do
-  after do
-    Search::FeedContent.refresh_index
-  end
-
-  it "indexes feed content(articles, comments, podcast episodes) and users to Elasticsearch", :aggregate_failures do
-    article = create(:article)
-    podcast_episode = create(:podcast_episode)
-    comment = create(:comment)
-
-    expect { article.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-    expect { podcast_episode.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-    expect { comment.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-
-    sidekiq_perform_enqueued_jobs { described_class.new.run }
-
-    expect(article.elasticsearch_doc).not_to be_nil
-    expect(podcast_episode.elasticsearch_doc).not_to be_nil
-    expect(comment.elasticsearch_doc).not_to be_nil
-
-    expect(article.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
-    expect(podcast_episode.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
-    expect(comment.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
-  end
-end

spec/lib/data_update_scripts/reindex_listing_search_column_spec.rb

@@ -1,11 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20200803142830_reindex_listing_search_column.rb")
-
-describe DataUpdateScripts::ReindexListingSearchColumn, elasticsearch: "Listing" do
-  it "indexes listings to Elasticsearch" do
-    listing = create(:listing)
-    expect { listing.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-    described_class.new.run
-    expect(listing.elasticsearch_doc).not_to be_nil
-  end
-end

spec/lib/data_update_scripts/reindex_users_for_username_search_spec.rb

@@ -1,12 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20201030134117_reindex_users_for_username_search.rb")
-
-describe DataUpdateScripts::ReindexUsersForUsernameSearch do
-  let(:user) { create :user }
-
-  it "reindexes users" do
-    sidekiq_assert_enqueued_with(job: Search::BulkIndexWorker, args: ["User", [user.id]], queue: "default") do
-      described_class.new.run
-    end
-  end
-end

spec/lib/data_update_scripts/remove_reaction_index_by_name_spec.rb

@@ -1,13 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20200924140813_remove_reaction_index_by_name.rb")
-
-describe DataUpdateScripts::RemoveReactionIndexByName do
-  it "removes reaction index" do
-    index_name = "reactions_#{Rails.env}"
-    Search::Client.indices.create(index: index_name)
-    expect(Search::Client.indices.get(index: "*").keys).to include(index_name)
-
-    described_class.new.run
-    expect(Search::Client.indices.get(index: "*").keys).not_to include(index_name)
-  end
-end

spec/lib/data_update_scripts/resync_elasticsearch_documents_spec.rb

@@ -1,56 +0,0 @@
-require "rails_helper"
-require Rails.root.join("lib/data_update_scripts/20200410152018_resync_elasticsearch_documents.rb")
-
-describe DataUpdateScripts::ResyncElasticsearchDocuments, elasticsearch: %w[FeedContent User Tag] do
-  after do
-    Article::SEARCH_CLASS.refresh_index
-    User::SEARCH_CLASS.refresh_index
-  end
-
-  it "indexes podcast episodes and tags to Elasticsearch" do
-    tag = create(:tag)
-    podcast_episode = create(:podcast_episode)
-    Sidekiq::Worker.clear_all
-
-    expect { tag.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-    expect { podcast_episode.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
-
-    sidekiq_perform_enqueued_jobs { described_class.new.run }
-    expect(tag.elasticsearch_doc).not_to be_nil
-    expect(podcast_episode.elasticsearch_doc).not_to be_nil
-  end
-
-  it "syncs articles, comments, and users to Elasticsearch", :aggregate_failures do
-    article = create(:article)
-    comment = create(:comment)
-    user = create(:user)
-    index_real_and_mock_documents(article, comment, user)
-
-    expect(Article::SEARCH_CLASS.articles_document_count).not_to eq(Article.count)
-    expect(Comment::SEARCH_CLASS.comments_document_count).not_to eq(Comment.count)
-    expect(User::SEARCH_CLASS.document_count).not_to eq(User.count)
-
-    sidekiq_perform_enqueued_jobs { described_class.new.run }
-    refresh_indexes
-
-    expect(article.elasticsearch_doc).not_to be_nil
-    expect(user.elasticsearch_doc).not_to be_nil
-    expect(comment.elasticsearch_doc).not_to be_nil
-    expect(Article::SEARCH_CLASS.articles_document_count).to eq(Article.count)
-    expect(Comment::SEARCH_CLASS.comments_document_count).to eq(Comment.count)
-    expect(User::SEARCH_CLASS.document_count).to eq(User.count)
-  end
-
-  def index_real_and_mock_documents(article, comment, user)
-    sidekiq_perform_enqueued_jobs
-    Article::SEARCH_CLASS.index("article_#{article.id + 100}", class_name: "Article")
-    Comment::SEARCH_CLASS.index("comment_#{comment.id + 100}", class_name: "Comment")
-    User::SEARCH_CLASS.index(user.id + 100, name: "User")
-    refresh_indexes
-  end
-
-  def refresh_indexes
-    User::SEARCH_CLASS.refresh_index
-    Article::SEARCH_CLASS.refresh_index
-  end
-end