maverick/docbrown
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Remove Elasticsearch ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ (#13606)

Browse source 
* Attempt number 1

* Fix rack_attack specs

* Fix users_searches_users spec

* Fix display_users_search_spec

* Fix comment typo

* Remove search:destroy task from cypress

* Remove port 9300 from gitpod

* Stub response in attack_spec
This commit is contained in:
Alex 2021-05-03 11:09:45 -04:00 • committed by GitHub
parent b9ecfef149
commit 393ba00221
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
167 changed files with 178 additions and 4996 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
.env_sample
View file

@ -33,9 +33,6 @@ REDIS_SIDEKIQ_URL="redis://localhost:6379"
# Redis Devices/Rpush storage
REDIS_RPUSH_URL="redis://localhost:6379"
# Elasticsearch
ELASTICSEARCH_URL="http://localhost:9200"
# OpenResty
OPENRESTY_URL=""
@ -164,4 +161,4 @@ SENDGRID_API_KEY="Optional, Production only"
# Disable simplecov coverage testing when running rspec locally
COVERAGE="false"
COVERAGE="false"

9
.gitpod.dockerfile
View file

@ -18,12 +18,3 @@ RUN sudo apt-get update \
&& sudo apt-get install -y \
redis-server \
&& sudo rm -rf /var/lib/apt/lists/*
# Install Elasticsearch
ARG ES_REPO=https://artifacts.elastic.co/downloads/elasticsearch
ARG ES_ARCHIVE=elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
RUN wget "${ES_REPO}/${ES_ARCHIVE}" \
&& wget "${ES_REPO}/${ES_ARCHIVE}.sha512" \
&& shasum -a 512 -c ${ES_ARCHIVE}.sha512 \
&& tar -xzf ${ES_ARCHIVE} \
&& rm ${ES_ARCHIVE} ${ES_ARCHIVE}.sha512

5
.gitpod.yml
View file

@ -10,16 +10,11 @@ ports:
onOpen: ignore
- port: 6379
onOpen: ignore
- port: 9200
onOpen: ignore
- port: 9300
onOpen: ignore
tasks:
- name: Forem Server
before: |
redis-server &
/home/gitpod/elasticsearch-7.5.2/bin/elasticsearch &
init: >
cp .env_sample .env &&
gem install solargraph &&

9
.travis.yml
View file

@ -39,20 +39,11 @@ env:
- KNAPSACK_PRO_CI_NODE_TOTAL=3
- COVERAGE_REPORTS_TOTAL=4
- FOREM_OWNER_SECRET="secret" # test secret so e2e tests can run properly.
- ELASTICSEARCH_URL="http://localhost:9200"
before_install:
- gem install bundler:"<2.3"
install:
- date --rfc-3339=seconds
- nvm install
- pkill -9 -f elasticsearch || true
- curl -s -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.2-amd64.deb
- sudo dpkg -i --force-confnew elasticsearch-7.5.2-amd64.deb
- sudo sed -i.old 's/-Xms1g/-Xms128m/' /etc/elasticsearch/jvm.options
- sudo sed -i.old 's/-Xmx1g/-Xmx128m/' /etc/elasticsearch/jvm.options
- echo -e '-XX:+DisableExplicitGC\n-Djdk.io.permissionsUseCanonicalPath=true\n-Dlog4j.skipJansi=true\n-server\n' | sudo tee -a /etc/elasticsearch/jvm.options
- sudo chown -R elasticsearch:elasticsearch /etc/default/elasticsearch
- sudo systemctl start elasticsearch
- bundle config set path 'vendor/bundle'
- cp .env_sample .env
- bin/ci-bundle

3
Gemfile
View file

@ -28,7 +28,6 @@ gem "devise" # Flexible authentication solution for Rails
gem "devise_invitable", "~> 2.0.5" # Allows invitations to be sent for joining
gem "dogstatsd-ruby", "~> 4.8" # A client for DogStatsD, an extension of the StatsD metric server for Datadog
gem "doorkeeper", "~> 5.5" # Oauth 2 provider
gem "elasticsearch", "~> 7.12" # Powers DEVs core search functionality
gem "email_validator", "~> 2.2" # Email validator for Rails and ActiveModel
gem "emoji_regex", "~> 3.2" # A pair of Ruby regular expressions for matching Unicode Emoji symbols
gem "fastly", "~> 3.0" # Client library for the Fastly acceleration system
@ -65,7 +64,7 @@ gem "omniauth-github", "~> 2.0" # OmniAuth strategy for GitHub
gem "omniauth-rails_csrf_protection", "~> 1.0" # Provides CSRF protection on OmniAuth request endpoint on Rails application.
gem "omniauth-twitter", "~> 1.4" # OmniAuth strategy for Twitter
gem "parallel", "~> 1.20" # Run any kind of code in parallel processes
gem "patron", "~> 0.13.3" # HTTP client library based on libcurl, used with Elasticsearch to support http keep-alive connections
gem "patron", "~> 0.13.3" # HTTP client library based on libcurl, used with GitHub OAuth client
gem "pg", "~> 1.2" # Pg is the Ruby interface to the PostgreSQL RDBMS
gem "pg_search", "~> 2.3.5" # PgSearch builds Active Record named scopes that take advantage of PostgreSQL's full text search
gem "puma", "~> 5.2.2" # Puma is a simple, fast, threaded, and highly concurrent HTTP 1.1 server

9
Gemfile.lock
View file

@ -223,14 +223,6 @@ GEM
dotenv-rails (2.7.6)
dotenv (= 2.7.6)
railties (>= 3.2)
elasticsearch (7.12.0)
elasticsearch-api (= 7.12.0)
elasticsearch-transport (= 7.12.0)
elasticsearch-api (7.12.0)
multi_json
elasticsearch-transport (7.12.0)
faraday (~> 1)
multi_json
em-websocket (0.5.2)
eventmachine (>= 0.12.9)
http_parser.rb (~> 0.6.0)
@ -880,7 +872,6 @@ DEPENDENCIES
dogstatsd-ruby (~> 4.8)
doorkeeper (~> 5.5)
dotenv-rails (~> 2.7.6)
elasticsearch (~> 7.12)
email_validator (~> 2.2)
emoji_regex (~> 3.2)
erb_lint (~> 0.0.37)

1
README.md
View file

@ -116,7 +116,6 @@ A more complete overview of our stack is available in
- [ImageMagick](https://imagemagick.org/): please refer to ImageMagick's
[installation instructions](https://imagemagick.org/script/download.php).
- [Redis](https://redis.io/) 4 or higher.
- [Elasticsearch](https://www.elastic.co) 7 or higher.
#### Containers

8
app/controllers/api/v0/health_checks_controller.rb
View file

@ -7,14 +7,6 @@ module Api
render json: { message: "App is up!" }, status: :ok
end
def search
if Search::Client.ping
render json: { message: "Search ping succeeded!" }, status: :ok
else
render json: { message: "Search ping failed!" }, status: :internal_server_error
end
end
def database
if ActiveRecord::Base.connected?
render json: { message: "Database connected" }, status: :ok

1
app/controllers/chat_channels_controller.rb
View file

@ -159,7 +159,6 @@ class ChatChannelsController < ApplicationController
authorize chat_channel
chat_channel.status = "blocked"
chat_channel.save
chat_channel.chat_channel_memberships.map(&:index_to_elasticsearch)
render json: { status: "success", message: "chat channel blocked" }, status: :ok
end

8
app/controllers/search_controller.rb
View file

@ -93,10 +93,6 @@ class SearchController < ApplicationController
render json: { result: result }
end
def users
render json: { result: user_search }
end
def usernames
result = Search::Postgres::Username.search_documents(params[:username])
@ -197,10 +193,6 @@ class SearchController < ApplicationController
)
end
def user_search
Search::User.search_documents(params: user_params.to_h)
end
def chat_channel_params
params.permit(CHAT_CHANNEL_PARAMS)
end

11
app/lib/acts_as_taggable_on/tag.rb
View file

@ -1,11 +0,0 @@
module ActsAsTaggableOn
class Tag
after_commit on: %i[create update] do
::Tag.find(id).index_to_elasticsearch
end
after_commit on: :update do
DataSync::Elasticsearch::Tag.new(self).call
end
end
end

1
app/lib/forem_stats_drivers/datadog_driver.rb
View file

@ -9,7 +9,6 @@ module ForemStatsDrivers
c.tracer enabled: ENV["DD_API_KEY"].present?
c.tracer partial_flush: true
c.tracer priority_sampling: true
c.use :elasticsearch
c.use :sidekiq
c.use :redis, service_name: "redis", describes: { url: ENV["REDIS_URL"] }
c.use :redis, service_name: "redis-sessions", describes: { url: ENV["REDIS_SESSIONS_URL"] }

9
app/models/article.rb
View file

@ -3,14 +3,9 @@ class Article < ApplicationRecord
include ActionView::Helpers
include Storext.model
include Reactable
include Searchable
include UserSubscriptionSourceable
include PgSearch::Model
SEARCH_SERIALIZER = Search::ArticleSerializer
SEARCH_CLASS = Search::FeedContent
DATA_SYNC_CLASS = DataSync::Elasticsearch::Article
acts_as_taggable_on :tags
resourcify
@ -112,9 +107,6 @@ class Article < ApplicationRecord
}
after_commit :async_score_calc, :touch_collection, on: %i[create update]
after_commit :index_to_elasticsearch, on: %i[create update]
after_commit :sync_related_elasticsearch_docs, on: %i[update]
after_commit :remove_from_elasticsearch, on: [:destroy]
# The trigger `update_reading_list_document` is used to keep the `articles.reading_list_document` column updated.
#
@ -361,7 +353,6 @@ class Article < ApplicationRecord
def touch_by_reaction
async_score_calc
index_to_elasticsearch
end
def comments_blob

7
app/models/chat_channel_membership.rb
View file

@ -1,10 +1,6 @@
class ChatChannelMembership < ApplicationRecord
attr_accessor :invitation_usernames
include Searchable
SEARCH_SERIALIZER = Search::ChatChannelMembershipSerializer
SEARCH_CLASS = Search::ChatChannelMembership
ROLES = %w[member mod].freeze
STATUSES = %w[active inactive pending rejected left_channel removed_from_channel joining_request].freeze
@ -18,9 +14,6 @@ class ChatChannelMembership < ApplicationRecord
validate :permission
after_commit :index_to_elasticsearch, on: %i[create update]
after_commit :remove_from_elasticsearch, on: [:destroy]
delegate :channel_type, to: :chat_channel
scope :eager_load_serialized_data, -> { includes(:user, :channel) }

13
app/models/comment.rb
View file

@ -4,10 +4,6 @@ class Comment < ApplicationRecord
include PgSearch::Model
include Reactable
include Searchable
SEARCH_SERIALIZER = Search::CommentSerializer
SEARCH_CLASS = Search::FeedContent
BODY_MARKDOWN_SIZE_RANGE = (1..25_000).freeze
@ -63,18 +59,9 @@ class Comment < ApplicationRecord
after_create_commit :send_to_moderator
after_commit :calculate_score, on: %i[create update]
after_commit :index_to_elasticsearch, on: %i[create update]
after_update_commit :update_notifications, if: proc { |comment| comment.saved_changes.include? "body_markdown" }
after_commit :remove_from_elasticsearch, on: [:destroy]
# [@atsmith813] this is adapted from the `search_field` property in
# `config/elasticsearch/mappings/feed_content.json` and
# `app/serializers/search/comment_serializer.rb`
#
# highlighter settings are taken from
# Search::QueryBuildersFeedContent#add_highlight_fields
pg_search_scope :search_comments,
against: %i[body_markdown],
using: {

32
app/models/concerns/searchable.rb
View file

@ -1,32 +0,0 @@
module Searchable
def search_id
id
end
def index_to_elasticsearch
Search::IndexWorker.perform_async(self.class.name, id)
end
def index_to_elasticsearch_inline
self.class::SEARCH_CLASS.index(search_id, serialized_search_hash)
end
def remove_from_elasticsearch
# Callbacks can cause index and removal jobs to be enqueued at the same time
# to avoid indexing a document after removing it we delay the removal job by 5 seconds to
# ensure it is run last
Search::RemoveFromIndexWorker.perform_in(5.seconds, self.class::SEARCH_CLASS.to_s, search_id)
end
def serialized_search_hash
self.class::SEARCH_SERIALIZER.new(self).serializable_hash.dig(:data, :attributes)
end
def elasticsearch_doc
self.class::SEARCH_CLASS.find_document(search_id)
end
def sync_related_elasticsearch_docs
self.class::DATA_SYNC_CLASS.new(self).call
end
end

8
app/models/listing.rb
View file

@ -3,12 +3,8 @@ class Listing < ApplicationRecord
# We standardized on the latter, but keeping the table name was easier.
self.table_name = "classified_listings"
include Searchable
include PgSearch::Model
SEARCH_SERIALIZER = Search::ListingSerializer
SEARCH_CLASS = Search::Listing
attr_accessor :action
# NOTE: categories were hardcoded at first and the model was only added later.
@ -19,8 +15,6 @@ class Listing < ApplicationRecord
before_validation :modify_inputs
before_save :evaluate_markdown
before_create :create_slug
after_commit :index_to_elasticsearch, on: %i[create update]
after_commit :remove_from_elasticsearch, on: [:destroy]
acts_as_taggable_on :tags
has_many :credits, as: :purchase, inverse_of: :purchase, dependent: :nullify
@ -33,8 +27,6 @@ class Listing < ApplicationRecord
validate :restrict_markdown_input
validate :validate_tags
# [@atsmith813] this is adapted from the `listing_search` property in
# `config/elasticsearch/mappings/listings.json`
pg_search_scope :search_listings,
against: %i[body_markdown cached_tag_list location slug title],
using: { tsearch: { prefix: true } }

1
app/models/message.rb
View file

@ -31,7 +31,6 @@ class Message < ApplicationRecord
def update_chat_channel_last_message_at
chat_channel.touch(:last_message_at)
ChatChannels::IndexesMembershipsWorker.perform_async(chat_channel.id)
end
def update_all_has_unopened_messages_statuses

5
app/models/organization.rb
View file

@ -30,7 +30,6 @@ class Organization < ApplicationRecord
# This callback will eventually invoke Article.update_cached_user to update the organization.name
# only when it has been changed, thus invoking the trigger on Article.reading_list_document
after_update_commit :update_articles_cached_organization
after_update_commit :sync_related_elasticsearch_docs
after_destroy_commit :bust_cache, :article_sync
has_many :articles, dependent: :nullify
@ -173,10 +172,6 @@ class Organization < ApplicationRecord
errors.add(:slug, "is taken.") if slug_taken
end
def sync_related_elasticsearch_docs
DataSync::Elasticsearch::Organization.new(self).call
end
def article_sync
# Syncs article cached organization and updates Elasticsearch docs
Article.where(id: cached_article_ids).find_each(&:save)

11
app/models/podcast_episode.rb
View file

@ -4,10 +4,6 @@ class PodcastEpisode < ApplicationRecord
]
include PgSearch::Model
include Searchable
SEARCH_SERIALIZER = Search::PodcastEpisodeSerializer
SEARCH_CLASS = Search::FeedContent
acts_as_taggable
@ -38,13 +34,6 @@ class PodcastEpisode < ApplicationRecord
after_destroy :purge, :purge_all
after_save :bust_cache
after_commit :index_to_elasticsearch, on: %i[update]
after_commit :remove_from_elasticsearch, on: [:destroy]
# [@atsmith813] this is adapted from the `search_field` property in
# `config/elasticsearch/mappings/feed_content.json` and
# `app/serializers/search/podcast_episode_serializer.rb` with a couple of
# extras
pg_search_scope :search_podcast_episodes,
against: %i[body subtitle title],
using: { tsearch: { prefix: true } }

11
app/models/tag.rb
View file

@ -1,5 +1,3 @@
require_relative "../lib/acts_as_taggable_on/tag"
class Tag < ActsAsTaggableOn::Tag
attr_accessor :points, :tag_moderator_id, :remove_moderator_id
@ -8,8 +6,6 @@ class Tag < ActsAsTaggableOn::Tag
# This model doesn't inherit from ApplicationRecord so this has to be included
include Purgeable
include Searchable
include PgSearch::Model
ALLOWED_CATEGORIES = %w[uncategorized language library tool site_mechanic location subcommunity].freeze
@ -39,9 +35,6 @@ class Tag < ActsAsTaggableOn::Tag
before_save :mark_as_updated
after_commit :bust_cache
after_commit :index_to_elasticsearch, on: %i[create update]
after_commit :sync_related_elasticsearch_docs, on: [:update]
after_commit :remove_from_elasticsearch, on: [:destroy]
pg_search_scope :search_by_name,
against: :name,
@ -49,10 +42,6 @@ class Tag < ActsAsTaggableOn::Tag
scope :eager_load_serialized_data, -> {}
SEARCH_SERIALIZER = Search::TagSerializer
SEARCH_CLASS = Search::Tag
DATA_SYNC_CLASS = DataSync::Elasticsearch::Tag
# possible social previews templates for articles with a particular tag
def self.social_preview_templates
Rails.root.join("app/views/social_previews/articles").children.map { |ch| File.basename(ch, ".html.erb") }

19
app/models/user.rb
View file

@ -1,8 +1,8 @@
class User < ApplicationRecord
resourcify
rolify
include CloudinaryHelper
include Searchable
include Storext.model
# @citizen428 Preparing to drop profile columns from the users table
@ -100,12 +100,6 @@ class User < ApplicationRecord
:add_credits, :remove_credits, :add_org_credits, :remove_org_credits, :ip_address,
:current_password
rolify after_add: :index_roles, after_remove: :index_roles
SEARCH_SERIALIZER = Search::UserSerializer
SEARCH_CLASS = Search::User
DATA_SYNC_CLASS = DataSync::Elasticsearch::User
acts_as_followable
acts_as_follower
@ -310,9 +304,6 @@ class User < ApplicationRecord
after_create_commit :send_welcome_notification
after_commit :bust_cache
after_commit :index_to_elasticsearch, on: %i[create update]
after_commit :sync_related_elasticsearch_docs, on: %i[create update]
after_commit :remove_from_elasticsearch, on: [:destroy]
def self.dev_account
find_by(id: Settings::Community.staff_user_id)
@ -676,14 +667,6 @@ class User < ApplicationRecord
follows.destroy_all
end
def index_roles(_role)
return unless persisted?
index_to_elasticsearch_inline
rescue StandardError => e
Honeybadger.notify(e, context: { user_id: id })
end
def can_send_confirmation_email
return if changes[:email].blank? || id.blank?

16
app/services/data_sync/elasticsearch/article.rb
View file

@ -1,16 +0,0 @@
module DataSync
module Elasticsearch
class Article < Base
RELATED_DOCS = %i[
comments
].freeze
SHARED_FIELDS = %i[
published
title
].freeze
delegate :comments, to: :@updated_record
end
end
end

33
app/services/data_sync/elasticsearch/base.rb
View file

@ -1,33 +0,0 @@
module DataSync
module Elasticsearch
class Base
attr_reader :updated_record
def initialize(updated_record)
@updated_record = updated_record
end
def call
return unless sync_needed?
sync_related_documents
end
private
def sync_needed?
updated_fields.slice(*self.class::SHARED_FIELDS).any?
end
def sync_related_documents
self.class::RELATED_DOCS.each do |relation_name|
__send__(relation_name).find_each(&:index_to_elasticsearch)
end
end
def updated_fields
updated_record.saved_changes
end
end
end
end

17
app/services/data_sync/elasticsearch/organization.rb
View file

@ -1,17 +0,0 @@
module DataSync
module Elasticsearch
class Organization < Base
RELATED_DOCS = %i[
articles
].freeze
SHARED_FIELDS = %i[
slug
name
profile_image
].freeze
delegate :articles, to: :@updated_record
end
end
end

30
app/services/data_sync/elasticsearch/tag.rb
View file

@ -1,30 +0,0 @@
module DataSync
module Elasticsearch
class Tag < Base
RELATED_DOCS = %i[
articles
podcast_episodes
].freeze
SHARED_FIELDS = %i[
keywords_for_search
].freeze
private
def articles
::Article.cached_tagged_with(updated_record.name)
end
def reactions
::Reaction.readinglist.where(reactable: articles)
end
def podcast_episodes
::PodcastEpisode.where(
id: updated_record.taggings.where(taggable_type: "PodcastEpisode").select(:taggable_id),
)
end
end
end
end

30
app/services/data_sync/elasticsearch/user.rb
View file

@ -1,30 +0,0 @@
module DataSync
module Elasticsearch
class User < Base
RELATED_DOCS = %i[
articles
podcast_episodes
chat_channel_memberships
comments
].freeze
SHARED_FIELDS = %i[
username
name
profile_image_url
].freeze
delegate :articles, :chat_channel_memberships, :comments, to: :@updated_record
private
def reactions
updated_record.reactions.readinglist
end
def podcast_episodes
PodcastEpisode.for_user(updated_record)
end
end
end
end

1
app/services/edge_cache/bust_commentable.rb
View file

@ -7,7 +7,6 @@ module EdgeCache
cache_bust = EdgeCache::Bust.new
cache_bust.call("#{commentable.path}/comments")
commentable.index_to_elasticsearch_inline
end
end
end

1
app/services/podcasts/create_episode.rb
View file

@ -69,7 +69,6 @@ module Podcasts
def finalize(episode)
episode.purge_all
episode.index_to_elasticsearch
episode.save if episode.processed_html.blank?
end
end

143
app/services/search/base.rb
View file

@ -1,143 +0,0 @@
module Search
class Base
class << self
def index(doc_id, serialized_data)
Search::Client.index(
id: doc_id,
index: self::INDEX_ALIAS,
body: serialized_data.merge(last_indexed_at: Time.current),
)
end
def bulk_index(data_hashes)
indexing_hashes = data_hashes.map do |data_hash|
model_hash = data_hash.with_indifferent_access
model_hash[:last_indexed_at] = Time.current
{
index: {
_index: self::INDEX_ALIAS,
_id: model_hash[:id],
data: model_hash
}
}
end
process_hashes(indexing_hashes)
end
def find_document(doc_id)
Search::Client.get(id: doc_id, index: self::INDEX_ALIAS)
end
def delete_document(doc_id)
Search::Client.delete(id: doc_id, index: self::INDEX_ALIAS)
end
def index_exists?(index_name: self::INDEX_NAME)
Search::Client.indices.exists(index: index_name)
end
def create_index(index_name: self::INDEX_NAME)
Search::Client.indices.create(index: index_name, body: settings)
end
def update_index(index_name: self::INDEX_NAME)
return unless dynamic_index_settings.any?
Search::Client.indices.put_settings(index: index_name, body: dynamic_settings)
end
def delete_index(index_name: self::INDEX_NAME)
Search::Client.indices.delete(index: index_name)
end
def refresh_index(index_name: self::INDEX_ALIAS)
Search::Client.indices.refresh(index: index_name)
end
def add_alias(index_name: self::INDEX_NAME, index_alias: self::INDEX_ALIAS)
Search::Client.indices.put_alias(index: index_name, name: index_alias)
end
def update_mappings(index_alias: self::INDEX_ALIAS)
Search::Client.indices.put_mapping(index: index_alias, body: self::MAPPINGS)
end
def document_count
Search::Client.count(index: self::INDEX_ALIAS)["count"]
end
def search_documents(params:)
set_query_size(params)
query_hash = "Search::QueryBuilders::#{name.demodulize}".safe_constantize.new(params: params).as_hash
results = search(body: query_hash)
hits = results.dig("hits", "hits").map { |hit| prepare_doc(hit) }
paginate_hits(hits, params)
end
private
def prepare_doc(hit)
hit["_source"]
end
def search(body:)
Search::Client.search(index: self::INDEX_ALIAS, body: body)
end
def set_query_size(params)
params[:page] ||= self::DEFAULT_PAGE
params[:per_page] ||= self::DEFAULT_PER_PAGE
# pages start at 0
params[:size] = params[:per_page].to_i * (params[:page].to_i + 1)
end
def paginate_hits(hits, params)
start = params[:per_page] * params[:page]
hits[start, params[:per_page]] || []
end
def settings
{ settings: { index: index_settings } }
end
def index_settings
raise "Search classes must implement their own index settings"
end
def dynamic_settings
{ settings: { index: dynamic_index_settings } }
end
def dynamic_index_settings
{}
end
def process_hashes(indexing_hashes)
indexing_hashes.in_groups_of(1000, false).flat_map do |hashes|
indexing_chunks(hashes).filter_map { |chunk| Search::Client.bulk(body: chunk) }
end
end
def indexing_chunks(hashes)
return [] unless hashes.any?
return to_enum(__method__, hashes) unless block_given?
size = 0
chunk = []
hashes.each do |hash|
chunk << hash
size += hash.to_s.bytesize
next unless size > 5.megabytes
yield chunk
size = 0
chunk = []
end
yield chunk
end
end
end
end

28
app/services/search/chat_channel_membership.rb
View file

@ -1,28 +0,0 @@
module Search
class ChatChannelMembership < Base
INDEX_NAME = "chat_channel_memberships_#{Rails.env}".freeze
INDEX_ALIAS = "chat_channel_memberships_#{Rails.env}_alias".freeze
MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/chat_channel_memberships.json"),
symbolize_names: true).freeze
DEFAULT_PAGE = 0
DEFAULT_PER_PAGE = 30
class << self
private
def index_settings
if Rails.env.production?
{
number_of_shards: 3,
number_of_replicas: 1
}
else
{
number_of_shards: 1,
number_of_replicas: 0
}
end
end
end
end
end

65
app/services/search/client.rb
View file

@ -1,65 +0,0 @@
module Search
# Search client (uses Elasticsearch as a backend)
class Client
class << self
# adapted from https://api.rubyonrails.org/classes/Module.html#method-i-delegate_missing_to
def method_missing(method, *args, &block)
return super unless target.respond_to?(method, false)
# define for re-use
self.class.define_method(method) do |*new_args, &new_block|
request do
target.public_send(method, *new_args, &new_block)
end
end
# call the original method, this will only be called the first time
# as in subsequent calls, the newly defined method will prevail
request do
target.public_send(method, *args, &block)
end
end
# adapted from https://api.rubyonrails.org/classes/Module.html#method-i-delegate_missing_to
def respond_to_missing?(method, _include_all = false)
target.respond_to?(method, false) || super
end
private
TRANSPORT_EXCEPTIONS = [
Elasticsearch::Transport::Transport::Errors::BadRequest,
Elasticsearch::Transport::Transport::Errors::NotFound,
].freeze
def request
yield
rescue *TRANSPORT_EXCEPTIONS => e
class_name = e.class.name.demodulize
record_error(e.message, class_name)
# raise specific error if known, generic one if unknown
error_class = "::Search::Errors::Transport::#{class_name}".safe_constantize
raise error_class, e.message if error_class
raise ::Search::Errors::TransportError, e.message
end
def record_error(error_message, class_name)
Honeycomb.add_field("elasticsearch.result", "error")
Honeycomb.add_field("elasticsearch.error", class_name)
ForemStatsClient.increment("elasticsearch.errors", tags: ["error:#{class_name}", "message:#{error_message}"])
end
def target
@target ||= Elasticsearch::Client.new(
url: ApplicationConfig["ELASTICSEARCH_URL"],
retry_on_failure: 5,
request_timeout: 30,
adapter: :patron,
log: Rails.env.development?,
)
end
end
end
end

72
app/services/search/cluster.rb
View file

@ -1,72 +0,0 @@
module Search
class Cluster
SEARCH_CLASSES = [
Search::ChatChannelMembership,
Search::Listing,
Search::FeedContent,
Search::Tag,
Search::User,
].freeze
class << self
def recreate_indexes
delete_indexes
setup_indexes
end
def setup_indexes
update_settings
create_indexes
update_indexes
add_aliases
update_mappings
end
def update_settings
Search::Client.cluster.put_settings(body: default_settings)
end
def create_indexes
SEARCH_CLASSES.each do |search_class|
next if search_class.index_exists?
search_class.create_index
end
end
def update_indexes
SEARCH_CLASSES.each(&:update_index)
end
def add_aliases
SEARCH_CLASSES.each(&:add_alias)
end
def update_mappings
SEARCH_CLASSES.each(&:update_mappings)
end
def delete_indexes
return if Rails.env.production?
SEARCH_CLASSES.each do |search_class|
next unless Search::Client.indices.exists(index: search_class::INDEX_NAME)
search_class.delete_index
end
end
private
def default_settings
{
persistent: {
action: {
auto_create_index: false
}
}
}
end
end
end
end

83
app/services/search/feed_content.rb
View file

@ -1,83 +0,0 @@
module Search
class FeedContent < Base
INDEX_NAME = "feed_content_#{Rails.env}".freeze
INDEX_ALIAS = "feed_content_#{Rails.env}_alias".freeze
MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/feed_content.json"), symbolize_names: true).freeze
DEFAULT_PAGE = 0
DEFAULT_PER_PAGE = 60
INCLUDED_CLASS_NAMES = %w[Article Comment PodcastEpisode].freeze
class << self
INCLUDED_CLASS_NAMES.each do |class_name|
define_method("#{class_name.underscore.pluralize}_document_count") do
Search::Client.count(
index: self::INDEX_ALIAS, body: count_filter(class_name),
)["count"]
end
end
private
def prepare_doc(hit)
source = hit["_source"]
source["tag_list"] = hit["tags"]&.map { |t| t["name"] } || []
source["id"] = source["id"].split("_").last.to_i
source["tag_list"] = source["tags"]&.map { |t| t["name"] } || []
source["flare_tag"] = source["flare_tag_hash"]
source["user_id"] = source.dig("user", "id")
source["highlight"] = hit["highlight"]
source["readable_publish_date"] = source["readable_publish_date_string"]
source["podcast"] = {
"slug" => source["slug"],
"image_url" => source["main_image"],
"title" => source["title"]
}
source["_score"] = hit["_score"]
source.merge(timestamps_hash(hit))
end
def timestamps_hash(hit)
published_at = hit.dig("_source", "published_at")
published_at_timestamp = Time.zone.parse(published_at || "")
{
"published_at_int" => published_at_timestamp.to_i,
"published_timestamp" => published_at
}
end
def count_filter(class_name)
{
query: {
bool: {
filter: { term: { class_name: class_name } }
}
}
}
end
def index_settings
if Rails.env.production?
{
number_of_shards: 10,
number_of_replicas: 1
}
else
{
number_of_shards: 1,
number_of_replicas: 0
}
end
end
def dynamic_index_settings
if Rails.env.production?
{ refresh_interval: "5s" }
else
{ refresh_interval: "1s" }
end
end
end
end
end

29
app/services/search/listing.rb
View file

@ -1,29 +0,0 @@
module Search
class Listing < Base
# We used to use both "classified listing" and "listing" throughout the app.
# We standardized on the latter in most places, but kept the index name here.
INDEX_NAME = "classified_listings_#{Rails.env}".freeze
INDEX_ALIAS = "classified_listings_#{Rails.env}_alias".freeze
MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/listings.json"), symbolize_names: true).freeze
DEFAULT_PAGE = 0
DEFAULT_PER_PAGE = 75
class << self
private
def index_settings
if Rails.env.production?
{
number_of_shards: 2,
number_of_replicas: 1
}
else
{
number_of_shards: 1,
number_of_replicas: 0
}
end
end
end
end
end

57
app/services/search/query_builders/chat_channel_membership.rb
View file

@ -1,57 +0,0 @@
module Search
module QueryBuilders
class ChatChannelMembership < QueryBase
FILTER_KEYS = %i[
channel_status
channel_type
status
viewable_by
].freeze
TERMS_FILTER_KEYS = %i[viewable_by status].freeze
QUERY_KEYS = %i[
channel_text
].freeze
DEFAULT_PARAMS = {
sort_by: "channel_last_message_at",
sort_direction: "desc",
size: 0
}.freeze
def initialize(params:)
super()
@params = params.deep_symbolize_keys
@params[:viewable_by] = @params[:user_id]
# TODO: @mstruve: When we want to allow people like admins to
# search ALL memberships this will need to change
@params[:status] = %w[active joining_request]
build_body
end
private
def build_queries
@body[:query] = {}
@body[:query][:bool] = { filter: filter_conditions }
@body[:query][:bool][:must] = query_conditions if query_keys_present?
end
def filter_conditions
FILTER_KEYS.filter_map do |filter_key|
next if @params[filter_key].blank? || @params[filter_key] == "all"
if TERMS_FILTER_KEYS.include?(filter_key)
{ terms: { filter_key => @params[filter_key] } }
else
{ term: { filter_key => @params[filter_key] } }
end
end
end
end
end
end

202
app/services/search/query_builders/feed_content.rb
View file

@ -1,202 +0,0 @@
module Search
module QueryBuilders
class FeedContent < QueryBase
# In order for highlighting to work properly we have to search the fields we want to highlight
QUERY_KEYS = {
search_fields: [
"tags.keywords_for_search",
"tags.name^3",
"body_text^2",
"title^6",
"user.name",
"user.username",
"organization.name",
]
}.freeze
# Search keys from our controllers may not match what we have stored in Elasticsearch so we map them here,
# this allows us to change our Elasticsearch docs without worrying about the frontend
TERM_KEYS = {
id: "id", # NOTE: FeedContent ids are formatted article_#, podcast_episode_#, comment_#
tag_names: "tags.name",
approved: "approved",
user_id: "user.id",
class_name: "class_name",
published: "published",
organization_id: "organization.id"
}.freeze
RANGE_KEYS = %i[
published_at
].freeze
DEFAULT_PARAMS = {
sort: [
:_score,
{ score: "desc" },
{ hotness_score: "desc" },
{ comments_count: "desc" },
],
size: 0
}.freeze
HIGHLIGHT_FIELDS = %w[
body_text
].freeze
SOURCE = %i[
id
class_name
cloudinary_video_url
comments_count
flare_tag_hash
main_image
path
public_reactions_count
published_at
readable_publish_date_string
reading_time
slug
tags
title
video_duration_in_minutes
video_duration_string
user
organization
].freeze
attr_accessor :params, :body
def initialize(params:)
super()
@params = params.deep_symbolize_keys
# Default to only showing published articles to start
@params[:published] = true
build_body
add_function_scoring unless sort_params_present?
end
private
def add_sort
return @body[:sort] = DEFAULT_PARAMS[:sort] unless sort_params_present?
@body[:sort] = { @params[:sort_by] => @params[:sort_direction] }
end
def add_highlight_fields
highlight_fields = { encoder: "html", pre_tags: "<mark>", post_tags: "</mark>", fields: {} }
HIGHLIGHT_FIELDS.each do |field_name|
# This hash can be filled with options to further customize our highlighting
# https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-highlighting
highlight_fields[:fields][field_name] = { order: :score, number_of_fragments: 2, fragment_size: 75 }
end
@body[:highlight] = highlight_fields
end
def filter_source
@body[:_source] = SOURCE
end
def build_queries
@body[:query] = { bool: {} }
@body[:query][:bool][:filter] = filter_conditions if filter_keys_present?
return unless query_keys_present?
@body[:query][:bool][:must] = query_conditions
# Boost the score of queries that match these conditions but if they dont match any,
# minimum_should_match: 0, then that is OK
@body[:query][:bool][:should] = match_phrase_conditions
@body[:query][:bool][:minimum_should_match] = 0
end
def filter_conditions
filter_conditions = []
filter_conditions.concat(term_keys) if terms_keys_present?
filter_conditions.concat(range_keys) if range_keys_present?
filter_conditions
end
def filter_keys_present?
range_keys_present? || terms_keys_present?
end
def terms_keys_present?
TERM_KEYS.detect { |key, _| @params[key].present? }
end
def range_keys_present?
RANGE_KEYS.detect { |key| @params[key].present? }
end
def term_keys
TERM_KEYS.filter_map do |term_key, search_key|
next unless @params.key? term_key
values = Array.wrap(@params[term_key])
if params[:tag_boolean_mode] == "all" && term_key == :tag_names
values.map { |val| { term: { search_key => val } } }
else
{ terms: { search_key => values } }
end
end.compact
end
def range_keys
RANGE_KEYS.filter_map do |range_key|
next unless @params.key? range_key
{ range: { range_key => @params[range_key] } }
end
end
def query_hash(key, fields)
{
simple_query_string: {
query: key.downcase,
fields: fields,
lenient: true,
analyze_wildcard: true,
minimum_should_match: 2
}
}
end
def add_function_scoring
@body[:query] = { function_score: { query: @body[:query] } }
@body[:query][:function_score][:functions] = scoring_functions
end
def scoring_functions
[
scoring_filter(1000, 2.5),
scoring_filter(500, 2),
scoring_filter(100, 1.5),
]
end
def scoring_filter(score, weight)
{
filter: {
range: {
score: {
gte: score
}
}
},
weight: weight
}
end
def sort_params_present?
@params[:sort_by] && @params[:sort_direction]
end
end
end
end

91
app/services/search/query_builders/listing.rb
View file

@ -1,91 +0,0 @@
module Search
module QueryBuilders
class Listing < QueryBase
TERM_KEYS = %i[
category
contact_via_connect
location
published
slug
tags
title
user_id
].freeze
RANGE_KEYS = %i[
bumped_at
expires_at
].freeze
QUERY_KEYS = %i[
listing_search
].freeze
DEFAULT_PARAMS = {
published: true,
sort_by: "bumped_at",
sort_direction: "desc",
size: 0
}.freeze
def initialize(params:)
super()
@params = params.deep_symbolize_keys
# For now, we're not allowing searches for Listings that are
# not published. If we want to change this in the future we can do:
# @params[:published] = DEFAULT_PARAMS[:published] unless @params.key?(:published)
@params[:published] = DEFAULT_PARAMS[:published]
build_body
end
private
def build_queries
@body[:query] = { bool: {} }
@body[:query][:bool][:filter] = filter_conditions
@body[:query][:bool][:must] = query_conditions if query_keys_present?
end
def filter_conditions
filter_conditions = []
# term_keys are always present because we are setting a filter of
# published: true by default
filter_conditions.concat term_keys
filter_conditions.concat range_keys if range_keys_present?
filter_conditions
end
def term_keys
TERM_KEYS.flat_map do |term_key|
next unless @params.key? term_key
values = Array.wrap(@params[term_key])
if params[:tag_boolean_mode] == "all" && term_key == :tags
values.map { |tag| { terms: { term_key => Array.wrap(tag) } } }
else
{ terms: { term_key => values } }
end
end.compact
end
def range_keys
RANGE_KEYS.filter_map do |range_key|
next unless @params.key? range_key
{ range: { range_key => @params[range_key] } }
end
end
def range_keys_present?
RANGE_KEYS.detect { |key| @params[key].present? }
end
end
end
end

80
app/services/search/query_builders/query_base.rb
View file

@ -1,80 +0,0 @@
module Search
module QueryBuilders
class QueryBase
attr_accessor :params, :body
def as_hash
@body
end
private
def build_body
@body = ActiveSupport::HashWithIndifferentAccess.new
build_queries
add_sort
set_size
add_highlight_fields
filter_source
end
def add_highlight_fields; end
def filter_source; end
def add_sort
sort_key = @params[:sort_by] || self.class::DEFAULT_PARAMS[:sort_by]
sort_direction = @params[:sort_direction] || self.class::DEFAULT_PARAMS[:sort_direction]
@body[:sort] = {
sort_key => sort_direction
}
end
def set_size
# By default we will return 0 documents if size is not specified
@body[:size] = @params[:size] || self.class::DEFAULT_PARAMS[:size]
end
def query_keys_present?
self.class::QUERY_KEYS.detect { |key, _| @params[key].present? }
end
def query_conditions
self.class::QUERY_KEYS.filter_map do |query_key, query_fields|
next if @params[query_key].blank?
fields = query_fields.presence || [query_key]
query_hash(@params[query_key], fields)
end
end
def match_phrase_conditions
self.class::QUERY_KEYS.filter_map do |query_key, _fields|
next if @params[query_key].blank?
match_phrase(@params[query_key], query_key)
end
end
def match_phrase(phrase, query_key)
{
match_phrase: {
query_key => { query: phrase, slop: 0 }
}
}
end
def query_hash(key, fields)
{
simple_query_string: {
query: "#{key}*",
fields: fields,
lenient: true,
analyze_wildcard: true
}
}
end
end
end
end

52
app/services/search/query_builders/user.rb
View file

@ -1,52 +0,0 @@
module Search
module QueryBuilders
class User < QueryBase
QUERY_KEYS = %i[
search_fields
].freeze
# In the event we want to search for documents that do NOT contain certain values
EXCLUDED_TERM_KEYS = {
exclude_roles: "roles"
}.freeze
DEFAULT_PARAMS = {
sort_by: "hotness_score",
sort_direction: "desc",
size: 0
}.freeze
def initialize(params:)
super()
@params = params.deep_symbolize_keys
# default to excluding users who are suspended
# TODO: [@jacobherrington] banned can be removed once the data scripts have succesfully run on all Forems
@params[:exclude_roles] = %w[suspended banned]
build_body
end
private
def build_queries
@body[:query] = { bool: {} }
@body[:query][:bool][:must] = query_conditions if query_keys_present?
@body[:query][:bool][:must_not] = excluded_term_keys if excluded_term_keys_present?
end
def excluded_term_keys_present?
self.class::EXCLUDED_TERM_KEYS.detect { |key, _| @params[key].present? }
end
def excluded_term_keys
EXCLUDED_TERM_KEYS.filter_map do |term_key, search_key|
next unless @params.key? term_key
{ terms: { search_key => Array.wrap(@params[term_key]) } }
end
end
end
end
end

78
app/services/search/reading_list.rb
View file

@ -1,78 +0,0 @@
module Search
# This class does not inherit from Search::Base like our other search classes bc it
# is using the FeedContent index to search for ReadingList reaction articles rather than its
# own separate index. The primary function of this class is to help combine and parse reaction
# data with article Elasticsearch data to populate the Reading List view.
class ReadingList
DEFAULT_PAGE = 0
DEFAULT_PER_PAGE = 60
DEFAULT_STATUS = %w[valid confirmed].freeze
# Using a class method here to follow the pattern of the other Search classes
def self.search_documents(params:, user:)
new(params: params, user: user).reading_list_reactions
end
def initialize(params:, user:)
self.status = params.delete(:status) || DEFAULT_STATUS
self.view_page = params.delete(:page) || DEFAULT_PAGE
self.view_per_page = params.delete(:per_page) || DEFAULT_PER_PAGE
self.user = user
self.search_params = params
end
def reading_list_reactions
ordered_articles = parse_and_order_articles(article_docs)
{ "reactions" => paginate_articles(ordered_articles), "total" => total }
end
private
attr_accessor :search_params, :user, :status, :view_page, :view_per_page, :total
def paginate_articles(ordered_articles)
start = view_per_page * view_page
ordered_articles[start, view_per_page] || []
end
def article_docs
return @article_docs if @article_docs
# Gather articles from Elasticsearch based on search criteria containing
# tags, text search, status, and the list of IDs of all articles in a user's
# reading list
docs = FeedContent.search_documents(
params: search_params.merge(
id: search_ids,
class_name: "Article",
page: 0,
per_page: reading_list_article_ids.count,
),
)
self.total = docs.count
@article_docs = docs.index_by { |doc| doc["id"] }
end
def reading_list_article_ids
# Collect all reading list IDs and article IDs for a user
@reading_list_article_ids ||= user.reactions.readinglist.where(status: status).order(id: :desc).pluck(
:reactable_id, :id
).to_h
end
def search_ids
reading_list_article_ids.keys.map { |id| "article_#{id}" }
end
def parse_and_order_articles(articles)
# Combines reaction and article data to create hashes that contain the fields
# the reading list view needs. Ensures articles are returned in order of reaction ID
reading_list_article_ids.filter_map do |article_id, reaction_id|
found_article_doc = articles[article_id]
next unless found_article_doc
{ "id" => reaction_id, "user_id" => user.id, "reactable" => articles[article_id] }
end
end
end
end

45
app/services/search/tag.rb
View file

@ -1,45 +0,0 @@
module Search
class Tag < Base
INDEX_NAME = "tags_#{Rails.env}".freeze
INDEX_ALIAS = "tags_#{Rails.env}_alias".freeze
MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/tags.json"), symbolize_names: true).freeze
class << self
def search_documents(query_string)
results = search(body: query(query_string))
results.dig("hits", "hits").map { |tag_doc| tag_doc["_source"] }
end
private
def query(query_string)
{
query: {
query_string: {
query: query_string,
analyze_wildcard: true,
allow_leading_wildcard: false
}
},
sort: {
hotness_score: "desc"
}
}
end
def index_settings
if Rails.env.production?
{
number_of_shards: 1,
number_of_replicas: 1
}
else
{
number_of_shards: 1,
number_of_replicas: 0
}
end
end
end
end
end

80
app/services/search/user.rb
View file

@ -1,80 +0,0 @@
module Search
class User < Base
INDEX_NAME = "users_#{Rails.env}".freeze
INDEX_ALIAS = "users_#{Rails.env}_alias".freeze
MAPPINGS = JSON.parse(File.read("config/elasticsearch/mappings/users.json"), symbolize_names: true).freeze
DEFAULT_PAGE = 0
DEFAULT_PER_PAGE = 20
class << self
def search_usernames(username)
results = search(body: username_query(username))
results.dig("hits", "hits").map do |doc|
source = doc["_source"]
{
"username" => source["username"],
"name" => source["name"],
"profile_image_90" => source["profile_image_90"]
}
end
end
private
def username_query(username)
{
query: {
query_string: {
query: "username:#{username}*",
analyze_wildcard: true,
allow_leading_wildcard: false
}
},
size: Search::Postgres::Username::MAX_RESULTS # Limit the number of results we return to the frontend
}
end
def prepare_doc(hit)
source = hit["_source"]
{
"user" => {
"username" => source["username"],
"name" => source["username"],
"profile_image_90" => source["profile_image_90"]
},
"title" => source["name"],
"path" => source["path"],
"id" => source["id"],
"class_name" => "User",
"public_reactions_count" => source["public_reactions_count"],
"comments_count" => source["comments_count"],
"badge_achievements_count" => source["badge_achievements_count"],
"last_comment_at" => source["last_comment_at"],
"user_id" => source["id"]
}
end
def index_settings
if Rails.env.production?
{
number_of_shards: 10,
number_of_replicas: 1
}
else
{
number_of_shards: 1,
number_of_replicas: 0
}
end
end
def dynamic_index_settings
if Rails.env.production?
{ refresh_interval: "10s" }
else
{ refresh_interval: "1s" }
end
end
end
end
end

2
app/services/users/delete_activity.rb
View file

@ -2,8 +2,6 @@ module Users
module DeleteActivity
module_function
# If you're removing data that is in Elasticsearch, make sure to use
# .destroy_all to trigger the callback to remove the document(s)
def call(user)
delete_social_media(user)
delete_profile_info(user)

2
app/services/users/delete_articles.rb
View file

@ -12,10 +12,8 @@ module Users
comment.reactions.delete_all
EdgeCache::BustComment.call(comment.commentable)
EdgeCache::BustUser.call(comment.user)
comment.remove_from_elasticsearch
comment.delete
end
article.remove_from_elasticsearch
article.delete
article.purge
end

1
app/services/users/delete_comments.rb
View file

@ -9,7 +9,6 @@ module Users
comment.reactions.delete_all
EdgeCache::BustComment.call(comment.commentable)
comment.remove_notifications
comment.remove_from_elasticsearch
comment.delete
end
EdgeCache::BustUser.call(user)

1
app/workers/articles/score_calc_worker.rb
View file

@ -9,7 +9,6 @@ module Articles
return unless article
article.update_score
article.index_to_elasticsearch_inline
end
end
end

12
app/workers/chat_channels/indexes_memberships_worker.rb
View file

@ -1,12 +0,0 @@
module ChatChannels
class IndexesMembershipsWorker
include Sidekiq::Worker
sidekiq_options queue: :high_priority, lock: :until_executing
def perform(chat_channel_id)
chat_channel = ChatChannel.find(chat_channel_id)
chat_channel.chat_channel_memberships.each(&:index_to_elasticsearch)
end
end
end

10
app/workers/metrics/record_data_counts_worker.rb
View file

@ -14,16 +14,6 @@ module Metrics
Rails.logger.info(message: "db_table_size", table_info: { table_name: model.table_name, table_size: db_count })
ForemStatsClient.gauge("postgres.db_table_size", db_count, tags: ["table_name:#{model.table_name}"])
next unless model.const_defined?(:SEARCH_CLASS)
document_count = if model::SEARCH_CLASS.respond_to?("#{model.to_s.underscore.pluralize}_document_count")
model::SEARCH_CLASS.public_send("#{model.to_s.underscore.pluralize}_document_count")
else
model::SEARCH_CLASS.document_count
end
ForemStatsClient.gauge("elasticsearch.document_count", document_count,
tags: ["table_name:#{model.table_name}"])
end
end
end

1
app/workers/rating_votes/assign_rating_worker.rb
View file

@ -16,7 +16,6 @@ module RatingVotes
experience_level_rating_distribution: ratings.max - ratings.min,
last_experience_level_rating_at: Time.current,
)
article.index_to_elasticsearch_inline
end
end
end

18
app/workers/search/bulk_index_worker.rb
View file

@ -1,18 +0,0 @@
module Search
class BulkIndexWorker
include Sidekiq::Worker
sidekiq_options queue: :high_priority, lock: :until_executing
def perform(object_class, ids)
data_hashes = object_class.constantize
.eager_load_serialized_data
.where(id: ids)
.find_each.map(&:serialized_search_hash)
search_class = object_class.constantize::SEARCH_CLASS
search_class.bulk_index(data_hashes)
end
end
end

12
app/workers/search/index_worker.rb
View file

@ -1,12 +0,0 @@
module Search
class IndexWorker
include Sidekiq::Worker
sidekiq_options queue: :high_priority, lock: :until_executing
def perform(object_class, id)
object = object_class.constantize.find_by(id: id)
object&.index_to_elasticsearch_inline
end
end
end

17
app/workers/search/remove_from_index_worker.rb
View file

@ -1,17 +0,0 @@
module Search
class RemoveFromIndexWorker
include Sidekiq::Worker
sidekiq_options queue: :medium_priority, lock: :until_executing
def perform(search_class, id)
search_class.safe_constantize.delete_document(id)
rescue Search::Errors::Transport::NotFound
# Often race conditions cause us never to index a document and
# we end up with an error when trying to remove it. Because
# we have count checks that run every hour to track the counts in
# Elasticsearch and db to ensure they match we can ignore this error
nil
end
end
end

3
bin/setup
View file

@ -30,9 +30,6 @@ FileUtils.chdir APP_ROOT do
FileUtils.cp "config/database.yml.sample", "config/database.yml"
end
puts "\n== Preparing Test Elasticsearch =="
system! 'RAILS_ENV="test" bin/rails search:setup'
puts "\n== Initializing the application =="
system! "bin/rails app_initializer:setup forem:setup"

61
config/elasticsearch/mappings/chat_channel_memberships.json
View file

@ -1,61 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"status": {
"type": "keyword"
},
"viewable_by": {
"type": "keyword"
},
"chat_channel_id": {
"type": "keyword"
},
"last_opened_at": {
"type": "date"
},
"channel_text": {
"type": "text"
},
"channel_last_message_at": {
"type": "date"
},
"channel_status": {
"type": "keyword"
},
"channel_status": {
"type": "keyword"
},
"channel_type": {
"type": "keyword"
},
"channel_username": {
"type": "keyword"
},
"channel_name": {
"type": "text"
},
"channel_image": {
"type": "keyword"
},
"channel_modified_slug": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"channel_messages_count": {
"type": "integer"
},
"channel_discoverable": {
"type": "boolean"
},
"last_indexed_at": {
"type": "date"
}
}
}

186
config/elasticsearch/mappings/feed_content.json
View file

@ -1,186 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"approved": {
"type": "boolean"
},
"body_text": {
"type": "text",
"copy_to": "search_fields"
},
"class_name": {
"type": "keyword"
},
"cloudinary_video_url": {
"type": "keyword"
},
"comments_count": {
"type": "integer"
},
"experience_level_rating": {
"type": "integer"
},
"experience_level_rating_distribution": {
"type": "integer"
},
"featured": {
"type": "boolean"
},
"featured_number": {
"type": "integer"
},
"flare_tag_hash": {
"dynamic": "strict",
"properties": {
"name": {
"type": "keyword"
},
"bg_color_hex": {
"type": "keyword"
},
"text_color_hex": {
"type": "keyword"
}
}
},
"hotness_score": {
"type": "integer"
},
"last_indexed_at": {
"type": "date"
},
"main_image": {
"type": "keyword"
},
"organization": {
"dynamic": "strict",
"properties": {
"slug": {
"type": "keyword"
},
"name": {
"type": "text",
"copy_to": "search_fields",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"id": {
"type": "keyword"
},
"profile_image_90": {
"type": "keyword"
}
}
},
"path": {
"type": "keyword"
},
// deprecated in favor of public_reactions_count, to be removed
"positive_reactions_count": {
"type": "integer"
},
"public_reactions_count": {
"type": "integer"
},
"published": {
"type": "boolean"
},
"published_at": {
"type": "date"
},
"quote": {
"type": "text"
},
"reactions_count": {
"type": "integer"
},
"readable_publish_date": {
"type": "date"
},
"readable_publish_date_string": {
"type": "keyword"
},
"reading_time": {
"type": "integer"
},
"score": {
"type": "integer"
},
"search_fields": {
"type": "text"
},
"search_score": {
"type": "integer"
},
"slug": {
"type": "keyword"
},
"subtitle": {
"type": "text"
},
"summary": {
"type": "text"
},
"tags": {
"dynamic": "strict",
"properties": {
"name": {
"type": "keyword",
"copy_to": "search_fields"
},
"keywords_for_search": {
"type": "keyword",
"copy_to": "search_fields"
}
}
},
"title": {
"type": "text",
"copy_to": "search_fields",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"user": {
"dynamic": "strict",
"properties": {
"username": {
"type": "keyword",
"copy_to": "search_fields"
},
"name": {
"type": "text",
"copy_to": "search_fields",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"id": {
"type": "keyword"
},
"profile_image_90": {
"type": "keyword"
}
}
},
"video_duration_in_minutes": {
"type": "integer"
},
"video_duration_string": {
"type": "keyword"
},
"website_url": {
"type": "keyword"
}
}
}

87
config/elasticsearch/mappings/listings.json
View file

@ -1,87 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"author": {
"dynamic": "strict",
"properties": {
"username": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"profile_image_90": {
"type": "keyword"
}
}
},
"body_markdown": {
"type": "text",
"copy_to": "listing_search"
},
"originally_published_at": {
"type": "date"
},
"bumped_at": {
"type": "date"
},
"category": {
"type": "keyword"
},
"listing_search": {
"type": "text"
},
"contact_via_connect": {
"type": "boolean"
},
"expires_at": {
"type": "date"
},
"last_indexed_at": {
"type": "date"
},
"location": {
"type": "text",
"copy_to": "listing_search",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"processed_html": {
"type": "keyword"
},
"published": {
"type": "boolean"
},
"slug": {
"type": "text",
"copy_to": "listing_search",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"tags": {
"type": "keyword",
"copy_to": "listing_search"
},
"title": {
"type": "text",
"copy_to": "listing_search",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"user_id": {
"type": "keyword"
}
}
}

96
config/elasticsearch/mappings/reactions.json
View file

@ -1,96 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"category": {
"type": "keyword"
},
"created_at": {
"type": "date"
},
"last_indexed_at": {
"type": "date"
},
"reactable": {
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"body_text": {
"type": "text",
"copy_to": "search_fields"
},
"class_name": {
"type": "keyword"
},
"path": {
"type": "keyword"
},
"published_date_string": {
"type": "keyword"
},
"reading_time": {
"type": "integer"
},
"tags": {
"dynamic": "strict",
"properties": {
"name": {
"type": "keyword",
"copy_to": "search_fields"
},
"keywords_for_search": {
"type": "keyword",
"copy_to": "search_fields"
}
}
},
"title": {
"type": "text",
"copy_to": "search_fields",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"user": {
"dynamic": "strict",
"properties": {
"username": {
"type": "keyword",
"copy_to": "search_fields"
},
"name": {
"type": "text",
"copy_to": "search_fields",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"id": {
"type": "keyword"
},
"profile_image_90": {
"type": "keyword"
}
}
}
}
},
"search_fields": {
"type": "text"
},
"status": {
"type": "keyword"
},
"user_id": {
"type": "keyword"
}
}
}

31
config/elasticsearch/mappings/tags.json
View file

@ -1,31 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"last_indexed_at": {
"type": "date"
},
"name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"hotness_score": {
"type": "integer"
},
"supported": {
"type": "boolean"
},
"short_summary": {
"type": "text"
},
"rules_html": {
"type": "text"
}
}
}

87
config/elasticsearch/mappings/users.json
View file

@ -1,87 +0,0 @@
{
"dynamic": "strict",
"properties": {
"id": {
"type": "keyword"
},
"available_for": {
"type": "keyword",
"copy_to": "search_fields"
},
"badge_achievements_count": {
"type": "integer"
},
"comments_count": {
"type": "integer"
},
"employer_name": {
"type": "keyword",
"copy_to": "search_fields"
},
"hotness_score": {
"type": "integer"
},
"last_comment_at": {
"type": "date"
},
"last_indexed_at": {
"type": "date"
},
"mostly_work_with": {
"type": "keyword",
"copy_to": "search_fields"
},
"name": {
"type": "text",
"copy_to": "search_fields"
},
"path": {
"type": "keyword"
},
// deprecated in favor of public_reactions_count, to be removed
"positive_reactions_count": {
"type": "integer"
},
"public_reactions_count": {
"type": "integer"
},
"profile_image_90": {
"type": "keyword"
},
"reactions_count": {
"type": "integer"
},
"roles": {
"type": "keyword"
},
"search_fields": {
"type": "text"
},
"username": {
"type": "keyword",
"copy_to": "search_fields"
},
"profile_fields": {
"type": "nested",
"properties": {
"name": {
"type": "keyword"
},
"value": {
"type": "keyword"
}
}
},
"custom_profile_fields": {
"type": "nested",
"properties": {
"name": {
"type": "keyword"
},
"value": {
"type": "keyword"
}
}
}
}
}

1
config/initializers/cypress_rails.rb
View file

@ -22,7 +22,6 @@ end
CypressRails.hooks.before_server_stop do
# Called once, at_exit
puts "Cleaning up and stopping server for end to end tests."
Rake::Task["search:destroy"].invoke
Rake::Task["db:truncate_all"].invoke
puts "The end to end test server shutdown gracefully."
end

2
config/routes.rb
View file

@ -89,7 +89,6 @@ Rails.application.routes.draw do
resources :health_checks, only: [] do
collection do
get :app
get :search
get :database
get :cache
end
@ -212,7 +211,6 @@ Rails.application.routes.draw do
get "/search/tags", to: "search#tags"
get "/search/chat_channels", to: "search#chat_channels"
get "/search/listings", to: "search#listings"
get "/search/users", to: "search#users"
get "/search/usernames", to: "search#usernames"
get "/search/feed_content", to: "search#feed_content"
get "/search/reactions", to: "search#reactions"

33
container-compose.yml
View file

@ -8,13 +8,11 @@ services:
depends_on:
- bundle
- db
- elasticsearch
- redis
- yarn
environment:
RAILS_ENV: development
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
@ -24,7 +22,7 @@ services:
APP_DOMAIN: rails
volumes:
- .:/opt/apps/forem:z
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s"]
command: [ "bash", "-c", "./scripts/entrypoint.sh bootstrap && bundle exec rails server -b 0.0.0.0 -p 3000"]
bundle:
@ -36,7 +34,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:z
command: ["./scripts/bundle.sh"]
@ -50,7 +47,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:z
command: [ "bash", "-c", "yarn install --dev"]
@ -67,7 +63,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:z
entrypoint: ["dockerize", "-wait", "file:///opt/apps/forem/node_modules/.bin/webpack-dev-server", "-timeout", "300s"]
@ -80,17 +75,15 @@ services:
- rails
- redis
- db
- elasticsearch
environment:
RAILS_ENV: development
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:z
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
command: ["bundle", "exec", "rake","db:seed"]
sidekiq:
@ -100,17 +93,15 @@ services:
- rails
- redis
- db
- elasticsearch
environment:
RAILS_ENV: development
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:z
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "300s"]
command: ["bundle", "exec", "sidekiq","-c","2"]
db:
@ -131,23 +122,5 @@ services:
ports:
- "6379:6379"
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: forem_elasticsearch
environment:
- cluster.name=forem
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.type=single-node"
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.graph.enabled=false
- xpack.watcher.enabled=false
ports:
- "9200:9200"
# volumes:
# - es_data:/usr/share/elasticsearch/data:Z
volumes:
db_data:
# es_data:

38
docker-compose.yml
View file

@ -9,13 +9,11 @@ services:
depends_on:
- bundle
- db
- elasticsearch
- redis
- yarn
environment:
RAILS_ENV: development
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
@ -25,7 +23,7 @@ services:
APP_DOMAIN: rails
volumes:
- .:/opt/apps/forem:delegated
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
command: [ "bash", "-c", "./scripts/entrypoint.sh bootstrap && bundle exec rails server -b 0.0.0.0 -p 3000"]
bundle:
@ -38,7 +36,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:delegated
command: ["./scripts/bundle.sh"]
@ -53,7 +50,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:delegated
command: yarn install --dev
@ -72,7 +68,6 @@ services:
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:delegated
entrypoint: ["dockerize", "-wait", "file:///opt/apps/forem/node_modules/.bin/webpack-dev-server", "-wait", "file:///opt/apps/forem/vendor/bundle/.bundle_finished", "-timeout", "2700s", "-wait-retry-interval", "10s"]
@ -86,17 +81,15 @@ services:
- rails
- redis
- db
- elasticsearch
environment:
RAILS_ENV: development
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:delegated
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
command: ["bundle", "exec", "rake","db:seed"]
sidekiq:
@ -107,17 +100,15 @@ services:
- rails
- redis
- db
- elasticsearch
environment:
RAILS_ENV: development
REDIS_SESSIONS_URL: redis://redis:6379
REDIS_SIDEKIQ_URL: redis://redis:6379
REDIS_URL: redis://redis:6379
DATABASE_URL: postgresql://forem:forem@db:5432/PracticalDeveloper_development
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
- .:/opt/apps/forem:delegated
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "http://elasticsearch:9200", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
entrypoint: ["dockerize", "-wait", "tcp://db:5432", "-wait", "tcp://redis:6379", "-wait", "http://rails:3000", "-timeout", "2700s", "-wait-retry-interval", "20s"]
command: ["bundle", "exec", "sidekiq","-c","2"]
db:
@ -138,28 +129,5 @@ services:
ports:
- "6379:6379"
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: forem_elasticsearch
environment:
- cluster.name=forem
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.type=single-node"
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.graph.enabled=false
- xpack.watcher.enabled=false
volumes:
- es_data:/usr/share/elasticsearch/data:delegated
ports:
- "9200:9200"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
db_data:
es_data:

4
docs/admin/admin-search.md
View file

@ -16,9 +16,9 @@ The view responsible for managing user reports can be found at
action of the [`admin/feedback_messages_controller`][feedback_messages].
For Forem, Ransack is being used exclusively in admin, for search problems in
other parts of the app we use [Elasticsearch][elasticsearch].
other parts of the app we use [PostgreSQL Full Text Search][postgres_fts].
[feedback_messages]:
https://github.com/forem/forem/blob/4e41e4a2ac893fa2a6c36990cfe475858ffb086a/app/controllers/admin/feedback_messages_controller.rb#L4
[ransack]: https://github.com/activerecord-hackery/ransack
[elasticsearch]: /backend/elasticsearch
[postgres_fts]: https://www.postgresql.org/docs/11/textsearch.html

4
docs/backend/data-update-scripts.md
View file

@ -8,9 +8,7 @@ Data Update Scripts were introduced in
[this PR](https://github.com/forem/forem/pull/6025) and allow us to run any data
updates we might need. For example, if we added a column to the database and
then wanted to backfill that column with data, rather than going and manually
doing it in a console, we would use a DataUpdateScript. Another example might be
adding a new attribute to Elasticsearch. We could then use a DataUpdateScript to
reindex all of our models.
doing it in a console, we would use a DataUpdateScript.
## How it works

92
docs/backend/elasticsearch.md
View file

@ -1,92 +0,0 @@
---
title: Elasticsearch
---
# What is [Elasticsearch](https://www.elastic.co/what-is/elasticsearch)?
> Elasticsearch is a distributed, open-source search and analytics engine for
> all types of data, including textual, numerical, geospatial, structured, and
> unstructured. Elasticsearch is built on Apache Lucene and was first released
> in 2010 by Elastic.
# How Forem uses Elasticsearch
### Searchable Models
At Forem we use Elasticsearch for all of our user-facing searching needs. Models
that are searched using Elasticsearch are:
- Tags
- Listings
- Chat Channels
- Users
- Podcast Episodes
- Comments
- Articles
### Index Setup
The above models are organized in Elasticsearch in different
[indexes](https://www.elastic.co/blog/what-is-an-elasticsearch-index) to make
searching more accurate and performant. The index breakdown is:
- Tags index -> Tags
- Listings index -> Listings
- Chat Channel Membership index -> Chat Channels
- User's index -> Users
- Feed Content index -> Podcast Episodes, Comments, and Articles
You will notice that Podcast Episodes, Comments, and Articles all share the same
index. This allows us to search and aggregate across all of these documents at
the same time. These documents also share a lot of similar fields so putting
them in the same index made sense. In the future, depending on our search needs
this index structure might change.
Each index that is listed above has a corresponding
[mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html)
that defines what fields can and cannot be indexed to it. These mappings are
similar to a db schema. In our case, all of our index mappings are defined as
[`strict`](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic)
which means if we try to index a field that is not defined it will throw an
error. Mappings for each index can be found in
`app/config/elasticsearch/mappings`.
### Indexing documents into Elasticsearch
In order to index the data from Postgres into Elasticsearch, we rely on
callbacks in our ActiveRecord models. Whenever a model is created or updated we
use an `after_commit` callback to enqueue a `Search::IndexWorker` which handles
indexing the document into Elasticsearch. In order to translate our ActiveRecord
data to Elasticsearch we use serializers. Each model has its own serializer
which can be found in `app/serializers/search`
### Searching Elasticsearch
Once the documents are in Elasticsearch then we can search for them. This is
handled by code in our `app/services/search` directory. Here you will see a
collection of classes used to help us index and search documents in
Elasticsearch. The nested `query_builders` directory contains all the logic we
need to help us build complex search queries to send to Elasticsearch.
# Working With Elasticsearch
## How to add a new field to Elasticsearch
When adding a new field to Elasticsearch here are the steps you have to follow:
1. Add field to mapping by editing the appropriate JSON file in
`app/config/elasticsearch/mappings`. If you plan to test locally you will
need to run `bin/setup` to update your local Elasticsearch mappings.
2. Add field to serializer for model located in `app/serializers/search`.
3. If the new field needs to be backfilled write a DataUpdateScript to reindex
all affected models so the field is available and populated in Elasticsearch
4. DEPLOY - Before you can start using a new field you need to ensure that it is
added to Elasticsearch and populated properly. The worker that runs the
DataUpdateScript will run 5 min after a deploy completes. This means you have
to deploy the code for adding the field separate from the code that will use
it.
5. Add code to search for your new field. You likely will need to update the
allowed params in the searches controller as well as the pertinent query
builder.
6. DEPLOY new search code
7. DONE!

1
docs/backend/readme.md
View file

@ -10,7 +10,6 @@ items:
- authorization.md
- configuration.md
- data-update-scripts.md
- elasticsearch.md
- emails.md
- fastly.md
- internationalization.md

6
docs/installation/containers.md
View file

@ -108,12 +108,6 @@ The script executes the following steps:
### Docker on Mac
- Should you experience problems with the Elasticsearch container, try to
increase the memory and/or swap allocation for Docker. On macOS this can be
done via the GUI:
![docker gui](https://user-images.githubusercontent.com/47985/74210448-b63b7c80-4c83-11ea-959b-02249b2a6952.png)
- In case `rails server` starts with the following message:
```shell

15
docs/installation/linux.md
View file

@ -66,21 +66,6 @@ We recommend following Digital Ocean's extensive guides
[How to Install and Secure Redis](https://www.digitalocean.com/community/tutorial_collections/how-to-install-and-secure-redis)
to setup Redis.
### Elasticsearch
Forem requires a version of Elasticsearch between 7.1 and 7.5. Version 7.6 is
not supported. We recommend version 7.5.2.
We recommend following
[Elasticsearch's guide for installing on Linux](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-linux).
Elasticsearch is also available as as
[Debian package](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/deb.html)
or a
[RPM package](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/rpm.html).
NOTE: Make sure to download **the OSS version**, `elasticsearch-oss`.
## Installing Forem
1. Fork Forem's repository, e.g. <https://github.com/forem/forem/fork>

164
docs/installation/mac.md
View file

@ -8,18 +8,25 @@ title: macOS
### Ruby
1. **Note:** MacOS ships with a version of Ruby, needed for various operating systems.
To avoid causing an issue with your operating system you should use a version manager for Ruby.
1. **Note:** MacOS ships with a version of Ruby, needed for various operating
systems. To avoid causing an issue with your operating system you should use
a version manager for Ruby.
If you don't already have a Ruby version manager, we highly recommend
[rbenv](https://github.com/rbenv/rbenv). This will allow you to have
different versions running on a per project basis. The MacOS system version
of Ruby will stay intact while giving you the ability to use the version
needed for this Forem project. Please follow their
[installation guide](https://github.com/rbenv/rbenv#installation).
If you don't already have a Ruby version manager, we highly recommend [rbenv](https://github.com/rbenv/rbenv). This will allow you to have different versions running on a per project basis. The MacOS system version of Ruby will stay intact while giving you the ability to use the version needed for this Forem project.
Please follow their [installation guide](https://github.com/rbenv/rbenv#installation).
2. With the Ruby version manager, install the Ruby version listed on our badge.
(i.e. with rbenv: `rbenv install $(cat .ruby-version)`)
**Note:** The repository must be forked and cloned before running the `rbenv install $(cat .ruby-version)` command.
**Note:** The repository must be forked and cloned before running the
`rbenv install $(cat .ruby-version)` command.
### Yarn
Please refer to their [installation guide](https://yarnpkg.com/en/docs/install).
### PostgreSQL
@ -63,146 +70,6 @@ You can test if it's up and running by issuing the following command:
redis-cli ping
```
### Elasticsearch
Forem requires Elasticsearch 7.x to run. We recommend version 7.5.2.
You have the option of installing Elasticsearch with Homebrew or through an
archive. We **recommend** installing from archive on Mac.
### Installing Elasticsearch from the archive
We recommend that you **do not** install Elasticsearch in the app directory.
Instead, we recommend installing it in your home directory (for example,
`cd $HOME`). (This also ensures that we don't accidentally commit Elasticsearch
code to the project's repository!)
The following directions were
[taken from the Elasticsearch docs themselves](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-macos),
so check those out if you run into any issues or want further information. Make
sure to download **the OSS version** of Elasticsearch, `elasticsearch-oss`.
Please note that you will need `wget` in order to proceed with this installation
(`brew install wget`).
```shell
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz.sha512
tar -xzf elasticsearch-oss-7.5.2-darwin-x86_64.tar.gz
```
To start elasticsearch, make sure you are in the correct directory:
```shell
cd elasticsearch-7.5.2
```
You can then start it by running:
```shell
./bin/elasticsearch
```
To start elasticsearch as a daemonized process:
```shell
./bin/elasticsearch -d
```
### Installing Elasticsearch with Homebrew
To install Elasticsearch with Homebrew we will use the following commands to:
- tap the Elastic Homebrew repository
- install the latest OSS distribution
- pin the latest OSS distribution.
```shell
brew tap elastic/tap
brew install elastic/tap/elasticsearch-oss
brew pin elasticsearch-oss
```
After installation you can manually test if the Elasticsearch server starts by
issuing the command `elasticsearch` in the shell. You can then start the server
as a service with `brew services start elasticsearch-oss`.
You can find further info on your local Elasticsearch installation by typing
`brew info elastic/tap/elasticsearch-oss`.
#### Troubleshooting startup issues
Two possible startup issues you might encounter:
- `java.nio.file.FileSystemLoopException`:
```text
Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.FileSystemLoopException: /usr/local/etc/elasticsearch/elasticsearch
Likely root cause: java.nio.file.FileSystemLoopException: /usr/local/etc/elasticsearch/elasticsearch
```
This happens because the installation of Elasticsearch might have a recursive
link in the configuration directory causing the infinite loop:
```shell
> ll /usr/local/etc/elasticsearch
elasticsearch -> /usr/local/etc/elasticsearch
```
By manually removing the link with
`rm -i /usr/local/etc/elasticsearch/elasticsearch` the issue should be fixed.
- `java.lang.IllegalStateException`:
```text
java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [plugins]
Likely root cause: java.nio.file.NoSuchFileException: /usr/local/Cellar/elasticsearch-oss/7.6.0/libexec/plugins/plugins/plugin-descriptor.properties
```
This happens for a similar reason as the previous error, the installation might
create a recursive link in the plugins directory.
```shell
> ll /usr/local/var/elasticsearch/plugins
plugins -> /usr/local/var/elasticsearch/plugins
```
By manually removing the link with
`rm -i /usr/local/var/elasticsearch/plugins/plugins` the issue should be fixed.
### Testing if Elasticsearch is running
Once installed and started you can test if it's up and running correctly by
issuing the following command:
```shell
curl http://localhost:9200
```
You should receive in response a JSON document containing some information about
your local Elasticsearch installation, for example:
```json
{
"name": "hostname",
"cluster_name": "elasticsearch_...",
"cluster_uuid": "...",
"version": {
"number": "7.5.2",
"build_flavor": "oss",
"build_type": "tar",
"build_hash": "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
"build_date": "2020-01-15T12:11:52.313576Z",
"build_snapshot": false,
"lucene_version": "8.3.0",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}
```
## Installing Forem
1. Fork Forem's repository, e.g. <https://github.com/forem/forem/fork>
@ -240,8 +107,7 @@ your local Elasticsearch installation, for example:
### Possible error messages
**Error:**
`rbenv install hangs at ruby-build: using readline from homebrew`
**Error:** `rbenv install hangs at ruby-build: using readline from homebrew`
**_Solution:_**
[Stackoverflow answer](https://stackoverflow.com/questions/63599818/rbenv-install-hangs-at-ruby-build-using-readline-from-homebrew)

62
docs/installation/windows.md
View file

@ -176,60 +176,6 @@ We recommend to follow
[this guide](https://redislabs.com/blog/redis-on-windows-10/) to run Redis under
WSL.
### Elasticsearch
Forem requires a version of Elasticsearch between 7.1 and 7.5. Version 7.6 is
not supported. We recommend version 7.5.2.
We recommend that you **do not** install Elasticsearch in the app directory.
Instead, we recommend installing it in your home directory (for example,
`cd $HOME`). (This also ensures that we don't accidentally commit Elasticsearch
code to the project's repository!)
The following directions were taken from
[the Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#install-linux),
check them out to learn more about the installation process and troubleshooting
issues. Make sure to refer to **the OSS version**, `elasticsearch-oss` while
going through the Elasticsearch docs.
To install Elasticsearch perform the following steps:
1. Execute the following commands:
```shell
cd
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.2-linux-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-oss-7.5.2-linux-x86_64.tar.gz.sha512
tar -xzf elasticsearch-oss-7.5.2-linux-x86_64.tar.gz
```
2. Next, switch to the correct directory with:
```shell
cd elasticsearch-7.5.2/
```
3. To start Elasticsearch, run the following command:
```shell
./bin/elasticsearch
```
or, start it as a daemonized process with:
```shell
./bin/elasticsearch -d
```
4. Once Elasticsearch is running,
[verify Elasticsearch's installation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/targz.html#_checking_that_elasticsearch_is_running)
by executing the `cURL` command as follows:
```shell
curl -X GET "localhost:9200/?pretty"
```
## Installing Forem
1. Fork Forem's repository, eg. <https://github.com/forem/forem/fork>
@ -277,8 +223,8 @@ To install Elasticsearch perform the following steps:
- You do not need "real" keys for basic development. Some features require
certain keys, so you may be able to add them as you go.
1. After ensuring that Elasticsearch, the PostgreSQL server, and the Redis
server are running, run `bin/setup`.
1. After ensuring that the PostgreSQL server and the Redis server are running,
run `bin/setup`.
> The `bin/setup` script is responsible for installing a varienty of
> dependencies. One can find it inside the `bin` folder by the name of
@ -290,8 +236,8 @@ To install Elasticsearch perform the following steps:
> - It then installs JavaScript dependencies using the script in `bin/yarn`
> file. These dependencies are located in `package.json` in the root of the
> repository.
> - Next, it uses various Rake files located inside the `lib` folder to setup
> ElasticSearch environment, PostgreSQL database creation and updation.
> - Next, it uses various Rake files located inside the `lib` folder for the
> PostgreSQL database creation and updation.
> - Finally it cleans up all the log files and restarts the Puma server.
### Possible error messages

17
docs/maintainers/deploying.md
View file

@ -57,10 +57,10 @@ stage will do nothing. If the branch has been merged into main, then this stage
will kick off a deploy. The deploy will run in its own job deploying our
application to Heroku.
Prior to deploying the code, Heroku will run database migrations, Elasticsearch
updates, and do some final checks (more information on that below) to make sure
everything is working as expected. If these all succeed, then the deploy
completes and our team is notified.
Prior to deploying the code, Heroku will run database migrations and do some
final checks (more information on that below) to make sure everything is working
as expected. If these all succeed, then the deploy completes and our team is
notified.
## Deploying to Heroku
@ -84,13 +84,8 @@ directory. During this release step we do a few checks.
successfully before the code that uses it goes live.
3. We run any data update scripts that need to be run. A data update script is
one that allows us to update data in the background separate from a
migration. For example, if we add a new field to Elasticsearch and need to
reindex all of our documents we would use a data update script.
4. We update Elasticsearch. Elasticsearch contains indexes which have mappings.
Mappings are similar to database schema. The same way we run a migration to
update our database we have to run a setup task to update any Elasticsearch
mappings.
5. Following updating all of our datastores we use the Rails runner to output a
migration.
4. Following updating all of our datastores we use the Rails runner to output a
simple string. Executing a Rails runner command ensures that we can boot up
the entire app successfully before it is deployed. We deploy asynchronously,
so the website is running the new code a few minutes after deploy. A new

8
docs/technical-overview/architecture.md
View file

@ -72,10 +72,10 @@ Rails][fastly_rails] docs, but we bust specific URLs via `EdgeCache::Bust#call`.
The home feed is based on a combination of recent collective posts that are
cached and delivered the same to everyone in the HTML, and additional articles
fetched from an Elasticsearch index after page load. To determine which posts a
user sees, they are ranked based on the user's followed tags, followed users,
and relative weights for each tag. Additional fetched articles also follow this
general pattern.
fetched from a PostgreSQL Full Text Search index after page load. To determine
which posts a user sees, they are ranked based on the user's followed tags,
followed users, and relative weights for each tag. Additional fetched articles
also follow this general pattern.
Currently, the top post on the home feed, which must have a cover image, is
shared among all users.

4
docs/technical-overview/stack.md
View file

@ -7,7 +7,8 @@ title: Stack
For the Forem tech stack we use:
- [_Puma_](https://github.com/puma/puma) as the web server
- [_PostgreSQL_](https://www.postgresql.org/) as the primary database
- [_PostgreSQL_](https://www.postgresql.org/) as the primary database and for
Full Text Search
- [_Redis_](https://redis.io/) to store cached data
- [_Fastly_](https://www.fastly.com/) for
[edge caching](https://dev.to/ben/making-devto-insanely-fast)
@ -18,7 +19,6 @@ For the Forem tech stack we use:
- [_Sidekiq_](https://github.com/mperham/sidekiq) and
[_Active Job_](https://guides.rubyonrails.org/active_job_basics.html) for
background workers
- [_Elasticsearch_](https://www.elastic.co/elasticsearch/) for in app searching
- [Ransack](https://github.com/activerecord-hackery/ransack) for internal search
- [_Redcarpet_](https://github.com/vmg/redcarpet) and
[_Rouge_](https://github.com/jneen/rouge) to parse Markdown

5
docs/tests/readme.md
View file

@ -69,8 +69,3 @@ retry the individual job that failed rather than the entire suite in order to
save some time. When you retry the individual job, make sure to also retry the
Deploy job. Even though you may not be deploying that job must complete for the
entire build to pass.
Please note that you will need to have Elasticsearch installed and running for
certain tests in our test suite. You can find instructions on how to install and
run Elasticsearch specific to your environment in the
[Installation Guide](/installation).

46
docs/tests/test-flags.md
View file

@ -11,59 +11,15 @@ test example.
Some flags that we use are:
- `js: true`
- `elasticsearch_reset: true`
- `elasticsearch: <search class name>`
- `stub_elasticsearch: true`
- `throttle: true`
- `type: <test type>`
### Elasticsearch Flags
Two Elasticsearch flags that can be used when dealing with specs that will be
hitting Elasticsearch. When running a spec that is going to interact with
Elasticsearch, you want Elasticsearch to be clean like your database. Since we
don't have a "database cleaner" for Elasticsearch, we have to do it manually.
There are two ways to do this:
1. `elasticsearch: reset` - This will trigger a complete tear down and rebuild
of Elasticsearch via a block. This takes time so it's not something you want
to be doing unless you absolutely have to.
```ruby
config.around(:each, elasticsearch_reset: true) do |example|
Search::Cluster.recreate_indexes
example.run
Search::Cluster.recreate_indexes
end
```
2. `elasticsearch: <search class name>` - This will clear the data for the given
search class index. For example, if you passed "User" as your search class
name it would clear out the user index data using code from this block.
```ruby
config.around(:each, :elasticsearch) do |ex|
klasses = Array.wrap(ex.metadata[:elasticsearch]).map do |search_class|
Search.const_get(search_class)
end
klasses.each { |klass| clear_elasticsearch_data(klass) }
ex.run
end
```
### `js: true` Flag
`js: true` indicates that we want the JavaScript on the page to be executed when
the page is rendered, and a headless chrome instance will be initialized to do
so (instead of the default
[rack_test](https://github.com/teamcapybara/capybara#racktest) driver). One side
effect of running our JavaScript in our specs is that a lot of pages will hit
Elasticsearch. Since we don't clean out Elasticsearch between every single spec
(because it is very costly) this can lead to unexpected data being loaded for a
spec. To prevent this from happening, we can use the `:stub_elasticsearch` flag.
The `:stub_elasticsearch` flag will stub all index and search requests made to
Elasticsearch and return an empty response. This will ensure that no unwanted
data shows up on your spec's page.
[rack_test](https://github.com/teamcapybara/capybara#racktest) driver).
If you are debugging a `js: true` spec and want to see the browser, you can set
`HEADLESS=false` before running a spec:

60
docs/troubleshooting.md
View file

@ -33,66 +33,6 @@ The first command creates the user **ec2-user** and the second one creates the
database for this user because every user needs its database. Even if the first
command fails, run the second command to create the missing database.
## Elasticsearch
### Index read-only error
If you encounter an error similar to the following:
```shell
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [tags_development] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"index [tags_development] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}
```
it means that Elasticsearch went into read only mode because its disk allocator
noticed how the disk drive is nearly full:
> Elasticsearch enforces a read-only index block
> (`index.blocks.read_only_allow_delete`) on every index that has one or more
> shards allocated on the node that has at least one disk exceeding the flood
> stage.
This is an indication that you might not have enough space for ES to work
correctly.
If you want to disable the threshold check on your local machine you can open a
Rails console (with `rails console`) and issue the following command:
```ruby
Search::Client.cluster.put_settings(body: {
persistent: {
"cluster.routing.allocation.disk.threshold_enabled" => false,
}
})
```
Alternatively you can use the following command in a terminal session:
```shell
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
```
To disable the "read only" mode to allow operations on the Elasticsearch indexes
you can issue the following command, similary in the Rails console:
```ruby
Search::Client.indices.get(index: "*").keys.each do |index_name|
Search::Client.indices.put_settings(
index: index_name,
body: { "index.blocks.read_only_allow_delete" => nil }
)
end
```
Or alternatively:
```shell
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
```
If instead you want to tune the Elasticsearch disk allocator's settings, please
refer to
[Disk-based shard allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html#disk-allocator).
## CORS
If you are experiencing CORS issues locally or need to display more information

2
lib/data_update_scripts/20200218195023_index_chat_channel_memberships_to_elasticsearch.rb
View file

@ -1,7 +1,7 @@
module DataUpdateScripts
class IndexChatChannelMembershipsToElasticsearch
def run
ChatChannelMembership.find_each(&:index_to_elasticsearch_inline)
# ChatChannelMembership.find_each(&:index_to_elasticsearch_inline)
end
end
end

10
lib/data_update_scripts/20200410152018_resync_elasticsearch_documents.rb
View file

@ -2,13 +2,13 @@ module DataUpdateScripts
class ResyncElasticsearchDocuments
def run
# Previous bug: Not getting removed properly
sync_docs(Article.ids, "Article")
sync_docs(Comment.ids, "Comment")
sync_docs(User.ids, "User")
# sync_docs(Article.ids, "Article")
# sync_docs(Comment.ids, "Comment")
# sync_docs(User.ids, "User")
# Previous bug: Not getting indexed on creation properly
index_docs(PodcastEpisode.ids, "PodcastEpisode")
index_docs(Tag.ids, "Tag")
# index_docs(PodcastEpisode.ids, "PodcastEpisode")
# index_docs(Tag.ids, "Tag")
end
private

30
lib/data_update_scripts/20200519142908_re_index_feed_content_and_users_to_elasticsearch.rb
View file

@ -1,21 +1,21 @@
module DataUpdateScripts
class ReIndexFeedContentAndUsersToElasticsearch
def run
Article.select(:id).in_batches(of: 100) do |batch|
Search::BulkIndexWorker.set(queue: :default).perform_async(
"Article", batch.ids
)
end
Comment.select(:id).in_batches(of: 100) do |batch|
Search::BulkIndexWorker.set(queue: :default).perform_async(
"Comment", batch.ids
)
end
PodcastEpisode.select(:id).in_batches(of: 100) do |batch|
Search::BulkIndexWorker.set(queue: :default).perform_async(
"PodcastEpisode", batch.ids
)
end
# Article.select(:id).in_batches(of: 100) do |batch|
# Search::BulkIndexWorker.set(queue: :default).perform_async(
# "Article", batch.ids
# )
# end
# Comment.select(:id).in_batches(of: 100) do |batch|
# Search::BulkIndexWorker.set(queue: :default).perform_async(
# "Comment", batch.ids
# )
# end
# PodcastEpisode.select(:id).in_batches(of: 100) do |batch|
# Search::BulkIndexWorker.set(queue: :default).perform_async(
# "PodcastEpisode", batch.ids
# )
# end
# See: https://github.com/forem/forem/pull/10313#discussion_r487646864
# User.select(:id).in_batches(of: 200) do |batch|

2
lib/data_update_scripts/20200803142830_reindex_listing_search_column.rb
View file

@ -3,7 +3,7 @@ module DataUpdateScripts
def run
# Choose to do inline so development envs are ready immediately after
# this is run
Listing.find_each(&:index_to_elasticsearch_inline)
# Listing.find_each(&:index_to_elasticsearch_inline)
end
end
end

126
lib/data_update_scripts/20200904132553_remove_draft_articles_with_duplicate_feed_source_url.rb
View file

@ -5,76 +5,76 @@ module DataUpdateScripts
# This statement deletes all draft articles in excess found to be duplicate over feed_source_url,
# excluding those whose body_markdown is different from the other duplicate occurrences
result_same_body = ActiveRecord::Base.connection.execute(
<<-SQL,
WITH duplicates_draft_articles AS
(SELECT id
FROM
(SELECT id,
published,
body_markdown,
LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
ORDER BY id ASC) AS previous_body_markdown,
ROW_NUMBER() OVER(PARTITION BY feed_source_url
ORDER BY id ASC) AS row_number
FROM articles
WHERE feed_source_url IS NOT NULL ) duplicates
WHERE duplicates.row_number > 1
AND published = 'f' -- drafts
AND body_markdown = previous_body_markdown -- with the same body
)
DELETE
FROM articles
WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
SQL
)
# result_same_body = ActiveRecord::Base.connection.execute(
# <<-SQL,
# WITH duplicates_draft_articles AS
# (SELECT id
# FROM
# (SELECT id,
# published,
# body_markdown,
# LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
# ORDER BY id ASC) AS previous_body_markdown,
# ROW_NUMBER() OVER(PARTITION BY feed_source_url
# ORDER BY id ASC) AS row_number
# FROM articles
# WHERE feed_source_url IS NOT NULL ) duplicates
# WHERE duplicates.row_number > 1
# AND published = 'f' -- drafts
# AND body_markdown = previous_body_markdown -- with the same body
# )
# DELETE
# FROM articles
# WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
# SQL
# )
#
# Now that all duplicates with the same body are gone, we need to deal with duplicate feed source URLs
# with different bodies.
# We thus select the oldest for removal preserving the most recent one
result_different_bodies = ActiveRecord::Base.connection.execute(
<<-SQL,
WITH duplicates_draft_articles AS
(SELECT id
FROM
(SELECT id,
published,
body_markdown,
LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
ORDER BY created_at DESC) AS previous_body_markdown,
ROW_NUMBER() OVER(PARTITION BY feed_source_url
ORDER BY created_at DESC) AS row_number
FROM articles
WHERE feed_source_url IS NOT NULL ) duplicates
WHERE duplicates.row_number > 1
AND published = 'f' -- drafts
AND body_markdown != previous_body_markdown -- with different bodies
)
DELETE
FROM articles
WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
SQL
)
result_same_body_ids = result_same_body.map { |row| row["id"] }
result_different_bodies_ids = result_different_bodies.map { |row| row["id"] }
# result_different_bodies = ActiveRecord::Base.connection.execute(
# <<-SQL,
# WITH duplicates_draft_articles AS
# (SELECT id
# FROM
# (SELECT id,
# published,
# body_markdown,
# LAG(body_markdown, 1) OVER(PARTITION BY feed_source_url
# ORDER BY created_at DESC) AS previous_body_markdown,
# ROW_NUMBER() OVER(PARTITION BY feed_source_url
# ORDER BY created_at DESC) AS row_number
# FROM articles
# WHERE feed_source_url IS NOT NULL ) duplicates
# WHERE duplicates.row_number > 1
# AND published = 'f' -- drafts
# AND body_markdown != previous_body_markdown -- with different bodies
# )
# DELETE
# FROM articles
# WHERE id IN (SELECT id FROM duplicates_draft_articles) RETURNING id;
# SQL
# )
#
# result_same_body_ids = result_same_body.map { |row| row["id"] }
# result_different_bodies_ids = result_different_bodies.map { |row| row["id"] }
# Remove all articles from Elasticsearch
(result_same_body_ids + result_different_bodies_ids).each do |id|
Search::RemoveFromIndexWorker.perform_in(5.seconds, "Search::FeedContent", "article_#{id}")
end
# (result_same_body_ids + result_different_bodies_ids).each do |id|
# Search::RemoveFromIndexWorker.perform_in(5.seconds, "Search::FeedContent", "article_#{id}")
# end
# Store deleted IDs temporarily in Redis for safe keeping
Rails.cache.write(
"DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::SameBody",
result_same_body_ids,
expires_in: 2.weeks,
)
Rails.cache.write(
"DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::DifferentBodies",
result_same_body_ids,
expires_in: 2.weeks,
)
# Rails.cache.write(
# "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::SameBody",
# result_same_body_ids,
# expires_in: 2.weeks,
# )
# Rails.cache.write(
# "DataUpdateScripts::RemoveDraftArticlesWithDuplicateFeedSourceUrl::DifferentBodies",
# result_same_body_ids,
# expires_in: 2.weeks,
# )
end
end
end

2
lib/data_update_scripts/20200910140109_cleanup_published_articles_with_duplicate_user_id_title_body_markdown.rb
View file

@ -34,7 +34,7 @@ module DataUpdateScripts
# save all the magic
article.save
article.index_to_elasticsearch_inline
# article.index_to_elasticsearch_inline
articles_to_delete_ids += articles_to_graft.map { |a| a["id"] }
end

4
lib/data_update_scripts/20200911045602_reindex_articles_with_videos.rb
View file

@ -1,8 +1,8 @@
module DataUpdateScripts
class ReindexArticlesWithVideos
def run
articles = Article.where.not(video: nil).or(Article.where.not(video: ""))
articles.find_each(&:index_to_elasticsearch_inline)
# articles = Article.where.not(video: nil).or(Article.where.not(video: ""))
# articles.find_each(&:index_to_elasticsearch_inline)
end
end
end

6
lib/data_update_scripts/20200924140813_remove_reaction_index_by_name.rb
View file

@ -1,10 +1,10 @@
module DataUpdateScripts
class RemoveReactionIndexByName
def run
index_name = "reactions_#{Rails.env}"
return unless Search::Client.indices.exists(index: index_name)
# index_name = "reactions_#{Rails.env}"
# return unless Search::Client.indices.exists(index: index_name)
Search::Client.indices.delete(index: index_name)
# Search::Client.indices.delete(index: index_name)
end
end
end

6
lib/data_update_scripts/20201030134117_reindex_users_for_username_search.rb
View file

@ -1,9 +1,9 @@
module DataUpdateScripts
class ReindexUsersForUsernameSearch
def run
User.select(:id).in_batches(of: 200) do |batch|
Search::BulkIndexWorker.set(queue: :default).perform_async("User", batch.ids)
end
# User.select(:id).in_batches(of: 200) do |batch|
# Search::BulkIndexWorker.set(queue: :default).perform_async("User", batch.ids)
# end
end
end
end

6
lib/data_update_scripts/20210118194138_resync_unpublished_articles_comments_elasticsearch_document.rb
View file

@ -1,9 +1,9 @@
module DataUpdateScripts
class ResyncUnpublishedArticlesCommentsElasticsearchDocument
def run
Article.unpublished.where.not(comments_count: 0).each do |article|
article.comments.each(&:index_to_elasticsearch)
end
# Article.unpublished.where.not(comments_count: 0).each do |article|
# article.comments.each(&:index_to_elasticsearch)
# end
end
end
end

3
lib/tasks/app_initializer.rake
View file

@ -1,9 +1,6 @@
namespace :app_initializer do
desc "Prepare Application on Boot Up"
task setup: :environment do
puts "\n== Preparing Elasticsearch =="
Rake::Task["search:setup"].execute
puts "\n== Preparing database =="
system("bin/rails db:prepare") || exit!(1)
Rake::Task["db:migrate"].execute # it'll re-alphabetize the columns in `schema.rb`

26
lib/tasks/search.rake
View file

@ -1,26 +0,0 @@
namespace :search do
desc "set up Elasticsearch indexes"
task setup: :environment do
Search::Cluster.setup_indexes
end
desc "update Elasticsearch index mappings"
task update_mappings: :environment do
Search::Cluster.update_mappings
end
desc "tear down Elasticsearch indexes"
task destroy: :environment do
if Rails.env.production?
puts "Will NOT destroy indexes in production"
next
end
Search::Cluster.delete_indexes
end
end
if Rails.env.development?
Rake::Task["db:create"].enhance(["search:setup"])
Rake::Task["db:drop"].enhance(["search:destroy"])
end

3
release-tasks.sh
View file

@ -14,8 +14,7 @@ set -Eex
# abort release if deploy status equals "blocked"
[[ $DEPLOY_STATUS = "blocked" ]] && echo "Deploy blocked" && exit 1
# runs migration for Postgres, setups/updates Elasticsearch
# and boots the app to check there are no errors
# runs migration for Postgres and boots the app to check there are no errors
STATEMENT_TIMEOUT=4500000 bundle exec rails app_initializer:setup
bundle exec rake fastly:update_configs
bundle exec rails runner "puts 'app load success'"

4
spec/factories/tags.rb
View file

@ -4,9 +4,5 @@ FactoryBot.define do
factory :tag do
name { generate :name }
supported { true }
trait :search_indexed do
after(:create, &:index_to_elasticsearch_inline)
end
end
end

9
spec/initializers/rack/attack_spec.rb
View file

@ -11,12 +11,13 @@ describe Rack::Attack, type: :request, throttle: true do
describe "search_throttle" do
it "throttles /search endpoints based on IP" do
Timecop.freeze do
allow(Search::User).to receive(:search_documents).and_return({})
allow(Search::Postgres::Username).to receive(:search_documents).and_return({})
valid_responses = Array.new(5).map do
get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
end
throttled_response = get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
new_ip_response = get "/search/users", headers: { "HTTP_FASTLY_CLIENT_IP" => "1.1.1.1" }
throttled_response = get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "5.6.7.8" }
new_ip_response = get "/search/usernames", headers: { "HTTP_FASTLY_CLIENT_IP" => "1.1.1.1" }
valid_responses.each { |r| expect(r).not_to eq(429) }
expect(throttled_response).to eq(429)

31
spec/lib/acts_as_taggable_on/tag_spec.rb
View file

@ -1,31 +0,0 @@
require "rails_helper"
RSpec.describe ActsAsTaggableOn::Tag, type: :lib do
describe "#after_commit" do
it "on create indexes tag to elasticsearch" do
tag_name = "muffintag"
create(:article, body_markdown: "---\ntitle: Me#{rand(1000)}\ntags: #{tag_name}\n---\n\nMeMeMe")
sidekiq_perform_enqueued_jobs
tag = Tag.find_by(name: tag_name)
expect(tag.elasticsearch_doc).not_to be_nil
end
it "syncs related elasticsearch documents" do
article = create(:article)
podcast_episode = create(:podcast_episode)
tag = article.tags.first
podcast_episode.tags << tag
new_keywords = "keyword1, keyword2, keyword3"
sidekiq_perform_enqueued_jobs
tag.update(keywords_for_search: new_keywords)
sidekiq_perform_enqueued_jobs
expect(collect_keywords(article)).to include(new_keywords)
expect(collect_keywords(podcast_episode)).to include(new_keywords)
end
end
def collect_keywords(record)
record.elasticsearch_doc.dig("_source", "tags").flat_map { |t| t["keywords_for_search"] }
end
end

11
spec/lib/data_update_scripts/index_chat_channel_memberships_to_elasticsearch_spec.rb
View file

@ -1,11 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20200218195023_index_chat_channel_memberships_to_elasticsearch.rb")
describe DataUpdateScripts::IndexChatChannelMembershipsToElasticsearch, elasticsearch: "ChatChannelMembership" do
it "indexes chat channel memberships to Elasticsearch" do
chat_channel_membership = FactoryBot.create(:chat_channel_membership)
expect { chat_channel_membership.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
described_class.new.run
expect(chat_channel_membership.elasticsearch_doc).not_to be_nil
end
end

28
spec/lib/data_update_scripts/re_index_feed_content_and_users_to_elasticsearch_spec.rb
View file

@ -1,28 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20200519142908_re_index_feed_content_and_users_to_elasticsearch.rb")
describe DataUpdateScripts::ReIndexFeedContentAndUsersToElasticsearch do
after do
Search::FeedContent.refresh_index
end
it "indexes feed content(articles, comments, podcast episodes) and users to Elasticsearch", :aggregate_failures do
article = create(:article)
podcast_episode = create(:podcast_episode)
comment = create(:comment)
expect { article.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
expect { podcast_episode.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
expect { comment.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
sidekiq_perform_enqueued_jobs { described_class.new.run }
expect(article.elasticsearch_doc).not_to be_nil
expect(podcast_episode.elasticsearch_doc).not_to be_nil
expect(comment.elasticsearch_doc).not_to be_nil
expect(article.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
expect(podcast_episode.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
expect(comment.elasticsearch_doc["_source"].keys).to include "public_reactions_count"
end
end

11
spec/lib/data_update_scripts/reindex_listing_search_column_spec.rb
View file

@ -1,11 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20200803142830_reindex_listing_search_column.rb")
describe DataUpdateScripts::ReindexListingSearchColumn, elasticsearch: "Listing" do
it "indexes listings to Elasticsearch" do
listing = create(:listing)
expect { listing.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
described_class.new.run
expect(listing.elasticsearch_doc).not_to be_nil
end
end

12
spec/lib/data_update_scripts/reindex_users_for_username_search_spec.rb
View file

@ -1,12 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20201030134117_reindex_users_for_username_search.rb")
describe DataUpdateScripts::ReindexUsersForUsernameSearch do
let(:user) { create :user }
it "reindexes users" do
sidekiq_assert_enqueued_with(job: Search::BulkIndexWorker, args: ["User", [user.id]], queue: "default") do
described_class.new.run
end
end
end

13
spec/lib/data_update_scripts/remove_reaction_index_by_name_spec.rb
View file

@ -1,13 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20200924140813_remove_reaction_index_by_name.rb")
describe DataUpdateScripts::RemoveReactionIndexByName do
it "removes reaction index" do
index_name = "reactions_#{Rails.env}"
Search::Client.indices.create(index: index_name)
expect(Search::Client.indices.get(index: "*").keys).to include(index_name)
described_class.new.run
expect(Search::Client.indices.get(index: "*").keys).not_to include(index_name)
end
end

56
spec/lib/data_update_scripts/resync_elasticsearch_documents_spec.rb
View file

@ -1,56 +0,0 @@
require "rails_helper"
require Rails.root.join("lib/data_update_scripts/20200410152018_resync_elasticsearch_documents.rb")
describe DataUpdateScripts::ResyncElasticsearchDocuments, elasticsearch: %w[FeedContent User Tag] do
after do
Article::SEARCH_CLASS.refresh_index
User::SEARCH_CLASS.refresh_index
end
it "indexes podcast episodes and tags to Elasticsearch" do
tag = create(:tag)
podcast_episode = create(:podcast_episode)
Sidekiq::Worker.clear_all
expect { tag.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
expect { podcast_episode.elasticsearch_doc }.to raise_error(Search::Errors::Transport::NotFound)
sidekiq_perform_enqueued_jobs { described_class.new.run }
expect(tag.elasticsearch_doc).not_to be_nil
expect(podcast_episode.elasticsearch_doc).not_to be_nil
end
it "syncs articles, comments, and users to Elasticsearch", :aggregate_failures do
article = create(:article)
comment = create(:comment)
user = create(:user)
index_real_and_mock_documents(article, comment, user)
expect(Article::SEARCH_CLASS.articles_document_count).not_to eq(Article.count)
expect(Comment::SEARCH_CLASS.comments_document_count).not_to eq(Comment.count)
expect(User::SEARCH_CLASS.document_count).not_to eq(User.count)
sidekiq_perform_enqueued_jobs { described_class.new.run }
refresh_indexes
expect(article.elasticsearch_doc).not_to be_nil
expect(user.elasticsearch_doc).not_to be_nil
expect(comment.elasticsearch_doc).not_to be_nil
expect(Article::SEARCH_CLASS.articles_document_count).to eq(Article.count)
expect(Comment::SEARCH_CLASS.comments_document_count).to eq(Comment.count)
expect(User::SEARCH_CLASS.document_count).to eq(User.count)
end
def index_real_and_mock_documents(article, comment, user)
sidekiq_perform_enqueued_jobs
Article::SEARCH_CLASS.index("article_#{article.id + 100}", class_name: "Article")
Comment::SEARCH_CLASS.index("comment_#{comment.id + 100}", class_name: "Comment")
User::SEARCH_CLASS.index(user.id + 100, name: "User")
refresh_indexes
end
def refresh_indexes
User::SEARCH_CLASS.refresh_index
Article::SEARCH_CLASS.refresh_index
end
end

Some files were not shown because too many files have changed in this diff Show more