Commit graph

77 commits

Author SHA1 Message Date
Molly Struve
ef9611d745
Create DataUpdateScript to Index all Tags (#6078) [deploy] 2020-02-14 14:17:52 -05:00
Molly Struve
65eba740ab
Document DataUpdateScripts, Add to Deploy Process, Add Development Check (#6068) [deploy] 2020-02-14 11:45:27 -05:00
Molly Struve
0bfbd9bb1f
Set Up Framework For Running Data Update Scripts (#6025) [deploy] 2020-02-13 10:48:23 -05:00
Molly Struve
acfdd11b50
update cluster settings to prevent auto index creation, update destroy to remove ALL indexes (#6042) 2020-02-12 15:21:38 -05:00
Juanito Fatas
33b8014677
Fix indentation in search.rake (#5971)
lib/tasks/search.rake:9:3: C: Layout/IndentationWidth: Use 2 (not 3) spaces for indentation. (https://rubystyle.guide#spaces-indentation)
     Search::Cluster.update_mappings
  ^^^
2020-02-10 09:09:23 -05:00
Molly Struve
4691ae1e34
Log in Datadog When Sidekiq Worker Fails for Final Time (#5942) [deploy] 2020-02-10 08:56:41 -05:00
Maykon Menezes
01dc81afa9
Move Podcasts::GetEpisodesJob to Sidekiq (#5443) [deploy]
* Refactored Podcast::GetEpisodesJob to sidekiq

* fixing specs

* removing env keys

* Change arguments to Hash

* Update specs and add spec for queue

* Update references for Hash argument

* Add comment to explain parameters of #perform

Co-authored-by: Alex <alexandersmith223@gmail.com>
2020-02-06 09:42:55 -08:00
Molly Struve
00fefe5865
Add Code To Setup the Elasticsearch Tag Index (#5919) [deploy] 2020-02-06 11:46:11 -05:00
rhymes
3260270103
Have counters:update_users use a transaction per user and customizable batch size (#5912) [deploy] 2020-02-05 12:26:03 -05:00
rhymes
a1835ae5b9
User counters using PostgreSQL JSON (#5373) [deploy]
Co-authored-by: Ben Halpern <bendhalpern@gmail.com>
2020-02-03 10:35:08 -05:00
Ben Halpern
a4b86e3ff1
Add initial usage metrics for datatog (#5544) [deploy]
* Add initial usage metrics for datatog

* Make measurable name more clear

* Change name of daily usage worker

* Fix test names
2020-01-16 12:59:17 -05:00
Molly Struve
1a3d59c97f
Send Sidekiq data to Honeycomb (#5517) [deploy] 2020-01-16 12:30:43 -05:00
Molly Struve
5962d52a61
record background queue stats in a worker (#5533) [deploy] 2020-01-15 14:44:57 -05:00
Molly Struve
2e5a36e538
Log DB Table sizes in a Sidekiq Worker (#5518) 2020-01-15 12:33:14 -05:00
rhymes
9b2886979a ActiveRecord count optimizations (#5478) [deploy]
* Do not load messages in memory, just to count them
* Add Model.estimated_count
* Use Model.estimated_count in task record_db_table_counts
* Use size and estimated_count
* Use table_name so that .estimated_count works with all models
* Add comment to explain the .load.size + .each pattern
2020-01-14 12:03:38 -05:00
Michael Kohl
dba7e6ca60 Move Twitch webhook registration from ActiveJob to Sidekiq (#5414) 2020-01-14 11:55:35 -05:00
rhymes
c267526ed2 Remove temporary site config script and unused ENV var (#5455) [deploy] 2020-01-14 09:49:31 -06:00
rhymes
102e1a6b9c Routine rubocop fixes (#5386) 2020-01-08 09:05:12 -05:00
Molly Struve
6b23e327aa
Track Queue sizes and latency for Sidekiq (#5352) [deploy] 2020-01-07 11:57:13 -05:00
Molly Struve
bc8c177966
Record db table sizes in datadog where we can use them to generate relevant alerts (#5377) 2020-01-06 15:52:47 -05:00
Molly Struve
660852b711 record counts for important db tables to help track growth (#5253) [deploy] 2019-12-27 11:29:47 -05:00
Molly Struve
f86f93bc8a
Create rake Task to Destroy Old Notifications (#5180) [deploy] 2019-12-26 13:17:34 -06:00
Molly Struve
fb17d0e079
fix title_with_timeframe to handle invalid timeframe and update rake task to not use return (#5241) [deploy] 2019-12-26 08:43:46 -06:00
Ben Halpern
cd9805bc22 Fix tag badge task logic (#5189) [deploy] 2019-12-20 14:39:28 -05:00
Ben Halpern
c6d94d4049
Add rake task to reward tag badges weekly (#5183) [deploy] 2019-12-19 16:34:22 -05:00
Molly Struve
ce1bc8d2a4
Revert "Setup dual Redis store for rolling over fragment cacheing to Redis (#4991)" (#5014) [deploy]
This reverts commit e871a01c01.
2019-12-05 14:03:41 -06:00
Molly Struve
e871a01c01
Setup dual Redis store for rolling over fragment cacheing to Redis (#4991) [deploy] 2019-12-05 13:19:01 -06:00
rhymes
c9a07faf6a Add script to display and migrate environment variables to SiteConfig (#4865) [deploy] 2019-11-22 10:18:22 -05:00
Michael Kohl
b57ced5466 CacheBuster refactoring (#4766)
* Turn CacheBuster into a module

This class used no internal state, so repeatedly creating short-lived objects seems wasteful.

* Consistently use string interpolation and parenthesis

* Destructure arrays into meaningful names, formatting

* Fix request spec for internal classified listings controller

Interestingly this works when asserting directly on the module, but not on a double.
Asserting directly in the module seems sufficient for this test so the indirection
was removed.

* Turn CacheBuster into a module

This class used no internal state, so repeatedly creating short-lived objects seems wasteful.

* Fix specs after rebasing
2019-11-13 10:51:23 -05:00
Molly Struve
00e3ee811b Use ActiveSupport::Cache::RedisCacheStore As RedisRailsCache (#4676) [deploy] 2019-11-01 12:19:12 -04:00
Molly Struve
cbc85100e3 Create RedisRailsCache Object to Mimic Rails.cache And Apply (#4667)
* Create RedisRailsCache object to mimic Rails.cache

* Do not use prefix _ for a variable that is used

* dont shadow outer local variable
2019-10-31 12:12:21 -04:00
rhymes
c8abbe9d60 Pro: memberships (#3461)
* Add ProMembership model

* Implement ProMembershipsController#create

* Implement basic ProMembershipsController#show

* Add ProMembership to ledger

* Populate user history after pro subscription is created

* Add fields for expiration notifications

* Add ProMemberships::ExpirationNotifier to notify users of expiring memberships

* Add tasks for recurring jobs to notify users of expiration

* Add auto_recharge column to ProMembership

* Add ProMemberships::Biller (incomplete)

* Fix specs

* Add ProMembership to Administrate

* Fix spec

* Add has_enough_credits? to User and Organization

* Add Payments::Customer class

* Finish ProMembership::Biller functionality

* Fix ProMemberships::Creator check for credits

* Disable destroy actions for ProMembershipsController

* Correctly authenticate ProMembershipsController actions

* Make sure only pro user's history can be indexed

* Add ProMembershipsController#update action for auto recharge

* Use regular AR to save new credits and add touch to the purchaser

* Clarify Pro membership create policy

* Display information about an existing pro membership

* Add UI to show page

* Add system test for Pro membership creation

* Implement edit membership

* Make sure users with pro memberships can access history and dashboard pro

* Fix padding issue

* Show a different text for a user that has credits but not enough for Pro

* Move Pro Membership functionality inside settings

* Update Pro Membership link in email notifications

* Bust all relevant caches

* Add the Pro checkmark around the website

* Use Users::ResaveArticlesJob instead of delay

* Add/remove user from pro-members chat channel

* Use the appropriate Pro checkmark

* Remove unfinished pro elements

* Remove checkmark JS
2019-09-24 10:38:54 -07:00
Jess Lee
65fccf00a9 add rake task for fixing credit counter cache (#3962) 2019-09-06 13:08:39 -04:00
Ben Halpern
66f72c091a Fix rake task with repeated name (#3845) [ci skip] 2019-08-26 17:33:00 -04:00
Mario See
47c2885810 Listings set custom expiry date (#3770)
* add expire_on column to classified_listings

* add expire_on to valid listing params

* add expire_on to model

* add expire date to new listing form

* add field in edit form

* add expiry dates to dashboard

* update expiry date in listing toolkit

* WIP unpublish job and test

test still has to run properly

* add job to class name

* change job to rake task

* revert snap change

* delete migration file

will add new one and revert schema changes to rename column

* reset schema

* add new migration to schema 'expires_at'

* change expire_on to expires_at
2019-08-23 17:33:50 -04:00
Anna Buianova
fabaeb5461 Introduced published field to podcasts (#3593)
* Published field for podcasts

* Add podcasts migration

* More specs for podcasts availability

* Show published in podcasts internal

* Update schema.rb with db:migrate
2019-08-01 09:29:13 -04:00
Anna Buianova
95a9e13922 Remove logic relying on old podcast status (#3505) 2019-07-19 09:30:51 -04:00
Anna Buianova
a8227b3048 A temp rake task to update podcast statuses (#3496) 2019-07-17 11:10:02 -04:00
Anna Buianova
94ab4e23fe Update podcast episode statuses #2952 (#3450) 2019-07-16 13:36:25 -04:00
rhymes
611f4b8675 Sponsorship: remove old columns (step 3) (#3446)
* Remove old sponsorship columns

* Rename redundant param names
2019-07-11 15:35:24 -04:00
rhymes
ad79116366 Sponsorship: add model (step 1) (#3438)
* Add Sponsorship model

* Add Sponsorship to admin

* Add correct relationships between models

* Add temporary script to migrate sponsorship data
2019-07-10 15:58:37 -04:00
rhymes
97b288330e Remove temp page's views referrers rake task (#3426) [ci skip] 2019-07-09 15:05:08 -04:00
rhymes
7da7a16d8d Pro: add referrers to dashboard and single article stats (#3295)
* Reorganize PageViewsController

* Add domain and path to PageView model

* Add before_create callback to populate domain and path

* Add list of referrers to AnalyticsService

* Add referrers to the UI

* Remove useless referrers card and tweak table line height

* Add referrer stats to article stats page

* Add not null and empty default to domain and path

* Refactor JS analytics client

* create_list is a step back here

* Revert "Add not null and empty default to domain and path"

This reverts commit bc02440076047a887c65d300bccd4661ecc8ffd0.

* Add index on domain concurrently

* Make the script more robust
2019-06-25 13:58:09 -04:00
Anna Buianova
9751e84549 Moved Podcast#pull_all_episodes to a separate ActiveJob #2952 #2497 (#3057)
* Use one PodcastFeed instance while fetching podcasts

* Moved Podcast#pull_all_episodes to a separate ActiveJob
2019-06-06 09:49:10 -04:00
Corey Alexander
ec388804df Embed Twitch Live Streaming (#2591)
* Get a job created that can create a webhook subscription for a twitch user login

* Remove ngork url

* Refactor add store the access token in the cache

* Get a controller stood up to recieve the webhooks. Now they just need to be processed

* Get User columns added and got webhook controller bones working

* Update the webhook job to use the User

* Add a way for the User to input their Twitch User Name. Plus a linter fix

* Delay webhook registration when profile is updated

* Don't add _ in username

* Use String columns

* Quick fix and add some more requests specs

* Specs for the webhook job

* Get a show page Twitch Live Streams. Just a straight embed of the Twitch Everything Embed UI. Works surprisingly well responsively, and works on all screen sizes

* Fix Gemfile.lock from merge issues

* Add support for expired tokens and add spec

* Add secrets to webhook registration and clean up spec to remove token logic

* Verify webhook secret and spec it

* Add rake task to enqueue webhook registration for all Users. This can be used from Heroku Scheduler

* Update the lease seconds to be for 5 days

* Actually lets do 7 so we can refresh twice a week and try to make sure that we can always miss one

* Hijack the existing Twitch logo instead of making a duplicate one

* Remove comment and replace with log line

* Remove some white space

* Log to Airbrake when webhook errors occur

* Move to passing in an id instead of User object

* Extract logic from Job to Service object

* Capitilize in the view

* Move out of models and into services

* Remove letover stub

* Remove one usage of Faraday

* Use HTTParty for all the HTTP here
2019-04-29 18:11:53 -04:00
Ben Halpern
bfedee0561
Fix some listing/credit bugs (#2584)
* Fix some listing/credit bugs

* Tack on codepen team link fix

* Fix test

* Actually fix test
2019-04-26 19:01:35 -04:00
rhymes
5782ffd7e0 Use published scope for articles (#2374)
* Use published scope for articles

* Fix variable name

* The sequencing in the union query matters

* Fix spec

* Restore a published: true condition

* This test is actually misleading due to the union

* Revert this change, not sure why it's not a relation

* Fix useless diff
2019-04-11 13:18:26 -04:00
Andy Zhao
8ef611f879 Remove Honeycomb from the app (#2366)
* Remove Honeycomb related gems

* Remove Honeycomb related code from controllers

* Remove Honeycomb from RSS Reader

* Remove Honeycomb related files

* Remove Honeycomb related tests

* Remove leftover Honeycomb methods
2019-04-11 10:49:35 -04:00
rhymes
3b4bda89df Upgrade Honeycomb gem, fix and improve RssReader tests (#2319)
* Switch from deprecated honeycomb-rails to honeycomb-beeline

* Improve error handling, tests and add refactoring

* Fix multiple fetching test and add one more

The test wasn't actually testing the correct behavior, because the class
by default force fetches. It worked because it wasn't reloading the user
from the database.

* Test that Honeycomb's client is doing its thing

* Use some defensive programming to avoid any possible problem during shutdown

* Fix typo

* Use the new API to add fields to the current measured event

* Use default instrumentation and add user info to event

* Replace user.email with user.username, less personal data around

* Use mocking instead of calling Honeycomb.init

* Freeze time before traveling

* Remove a flaky test dependent on rand
2019-04-07 15:27:53 -04:00
Tim Lange
3494264404 Enabled rubocop/lint (#2130)
* WIP: Enabled rubocop/lint

* FIX: Fixed failing test

* FIX: Small change to return

* FIX: Changed indent
2019-03-21 10:53:44 -04:00