Commit graph

70 commits

Author SHA1 Message Date
rhymes
e7f9735354
Remove RssReader (#12169)
* Add script to remove feature flag

* Remove :feeds_import feature flag and RssReader and related classes

* Change Feeds::ImportArticlesWorker signature to support Sidekiq Cron serialization

* Replace RssReader::Assembler with Feeds::AssembleArticleMarkdown

* Removing Assembler
2021-01-08 11:45:15 -05:00
rhymes
4aaeb3e955
Feeds::Import: only import feeds that haven't been recently updated (#12126)
* Add earlier_than to Feeds::Import

* Ignore feeds that have been imported in the last 4 hours

* Make earlier_than mandatory in Feeds::ImportArticlesWorker
2021-01-06 16:58:42 +01:00
rhymes
1741e52e83
Add missing features to new feed importer (#11501)
* Add explanatory comment to how RssReaderFetchUserWorker works

* Add Feeds::ImportWorker worker

* Enable Feeds::Import for single user feed fetching behind a feature flag

* Add Feeds::ValidateUrl service

* Add feature flag to user's validate feed URL feature

* Remove todo notices

* Add feature flag to fetch_all_rss task

* Add feature flag to RssReaderWorker

* Bring back short-circuit for Articles::RssReaderWorker
2020-11-24 18:11:13 +01:00
rhymes
51a4a35448
Add Feeds::Import service class (#10998)
* Add the parallel gem

* Add prototype version of Feeds::Import with parallel URL fetching and parsing

* Tune Feeds::Import and add rake task for local tests

* Add rough measurer tool

* Add specs

* Apply suggestions by @citizen428

* Replace silence with silent

* No need for eager loading

* Add temporary Articles::DevFeedsImportWorker

* Remove temporary rake task

* Add basic error handling, copied from RssReader

* Fix error handling

* Remove temporary measuring rake task

* Remove logging added for development purposes

* Add info and error logging on error level
2020-10-30 15:01:44 +01:00
Molly Struve
7891db594f
[deploy] Bug Fix:Add send_email_digest Rake Task for DEV (#10065) 2020-08-28 14:12:10 -05:00
Molly Struve
b23374e737
[deploy] Revert "Refactor:Move RssReader Fetch Article task to Sidekiq (#9976)" (#9998) 2020-08-25 14:57:33 -05:00
Molly Struve
ff06454151
[deploy] Refactor:Move RssReader Fetch Article task to Sidekiq (#9976) 2020-08-25 13:47:13 -05:00
Molly Struve
0947256f13
[deploy] Refactor:Fetch New Github Repos From Sidekiq Cron (#9986) 2020-08-25 12:42:33 -05:00
Molly Struve
e0c37442c3
[deploy] Refactor:Move Enqueueing Podcast Workers to Sidekiq Cron (#9983) 2020-08-24 16:28:57 -05:00
Molly Struve
513717e680
Refactor:Move Credit Counter Cache Sync to Sidekiq (#9972) 2020-08-24 14:20:11 -05:00
Molly Struve
972253a17b
Refactor:Remove save_nil_hotness_scores Task (#9930) 2020-08-24 10:20:19 -05:00
Molly Struve
8574153b07
[deploy] Refactor:Move Remove Old HTML Variant Data task to Sidekiq (#9925) 2020-08-22 18:36:22 -05:00
Molly Struve
4308914fe8
[deploy] Refactor:Move send_email_digest Rake Task to Sidekiq (#9929) 2020-08-22 08:26:30 -05:00
Molly Struve
e8b14dec83
Refactor:Move prune_old_field_tests Rake to Sidekiq Cron (#9924) 2020-08-21 12:56:42 -04:00
Molly Struve
f00bb8eb8c
[deploy] Refactor:Move Listing and Support Tag Tasks to Sidekiq (#9898) 2020-08-20 14:05:34 -05:00
rhymes
085c60992b
Rubocop: Enable and fix Style/OptionalBooleanParameter cop (#9711)
* Enable and fix Style/OptionalBooleanParameter cop

* Keep regular parameters for workers

* Keep regular parameters for workers

* Fix spec

* Trigger Travis correctly
2020-08-10 19:35:55 +02:00
rhymes
71d53873d1
Use Rails 6 new features in the code (#8342)
* Use pick instead of pluck first

* Use touch_all instead of update_all

* Use destroy_by instead of where.destroy_all

* Use scopes

* Fix db:seed:replant

* Remove ActiveSupport::Cache::RedisCacheStore expires_in cache
2020-06-08 16:06:29 -04:00
Michael Kohl
7f75f99560
[deploy] Rename classified listings (#7910)
* Change models and related files

* Update controllers and specs

* More renaming

* Seek and destroy, I mean search and replace

* Round up the stragglers

* Ground control to Major Travis...

* More fixes

* PR feedback

* Various fixes

* Rename view

* Fix list query builder

* Unify request specs

* Fix some API spec errors

* Fix remaining API specs

* Make spec conform to API

* Fix leftover problems

* Fix JS tests

* Fix column name in select

* Fix API specs

* Fix search specs

* Paging Mr. Travis
2020-05-27 13:35:09 +00:00
Mac Siri
217ee6c3f4
Organize rake tasks (#7005) 2020-04-01 12:39:40 -04:00
Alex
129cd3ba83
Log Elasticsearch index counts to Datadog (#6845) [deploy] 2020-03-25 15:52:54 -05:00
Ben Halpern
5bd042e7d6
Prune old field tests for ongoing tests (#6540) [deploy] 2020-03-09 16:22:42 -04:00
Alex
bbec492844
Remove more Algolia from ClassifiedListing (#6463) [deploy]
* Remove ClassifiedListing from algolia initializer

* Remove reference to remove_from_index
2020-03-05 09:23:22 -08:00
rhymes
f3c2adfa45
Routine rubocop fixes (#6138) 2020-02-18 11:21:42 -05:00
Maykon Menezes
01dc81afa9
Move Podcasts::GetEpisodesJob to Sidekiq (#5443) [deploy]
* Refactored Podcast::GetEpisodesJob to sidekiq

* fixing specs

* removing env keys

* Change arguments to Hash

* Update specs and add spec for queue

* Update references for Hash argument

* Add comment to explain parameters of #perform

Co-authored-by: Alex <alexandersmith223@gmail.com>
2020-02-06 09:42:55 -08:00
Ben Halpern
a4b86e3ff1
Add initial usage metrics for datatog (#5544) [deploy]
* Add initial usage metrics for datatog

* Make measurable name more clear

* Change name of daily usage worker

* Fix test names
2020-01-16 12:59:17 -05:00
Molly Struve
5962d52a61
record background queue stats in a worker (#5533) [deploy] 2020-01-15 14:44:57 -05:00
Molly Struve
2e5a36e538
Log DB Table sizes in a Sidekiq Worker (#5518) 2020-01-15 12:33:14 -05:00
rhymes
9b2886979a ActiveRecord count optimizations (#5478) [deploy]
* Do not load messages in memory, just to count them
* Add Model.estimated_count
* Use Model.estimated_count in task record_db_table_counts
* Use size and estimated_count
* Use table_name so that .estimated_count works with all models
* Add comment to explain the .load.size + .each pattern
2020-01-14 12:03:38 -05:00
rhymes
102e1a6b9c Routine rubocop fixes (#5386) 2020-01-08 09:05:12 -05:00
Molly Struve
6b23e327aa
Track Queue sizes and latency for Sidekiq (#5352) [deploy] 2020-01-07 11:57:13 -05:00
Molly Struve
bc8c177966
Record db table sizes in datadog where we can use them to generate relevant alerts (#5377) 2020-01-06 15:52:47 -05:00
Molly Struve
660852b711 record counts for important db tables to help track growth (#5253) [deploy] 2019-12-27 11:29:47 -05:00
Molly Struve
f86f93bc8a
Create rake Task to Destroy Old Notifications (#5180) [deploy] 2019-12-26 13:17:34 -06:00
Molly Struve
fb17d0e079
fix title_with_timeframe to handle invalid timeframe and update rake task to not use return (#5241) [deploy] 2019-12-26 08:43:46 -06:00
Ben Halpern
cd9805bc22 Fix tag badge task logic (#5189) [deploy] 2019-12-20 14:39:28 -05:00
Ben Halpern
c6d94d4049
Add rake task to reward tag badges weekly (#5183) [deploy] 2019-12-19 16:34:22 -05:00
Michael Kohl
b57ced5466 CacheBuster refactoring (#4766)
* Turn CacheBuster into a module

This class used no internal state, so repeatedly creating short-lived objects seems wasteful.

* Consistently use string interpolation and parenthesis

* Destructure arrays into meaningful names, formatting

* Fix request spec for internal classified listings controller

Interestingly this works when asserting directly on the module, but not on a double.
Asserting directly in the module seems sufficient for this test so the indirection
was removed.

* Turn CacheBuster into a module

This class used no internal state, so repeatedly creating short-lived objects seems wasteful.

* Fix specs after rebasing
2019-11-13 10:51:23 -05:00
Jess Lee
65fccf00a9 add rake task for fixing credit counter cache (#3962) 2019-09-06 13:08:39 -04:00
Ben Halpern
66f72c091a Fix rake task with repeated name (#3845) [ci skip] 2019-08-26 17:33:00 -04:00
Mario See
47c2885810 Listings set custom expiry date (#3770)
* add expire_on column to classified_listings

* add expire_on to valid listing params

* add expire_on to model

* add expire date to new listing form

* add field in edit form

* add expiry dates to dashboard

* update expiry date in listing toolkit

* WIP unpublish job and test

test still has to run properly

* add job to class name

* change job to rake task

* revert snap change

* delete migration file

will add new one and revert schema changes to rename column

* reset schema

* add new migration to schema 'expires_at'

* change expire_on to expires_at
2019-08-23 17:33:50 -04:00
Anna Buianova
fabaeb5461 Introduced published field to podcasts (#3593)
* Published field for podcasts

* Add podcasts migration

* More specs for podcasts availability

* Show published in podcasts internal

* Update schema.rb with db:migrate
2019-08-01 09:29:13 -04:00
Anna Buianova
94ab4e23fe Update podcast episode statuses #2952 (#3450) 2019-07-16 13:36:25 -04:00
Anna Buianova
9751e84549 Moved Podcast#pull_all_episodes to a separate ActiveJob #2952 #2497 (#3057)
* Use one PodcastFeed instance while fetching podcasts

* Moved Podcast#pull_all_episodes to a separate ActiveJob
2019-06-06 09:49:10 -04:00
Ben Halpern
bfedee0561
Fix some listing/credit bugs (#2584)
* Fix some listing/credit bugs

* Tack on codepen team link fix

* Fix test

* Actually fix test
2019-04-26 19:01:35 -04:00
rhymes
5782ffd7e0 Use published scope for articles (#2374)
* Use published scope for articles

* Fix variable name

* The sequencing in the union query matters

* Fix spec

* Restore a published: true condition

* This test is actually misleading due to the union

* Revert this change, not sure why it's not a relation

* Fix useless diff
2019-04-11 13:18:26 -04:00
Ben Halpern
c901c7138d
Fix possible race condition with page view tracking (#2103)
* Fix possible race condition with page view tracking

* Change min number to check for html variant calculation
2019-03-18 15:53:13 -04:00
Keshav Biswa
8d57d45ecc Rubocop Style cops enabled (#2056)
* Rubocop enabled style/alias

* Enabled Style/ArrayJoin Cop

* Enabled Style/Attr

* Enabled Case Equality

* Enabled CharacterLiteral

*  Enabled ColonMethodCall Cop

* Enabled CommentAnnotation cop

* Enabled PreferredHashMethods Cop

*  Enabled DoubleNegation Cop

* Enabled EachWithObject Cop

* Enabled EmptyLiteral Cop

* Enabled EvenOdd Cop

* Enabled IfWithSemicolon Cop

* Enabled Lambda and LambdaCall Cop

* Enabled LineEndConcatenation Cop

* Enabled ModuleFunction Cop;

* Enable NegatedIf and NegatedWhile Cop

* Enabled NilComparison Cop

* Enabled Not Cop

* Enabled NumericLiterals Cop

* Enabled OneLineConditional Cop

* Enabled PercentLiteralDelimiters Cop

* Excluded internal/users_controller from negated_if cop

* Reverted the double negation change from github_issue_tag and github_issue.rb"

* Enabled PerlBackrefs Cop

* Changed Regexp.last_match(1) to Regexp.last_match(0)

* Enabled proc cop

* Enabled RaiseArgs Cop

* Reverted Regexp.last_match(0) to Regexp.last_match(1)

* Enabled SelfAssignment Cop

* Enabled SingleLineMethods Cop

* Enabled SpecialGlobalVars Cop

* Enabled VariableInterpolation Cop

* Enabled WhenThen Cop

* Enabled WhileUntilModifier Cop

* Enabled WordArray Cop

* Enabled IfUnlessModifier Cop

* Enabled GuardClause Cop
2019-03-15 18:33:54 -04:00
Ben Halpern
8149a72efb
Nevertheless, She Coded (#2000) 2019-03-07 18:52:09 -08:00
Ben Halpern
e5a2b25871
Add additional streak badge rewarding (#1896) 2019-02-25 15:37:16 -08:00
Ben Halpern
92b24a9978
Add html_variant split testing to /badge (#1763)
* Add /badge html_variant

* Fix some of the linting
2019-02-08 12:45:14 -04:00