Commit graph

217 commits

Author SHA1 Message Date
Joe Doss
aa49cb1cfe
Bugfix for #7663 and refactor of container setup (#7747)
* Move from Alpine Linux to Fedora Linux.

This changes the base container image away from Alpine Linux to
Fedora Linux to address some musl libc issues with some of our gems.

It also moved the main file to a Containerfile and symlinks the Dockerfile
to it. This makes our container setup less Docker-centric as Linux users
most likely will be using Podman as their container runtime.

Lastly, it moves the WORKDIR from /usr/src/app to /opt/apps/devto.
The Linux FHS states that /opt is the spot for Optional application
software packages and /usr/src is for kernel source code.

https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

/opt Optional application software packages
/usr/src Source code, e.g., the kernel source code with its header files.

Also, if SELinux becomes a thing in our future, moving this makes it
easier to manage contexts out of /opt rather than /usr/src.

* Adjust the Containerfile a bit to add some missing packages, make it
more generic and move env vars to the docker-compose.

* Move the Dockerfile to a symlink to the Containerfile.

We need to be able to support more than just Docker for a container
runtime. Moving everything to a Containerfile and symlinking the
Dockerfile helps users run runtimes such as Podman.

* Add in new entrypoint files and fix the app path in docker-entrypoint.sh

* Refactor the docker-compose.yml file so it doesn't build the main
application container three times in a row. We can use the same
container for web, webpacker, and sidekiq.

Also make the volume names very explicit on what their contents and add
in SELinux context support with :Z

* Fix ELASTICSEARCH_URL.

* Remove the absolute path on ip binary and add in iproute to Containerfile.

* Symlink the Dockerfile to Containerfile and add in a container-compose.yml file
for usage with podman-compose. We are waiting for this issue to get resolved

https://github.com/containers/libpod/issues/6153

and for podman-compose to mature a bit more. A user using podman-compose can use

podman-compose -f container-compose.yml

to launch the DEV container stack with Podman.

* Update the .gitignore file to reflect the new container volumes.

* Fix the entrypoint script on the Containerfile and clean up a bunch of things.

* Rework the Containerfile to prep it to run as a non-root user. We have to wait
for this issue to get fixed:

https://github.com/containers/libpod/issues/6153

for Linux users and then we can uncomment the USER line so we are running Rails
as a non-root user. :toot:

* This reworks the compose files so each task that is needed to start the app has
a correct wait command with dockerize. It also adds in containers for doing
yarn and bundle things for development. Since we mount the code directory inside
the container we lose our pre-containerized gems and node_modules.

This means users can run:

Linux
podman-compose -f container-compose.yml up

Mac
docker-compose up

and it should correctly build the app stack with containers!

* Clean up old container related things.

* Add in some container pre-reqs and my name to the "Core team" section! :toot:

* Adjust the seed and sidekiq compose entries so they do not use the web entrypoint
and set the REDIS_URL and REDIS_SESSIONS_URL env vars for seed.

Add in some echos to the entrypoint.sh.

Also set docker-compose to use :delegated on mount points to speed things up.

https://docs.docker.com/storage/bind-mounts/#configure-mount-consistency-for-macos

* Just call yarn install --dev on the yarn container.

* Update documentation to support Docker and Podman as Container Engines and
move the page from docker to containers to reflect the fact not all users run
their containers via Docker.

There are dozens of us... DOZENS!

* Set --local on bundle config so it just impacts this Ruby app.

* Renamed bin/docker-setup to bin/container-setup to reflect the fact that that
we can use Podman as a container engine. I also refactored it so it does some
basic pre-flight checks for docker and docker-compose or podman and
podman-compose to make sure users have a container engine installed.

* Set cache_all_platforms true for bundler.

* Bump docker-compose dockerize -timeout to 45min for slower macOS hardware.

* Move to the consolidated app_initializer:setup rake task for bootstrapping the
app.

* Update docs/installation/readme.md
2020-05-28 12:11:51 -05:00
Molly Struve
14c2372ae7
[deploy] Add API Key to API Write Throttle (#8110) 2020-05-28 09:45:01 -05:00
Josh Puetz
1a1b5b544e
Add Feature Flags support to DEV (#7996)
* Create tables for Flipper gem

* Enable Flipper gem

* Remove concurrent ruby from Gemfile

* Add Flipper UI

* Codeland landing page

* Remove codeland controller and sample page

* Remove oops

* Rename Flipper-UI endpoint

* Wrap Flipper in a FeatureFlag module

* [deploy] Lock node to 12.16 (#8057)

Co-authored-by: Mac Siri <krairit.siri@gmail.com>
2020-05-26 15:26:33 -05:00
rhymes
1112cbf316
[deploy] Update webpacker configuration (#8004) 2020-05-25 12:03:18 -04:00
Mac Siri
1abd7cd46b
[deploy] Add async_info to ReservedWords (#8022) 2020-05-21 17:02:26 -04:00
Molly Struve
6a4a99f162
[deploy] Track Deploys In Honeycomb (#7935) 2020-05-19 09:10:41 -05:00
Molly Struve
dbb5c752ab
[deploy] Sample Noisy Honeycomb Spans (#7891) 2020-05-18 10:36:01 -05:00
Molly Struve
7293df3191
[deploy] Return if there is no ip_address from Method (#7874) 2020-05-14 18:13:24 -05:00
Molly Struve
7dd249ac21
[deploy] Add Back Tracking API-keys in Honeycomb (#7868) 2020-05-14 15:10:15 -05:00
Molly Struve
8ae486a362
[deploy] Track and Limit API writes by IP Address (#7847) 2020-05-14 14:13:56 -05:00
Mac Siri
321b24d2a0
[deploy] Create write_articles oauth scope (#7834) 2020-05-14 09:55:53 -04:00
Molly Struve
3a49d3f0d1
[deploy] Revert "filter/sample noisy spans (#7833)" (#7837)
This reverts commit 716bb099fa.
2020-05-13 17:45:49 -05:00
Liz Fong-Jones
716bb099fa
[deploy] filter/sample noisy spans (#7833) 2020-05-13 17:07:09 -05:00
Molly Struve
fb04925c29
[deploy] Decrease General Site Request Limit (#7771) 2020-05-12 09:41:19 -05:00
Molly Struve
1570580279
[deploy] Return the First 300 characters for Redis Commands (#7778) 2020-05-11 15:01:23 -05:00
Molly Struve
bbff0f757e
[deploy] dont configure Honeycomb if its disabled (#6618) 2020-05-11 11:59:18 -05:00
Molly Struve
81c9b27a7f
Track IPs and API Tokens for Throttled Requets, Send Retry After Header (#7735) 2020-05-08 10:16:49 -04:00
Molly Struve
f190f4c6c0
[deploy] Track Client IPs to Help Block Bad Actors (#7730) 2020-05-07 12:45:30 -05:00
Molly Struve
2ba6c98e81
Limit 1 message every second for single user (#7631)
* limit 1 message every second for  single user

* increase message limit to 2 in a second
2020-04-30 19:06:19 -04:00
Molly Struve
4c779be2ed
[deploy] Remove ALL Algolia Code (#7577) 2020-04-29 10:11:57 -05:00
rhymes
8407bf9b4e
[deploy] Authentication refactoring: cleanups (#7576)
* Use Devise registry to list available providers

* Cleanups following Devise+Omniauth docs

* More cleanups

* Use helpers

* More cleanups

* Oops, forgot to remove these specs
2020-04-29 10:33:30 -04:00
rhymes
2d26318cf9
[deploy] Replace (black|white)list with more inclusive language (#7459)
* Replace isBlacklisted

* Update fastly.rake task

* More renames

* Rename Faslty config file

* More changes

* Replace isForbiddenFromPreloading with isNotPreloadable

* Skip problematic test

* Re-enable
2020-04-27 18:00:12 +02:00
Tam Chau
ffba42cf43
Add expires_in option for ActiveSupport::Cache::RedisStore#increment method (#7515)
* rails patch to fix expires_in for increment method

* fixup! rails patch to fix expires_in for increment method

correct test case

* fix redundant file name & class name

* add comment to explain monkey patch for RedisCacheStore#increment
2020-04-27 16:40:49 +02:00
rhymes
f5a6ff32cc
Finalize Slack messengers refactoring (#7484)
* Remove dead code

* Rename SlackBotPingWorker to Slack::Messengers::Worker

* Remove useless Slack mock

* Fix specs
2020-04-24 21:15:10 +02:00
Molly Struve
edcd019e9c
[deploy] Throttle General Site Requests (#7418) 2020-04-23 08:58:27 -05:00
Molly Struve
de88a08ebd
[deploy] handle rate limit errors gracefully (#7422) 2020-04-22 08:55:26 -05:00
Alex
b51852e866
[deploy] Remove fastly-rails gem (#7257)
* Remove fastly-rails gem

* Remove FastlyRails.configuration references

* Remove FastlyRails initializer
2020-04-14 10:11:27 -04:00
rhymes
34848a4209
Upgrade Rubocop to 0.81 (#7219)
* Upgrade rubocop to 0.81.0

* Reordered rubocop file and updated it

* Regenerate todo

* Correct some things

* Fix specs
2020-04-11 10:10:04 +02:00
rhymes
b8f2390cb5
Fix Datadog auth failure payload and add tests (#7150)
* Fix Datadog auth failure payload and add tests

* Use functional Ruby properly
2020-04-09 09:54:51 +07:00
Molly Struve
6f4e6fe064
[deploy] ensure that often run sidekiq jobs are unique until they execute (#7168) 2020-04-08 19:19:30 -04:00
Molly Struve
b652147e7a
throttle API get and write endpoints (#7167) 2020-04-08 19:17:51 -04:00
Ben Halpern
a4a14a5377
[deploy] Make force_ssl an environment variable to make sure new communities can get proper ssl config. (#7090)
* Add conditional for redirect

* Force ssl

* Redis secure

* More push

* Add FORCE_SSL_IN_RAILS env var
2020-04-08 14:23:50 -04:00
Ben Halpern
22ffc7febc
Cache onboarding at edge (#6966) 2020-03-31 16:12:14 +02:00
Molly Struve
55fe5a40a8
Add Rack::Attack Gem to Throttle Requests, Only Search to start (#6841) [deploy]
* throttle requests made to search endpoints

* stub elasticsearch requests in rack attack specs and disable for running specs unless tagged

* use original client IP from fastly to throttle requests to search endpoints
2020-03-30 12:09:40 -04:00
rhymes
e9e4059ad3
Add Slack messengers: part 1 (#6748)
* Add Slack::Announcer and SlackClient

* Add Slack::Messengers::Feedback class

* Use Slack::Messengers::Feedback and refactor test

* Should not raise an error if params are nil, let the job fail

* Add Slack::Messengers::Sponsorship

* Rename spec

* Use Slack::Messengers::Sponsorship

* Add Slack::Messengers::PotentialSpammer

* Use Slack::Messengers::PotentialSpammer

* Add Slack::Messengers::RateLimit and use it

* Refactor a bit and improve tests

* Add Slack::Messengers::Note and use it

* Use Slack::Announcer and fix spec

* Fix specs

* Update app/services/slack/announcer.rb

Co-Authored-By: Vaidehi Joshi <vaidehi.sj@gmail.com>

* Implement review suggestions

* Empty commit to trigger Travis

Co-authored-by: Vaidehi Joshi <vaidehi.sj@gmail.com>
2020-03-24 18:45:40 +01:00
Michael Kohl
99da2a0863
Run rails app:update for 5.2.4.2 (#6743) [deploy]
* Run rails app:update for 5.2.4.2

* Remove bin/update and disable ActiveStorage
2020-03-23 10:38:20 -04:00
Molly Struve
066e2885e8
Only Enable Hypershield if ENV variable is present (#6666) [deploy]
* only enable hypershield if ENV variable is present

* enable hypershield if in production
2020-03-16 15:28:46 -04:00
Molly Struve
bd10c9086c
Enable Hypershield in Production Take 2 (#6628) [deploy] 2020-03-16 09:10:01 -05:00
rhymes
5f831c9836
Remove extra HoneycombClient (#6573) 2020-03-11 10:34:14 +01:00
Mac Siri
8e4905522d
Implement KatexTag (#6237) [deploy]
* Install Katex gem

* Add Katex stylesheet and spec

* Implement basic error handling

* Remove comment

* Create more spec

* Update approval file

Changes is due to turning off RedCarpet's superscript feature

* Add instruction to _editor_guide_text

* Serve our own KaTeX css

* Apply Rhyme's suggestion

* Limit Katex's css use

* Update spec

* Add spec

Co-authored-by: Ben Halpern <bendhalpern@gmail.com>
2020-03-09 17:58:54 -04:00
Molly Struve
3fa1287c2d
remove hypershield (#6554) [deploy] 2020-03-09 13:41:20 -04:00
Molly Struve
b43df4aff7
only activate hypershield in production (#6541) [deploy] 2020-03-09 11:31:51 -04:00
Molly Struve
521cf57e63
Create Hypershield Schema to Shield Sensitive data in Production (#6461) [deploy] 2020-03-09 08:54:31 -04:00
Alex
35651b26e5
Remove Algolia from Comments (#6464) [deploy]
* Remove Algolia from comments

* Remove more references to ordered_comments index

* Remove more references to Comments Algolia index

* Fix index_worker_spec
2020-03-05 11:30:25 -08:00
Alex
bbec492844
Remove more Algolia from ClassifiedListing (#6463) [deploy]
* Remove ClassifiedListing from algolia initializer

* Remove reference to remove_from_index
2020-03-05 09:23:22 -08:00
rhymes
e27c9b77a3
Refactor Elasticsearch client access (#6307) [deploy]
* Add Transport module to Search
* Test public methods only
* Add Search::Client with exception handling
* Remove Transport and request do
* Replace SearchClient with Search::Client
2020-02-28 11:38:37 -05:00
rhymes
fc09716709
Use builtin Rails index_exists? for indices (#6293) 2020-02-26 11:05:23 -05:00
rhymes
c3e7179f5b
Bump to sprockets 4.0.0 (#6172) [deploy]
* Bump to sprocket 4.0.0

* Add manifest.js

* Link internal/layout.css

* Make it more similar to Rails 5.2 default
2020-02-24 18:16:08 -05:00
rhymes
a5588ae430
Enable strong-migrations in all environments (#6279) [deploy]
* Enable strong-migrations in production
* Disable environment checks in initializers
* Remove misleading description
2020-02-24 11:12:49 -05:00
rhymes
aa5f4b8d85
Remove leftover Stripe subscriptions code (#6243) 2020-02-24 08:33:36 -05:00