This "rebases" our application images off of the new Ruby base image
added in #19632, and fixes numerous problems and quirks with how the
images were built along the way. Notably:
- Issues where layers attempted to delete files in prior layers have
been resolved (this caused build failures on some Docker filesystem
drivers, notably overlay2).
- Bundler is no longer allowed to deviate from or modify the lockfile
(`BUNDLE_FROZEN` is now `true`).
- `git(1)` is no longer required to live inside the container and `.git`
is no longer required to be copied into the Docker build context, as
these were only used to calculate `FOREM_BUILD_SHA`, which is now
passed in as a Build Argument to the container build context.
- The entire source tree is no longer `chmod` in one giant swing, which
ran so long on my system (as just one example) that I gave up after
15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is
used more heavily and ensures the `APP_USER` will have access to the
requisite files.
This new container image appears to build successfully for
`linux/arm64`, which refs (but does not complete) #19626. Currently,
such builds aren't automated , and must be built on a developer
workstation. For example:
```sh
docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD)
```
In the meantime, the existing `linux/amd64`-only BuildKite scripts have
been updated to allow this PR to merge as a separate unit, and CI
refactors to enable the multiarch builds of `linux/arm64,linux/amd64`
can come later when more time is available.
This is one of several blockers on the path to getting #19603 merged.
The next step in that chronology will be rebasing that work on top of
this work, which *should* be, on the containerization side, as
straightforward as bumping `Containerfile.base` to reference the new
upstream image, rebuilding the base container, and then bumping the
reference in `Containerfile`.
* update to node version 16
* Remove canvas compiled node objects
* use f35 testing image as builder/base
* Update production builder base image to use Fedora 35
I missed this (it's not immediately obvious that there are 2 base
images declared in this file, one called builder, one called
production, this seems like it could be refactored to lift that
out (give it a name, a cmd, and do nothing else) `as base` perhaps,
while keeping separate install processes for testing- and pr-/production images.
* Add libpq dependency in production
We need this (if not the -devel header file, at least the library) to
start pg_ext.so
I think this might have been working because of the --cache-from
options when building in the build container script?
* Update .gitpod.dockerfile
* Add temporary cleanup for upgrade to bin/setup
This has the undesirable effect of requiring a yarn reinstall.
It would be better if there were a smart check to confirm the version
of the canvas.node file matched the node version or did not (so we
only do this as needed, rather than on every setup invocation until
this is removed from the code).
* Use check-files rather than force when rebuilding canvas
Optimal situation would be a `rebuild` command in yarn (I believe npm
has this option) to recompile canvas (all that's needed) rather than
fetch, install, and build.
Co-authored-by: Michael Kohl <me@citizen428.net>
We started seeing buildkite failures building after the rails 6.1
upgrade
While the bundler version probably isn't important by itself,
changing the containerfile invalidates any cached layers used to speed
up the build.
* Use premade development image.
* Use prebuilt container for podman-compose too!
* Fix bundle bundle_finished path.
* Fix docker-compose to work with the Multistage Containerfile
Add root user for Forem containers, fix bundle_finished path and make
webpacker wait for the bundle to finish.
Switch Redis to redis:6.0.9-alpine
* Remove build steps and fix bundle path.
* Merge dev entrypoint into the main one.
* Use merged entrypoint.sh, remove old entrypoint directives, and fix a dockerize bug.
* Make sure unset BUNDLE_WITHOUT and remove without in bundle config.
* Unset BUNDLE_WITHOUT="development:test" as a global ENV.
Also ensure that we remove the without directives for development and testing.
This will make sure that the right gems are installed for each environment.
Removed some unneeded COPYs and remove a commented out RUN.
* Fix ENTRYPOINT on development and testing containers.
* Fix entrypoint and command on rails container.
* Ignore bundle_finished file.
* Fix REDIS_ env vars and adjust podman-compose file.
* Change bundle_finished path and update Yarn docker-compose command.
* Fix bundle issues with the Multistage container
and adjusts a few other things:
- Disable ruby docs on bundler gem install
- Fix the bundle config
- chmod +x on scripts/build_containers.sh
* Add app/assets back into the builder image.
* Looks like running:
yarn install --frozen-lockfile \
&& RAILS_ENV=test NODE_ENV=test bundle exec rails webpacker:compile'
is super busted.
https://buildkite.com/forem/build-containers/builds/48#97451356-6552-40a1-8333-ac6fc0fccc3a
so let's just comment that out for now.
* WIP multi-stage container build.
* Move to a multistage build process in the Containerfile.
* Enable testing container
* Finalize the multistage build script and Containerfile. Ooof!
I got blocked for a bit with this bugger:
Step 27/32 : COPY --from=builder --chown=${APP_USER}:${APP_USER} ${APP_HOME} ${APP_HOME}
failed to copy files: failed to copy directory: Error processing tar file(exit status 1): Container ID 100999 cannot be mapped to a host ID
but https://github.com/phusion/passenger-docker/issues/235#issuecomment-636318827
was a good shove in the right direction!
* Fix bugs on container build script.
* Adjust compose files to use multistage builds.
* Add in Buildkite pipeline for building containers.
* Update CODEOWNERS to claim ownership over things that Systems Eng cares about
* Update to Ruby 2.7.2
* Update remaining files
* Remove VERBOSE from development.rb and test.rb
* Update Ruby version for dependabot and docs
* Readd accidentally removed code
* Create entrypoint-dev.sh to be used with devel workflows with docker/podman compose
and adjust the compose files to use it.
* Adjust the entrypoint script so we have better control over when we run
rake assets:precompile and rake app_initializer:setup.
* Copy vendor/cache so we can bundle install from cached gems instead of
fetching them from the Internet.
* Run rake assets:precompile in the container build.
* Add in clobber and clean tasks to entrypoint.
In the past it was always useful to have meta data within regards to the SHA and
build date/time. This adds in two new env vars that are written to disk as flat
files and then exported to environment variables when the container starts up.
This motivation comes from this comment:
https://github.com/forem/forem/pull/9987#discussion_r476123829
This post on https://forem.dev/avalerionv/difficulty-installing-forem-on-ubuntu-18-04-tls-4eh6
talks about issues running Forem on Linux. Digging into the issue it seems that
podman-compose does not pass the --userns=keep-id flag correctly to podman so
files and directories mounted as root and not as the user. This causes permission
denied errors.
This PR moves the away from the dev specific naming to forem w/r/t container
things. Also moves to the Forem quay.io org.
* Move to Forem from DEV
* Add in assets and packs directories.
* Set --disable-march-tune-native when building sassc.
https://github.com/sass/sassc-ruby/issues/146
* Let's ignore the .gems folder when building a Container.
* Let's purge the dnf cache!
* Move from Alpine Linux to Fedora Linux.
This changes the base container image away from Alpine Linux to
Fedora Linux to address some musl libc issues with some of our gems.
It also moved the main file to a Containerfile and symlinks the Dockerfile
to it. This makes our container setup less Docker-centric as Linux users
most likely will be using Podman as their container runtime.
Lastly, it moves the WORKDIR from /usr/src/app to /opt/apps/devto.
The Linux FHS states that /opt is the spot for Optional application
software packages and /usr/src is for kernel source code.
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
/opt Optional application software packages
/usr/src Source code, e.g., the kernel source code with its header files.
Also, if SELinux becomes a thing in our future, moving this makes it
easier to manage contexts out of /opt rather than /usr/src.
* Adjust the Containerfile a bit to add some missing packages, make it
more generic and move env vars to the docker-compose.
* Move the Dockerfile to a symlink to the Containerfile.
We need to be able to support more than just Docker for a container
runtime. Moving everything to a Containerfile and symlinking the
Dockerfile helps users run runtimes such as Podman.
* Add in new entrypoint files and fix the app path in docker-entrypoint.sh
* Refactor the docker-compose.yml file so it doesn't build the main
application container three times in a row. We can use the same
container for web, webpacker, and sidekiq.
Also make the volume names very explicit on what their contents and add
in SELinux context support with :Z
* Fix ELASTICSEARCH_URL.
* Remove the absolute path on ip binary and add in iproute to Containerfile.
* Symlink the Dockerfile to Containerfile and add in a container-compose.yml file
for usage with podman-compose. We are waiting for this issue to get resolved
https://github.com/containers/libpod/issues/6153
and for podman-compose to mature a bit more. A user using podman-compose can use
podman-compose -f container-compose.yml
to launch the DEV container stack with Podman.
* Update the .gitignore file to reflect the new container volumes.
* Fix the entrypoint script on the Containerfile and clean up a bunch of things.
* Rework the Containerfile to prep it to run as a non-root user. We have to wait
for this issue to get fixed:
https://github.com/containers/libpod/issues/6153
for Linux users and then we can uncomment the USER line so we are running Rails
as a non-root user. :toot:
* This reworks the compose files so each task that is needed to start the app has
a correct wait command with dockerize. It also adds in containers for doing
yarn and bundle things for development. Since we mount the code directory inside
the container we lose our pre-containerized gems and node_modules.
This means users can run:
Linux
podman-compose -f container-compose.yml up
Mac
docker-compose up
and it should correctly build the app stack with containers!
* Clean up old container related things.
* Add in some container pre-reqs and my name to the "Core team" section! :toot:
* Adjust the seed and sidekiq compose entries so they do not use the web entrypoint
and set the REDIS_URL and REDIS_SESSIONS_URL env vars for seed.
Add in some echos to the entrypoint.sh.
Also set docker-compose to use :delegated on mount points to speed things up.
https://docs.docker.com/storage/bind-mounts/#configure-mount-consistency-for-macos
* Just call yarn install --dev on the yarn container.
* Update documentation to support Docker and Podman as Container Engines and
move the page from docker to containers to reflect the fact not all users run
their containers via Docker.
There are dozens of us... DOZENS!
* Set --local on bundle config so it just impacts this Ruby app.
* Renamed bin/docker-setup to bin/container-setup to reflect the fact that that
we can use Podman as a container engine. I also refactored it so it does some
basic pre-flight checks for docker and docker-compose or podman and
podman-compose to make sure users have a container engine installed.
* Set cache_all_platforms true for bundler.
* Bump docker-compose dockerize -timeout to 45min for slower macOS hardware.
* Move to the consolidated app_initializer:setup rake task for bootstrapping the
app.
* Update docs/installation/readme.md
