From ffab3a43fc17f4d06d241179ae9e2e26fdea47df Mon Sep 17 00:00:00 2001 From: Adrian Priestley <47989725+a-priestley@users.noreply.github.com> Date: Thu, 17 Jul 2025 05:30:57 -0230 Subject: [PATCH] Docker: Add initial configuration for project (#4419) * feat(docker): Add initial Docker configuration for project - Add .dockerignore file to ignore unnecessary files - Create Dockerfile with basic build and deployment configuration * feat(docker): Updated Docker configuration for improved security and build efficiency - Removed sensitive files from .dockerignore - Moved WORKDIR to /app in Dockerfile - Added gunicorn==23.0.0 dependency in RUN command - Created new docker-compose.yml file for service definition * feat(deployment): Implement containerized deployment configuration - Add additional environment variables for Python optimization - Update Dockerfile with new dependencies: eventlet, gevent, tornado - Create docker-compose.yml and configure services for web and nginx - Implement example configurations for web host settings and gunicorn - Establish nginx configuration for reverse proxy - Remove outdated docker-compose.yml from root directory * feat(deploy): Introduce Docker Compose configuration for multi-world deployment - Separate web service into two containers, one for main process and one for gunicorn - Update container configurations for improved security and maintainability - Remove unused volumes and network configurations * docs: Add new documentation for deploying Archipelago using containers - Document standalone image build and run process - Include example Docker Compose file for container orchestration - Provide information on services defined in the `docker-compose.yaml` file - Mention optional Enemizer feature and Git requirements * fixup! feat(docker): Updated Docker configuration for improved security and build efficiency - Removed sensitive files from .dockerignore - Moved WORKDIR to /app in Dockerfile - Added gunicorn==23.0.0 dependency in RUN command - Created new docker-compose.yml file for service definition * feat(deploy): Updated gunicorn configuration example - Adjusted worker and thread counts - Switched worker class from sync to gthread - Changed log level to info - Added example code snippet for customizing worker count * fix(deploy): Adjust concurrency settings for self-launch configuration - Reduce the number of world generators from 8 to 3 - Decrease the number of hosters from 5 to 4 * docs(deploy using containers): Improve readability, fix broken links - Update links to other documentation pages - Improve formatting for better readability - Remove unnecessary sections and files - Add note about building the image requiring a local copy of ArchipelagoMW source code * Update deploy/example_config.yaml Co-authored-by: black-sliver <59490463+black-sliver@users.noreply.github.com> * Update deploy/example_selflaunch.yaml Co-authored-by: black-sliver <59490463+black-sliver@users.noreply.github.com> * Update Dockerfile Co-authored-by: black-sliver <59490463+black-sliver@users.noreply.github.com> * Update deploy/example_selflaunch.yaml Co-authored-by: black-sliver <59490463+black-sliver@users.noreply.github.com> * fixup! Update Dockerfile * fix(Dockerfile): Update package installations to use latest versions - Remove specific version pins for git and libc6-dev - Ensure compatibility with newer package updates * feat(ci): Add GitHub Actions workflow for building and publishing Docker images - Create a new workflow for Docker image build and publish - Configure triggers for push and pull_request on main branch - Set up QEMU and Docker Buildx for multi-platform builds - Implement Docker login for GitHub Container Registry - Include Docker image metadata extraction and tagging * feat(healthcheck): Update Dockerfile and docker-compose for health checks - Add health check for the Webhost service in Dockerfile - Modify docker-compose to include a placeholder health check for multiworld service - Standardize comments and remove unnecessary lines * Revert "feat(ci): Add GitHub Actions workflow for building and publishing Docker images" This reverts commit 32a51b272627d99ca9796cbfda2e821bfdd95c70. * feat(docker): Enhance Dockerfile with Cython build stage - Add Cython builder stage for compiling speedups - Update package installation and organization for efficiency - Improve caching by copying requirements before installing - Add documentation for rootless Podman * fixup! feat(docker): Enhance Dockerfile with Cython build stage - Add Cython builder stage for compiling speedups - Update package installation and organization for efficiency - Improve caching by copying requirements before installing - Add documentation for rootless Podman --------- Co-authored-by: Adrian Priestley Co-authored-by: black-sliver <59490463+black-sliver@users.noreply.github.com> Co-authored-by: Adrian Priestley --- .dockerignore | 210 ++++++++++++++++++++++++++++++++ Dockerfile | 97 +++++++++++++++ deploy/docker-compose.yml | 61 ++++++++++ deploy/example_config.yaml | 10 ++ deploy/example_gunicorn.conf.py | 19 +++ deploy/example_nginx.conf | 64 ++++++++++ deploy/example_selflaunch.yaml | 13 ++ docs/deploy using containers.md | 91 ++++++++++++++ 8 files changed, 565 insertions(+) create mode 100644 .dockerignore create mode 100644 Dockerfile create mode 100644 deploy/docker-compose.yml create mode 100644 deploy/example_config.yaml create mode 100644 deploy/example_gunicorn.conf.py create mode 100644 deploy/example_nginx.conf create mode 100644 deploy/example_selflaunch.yaml create mode 100644 docs/deploy using containers.md diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000..982e4110 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,210 @@ +.git +.github +.run +docs +test +typings +*Client.py + +.idea +.vscode + +*_Spoiler.txt +*.bmbp +*.apbp +*.apl2ac +*.apm3 +*.apmc +*.apz5 +*.aptloz +*.apemerald +*.pyc +*.pyd +*.sfc +*.z64 +*.n64 +*.nes +*.smc +*.sms +*.gb +*.gbc +*.gba +*.wixobj +*.lck +*.db3 +*multidata +*multisave +*.archipelago +*.apsave +*.BIN +*.puml + +setups +build +bundle/components.wxs +dist +/prof/ +README.html +.vs/ +EnemizerCLI/ +/Players/ +/SNI/ +/sni-*/ +/appimagetool* +/host.yaml +/options.yaml +/config.yaml +/logs/ +_persistent_storage.yaml +mystery_result_*.yaml +*-errors.txt +success.txt +output/ +Output Logs/ +/factorio/ +/Minecraft Forge Server/ +/WebHostLib/static/generated +/freeze_requirements.txt +/Archipelago.zip +/setup.ini +/installdelete.iss +/data/user.kv +/datapackage +/custom_worlds + +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so +*.dll + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt +installer.log + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# vim editor +*.swp + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv* +env/ +venv/ +/venv*/ +ENV/ +env.bak/ +venv.bak/ +*.code-workspace +shell.nix + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# Cython intermediates +_speedups.c +_speedups.cpp +_speedups.html + +# minecraft server stuff +jdk*/ +minecraft*/ +minecraft_versions.json +!worlds/minecraft/ + +# pyenv +.python-version + +#undertale stuff +/Undertale/ + +# OS General Files +.DS_Store +.AppleDouble +.LSOverride +Thumbs.db +[Dd]esktop.ini diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..0ed61c03 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,97 @@ +# hadolint global ignore=SC1090,SC1091 + +# Source +FROM scratch AS release +WORKDIR /release +ADD https://github.com/Ijwu/Enemizer/releases/latest/download/ubuntu.16.04-x64.zip Enemizer.zip + +# Enemizer +FROM alpine:3.21 AS enemizer +ARG TARGETARCH +WORKDIR /release +COPY --from=release /release/Enemizer.zip . + +# No release for arm architecture. Skip. +RUN if [ "$TARGETARCH" = "amd64" ]; then \ + apk add unzip=6.0-r15 --no-cache && \ + unzip -u Enemizer.zip -d EnemizerCLI && \ + chmod -R 777 EnemizerCLI; \ + else touch EnemizerCLI; fi + +# Cython builder stage +FROM python:3.12 AS cython-builder + +WORKDIR /build + +# Copy and install requirements first (better caching) +COPY requirements.txt WebHostLib/requirements.txt + +RUN pip install --no-cache-dir -r \ + WebHostLib/requirements.txt \ + setuptools + +COPY _speedups.pyx . +COPY intset.h . + +RUN cythonize -b -i _speedups.pyx + +# Archipelago +FROM python:3.12-slim AS archipelago +ARG TARGETARCH +ENV VIRTUAL_ENV=/opt/venv +ENV PYTHONUNBUFFERED=1 +WORKDIR /app + +# Install requirements +# hadolint ignore=DL3008 +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + git \ + gcc=4:12.2.0-3 \ + libc6-dev \ + libtk8.6=8.6.13-2 \ + g++=4:12.2.0-3 \ + curl && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +# Create and activate venv +RUN python -m venv $VIRTUAL_ENV; \ + . $VIRTUAL_ENV/bin/activate + +# Copy and install requirements first (better caching) +COPY WebHostLib/requirements.txt WebHostLib/requirements.txt + +RUN pip install --no-cache-dir -r \ + WebHostLib/requirements.txt \ + gunicorn==23.0.0 + +COPY . . + +COPY --from=cython-builder /build/*.so ./ + +# Run ModuleUpdate +RUN python ModuleUpdate.py -y + +# Purge unneeded packages +RUN apt-get purge -y \ + git \ + gcc \ + libc6-dev \ + g++ && \ + apt-get autoremove -y + +# Copy necessary components +COPY --from=enemizer /release/EnemizerCLI /tmp/EnemizerCLI + +# No release for arm architecture. Skip. +RUN if [ "$TARGETARCH" = "amd64" ]; then \ + cp /tmp/EnemizerCLI EnemizerCLI; \ + fi; \ + rm -rf /tmp/EnemizerCLI + +# Define health check +HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \ + CMD curl -f http://localhost:${PORT:-80} || exit 1 + +ENTRYPOINT [ "python", "WebHost.py" ] diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml new file mode 100644 index 00000000..14726674 --- /dev/null +++ b/deploy/docker-compose.yml @@ -0,0 +1,61 @@ +services: + multiworld: + # Build only once. Web service uses the same image build + build: + context: .. + # Name image for use in web service + image: archipelago-base + # Use locally-built image + pull_policy: never + # Launch main process without website hosting (config override) + entrypoint: python WebHost.py --config_override selflaunch.yaml + volumes: + # Mount application volume + - app_volume:/app + + # Mount configs + - ./example_config.yaml:/app/config.yaml + - ./example_selflaunch.yaml:/app/selflaunch.yaml + + # Expose on host network for access to dynamically mapped ports + network_mode: host + + # No Healthcheck in place yet for multiworld + healthcheck: + test: ["NONE"] + web: + # Use image build by multiworld service + image: archipelago-base + # Use locally-built image + pull_policy: never + # Launch gunicorn targeting WebHost application + entrypoint: gunicorn -c gunicorn.conf.py + volumes: + # Mount application volume + - app_volume:/app + + # Mount configs + - ./example_config.yaml:/app/config.yaml + - ./example_gunicorn.conf.py:/app/gunicorn.conf.py + environment: + # Bind gunicorn on 8000 + - PORT=8000 + + nginx: + image: nginx:stable-alpine + volumes: + # Mount application volume + - app_volume:/app + + # Mount config + - ./example_nginx.conf:/etc/nginx/nginx.conf + ports: + # Nginx listening internally on port 80 -- mapped to 8080 on host + - 8080:80 + depends_on: + - web + +volumes: + # Share application directory amongst multiworld and web services + # (for access to log files and the like), and nginx (for static files) + app_volume: diff --git a/deploy/example_config.yaml b/deploy/example_config.yaml new file mode 100644 index 00000000..d74f7f23 --- /dev/null +++ b/deploy/example_config.yaml @@ -0,0 +1,10 @@ +# Refer to ../docs/webhost configuration sample.yaml + +# We'll be hosting VIA gunicorn +SELFHOST: false +# We'll start a separate process for rooms and generators +SELFLAUNCH: false + +# Host Address. This is the address encoded into the patch that will be used for client auto-connect. +# Set as your local IP (192.168.x.x) to serve over LAN. +HOST_ADDRESS: localhost diff --git a/deploy/example_gunicorn.conf.py b/deploy/example_gunicorn.conf.py new file mode 100644 index 00000000..49f153df --- /dev/null +++ b/deploy/example_gunicorn.conf.py @@ -0,0 +1,19 @@ +workers = 2 +threads = 2 +wsgi_app = "WebHost:get_app()" +accesslog = "-" +access_log_format = ( + '%({x-forwarded-for}i)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"' +) +worker_class = "gthread" # "sync" | "gthread" +forwarded_allow_ips = "*" +loglevel = "info" + +""" +You can programatically set values. +For example, set number of workers to half of the cpu count: + +import multiprocessing + +workers = multiprocessing.cpu_count() / 2 +""" diff --git a/deploy/example_nginx.conf b/deploy/example_nginx.conf new file mode 100644 index 00000000..b0c0e8e5 --- /dev/null +++ b/deploy/example_nginx.conf @@ -0,0 +1,64 @@ +worker_processes 1; + +user nobody nogroup; +# 'user nobody nobody;' for systems with 'nobody' as a group instead +error_log /var/log/nginx/error.log warn; +pid /var/run/nginx.pid; + +events { + worker_connections 1024; # increase if you have lots of clients + accept_mutex off; # set to 'on' if nginx worker_processes > 1 + # 'use epoll;' to enable for Linux 2.6+ + # 'use kqueue;' to enable for FreeBSD, OSX + use epoll; +} + +http { + include mime.types; + # fallback in case we can't determine a type + default_type application/octet-stream; + access_log /var/log/nginx/access.log combined; + sendfile on; + + upstream app_server { + # fail_timeout=0 means we always retry an upstream even if it failed + # to return a good HTTP response + + # for UNIX domain socket setups + # server unix:/tmp/gunicorn.sock fail_timeout=0; + + # for a TCP configuration + server web:8000 fail_timeout=0; + } + + server { + # use 'listen 80 deferred;' for Linux + # use 'listen 80 accept_filter=httpready;' for FreeBSD + listen 80 deferred; + client_max_body_size 4G; + + # set the correct host(s) for your site + # server_name example.com www.example.com; + + keepalive_timeout 5; + + # path for static files + root /app/WebHostLib; + + location / { + # checks for static file, if not found proxy to app + try_files $uri @proxy_to_app; + } + + location @proxy_to_app { + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + # we don't want nginx trying to do something clever with + # redirects, we set the Host: header above already. + proxy_redirect off; + + proxy_pass http://app_server; + } + } +} diff --git a/deploy/example_selflaunch.yaml b/deploy/example_selflaunch.yaml new file mode 100644 index 00000000..41149dc1 --- /dev/null +++ b/deploy/example_selflaunch.yaml @@ -0,0 +1,13 @@ +# Refer to ../docs/webhost configuration sample.yaml + +# We'll be hosting VIA gunicorn +SELFHOST: false +# Start room and generator processes +SELFLAUNCH: true +JOB_THRESHOLD: 0 + +# Maximum concurrent world gens +GENERATORS: 3 + +# Rooms will be spread across multiple processes +HOSTERS: 4 diff --git a/docs/deploy using containers.md b/docs/deploy using containers.md new file mode 100644 index 00000000..bb779001 --- /dev/null +++ b/docs/deploy using containers.md @@ -0,0 +1,91 @@ +# Deploy Using Containers + +If you just want to play and there is a compiled version available on the [Archipelago releases page](https://github.com/ArchipelagoMW/Archipelago/releases), use that version. +To build the full Archipelago software stack, refer to [Running From Source](running%20from%20source.md). +Follow these steps to build and deploy a containerized instance of the web host software, optionally integrating [Gunicorn](https://gunicorn.org/) WSGI HTTP Server running behind the [nginx](https://nginx.org/) reverse proxy. + + +## Building the Container Image + +What you'll need: + * A container runtime engine such as: + * [Docker](https://www.docker.com/) + * [Podman](https://podman.io/) + * For running with rootless podman, you need to ensure all ports used are usable rootless, by default ports less than 1024 are root only. See [the official tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md) for details. + +Starting from the root repository directory, the standalone Archipelago image can be built and run with the command: +`docker build -t archipelago .` +Or: +`podman build -t archipelago .` + +It is recommended to tag the image using `-t` to more easily identify the image and run it. + + +## Running the Container + +Running the container can be performed using: +`docker run --network host archipelago` +Or: +`podman run --network host archipelago` + +The Archipelago web host requires access to multiple ports in order to host game servers simultaneously. To simplify configuration for this purpose, specify `--network host`. + +Given the default configuration, the website will be accessible at the hostname/IP address (localhost if run locally) of the machine being deployed to, at port 80. It can be configured by creating a YAML file and mapping a volume to the container when running initially: +`docker run archipelago --network host -v /path/to/config.yaml:/app/config.yaml` +See `docs/webhost configuration sample.yaml` for example. + + +## Using Docker Compose + +An example [docker compose](../deploy/docker-compose.yml) file can be found in [deploy](../deploy), along with example configuration files used by the services it orchestrates. Using these files as-is will spin up two separate archipelago containers with special modifications to their runtime arguments, in addition to deploying an `nginx` reverse proxy container. + +To deploy in this manner, from the ["deploy"](../deploy) directory, run: +`docker compose up -d` + +### Services + +The `docker-compose.yaml` file defines three services: + * multiworld: + * Executes the main `WebHost` process, using the [example config](../deploy/example_config.yaml), and overriding with a secondary [selflaunch example config](../deploy/example_selflaunch.yaml). This is because we do not want to launch the website through this service. + * web: + * Executes `gunicorn` using its [example config](../deploy/example_gunicorn.conf.py), which will bind it to the `WebHost` application, in effect launching it. + * We mount the main [config](../deploy/example_config.yaml) without an override to specify that we are launching the website through this service. + * No ports are exposed through to the host. + * nginx: + * Serves as a reverse proxy with `web` as its upstream. + * Directs all HTTP traffic from port 80 to the upstream service. + * Exposed to the host on port 8080. This is where we can reach the website. + +### Configuration + +As these are examples, they can be copied and modified. For instance setting the value of `HOST_ADDRESS` in [example config](../deploy/example_config.yaml) to host machines local IP address, will expose the service to its local area network. + +The configuration files may be modified to handle for machine-specific optimizations, such as: + * Web pages responding too slowly + * Edit [the gunicorn config](../deploy/example_gunicorn.conf.py) to increase thread and/or worker count. + * Game generation stalls + * Increase the generator count in [selflaunch config](../deploy/example_selflaunch.yaml) + * Gameplay lags + * Increase the hoster count in [selflaunch config](../deploy/example_selflaunch.yaml) + +Changes made to `docker-compose.yaml` can be applied by running `docker compose up -d`, while those made to other files are applied by running `docker compose restart`. + + +## Windows + +It is possible to carry out these deployment steps on Windows under [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install). + + +## Optional: A Link to the Past Enemizer + +Only required to generate seeds that include A Link to the Past with certain options enabled. You will receive an +error if it is required. +Enemizer can be enabled on `x86_64` platform architecture, and is included in the image build process. Enemizer requires a version 1.0 Japanese "Zelda no Densetsu" `.sfc` rom file to be placed in the application directory: +`docker run archipelago -v "/path/to/zelda.sfc:/app/Zelda no Densetsu - Kamigami no Triforce (Japan).sfc"`. +Enemizer is not currently available for `aarch64`. + + +## Optional: Git + +Building the image requires a local copy of the ArchipelagoMW source code. +Refer to [Running From Source](running%20from%20source.md#optional-git).