Homelab

A collection of Podman quadlet definitions for a personal homelab spanning multiple machines.

Podman quadlets merge systemd unit files with container definitions. This allows you to use the full power of systemd to manage your containers.

The repository is structured so each machine does a sparse checkout of only its own directory, then uses GNU Stow to symlink the relevant files into the systemd quadlet path. This keeps everything version-controlled and editable from any machine with a full checkout, while each server only sees the files it needs.

One constraint: the git repository and the symlink destinations must live on the same disk partition. Systemd resolves quadlets early during the boot sequence, cross-partition symlinks can cause units to fail to load because the links might still point to an unmounted filesystem.

Architecture

Machine	Role
janus	Public-facing VPS; TLS termination and traffic managed by Pangolin
circe	Home server; hosts internal services, reachable over VPN
hecate	Local workstation; runs development and local-only services

Each machine isolates its containers in a dedicated network named after it, using the pattern [hostname]-net

Repository structure

Each machine has a top-level directory named after its hostname. Services that need accompanying config files use a subdirectory (e.g., traefik/ holds both the container unit and its YAML configs) rather than placing everything at the top level.

A few special directories live alongside the machine directories:

• utilities/: reusable one-off containers not tied to any specific machine
• scripts/: utility scripts
• scripts/tls/: local certificate generation; generated certs are gitignored

Server setup

Clone the repository with a sparse checkout so each machine only pulls its own directory:

git clone --no-checkout <repo-url> /opt/quadlet
cd /opt/quadlet
git sparse-checkout init --cone
git sparse-checkout set <machine>/ # Sets the pattern to add to worktree
git sparse-checkout add <dir>/     # Adds a new pattern to the worktree
git sparse-checkout list           # Consult the current patterns
$EDITOR .git/info/sparse-checkout  # Or just edit the file directly
git sparse-checkout reapply        # Ensure the current config is applied

Then use Stow to symlink the machine's directory into the systemd quadlet path, treating each machine directory as a Stow package. Make sure to use the command from the repository root so Stow can find .stowrc.

stow -R <machine>

Re-run the same command whenever new files are added to the machine's directory, so the new files get symlinked too, -R should also handle stale links.

Once symlinks are in place, reload to generate the units and start the service:

# Generates the unit files from the quadlet files
systemctl daemon-reload # This is not necessary if the unit hasn't changed
# Restart the service to pick up on changes
systemctl start <service>.service # This also forces apps to pick up new configs

Secrets

Secrets are never stored in the repository. They are created on the host with podman secret create and referenced in the container unit:

Secret=secret-name,type=env,target=ENV_VAR_NAME

This injects the secret as an environment variable at runtime without it appearing in the unit file or the repository.

Use the provided script to create a secret. It prompts silently (no echo, no shell history) and saves the podman secret. Be mindful that system quadlets use Podman from the root user. Remember to run the script as root too.

./scripts/read-secret.sh secret-name

List existing secrets:

podman secret ls

Inspect a secret, including its value:

podman secret inspect --showsecret secret-name

Adding a service

Start from the template at the root of the repository:

cp template.container <machine>/<service-name>.container

Use the service name only as the filename (no version numbers).

Image: always use a fully qualified registry path. Never use short names:

# Correct
Image=docker.io/library/ubuntu:24.04

# Wrong
Image=ubuntu:latest

Network: attach to the machine's network, which follows the [hostname]-net naming convention:

Network=<machine>-net

Volumes: three conventions depending on what's being mounted:

# Persistent data (databases, state, application config)
Volume=/srv/containers/<service>/data:/data:Z

# Version-controlled config, mounted read-only from the repo
Volume=/opt/quadlet/<machine>/<service>/config.yml:/etc/service/config.yml:ro,Z

# Media libraries and large files stored on separate storage
Volume=/media/storage/containers/<service>/library:/library:Z

SELinux is enabled by default in most Red Hat based Linux distros, always include a context label on volume mounts:

• :Z for mounts used exclusively by this container
• :z for mounts shared between multiple containers

Versioning

Version pinning: always pin to an exact semver tag, never latest. For images that don't follow the typical semver pattern that most images follow, that is: major.minor.patch-tag for example, Pangolin uses such as ee-1.2.3, add an [X-Kluzo]section with a TagPattern= regex so the update tooling can parse the version:

[X-Kluzo]
TagPattern=^(?P<extra>ee)-(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)$

Kluzo is an external update tool, this config is inert without it. It has many more settings you might see in this repo or want to use, for more information check out its repository ieris19/kluzo.

Exceptions to version pinning: a handful of services are intentionally left on :latest instead of a semver pin, paired with a mechanism to keep the image from going stale.

There are two distinct update mechanisms that one can use with quadlets:

• AutoUpdate=registry (plus the matching label) relies on the podman-auto-update.timer and podman-auto-update.service units. It runs daily and only pulls newer images and restarts containers that are running at the moment the unit runs (daily at midnight with the default timer), so it fits long-running services, not ones started on demand.
• Pull=newer checks the registry and pulls if the digest for the given tag is different every time the container is started. It fits services that are only up occasionally, since it guarantees freshness at the moment they're actually used instead of depending on them happening to be running when auto-update runs. By using newer instead of always it still checks every time, but only pulls images if they are different, with always we constantly pull the same image even if it's already available.

The appropriate mechanism for each kind of service depends on how it runs:

• Hecate workstation services (e.g. open-webui) use both Pull=newer and AutoUpdate=registry. Hecate is a workstation, not a server, which means it isn't always on, so the timer alone can't be relied on to ever catch these running; even if it runs at the next available time (next boot), there is no guarantee the services are running when the timer runs. Pull=newer covers freshness at every start, and AutoUpdate=registry is kept too in case a session stays up long enough to catch a midnight refresh. The relaxed pinning trades convenience for slightly more unstable services that are easier to debug since they are only consumed locally within the machine that defines them.
• Oneshot/on-demand generators (e.g. vikunja-icons) use Type=oneshot and triggered on demand, rather than staying up as a long-running service. They exit the moment the script finishes, so they're never in a "running" state for podman-auto-update.timer to observe. AutoUpdate=registry is structurally incapable of ever firing here. Pull=newer is the correct mechanism instead.
• utilities/ dev tools (e.g. nginx-static, pg-sql, swagger-ui) use Pull=newer. These are disposable, reusable dev tooling, not production services, started on demand rather than long-lived processes running continuously. A timer is unlikely to catch them.

Monitoring:

Running uptime monitoring services is very handy, but it requires maintaining a disciplined list of endpoints to monitor. A lot of services have made their way into the repo without corresponding uptime monitoring.

In order to prevent that in the future, a system is now in place to catch and warn about these situation. The script contains a list of folders to ignore, things like scripts/ and scripts/tls/ but also machines like hecate/ which as a workstation, has no uptime checking. Every other .container in the repository should be listed in exactly one of the following places:

• An endpoint in <machine>/gatus/config/*.yml carries an x-container-file: <path>/<to>/<unit>.container field pointing at the unit it checks (relative to repo root). YAML x- prefixed keys are ignored by most parsers, like Gatus' own parser. Units listed multiple times are fine here.

  - name: Service
    group: Group
    url: "http://example.com/health/endpoint"
    x-container-file: machine/example.container
    conditions:
      - '[STATUS] == 200'
  

• The unit is listed in that machine's <machine>/.uptime-exclude, prefixed with exempt for services that deliberately have no check, e.g. agents, sidecars, one-shot jobs, Gatus itself, etc..., reason is optional, but it helps the file be self documenting.

• The unit is listed in that machine's <machine>/.uptime-exclude, prefixed with pending for services that should get a real endpoint, but have not gotten one yet. For example, new services that aren't stable enough for a monitor check or any other reason to postpone checks. It can also carry a reason to help the file be self-documenting, but it's less important here.

  # <exempt|pending>  <path>  [reason...]
  exempt  machine/example.container  This example container exposes nothing to the network.
  pending circe/linkding.container
  

Check coverage with scripts/check-monitoring-coverage.py. It fails if a unit is not referenced anywhere, or if a reference points at a unit that no longer exists. pending entries are reported but don't fail the run.

scripts/check-monitoring-coverage.py          # all machines
scripts/check-monitoring-coverage.py circe    # a specific machine