IT Support » DMZ-WEB Infrastructure

What is up right now?

~/deploy/check-sites.sh                          # every site-* container: status + HTTP answer through NPM

podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'

systemctl --user list-units 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' --no-pager

systemctl --user --failed                        # anything that died

What is defined (even if not running)?

ls ~/.config/containers/systemd/                 # the live units = the list of sites

systemctl --user list-unit-files 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*'

podman ps -a                                     # containers incl. stopped/exited

podman volume ls ; podman network ls ; podman secret ls

/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn'   # unit syntax problems

A site = net-<name>.network + db-<name>.container + site-<name>.container (+ .volume files). If the files are there and podman ps doesn't show the containers, the services are stopped or failed — see next.

Bringing a site back

Order matters (network → db → site); Requires=/After= do it for you when you start the site unit, but starting them one by one shows where it fails:

systemctl --user start net-<name>-network.service

systemctl --user start db-<name>.service   && sleep 5 && podman logs --tail 20 db-<name>

systemctl --user start site-<name>.service && sleep 5 && podman logs --tail 20 site-<name>

systemctl --user status site-<name>.service --no-pager

journalctl --user -u site-<name>.service -e --no-pager | tail -30

Typical causes: image pull failed (no egress / registry down) → podman pull <image>; secret missing → podman secret ls (a container whose Secret= is missing won't start); volume removed → the DB initialises empty on next start, stop immediately and restore from backup; Restart=always loop → podman logs.

NPM says 502 / site unreachable but container is up

podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys'   # must list net-<name>

podman network inspect net-<name> | jq '.[0].containers | map(.name)'            # db, site, nginx-proxy-manager

podman exec nginx-proxy-manager sh -c 'getent hosts site-<name>; curl -sI http://site-<name>/ | head -1'

NPM not in the network → the live nginx-proxy-manager.container lost the Network= line (see [[NPM]]) → fix the unit, cp, daemon-reload, restart nginx-proxy-manager.service. Name doesn't resolve inside NPM → NPM joined the network after it started → same restart. Restarting NPM interrupts all sites for ~10–20 s.

Does it come back after a reboot?

Yes, by design, if three things hold:

Lingering for websvcmaster (the user manager starts at boot without a login): loginctl show-user websvcmaster | grep Linger → Linger=yes (root: loginctl enable-linger websvcmaster).

Every unit has [Install] WantedBy=default.target — Quadlet units are enabled by that section, not by systemctl enable (which doesn't work on generated units). Check: grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container → must print nothing.

Restart=always in [Service] so a crash is retried.

Dry test without rebooting: systemctl --user stop site-<name>.service db-<name>.service && systemctl --user start default.target → both come back. Real test: reboot the host, then check-sites.sh after ~2 min (MariaDB + Puma + Apache need a moment).

After a host reboot — checklist

loginctl show-user websvcmaster | grep Linger     # yes

systemctl --user --failed                         # empty

~/deploy/check-sites.sh                           # all 200/30x

firewall-cmd --list-all                           # (root) 81/444 + 8181 rich rules still there

Data safety reminders

Secrets live only in ~/.local/share/containers/storage/secrets/ — back up that directory (or re-create the secrets from your password store); without snipeit-app-key the Snipe-IT DB is unreadable, without the DB secrets the containers won't start.

Volumes: podman volume export <vol> > file.tar / podman volume import <vol> file.tar, plus DB dumps ([[WordPress]], [[Redmine]], [[Snipe-IT]]).

podman system reset or removing ~/.local/share/containers deletes everything above; never run it on this host.

Source: dmz-web-podman-cheatsheet.md v1.2 in the dmz-web-deploy repo ([[Releases]]). Edit the wiki and the repo copy together.