What is up right now?¶

~/deploy/check-sites.sh                          # every site-* container: status + HTTP answer through NPM
podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'
systemctl --user list-units 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' --no-pager
systemctl --user --failed                        # anything that died

What is defined (even if not running)?¶

ls ~/.config/containers/systemd/                 # the live units = the list of sites
systemctl --user list-unit-files 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*'
podman ps -a                                     # containers incl. stopped/exited
podman volume ls ; podman network ls ; podman secret ls
/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn'   # unit syntax problems

A site = net-<name>.network + db-<name>.container + site-<name>.container
(+ .volume files). If the files are there and podman ps doesn't show the
containers, the services are stopped or failed — see next.

Bringing a site back¶

Order matters (network → db → site); Requires=/After= do it for you when
you start the site unit, but starting them one by one shows where it fails:

systemctl --user start net-<name>-network.service
systemctl --user start db-<name>.service   && sleep 5 && podman logs --tail 20 db-<name>
systemctl --user start site-<name>.service && sleep 5 && podman logs --tail 20 site-<name>
systemctl --user status site-<name>.service --no-pager
journalctl --user -u site-<name>.service -e --no-pager | tail -30

Typical causes: image pull failed (no egress / registry down) → podman pull <image>;
secret missing → podman secret ls (a container whose Secret= is missing
won't start); volume removed → the DB initialises empty on next start, stop
immediately and restore from backup; Restart=always loop → podman logs.

NPM says 502 / site unreachable but container is up¶

podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys'   # must list net-<name>
podman network inspect net-<name> | jq '.[0].containers | map(.name)'            # db, site, nginx-proxy-manager
podman exec nginx-proxy-manager sh -c 'getent hosts site-<name>; curl -sI http://site-<name>/ | head -1'

NPM not in the network → the live nginx-proxy-manager.container lost the
Network= line (see NPM) → fix the unit, cp, daemon-reload,
restart nginx-proxy-manager.service. Name doesn't resolve inside NPM → NPM
joined the network after it started → same restart. Restarting NPM interrupts
all sites for ~10–20 s.

Does it come back after a reboot?¶

Yes, by design, if three things hold:

1. Lingering for websvcmaster (the user manager starts at boot without a login):
   loginctl show-user websvcmaster | grep Linger → Linger=yes (root: loginctl enable-linger websvcmaster).
2. Every unit has [Install] WantedBy=default.target — Quadlet units are
   enabled by that section, not by systemctl enable (which doesn't work on
   generated units). Check: grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container
   → must print nothing.
3. Restart=always in [Service] so a crash is retried.

Dry test without rebooting: systemctl --user stop site-<name>.service db-<name>.service && systemctl --user start default.target
→ both come back. Real test: reboot the host, then check-sites.sh after ~2 min
(MariaDB + Puma + Apache need a moment).

After a host reboot — checklist¶

loginctl show-user websvcmaster | grep Linger     # yes
systemctl --user --failed                         # empty
~/deploy/check-sites.sh                           # all 200/30x
firewall-cmd --list-all                           # (root) 81/444 + 8181 rich rules still there

Data safety reminders¶

• Secrets live only in ~/.local/share/containers/storage/secrets/ — back up
  that directory (or re-create the secrets from your password store); without
  snipeit-app-key the Snipe-IT DB is unreadable, without the DB secrets the
  containers won't start.
• Volumes: podman volume export <vol> > file.tar / podman volume import <vol> file.tar,
  plus DB dumps (WordPress, Redmine, Snipe-IT).
• podman system reset or removing ~/.local/share/containers deletes
  everything above; never run it on this host.

Source: dmz-web-podman-cheatsheet.md v1.2 in the dmz-web-deploy repo (Releases). Edit the wiki and the repo copy together.