Troubleshooting » History » Revision 1
Revision 1/4
| Next »
Redmine Admin, 10/25/2026 07:23 PM
What is up right now?
~/deploy/check-sites.sh # every site-* container: status + HTTP answer through NPM
podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'
systemctl --user list-units 'net-' 'db-' 'site-' 'nginx-proxy-manager' --no-pager
systemctl --user --failed # anything that died
What is defined (even if not running)?
ls ~/.config/containers/systemd/ # the live units = the list of sites
systemctl --user list-unit-files 'net-' 'db-' 'site-' 'nginx-proxy-manager'
podman ps -a # containers incl. stopped/exited
podman volume ls ; podman network ls ; podman secret ls
/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn' # unit syntax problems
A site = net-.network + db-.container + site-.container (+ .volume files). If the files are there and podman ps doesn't show the containers, the services are stopped or failed — see next.
Bringing a site back
Order matters (network → db → site); Requires=/After= do it for you when you start the site unit, but starting them one by one shows where it fails:
systemctl --user start net--network.service
systemctl --user start db-.service && sleep 5 && podman logs --tail 20 db-
systemctl --user start site-.service && sleep 5 && podman logs --tail 20 site-
systemctl --user status site-.service --no-pager
journalctl --user -u site-.service -e --no-pager | tail -30
Typical causes: image pull failed (no egress / registry down) → podman pull ; secret missing → podman secret ls (a container whose Secret= is missing won't start); volume removed → the DB initialises empty on next start, stop immediately and restore from backup; Restart=always loop → podman logs.
NPM says 502 / site unreachable but container is up
podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys' # must list net-
podman network inspect net- | jq '.[0].containers | map(.name)' # db, site, nginx-proxy-manager
podman exec nginx-proxy-manager sh -c 'getent hosts site-; curl -sI http://site-/ | head -1'
NPM not in the network → the live nginx-proxy-manager.container lost the Network= line (see NPM) → fix the unit, cp, daemon-reload, restart nginx-proxy-manager.service. Name doesn't resolve inside NPM → NPM joined the network after it started → same restart. Restarting NPM interrupts all sites for ~10–20 s.
Does it come back after a reboot?
Yes, by design, if three things hold:
Lingering for websvcmaster (the user manager starts at boot without a login): loginctl show-user websvcmaster | grep Linger → Linger=yes (root: loginctl enable-linger websvcmaster).
Every unit has [Install] WantedBy=default.target — Quadlet units are enabled by that section, not by systemctl enable (which doesn't work on generated units). Check: grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container → must print nothing.
Restart=always in [Service] so a crash is retried.
Dry test without rebooting: systemctl --user stop site-.service db-.service && systemctl --user start default.target → both come back. Real test: reboot the host, then check-sites.sh after ~2 min (MariaDB + Puma + Apache need a moment).
After a host reboot — checklist
loginctl show-user websvcmaster | grep Linger # yes
systemctl --user --failed # empty
~/deploy/check-sites.sh # all 200/30x
firewall-cmd --list-all # (root) 81/444 + 8181 rich rules still there
Data safety reminders
Secrets live only in ~/.local/share/containers/storage/secrets/ — back up that directory (or re-create the secrets from your password store); without snipeit-app-key the Snipe-IT DB is unreadable, without the DB secrets the containers won't start.
Volumes: podman volume export > file.tar / podman volume import file.tar, plus DB dumps (WordPress, Redmine, Snipe-IT).
podman system reset or removing ~/.local/share/containers deletes everything above; never run it on this host.
Source: dmz-web-podman-cheatsheet.md v1.2 in the dmz-web-deploy repo (Releases). Edit the wiki and the repo copy together.
Updated by Redmine Admin 12 days ago · 4 revisions