Troubleshooting » History » Revision 2
Revision 1 (Redmine Admin, 10/25/2026 07:23 PM) → Revision 2/4 (Redmine Admin, 10/25/2026 07:23 PM)
## What is up right now? ```bash ~/deploy/check-sites.sh # every site-* container: status + HTTP answer through NPM podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}' systemctl --user list-units 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' --no-pager systemctl --user --failed # anything that died ``` ## What is defined (even if not running)? ```bash ls ~/.config/containers/systemd/ # the live units = the list of sites systemctl --user list-unit-files 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' podman ps -a # containers incl. stopped/exited podman volume ls ; podman network ls ; podman secret ls /usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn' # unit syntax problems ``` A site = `net-<name>.network` net-<name>.network + `db-<name>.container` db-<name>.container + `site-<name>.container` site-<name>.container (+ `.volume` .volume files). If the files are there and `podman ps` podman ps doesn't show the containers, the services are stopped or failed — see next. ## Bringing a site back Order matters (network → db → site); `Requires=`/`After=` Requires=/After= do it for you when you start the site unit, but starting them one by one shows where it fails: ```bash systemctl --user start net-<name>-network.service systemctl --user start db-<name>.service && sleep 5 && podman logs --tail 20 db-<name> systemctl --user start site-<name>.service && sleep 5 && podman logs --tail 20 site-<name> systemctl --user status site-<name>.service --no-pager journalctl --user -u site-<name>.service -e --no-pager | tail -30 ``` Typical causes: image pull failed (no egress / registry down) → `podman podman pull <image>`; <image>; secret missing → `podman podman secret ls` ls (a container whose `Secret=` Secret= is missing won't start); volume removed → the DB initialises **empty** empty on next start, stop immediately and restore from backup; `Restart=always` Restart=always loop → `podman logs`. podman logs. ## NPM says 502 / site unreachable but container is up ```bash podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys' # must list net-<name> podman network inspect net-<name> | jq '.[0].containers | map(.name)' # db, site, nginx-proxy-manager podman exec nginx-proxy-manager sh -c 'getent hosts site-<name>; curl -sI http://site-<name>/ | head -1' ``` NPM not in the network → the live `nginx-proxy-manager.container` nginx-proxy-manager.container lost the `Network=` Network= line (see [[NPM]]) → fix the unit, `cp`, `daemon-reload`, `restart nginx-proxy-manager.service`. cp, daemon-reload, restart nginx-proxy-manager.service. Name doesn't resolve inside NPM → NPM joined the network after it started → same restart. Restarting NPM interrupts **all** all sites for ~10–20 s. ## Does it come back after a reboot? Yes, by design, if three things hold: 1. **Lingering** Lingering for websvcmaster (the user manager starts at boot without a login): `loginctl loginctl show-user websvcmaster | grep Linger` Linger → `Linger=yes` Linger=yes (root: `loginctl loginctl enable-linger websvcmaster`). websvcmaster). 2. Every unit has `[Install] WantedBy=default.target` [Install] WantedBy=default.target — Quadlet units are *enabled enabled by that section*, section, not by `systemctl enable` systemctl enable (which doesn't work on generated units). Check: `grep grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container` ~/.config/containers/systemd/*.container → must print nothing. 3. `Restart=always` Restart=always in `[Service]` [Service] so a crash is retried. Dry test without rebooting: `systemctl systemctl --user stop site-<name>.service db-<name>.service && systemctl --user start default.target` default.target → both come back. Real test: reboot the host, then `check-sites.sh` check-sites.sh after ~2 min (MariaDB + Puma + Apache need a moment). ## After a host reboot — checklist ```bash loginctl show-user websvcmaster | grep Linger # yes systemctl --user --failed # empty ~/deploy/check-sites.sh # all 200/30x firewall-cmd --list-all # (root) 81/444 + 8181 rich rules still there ``` ## Data safety reminders * Secrets live only in `~/.local/share/containers/storage/secrets/` ~/.local/share/containers/storage/secrets/ — back up that directory (or re-create the secrets from your password store); without `snipeit-app-key` snipeit-app-key the Snipe-IT DB is unreadable, without the DB secrets the containers won't start. * Volumes: `podman podman volume export <vol> > file.tar` file.tar / `podman podman volume import <vol> file.tar`, file.tar, plus DB dumps ([[WordPress]], [[Redmine]], [[Snipe-IT]]). * `podman podman system reset` reset or removing `~/.local/share/containers` ~/.local/share/containers deletes **everything** everything above; never run it on this host. --- _Source: `dmz-web-podman-cheatsheet.md` Source: dmz-web-podman-cheatsheet.md v1.2 in the `dmz-web-deploy` dmz-web-deploy repo ([[Releases]]). Edit the wiki and the repo copy together._ together.