Troubleshooting » History » Revision 3
Revision 2 (Redmine Admin, 10/25/2026 07:23 PM) → Revision 3/4 (Redmine Admin, 10/26/2026 02:24 AM)
## What is up right now? ```bash ~/deploy/check-sites.sh # every site-* container: status + HTTP answer through NPM podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}' systemctl --user list-units 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' --no-pager systemctl --user --failed # anything that died ``` ## What is defined (even if not running)? ```bash ls ~/.config/containers/systemd/ # the live units = the list of sites systemctl --user list-unit-files 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' podman ps -a # containers incl. stopped/exited podman volume ls ; podman network ls ; podman secret ls /usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn' # unit syntax problems ``` A site = `net-<name>.network` + `db-<name>.container` + `site-<name>.container` (+ `.volume` files). If the files are there and `podman ps` doesn't show the containers, the services are stopped or failed — see next. ## Bringing a site back Order matters (network → db → site); `Requires=`/`After=` do it for you when you start the site unit, but starting them one by one shows where it fails: ```bash systemctl --user start net-<name>-network.service systemctl --user start db-<name>.service && sleep 5 && podman logs --tail 20 db-<name> systemctl --user start site-<name>.service && sleep 5 && podman logs --tail 20 site-<name> systemctl --user status site-<name>.service --no-pager journalctl --user -u site-<name>.service -e --no-pager | tail -30 ``` Typical causes: image pull failed (no egress / registry down) → `podman pull <image>`; secret missing → `podman secret ls` (a container whose `Secret=` is missing won't start); volume removed → the DB initialises **empty** on next start, stop immediately and restore from backup; `Restart=always` loop → `podman logs`. ## NPM says 502 / site unreachable but container is up ```bash podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys' # must list net-<name> podman network inspect net-<name> | jq '.[0].containers | map(.name)' # db, site, nginx-proxy-manager podman exec nginx-proxy-manager sh -c 'getent hosts site-<name>; curl -sI http://site-<name>/ | head -1' ``` NPM not in the network → the live `nginx-proxy-manager.container` lost the `Network=` line (see [[NPM]]) → fix the unit, `cp`, `daemon-reload`, `restart nginx-proxy-manager.service`. Name doesn't resolve inside NPM → NPM joined the network after it started → same restart. Restarting NPM interrupts **all** sites for ~10–20 s. ## Snipe-IT shows HTTP 500 — how do I find out why? The long `#67 /var/www/html/vendor/...` list is only the request path. The cause is the `production.ERROR:` line above it: ```bash podman exec site-snipeit sh -c 'grep -nE "production\.(ERROR|CRITICAL)" /var/www/html/storage/logs/laravel.log | cut -c1-400 | tail -8' ``` * `Email "${MAIL_ENV_FROM_ADDR}" does not comply with addr-spec of RFC 2822` → From/Reply-To were never set; the image's placeholder is not an address. Set `MAIL_FROM_ADDR` and `MAIL_REPLYTO_ADDR` in the unit (Snipe-IT chapter, "Mail"). It only breaks the email step; the record itself (e.g. the new user) is saved. * `Table 'snipeit.password_resets' doesn't exist` from `auth:clear-resets`, repeating every 15 minutes → the scheduler is running against a database whose tables were not created yet (pre-flight "Create database tables" not clicked). In this install the errors stopped once the tables existed. Check: `podman exec db-snipeit sh -c 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -e "show tables from snipeit like \"password%\""'`. ## Snipe-IT mail: `451 4.4.4 ... ATTR5` from Exchange Seen in the tinker test above when the recipient was external. It is **not** an authentication or TLS problem and not a Snipe-IT setting: Exchange accepted the connection, then refused a message to a recipient outside the tenant. First thing to establish is whether internal works: send the test to `sysadmin@mdsdefense.com` (passes here). If internal passes and external fails, look at the connector / relay permission in Exchange, not at the unit. If **internal also** fails, compare the From address with Redmine's (Administration → Settings → Email notifications → "Emission email address") and the host's egress IP with the connector's IP list (`curl -s https://api.ipify.org` on the host). ## Does it come back after a reboot? Yes, by design, if three things hold: 1. **Lingering** for websvcmaster (the user manager starts at boot without a login): `loginctl show-user websvcmaster | grep Linger` → `Linger=yes` (root: `loginctl enable-linger websvcmaster`). 2. Every unit has `[Install] WantedBy=default.target` — Quadlet units are *enabled by that section*, not by `systemctl enable` (which doesn't work on generated units). Check: `grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container` → must print nothing. 3. `Restart=always` in `[Service]` so a crash is retried. Dry test without rebooting: `systemctl --user stop site-<name>.service db-<name>.service && systemctl --user start default.target` → both come back. Real test: reboot the host, then `check-sites.sh` after ~2 min (MariaDB + Puma + Apache need a moment). ## After a host reboot — checklist ```bash loginctl show-user websvcmaster | grep Linger # yes systemctl --user --failed # empty ~/deploy/check-sites.sh # all 200/30x firewall-cmd --list-all # (root) 81/444 + 8181 rich rules still there ``` ## Data safety reminders * Secrets live only in `~/.local/share/containers/storage/secrets/` — back up that directory (or re-create the secrets from your password store); without `snipeit-app-key` the Snipe-IT DB is unreadable, without the DB secrets the containers won't start. * Volumes: `podman volume export <vol> > file.tar` / `podman volume import <vol> file.tar`, plus DB dumps ([[WordPress]], [[Redmine]], [[Snipe-IT]]). * `podman system reset` or removing `~/.local/share/containers` deletes **everything** above; never run it on this host. --- _Source: `dmz-web-podman-cheatsheet.md` v1.2 in the `dmz-web-deploy` repo ([[Releases]]). Edit the wiki and the repo copy together._