Chapter 7 of the dmz-web documentation · headings are numbered like the cheat sheets · index: Wiki

• Table of contents
• 7.1 What is up right now?
• 7.2 What is defined (even if not running)?
• 7.3 Bringing a site back
• 7.4 NPM says 502 / site unreachable but container is up
• 7.5 Snipe-IT shows HTTP 500 — how do I find out why?
• 7.6 Snipe-IT mail: 451 4.4.4 ... ATTR5 from Exchange
• 7.7 Does it come back after a reboot?
• 7.8 After a host reboot — checklist
• 7.9 Data safety reminders

7.1 What is up right now?¶

~/deploy/check-sites.sh                          # every site-* container: status + HTTP answer through NPM
podman ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'
systemctl --user list-units 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*' --no-pager
systemctl --user --failed                        # anything that died

7.2 What is defined (even if not running)?¶

ls ~/.config/containers/systemd/                 # the live units = the list of sites
systemctl --user list-unit-files 'net-*' 'db-*' 'site-*' 'nginx-proxy-manager*'
podman ps -a                                     # containers incl. stopped/exited
podman volume ls ; podman network ls ; podman secret ls
/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -iE 'error|warn'   # unit syntax problems

A site = net-<name>.network + db-<name>.container + site-<name>.container
(+ .volume files). If the files are there and podman ps doesn't show the
containers, the services are stopped or failed — see next.

7.3 Bringing a site back¶

Order matters (network → db → site); Requires=/After= do it for you when
you start the site unit, but starting them one by one shows where it fails:

systemctl --user start net-<name>-network.service
systemctl --user start db-<name>.service   && sleep 5 && podman logs --tail 20 db-<name>
systemctl --user start site-<name>.service && sleep 5 && podman logs --tail 20 site-<name>
systemctl --user status site-<name>.service --no-pager
journalctl --user -u site-<name>.service -e --no-pager | tail -30

Typical causes: image pull failed (no egress / registry down) → podman pull <image>;
secret missing → podman secret ls (a container whose Secret= is missing
won't start); volume removed → the DB initialises empty on next start, stop
immediately and restore from backup; Restart=always loop → podman logs.

7.4 NPM says 502 / site unreachable but container is up¶

podman inspect nginx-proxy-manager | jq '.[0].NetworkSettings.Networks | keys'   # must list net-<name>
podman network inspect net-<name> | jq '.[0].containers | map(.name)'            # db, site, nginx-proxy-manager
podman exec nginx-proxy-manager sh -c 'getent hosts site-<name>; curl -sI http://site-<name>/ | head -1'

NPM not in the network → the live nginx-proxy-manager.container lost the
Network= line (see NPM) → fix the unit, cp, daemon-reload,
restart nginx-proxy-manager.service. Name doesn't resolve inside NPM → NPM
joined the network after it started → same restart. Restarting NPM interrupts
all sites for ~10–20 s.

7.5 Snipe-IT shows HTTP 500 — how do I find out why?¶

The long #67 /var/www/html/vendor/... list is only the request path. The cause is
the production.ERROR: line above it:

podman exec site-snipeit sh -c 'grep -nE "production\.(ERROR|CRITICAL)" /var/www/html/storage/logs/laravel.log | cut -c1-400 | tail -8'

• Email "${MAIL_ENV_FROM_ADDR}" does not comply with addr-spec of RFC 2822 → From/Reply-To
  were never set; the image's placeholder is not an address. Set MAIL_FROM_ADDR and
  MAIL_REPLYTO_ADDR in the unit (Snipe-IT chapter, "Mail"). It only breaks the email step;
  the record itself (e.g. the new user) is saved.
• Table 'snipeit.password_resets' doesn't exist from auth:clear-resets, repeating every
  15 minutes → the scheduler is running against a database whose tables were not created yet
  (pre-flight "Create database tables" not clicked). In this install the errors stopped once
  the tables existed. Check: podman exec db-snipeit sh -c 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -e "show tables from snipeit like \"password%\""'.

7.6 Snipe-IT mail: 451 4.4.4 ... ATTR5 from Exchange¶

Seen in the tinker test above when the recipient was external. It is not an authentication
or TLS problem and not a Snipe-IT setting: Exchange accepted the connection, then refused a
message to a recipient outside the tenant. First thing to establish is whether internal works:
send the test to sysadmin@mdsdefense.com (passes here). If internal passes and external
fails, look at the connector / relay permission in Exchange, not at the unit. If internal
also fails, compare the From address with Redmine's (Administration → Settings → Email
notifications → "Emission email address") and the host's egress IP with the connector's IP list
(curl -s https://api.ipify.org on the host).

7.7 Does it come back after a reboot?¶

Yes, by design, if three things hold:

1. Lingering for websvcmaster (the user manager starts at boot without a login):
   loginctl show-user websvcmaster | grep Linger → Linger=yes (root: loginctl enable-linger websvcmaster).
2. Every unit has [Install] WantedBy=default.target — Quadlet units are
   enabled by that section, not by systemctl enable (which doesn't work on
   generated units). Check: grep -L 'WantedBy=default.target' ~/.config/containers/systemd/*.container
   → must print nothing.
3. Restart=always in [Service] so a crash is retried.

Dry test without rebooting: systemctl --user stop site-<name>.service db-<name>.service && systemctl --user start default.target
→ both come back. Real test: reboot the host, then check-sites.sh after ~2 min
(MariaDB + Puma + Apache need a moment).

7.8 After a host reboot — checklist¶

loginctl show-user websvcmaster | grep Linger     # yes
systemctl --user --failed                         # empty
~/deploy/check-sites.sh                           # all 200/30x
firewall-cmd --list-all                           # (root) 81/444 + 8181 rich rules still there

7.9 Data safety reminders¶

• Secrets live only in ~/.local/share/containers/storage/secrets/ — back up
  that directory (or re-create the secrets from your password store); without
  snipeit-app-key the Snipe-IT DB is unreadable, without the DB secrets the
  containers won't start.
• Volumes: podman volume export <vol> > file.tar / podman volume import <vol> file.tar,
  plus DB dumps (WordPress, Redmine, Snipe-IT).
• podman system reset or removing ~/.local/share/containers deletes
  everything above; never run it on this host.

Source: dmz-web-podman-cheatsheet.md v1.2 in the dmz-web-deploy repo (Releases). Edit the wiki and the repo copy together.