IT Support » DMZ-WEB Infrastructure

**Version 1.2 — 2026-10-29** · companion of the host cheat sheet (conventions: [[Conventions]], [[NPM]], [[Operations]], [[Git_Workflow]])

Host `dmz-web` (Rocky Linux 10.2). Everything runs **rootless** under `websvcmaster` (no sudo) as Quadlet units. Design rule: **shared-nothing** — own networks, database, app container and volumes; the only shared piece is Nginx Proxy Manager (NPM), which joins the front network.

| Item | Value |

|---|---|

| Public name | `pf.sibernet.com.tr` (DNS A record → FortiGate WAN IP; Let's Encrypt HTTP-01) |

| App | `site-pf` — FastAPI/uvicorn on `:8000` (one worker), image `localhost/site-pf:<tag>` built locally from `~/apps/pf`; own login (password + TOTP), no public registration |

| Database | `db-pf` — TimescaleDB on PostgreSQL 16, `docker.io/timescale/timescaledb:2.31.0-pg16` |

| Networks | `net-pf` (NPM + app, outbound allowed — the app fetches market data) and `net-pf-db` (app + db only, `Internal=true`: the database has no route to the internet and NPM cannot reach it) |

| Volume | `db-pf-data` (PostgreSQL data) |

| Secrets | `pf-db-password`, `pf-app-secret` (both `openssl rand -hex 32`, created without a trailing newline) |

| Installer | `pf-install.sh` in `~/deploy` (this file's procedures are what it does) |

| Source | `pf-app-<ver>.tar.gz` (delivered with the release, not part of `dmz-web-deploy`); lives afterwards in its own repo |

Naming follows the host rule: containers `db-pf` / `site-pf`, secrets `pf-…`. The unit description `Description=Portfolio -- <domain>` is what `check-sites.sh` reads, so `./check-sites.sh | grep pf` works.

---

## 1. Install — `pf-install.sh` (websvcmaster)

First time, from `~/deploy` (shell setup: host cheat sheet §0), with the source tarball next to the script:

```bash

./pf-install.sh --app-src=pf-app-1.1.0.tar.gz --build --access-list=admin-segments

```

* `--app-src=FILE` unpacks the application (Containerfile, pinned `requirements.txt`, `app/`, `tests/`) into `~/apps/pf`; existing files are never overwritten, and an archive with absolute or `../` paths is refused. It is for the **first** install — later code comes from the app's own repo (§2).

* `--build` runs `podman build -t localhost/site-pf:1.1.0 ~/apps/pf` (the default tag; `--app-image=localhost/site-pf:1.2.0` for another).

* `--access-list=NAME` puts that NPM Access List on the new Proxy Host. The app has its own login (§5), so this is a second layer, not the only one — still recommended while the host is new.

* Other options: `--db-image`, `--app-dir`, `--timezone`, `--skip-npm-api`, `--force`; `./pf-install.sh -h`.

What it does, in order: generates the Quadlet files into `~/deploy/quadlet/` (`net-pf.network`, `net-pf-db.network`, `db-pf.volume`, `db-pf.container`, `site-pf.container`), creates the two secrets, adds `net-pf` to `nginx-proxy-manager.container`, installs the units, starts nets → db (waits until healthy) → app (waits for `/health`), **restarts NPM** (every site is down ~10–20 s — run it off-hours) and creates the Let's Encrypt certificate and Proxy Host through NPM's API.

Before it changes anything it compares the `Network=` lines of the **live** `nginx-proxy-manager.container` with the repo copy and stops if the live file knows a network the repo copy lacks (copying it over would cut that site off — the 2026-10-25 outage). If it stops: add the missing lines to the repo copy, commit, run again.

Prerequisite: the DNS name must already resolve to the FortiGate WAN IP, otherwise the certificate request fails (the rest still works; add cert and host in the NPM UI later: Forward Hostname `site-pf`, Forward Port `8000`, Force SSL, HTTP/2, Websockets, Block Common Exploits).

Then commit the generated units (root): `deploy-commit.sh -m "pf: portfolio stack on pf.sibernet.com.tr"`.

Create the first user (hidden password prompt) and enrol 2FA — §5. Verify:

```bash

curl -sI --resolve pf.sibernet.com.tr:444:127.0.0.1 https://pf.sibernet.com.tr:444 | head -1     # HTTP/2 200

curl -s  --resolve pf.sibernet.com.tr:444:127.0.0.1 https://pf.sibernet.com.tr:444/health        # {"status":"healthy"}

podman exec site-pf curl -s http://127.0.0.1:8000/health/db                                       # {"status":"db ok"}

./check-sites.sh | grep pf

podman network inspect net-pf net-pf-db | jq '.[].containers | keys'   # net-pf: site, nginx-proxy-manager · net-pf-db: db, site

```

---

## 2. Application code and images

The **no-host-packages rule** holds: no Python, pip or libraries on Rocky; everything lives in the image. The `Containerfile` is `python:3.12-slim-bookworm` + `curl` (for the healthcheck) + `pip install --only-binary=:all:` from a **fully pinned** `requirements.txt` (65 wheels) — no compiler in the image, and a dependency without a wheel fails the build loudly instead of compiling something unreviewed.

Source layout (`~/apps/pf`):

| File | Job |

|---|---|

| `app/config.py` | `PF_*` settings; DB URL built with `quote_plus`, so a password with `/ + = @ :` cannot break it |

| `app/database.py`, `app/models.py` | SQLAlchemy 2 (async, asyncpg); tables are created at start; `price_history` becomes a TimescaleDB hypertable when the extension is there |

| `app/auth.py` | bcrypt (SHA-256 pre-hash), TOTP with replay protection, JWT, per-IP rate limit, per-account lockout |

| `app/services/market_data.py` | Yahoo Finance (stocks, crypto, FX, dividends) and TEFAS (funds); raw, not dividend-adjusted prices |

| `app/services/drip.py` | dividend reinvestment (§5) |

| `app/services/portfolio.py`, `indicators.py` | positions, allocation, rebalancing hints, risk; RSI/MACD/SMA/EMA/Bollinger/ATR (pandas-ta) |

| `app/main.py`, `app/schemas.py`, `app/cli.py` | API, validation, admin commands |

| `tests/` | 21 tests (`pytest`), see "Tests" below |

* **Pinning:** `>=` ranges drift (on 2026-10-29 an unpinned install resolves to pandas 3.x and a beta of pandas-ta). The list was resolved for Python 3.12 and import-tested; it includes `greenlet`, which SQLAlchemy's async layer needs and which is easy to miss. Change versions on purpose, build a new tag, test, then switch the unit. Not used on purpose: `python-jose` and `passlib` (unmaintained; passlib breaks with current bcrypt) — the app uses `PyJWT` and `bcrypt` directly.

* **Image tags:** one tag per release (`site-pf:1.1.0`, `1.1.1`, …), never `latest` — the previous tag is the rollback.

* **Config (`PF_*`, set in `site-pf.container`):** `PF_BASE_CURRENCY` (default `TRY`), `PF_DIVIDEND_WITHHOLDING` (fraction taken at source, default `0` — set it to what your broker really withholds, e.g. `0.10`), `PF_DRIP_DEFAULT` (default `true`), `PF_DIVIDEND_PAY_LAG_DAYS` (default `0`), `PF_SYNC_ENABLED` / `PF_SYNC_INTERVAL_HOURS` (daily), `PF_REQUIRE_2FA` (default `true`), `PF_LOGIN_RATE_LIMIT` (default `5/minute`), `PF_ACCESS_TOKEN_MINUTES` (30), `PF_ENABLE_DOCS` (`0`: hides `/docs`, `/redoc`, `/openapi.json`; `1` only while developing, never behind the public Proxy Host). The app refuses to start when `PF_SECRET_KEY` (secret `pf-app-secret`) is shorter than 32 characters.

* **Source code is not in `dmz-web-deploy`.** Keep `~/apps/pf` in its own repository (e.g. a bare repo `/srv/git/pf-app.git`, same procedure as host cheat sheet §3 Git A–E) and tag it with the same number as the image.

* **Tables:** `create_all` at start creates what is missing; it never alters an existing table. A later schema change needs an explicit Alembic migration (installed, not wired): `podman exec site-pf alembic upgrade head`, run by hand after the app is up, never from the container start command.

* **Tests:** they need a throw-away PostgreSQL, never the production database (`PF_ENV=test PF_TIMESCALE=off PF_SECRET_KEY=<32+ chars> PF_DATABASE_URL_OVERRIDE=postgresql+asyncpg://user:pw@host/pf_test pytest`, with `requirements-dev.txt` installed in a venv away from the host's Python — e.g. on a workstation). Market data is faked in the tests: they prove the logic, not Yahoo/TEFAS behaviour.

---

## 3. Operations

### A. New code (database untouched)

```bash

cd ~/apps/pf                                         # edit / pull

podman build -t localhost/site-pf:1.1.1 .

# repo copy: change  Image=localhost/site-pf:1.1.1  in ~/deploy/quadlet/site-pf.container

# root:      deploy-commit.sh -m "pf: app 1.1.1"

cp ~/deploy/quadlet/site-pf.container ~/.config/containers/systemd/

systemctl --user daemon-reload && systemctl --user restart site-pf.service

~/deploy/check-sites.sh | grep pf

```

Rollback: put the previous tag back, same three lines.

### B. Logs and shell

```bash

journalctl --user -u site-pf.service -f

podman logs -f site-pf          podman logs -f db-pf

podman exec -it db-pf psql -U pf_user -d pf_db

```

### C. Backup and restore — TimescaleDB is not plain PostgreSQL

A plain dump restored the plain way breaks hypertables. Use the custom format and restore with the Timescale pre/post calls:

```bash

# backup (websvcmaster) — then copy the file OFF the host; a dump next to the database is not a backup

podman exec db-pf pg_dump -U pf_user -Fc pf_db > ~/tmp/pf_db-$(date +%F).dump

# restore into an EMPTY database (same TimescaleDB version or newer)

podman exec db-pf psql -U pf_user -d postgres -c "create database pf_restore"

podman exec db-pf psql -U pf_user -d pf_restore -c "create extension if not exists timescaledb"

podman exec db-pf psql -U pf_user -d pf_restore -c "select timescaledb_pre_restore()"

podman exec -i db-pf pg_restore -U pf_user -d pf_restore --no-owner < ~/tmp/pf_db-2026-10-29.dump

podman exec db-pf psql -U pf_user -d pf_restore -c "select timescaledb_post_restore()"

```

**Do one restore test before trusting the backups**, and again after each TimescaleDB upgrade. The application volume has nothing to back up (no uploads); the secrets are not needed for a restore (set a new password in the new environment).

### D. Password rotation

`POSTGRES_PASSWORD` is read only when the data directory is first created. To rotate: `podman exec db-pf psql -U pf_user -d pf_db -c "alter user pf_user password '<new>'"`, then `podman secret rm pf-db-password`, create it again with the new value (`printf '%s' '<new>' | podman secret create pf-db-password -`), restart `db-pf.service` and `site-pf.service`.

### E. Upgrading the database image

Dump first (C). Patch/minor (`2.31.0-pg16` → `2.31.1-pg16`): change the tag in the unit, `cp`, `daemon-reload`, restart `db-pf.service`, then `ALTER EXTENSION timescaledb UPDATE;` in `pf_db` (the first command in a fresh session). A PostgreSQL major jump (pg16 → pg17) is a dump-and-restore into a new volume, not a tag change.

---

## 4. Troubleshooting

* **App restarts in a loop right after boot** → it came up before the database accepted connections. The units wait for `db-pf` (Podman 5: `Notify=healthy`); check `podman healthcheck run db-pf` and `podman logs db-pf`.

* **`/health` is 200 but `/health/db` is 503** → the password in the secret is not the password the database was initialised with (secret recreated after first start): rotate as in 3.D, or drop the volume if the database is still empty.

* **NPM 502 for the pf host** → NPM is not on `net-pf` (`podman network inspect net-pf | jq '.[0].containers | keys'`); check the `Network=net-pf.network` line in the live `nginx-proxy-manager.container` and that NPM was restarted afterwards. Other sites also 502 → host cheat sheet §7 "NPM says 502".

* **Certificate request fails** → the DNS name does not point to the FortiGate WAN IP yet, or TCP 80 does not reach NPM; fix DNS, then request the certificate in the NPM UI.

* **App container exits at once, log says `PF_SECRET_KEY ... shorter than 32 characters`** → the `pf-app-secret` secret is missing or empty; recreate it (`openssl rand -hex 32 | tr -d '\n' | podman secret create pf-app-secret -` after `podman secret rm pf-app-secret`) and restart `site-pf.service`. Existing login tokens become invalid (30 minutes anyway).

* **Login answers 429** → either the per-IP limit (5 attempts per minute, the answer comes back after a minute) or the account lock (5 wrong passwords or codes lock the account for 15 minutes): `podman exec -it site-pf python -m app.cli unlock NAME`.

* **Rate limit hits everybody at once** → every request seems to come from one address (the proxy). Check that NPM passes `X-Forwarded-For` (it does by default) and that the app runs with `--proxy-headers` (it does in the `Containerfile`).

* **Authenticator code is always refused** → phone clock drift (turn automatic time on), or the code was already used (each 30-second code works once). Lost phone: `python -m app.cli reset-2fa NAME`, then enrol again.

* **A dividend is stuck in `pending_price`** → no price for its pay date yet (pay date in the future, or the data source was down); the daily sync completes it. Force it: `POST /api/v1/dividends/sync`.

* **Build fails on a dependency** → no wheel for Python 3.12 (the `--only-binary` guard); pick a version with a wheel, do not install a compiler into the image.

---

## 5. Using the application

### A. Users and two-factor login

There is no public registration. Create users inside the container (the password is typed at a hidden prompt, never given as an argument — it would land in the shell history):

```bash

podman exec -it site-pf python -m app.cli create-user kerem --admin

podman exec -it site-pf python -m app.cli list-users

podman exec -it site-pf python -m app.cli set-password kerem      # also: unlock, reset-2fa

```

Password rules: at least 12 characters, upper and lower case, a digit. First login returns a **restricted** token (`"scope": "setup"`) that can only enrol 2FA; after the code is confirmed you get a full token (30 minutes). From then on `login` needs the 6-digit code in the `otp` field.

```bash

U=https://pf.sibernet.com.tr

T=$(curl -s $U/api/v1/auth/login -d 'username=kerem' --data-urlencode 'password=...' | jq -r .access_token)

curl -s -X POST $U/api/v1/auth/2fa/setup   -H "Authorization: Bearer $T"          # -> secret + otpauth_uri: add to the authenticator app

curl -s -X POST $U/api/v1/auth/2fa/confirm -H "Authorization: Bearer $T" -H 'Content-Type: application/json' -d '{"code":"123456"}'

# every later login:

curl -s $U/api/v1/auth/login -d 'username=kerem' --data-urlencode 'password=...' -d 'otp=123456'

```

Data isolation: every query filters on the logged-in user's id; another user's transactions, dividends and settings answer 404, not 403. Instruments (`/assets`) are a shared catalogue; ownership exists only through transactions.

### B. Dividend reinvestment (DRIP)

| Rule | Behaviour |

|---|---|

| Switch | per holding (`PUT /api/v1/holdings/{asset_id}/reinvest`), else the user default (`PUT /api/v1/auth/profile` `drip_default`), else `PF_DRIP_DEFAULT` (on) |

| Entitlement | shares held at the close **before** the ex-date; a purchase on the ex-date earns nothing |

| Amount | `shares × per-share × (1 − withholding)` |

| Reinvest on | a BUY (`source: drip`, fee 0) is booked at the first close on/after the pay date; quantity = net ÷ price, rounded **down** to 6 decimals; the rest is kept as `cash_residual` |

| Reinvest off | the net amount is recorded as cash income (`status: paid`) |

| No price yet | `status: pending_price`; the daily sync completes it (the later dividends of that asset wait, so the share count stays right) |

| Compounding | reinvested shares are normal shares and earn the next dividend |

| Idempotent | one row per user, asset and ex-date — syncing twice does nothing |

| Manual entry | `POST /api/v1/dividends` for companies the feed does not know; optional `withholding`, `reinvest`, `reinvest_price` |

| Corrections | adding or deleting a manual trade recomputes that asset's dividends; `PUT .../reinvest?apply_to_history=true` and `POST /api/v1/dividends/rebuild/{asset_id}` do it on demand. DRIP purchases cannot be deleted by hand (409) |

| Returns | reinvested dividends raise the cost basis but are **not** new money: `money_invested` counts only your own purchases, `total_return = market value + sale proceeds + cash dividends − money invested` |

Limits to know: Yahoo reports ex-dates only, so the pay date is `ex-date + PF_DIVIDEND_PAY_LAG_DAYS` (0); the dividend is assumed to be in the asset's own currency; TEFAS funds and crypto have no dividend events; withholding is one global rate unless you pass it per manual entry.

### C. Endpoints (`/api/v1`, bearer token except `login`)

| Area | Calls |

|---|---|

| Auth | `POST auth/login`, `auth/2fa/setup`, `auth/2fa/confirm`, `auth/password`; `GET auth/me`; `PUT auth/profile` (`risk_profile`: conservative / balanced / growth, `drip_default`) |

| Instruments | `POST assets` (stock/crypto = Yahoo symbol such as `THYAO.IS`, `AAPL`, `BTC-USD`; fund = TEFAS code; looked up on creation, `?verify=false` skips it), `GET assets` |

| Trades | `POST transactions` (buy/sell, quantity, price, fee, trade_date; selling more than held is refused), `GET transactions` (`asset_id`, `source`, paging), `DELETE transactions/{id}` |

| Dividends | `GET dividends`, `POST dividends`, `POST dividends/sync` (6 per hour), `POST dividends/rebuild/{asset_id}`, `GET/PUT holdings/{asset_id}/reinvest` |

| Portfolio | `GET portfolio` (positions, totals in `PF_BASE_CURRENCY`, FX conversion), `portfolio/allocation`, `portfolio/rebalance`, `portfolio/risk` |

| Market | `GET market/{asset_id}/indicators`, `market/{asset_id}/history` |

`portfolio/rebalance` compares the asset-class mix (funds / stocks / crypto) with an illustrative target for the chosen risk profile (conservative 65/30/5, balanced 40/50/10, growth 20/65/15) and names the amount to move for any class more than 5 points off. Indicators and rebalancing are information, not personal investment advice.

---

## Document history

| Version | Date       | Status     | Changes |

|---------|------------|------------|---------|

| 1.2     | 2026-10-29 | Latest     | The real application replaces the starter: login with TOTP 2FA (rate limit, lockout, replay protection, no public registration), transactions, **dividend reinvestment (DRIP)**, Yahoo/TEFAS market data on a TimescaleDB hypertable, positions and allocation in the base currency, indicators, risk statistics, class-level rebalancing hints, admin CLI. `pf-install.sh` 1.1: `--app-src` instead of `--init-app`, image tag 1.1.0, longer start period. New §5 (users and 2FA, DRIP rules, endpoint list); §2 file map, config variables, tests; §4 entries for secret key, 429, authenticator codes, `pending_price`. Pins: `greenlet` added (SQLAlchemy async), `PyJWT`/`bcrypt` instead of python-jose/passlib. |

| 1.1     | 2026-10-29 | Superseded | Rewritten around `pf-install.sh` (one procedure instead of nine manual steps, with the live-vs-repo NPM `Network=` guard). Two networks (`net-pf-db` is `Internal=true`); DB healthcheck and `Notify=healthy`; explicit `.volume` unit; pinned images (`2.31.0-pg16`, `site-pf:<tag>`) and a fully pinned, wheels-only `requirements.txt`; secrets as hex (URL-safe) and the URL built with `quote`; `/docs` off; Access List option; Timescale-aware backup/restore, password rotation and DB upgrade; troubleshooting; history table. Fixes of the 1.0 draft: wrong `diff` expectation (`+` vs `>`), NPM snippet repeating section headers, unpinned `latest` tags. |

| 1.0     | 2026-10-29 | Superseded | Draft (manual nine-step procedure). Never committed. |