PF » History » Revision 3
Revision 2 (Redmine Admin, 10/31/2026 10:15 AM) → Revision 3/17 (Redmine Admin, 10/31/2026 02:23 PM)
**Version 1.4 1.3 — 2026-10-31** 2026-10-29** · companion of the host cheat sheet (conventions: [[Conventions]], [[NPM]], [[Operations]], [[Git_Workflow]]) Host `dmz-web` (Rocky Linux 10.2). Everything runs **rootless** under `websvcmaster` (no sudo) as Quadlet units. Design rule: **shared-nothing** — own networks, database, app container and volumes; the only shared piece is Nginx Proxy Manager (NPM), which joins the front network. | Item | Value | |---|---| | Public name | `pf.sibernet.com.tr` (DNS A record → FortiGate WAN IP; Let's Encrypt HTTP-01) | | App | `site-pf` — FastAPI/uvicorn on `:8000` (one worker), image `localhost/site-pf:<tag>` built locally from `~/apps/pf`; web GUI and API with own login (password + TOTP/QR, roles User and Admin), no public registration; app version 1.1.2 1.1.1 | | Database | `db-pf` — TimescaleDB on PostgreSQL 16, `docker.io/timescale/timescaledb:2.31.0-pg16` | | Networks | `net-pf` (NPM + app, outbound allowed — the app fetches market data) and `net-pf-db` (app + db only, `Internal=true`: the database has no route to the internet and NPM cannot reach it) | | Volume | `db-pf-data` (PostgreSQL data) | | Secrets | `pf-db-password`, `pf-app-secret` (both `openssl rand -hex 32`, created without a trailing newline) | | Installer | `pf-install.sh` in `~/deploy` (this file's procedures are what it does) | | Source | `pf-app-<ver>.tar.gz` (delivered with the release, not part of `dmz-web-deploy`); lives afterwards in its own repo | Naming follows the host rule: containers `db-pf` / `site-pf`, secrets `pf-…`. The unit description `Description=Portfolio -- <domain>` is what `check-sites.sh` reads, so `./check-sites.sh | grep pf` works. --- ## 1. Install — `pf-install.sh` (websvcmaster) First time, from `~/deploy` (shell setup: host cheat sheet §0), with the source tarball next to the script: ```bash ./pf-install.sh --app-src=pf-app-1.1.1.tar.gz --build --access-list=admin-segments ``` * `--app-src=FILE` unpacks the application (Containerfile, pinned `requirements.txt`, `app/`, `tests/`) into `~/apps/pf`; existing files are never overwritten, and an archive with absolute or `../` paths is refused. It is for the **first** install — later code comes from the app's own repo (§2). * `--build` runs `podman build -t localhost/site-pf:1.1.2 localhost/site-pf:1.1.1 ~/apps/pf` (the default tag; `--app-image=localhost/site-pf:<tag>` for another). * `--access-list=NAME` puts that NPM Access List on the new Proxy Host. The app has its own GUI login (§5), so this is a second layer, not the only one — still recommended while the host is new. * Other options: `--db-image`, `--app-dir`, `--timezone`, `--skip-npm-api`, `--force`; `./pf-install.sh -h`. What it does, in order: generates the Quadlet files into `~/deploy/quadlet/` (`net-pf.network`, `net-pf-db.network`, `db-pf.volume`, `db-pf.container`, `site-pf.container`), creates the two secrets, adds `net-pf` to `nginx-proxy-manager.container`, installs the units, starts nets → db (waits until healthy) → app (waits for `/health`), **restarts NPM** (every site is down ~10–20 s — run it off-hours) and creates the Let's Encrypt certificate and Proxy Host through NPM's API. Before it changes anything it compares the `Network=` lines of the **live** `nginx-proxy-manager.container` with the repo copy and stops if the live file knows a network the repo copy lacks (copying it over would cut that site off — the 2026-10-25 outage). If it stops: add the missing lines to the repo copy, commit, run again. Prerequisite: the DNS name must already resolve to the FortiGate WAN IP, otherwise the certificate request fails (the rest still works; add cert and host in the NPM UI later: Forward Hostname `site-pf`, Forward Port `8000`, Force SSL, HTTP/2, Websockets, Block Common Exploits). Then commit the generated units (root): `deploy-commit.sh -m "pf: portfolio stack on pf.sibernet.com.tr"`. Open the site and create the administrator on the first-run page — §5. Verify: ```bash curl -sI --resolve pf.sibernet.com.tr:444:127.0.0.1 https://pf.sibernet.com.tr:444 | head -1 # HTTP/2 200 curl -s --resolve pf.sibernet.com.tr:444:127.0.0.1 https://pf.sibernet.com.tr:444/health # {"status":"healthy"} podman exec site-pf curl -s http://127.0.0.1:8000/health/db # {"status":"db ok"} ./check-sites.sh | grep pf podman network inspect net-pf net-pf-db | jq '.[].containers | keys' # net-pf: site, nginx-proxy-manager · net-pf-db: db, site ``` --- ## 2. Application code and images The **no-host-packages rule** holds: no Python, pip or libraries on Rocky; everything lives in the image. The `Containerfile` is `python:3.12-slim-bookworm` + `curl` (for the healthcheck) + `pip install --only-binary=:all:` from a **fully pinned** `requirements.txt` (66 wheels) — no compiler in the image, and a dependency without a wheel fails the build loudly instead of compiling something unreviewed. Source layout (`~/apps/pf`): | File | Job | |---|---| | `app/config.py` | `PF_*` settings; DB URL built with `quote_plus`, so a password with `/ + = @ :` cannot break it | | `app/database.py`, `app/models.py` | SQLAlchemy 2 (async, asyncpg); tables are created at start; `price_history` becomes a TimescaleDB hypertable when the extension is there | | `app/auth.py` | bcrypt (SHA-256 pre-hash), TOTP with replay protection, JWT, per-IP rate limit, per-account lockout | | `app/services/market_data.py` | Yahoo Finance (stocks, crypto, FX, dividends) and TEFAS (funds); raw, not dividend-adjusted prices | | `app/services/drip.py` | dividend reinvestment (§5) | | `app/services/portfolio.py`, `indicators.py` | positions, allocation, ideal basket and rebalancing hints (four classes), hints, risk; RSI/MACD/SMA/EMA/Bollinger/ATR (pandas-ta) | | `app/services/suggest.py` | market-suggested basket: benchmark trend rules, tilt capped by `PF_SUGGEST_MAX_TILT` (§5 B-2) | | `app/services/mailer.py` | optional SMTP sender (notices, recovery links); silent when `PF_SMTP_HOST` is empty | | `app/main.py`, `app/schemas.py`, `app/cli.py` | API, validation, admin commands | | `app/static/` | the web GUI: `index.html`, `style.css`, `app.js` (no inline script or style, so the CSP stays strict; charts are drawn as SVG without libraries) and `fonts/` (Inter, SIL OFL, served from the app itself: the CSP allows `font-src 'self'` only, nothing is loaded from Google) | | `tests/` | 46 38 tests (`pytest`), see "Tests" below | * **Pinning:** `>=` ranges drift (on 2026-10-29 an unpinned install resolves to pandas 3.x and a beta of pandas-ta). The list was resolved for Python 3.12 and import-tested; it includes `greenlet`, which SQLAlchemy's async layer needs and which is easy to miss. Change versions on purpose, build a new tag, test, then switch the unit. Not used on purpose: `python-jose` and `passlib` (unmaintained; passlib breaks with current bcrypt) — the app uses `PyJWT` and `bcrypt` directly. * **Image tags:** one tag per release (`site-pf:1.1.0`, `1.1.1`, …), never `latest` — the previous tag is the rollback. * **Config (`PF_*`, set in `site-pf.container`):** `PF_BASE_CURRENCY` (default `TRY`; the default display currency of new users), `PF_CURRENCIES` (default `TRY,USD,EUR,GBP,QAR`: only `TRY,USD,EUR,GBP`: the starting list; afterwards the administrator edits it choices in the browser under **Currencies**, and that list is kept in the database), currency switch; any other 3-letter code with a Yahoo FX rate also works, e.g. via `PUT /auth/profile`), `PF_DIVIDEND_WITHHOLDING` (fraction taken at source, default `0` — set it to what your broker really withholds, e.g. `0.10`), `PF_DRIP_DEFAULT` (default `true`), `PF_DIVIDEND_PAY_LAG_DAYS` (default `0`), `PF_SYNC_ENABLED` / `PF_SYNC_INTERVAL_HOURS` (daily), `PF_REQUIRE_2FA` (default `true`: end users must enrol 2FA, administrators may; `false`: optional for everybody — an administrator can still require it per account), `PF_SUGGEST_MAX_TILT` (default `10`, hard cap `20`: the most percentage points the market-suggested basket may move one asset class away from the user's own basket; a server setting on purpose, not a per-user one), `PF_PW_LOCK_HOURS` (24), `PF_RECOVERY_TOKEN_MINUTES` (30), `PF_SMTP_HOST` / `PORT` (587) / `USER` / `PASSWORD` / `FROM` / `STARTTLS`, `PF_PUBLIC_URL` (links in e-mails; default `https://<PF_DOMAIN>`), `PF_LOGIN_RATE_LIMIT` (default `5/minute`), `PF_ACCESS_TOKEN_MINUTES` (30), `PF_ENABLE_DOCS` (`0`: hides `/docs`, `/redoc`, `/openapi.json`; `1` only while developing, never behind the public Proxy Host). The app refuses to start when `PF_SECRET_KEY` (secret `pf-app-secret`) is shorter than 32 characters. * **Source code is not in `dmz-web-deploy`.** Keep `~/apps/pf` in its own repository (e.g. a bare repo `/srv/git/pf-app.git`, same procedure as host cheat sheet §3 Git A–E) and tag it with the same number as the image. * **Tables:** `create_all` at start creates what is missing. Later columns are added by a short list of idempotent `ALTER TABLE … ADD COLUMN IF NOT EXISTS` statements in `app/database.py` that runs at every start (release 1.1.1 adds seven columns to `users`: `must_change_password`, `require_2fa`, `display_currency`, `is_owner`, `email`, `self_service_recovery`, `pw_locked_until`, and makes the oldest administrator the Owner). **Release 1.1.2 adds** `users.ideal_basket`; `transactions.rate_monthly`, `withholding_pct`, `term_months`, `maturity_date`, `parent_id` (deposits and interest); and the enum values `asset_type = deposit` and `tx_source = interest` (`ALTER TYPE … ADD VALUE IF NOT EXISTS`). Nothing is dropped or rewritten; **take a `pg_dump` (3.C) before the first start of a new version anyway.** * **Tests:** they need a throw-away PostgreSQL, never the production database (`PF_ENV=test PF_TIMESCALE=off PF_SECRET_KEY=<32+ chars> PF_DATABASE_URL_OVERRIDE=postgresql+asyncpg://user:pw@host/pf_test pytest`, with `requirements-dev.txt` installed in a venv away from the host's Python — e.g. on a workstation). Market data is faked in the tests: they prove the logic, not Yahoo/TEFAS behaviour. --- ## 3. Operations ### A. New application version The source is delivered as `pf-app-<ver>.tar.gz`. The old source stays aside as the rollback: ```bash # 0. backup first: 3.C (pg_dump -Fc), copy it off the host cd ~ && mv apps/pf apps/pf-1.1.1 apps/pf-1.1.0 # keep the old tree mkdir apps/pf && tar -xzf ~/deploy/pf-app-1.1.2.tar.gz ~/deploy/pf-app-1.1.1.tar.gz -C apps/pf --no-same-owner --no-same-permissions podman build -t localhost/site-pf:1.1.2 localhost/site-pf:1.1.1 ~/apps/pf # repo copy: change Image=localhost/site-pf:1.1.2 Image=localhost/site-pf:1.1.1 in ~/deploy/quadlet/site-pf.container (keep HealthStartPeriod=90s) # root: deploy-commit.sh -m "pf: app 1.1.2" 1.1.1" cp ~/deploy/quadlet/site-pf.container ~/.config/containers/systemd/ systemctl --user daemon-reload && systemctl --user restart site-pf.service # only site-pf, not db-pf and not NPM podman logs site-pf | tail -20 ~/deploy/check-sites.sh | grep pf ``` Check the result (websvcmaster): ```bash ~/deploy/check-sites.sh | grep pf # HTTP 200 / healthy podman exec db-pf psql -U pf_user -d pf_db -c "\d users" | grep ideal_basket podman exec db-pf psql -U pf_user -d pf_db -c "\d transactions" | grep -E "rate_monthly|withholding_pct|term_months|maturity_date|parent_id" podman exec db-pf psql -U pf_user -d pf_db -c "select enum_range(null::asset_type), enum_range(null::tx_source)" # deposit and interest are in the lists podman exec db-pf psql -U pf_user -d pf_db -c "select hypertable_name from timescaledb_information.hypertables" # price_history ``` In the browser press **Ctrl+F5** once (new `app.js`, `style.css` and the Inter font files). The first dividend sync runs about 60 seconds after the start; fractional shares are now kept to **9 decimals** (as Midas; the three share columns are widened by an idempotent migration), so existing dividend purchases (rounded to 6) change only after *apply to past* on a holding (Dividends page). Rollback: put the previous tag back in the unit (same four lines). The added columns are harmless to the old version, **but only as long as no deposit or interest line exists**: the old code does not know the enum values `deposit` / `interest` and fails on those rows. After deposits were entered, roll back by restoring the dump from step 0, not by changing the tag alone. version. ### B. Logs and shell ```bash journalctl --user -u site-pf.service -f podman logs -f site-pf podman logs -f db-pf podman exec -it db-pf psql -U pf_user -d pf_db ``` ### C. Backup and restore — TimescaleDB is not plain PostgreSQL A plain dump restored the plain way breaks hypertables. Use the custom format and restore with the Timescale pre/post calls: ```bash # backup (websvcmaster) — then copy the file OFF the host; a dump next to the database is not a backup podman exec db-pf pg_dump -U pf_user -Fc pf_db > ~/tmp/pf_db-$(date +%F).dump # restore into an EMPTY database (same TimescaleDB version or newer) podman exec db-pf psql -U pf_user -d postgres -c "create database pf_restore" podman exec db-pf psql -U pf_user -d pf_restore -c "create extension if not exists timescaledb" podman exec db-pf psql -U pf_user -d pf_restore -c "select timescaledb_pre_restore()" podman exec -i db-pf pg_restore -U pf_user -d pf_restore --no-owner < ~/tmp/pf_db-2026-10-29.dump podman exec db-pf psql -U pf_user -d pf_restore -c "select timescaledb_post_restore()" ``` **Do one restore test before trusting the backups**, and again after each TimescaleDB upgrade. The application volume has nothing to back up (no uploads); the secrets are not needed for a restore (set a new password in the new environment). ### D. Password rotation `POSTGRES_PASSWORD` is read only when the data directory is first created. To rotate: `podman exec db-pf psql -U pf_user -d pf_db -c "alter user pf_user password '<new>'"`, then `podman secret rm pf-db-password`, create it again with the new value (`printf '%s' '<new>' | podman secret create pf-db-password -`), restart `db-pf.service` and `site-pf.service`. ### E. Upgrading the database image Dump first (C). Patch/minor (`2.31.0-pg16` → `2.31.1-pg16`): change the tag in the unit, `cp`, `daemon-reload`, restart `db-pf.service`, then `ALTER EXTENSION timescaledb UPDATE;` in `pf_db` (the first command in a fresh session). A PostgreSQL major jump (pg16 → pg17) is a dump-and-restore into a new volume, not a tag change. --- ## 4. Troubleshooting * **App restarts in a loop right after boot** → it came up before the database accepted connections. The units wait for `db-pf` (Podman 5: `Notify=healthy`); check `podman healthcheck run db-pf` and `podman logs db-pf`. * **`/health` is 200 but `/health/db` is 503** → the password in the secret is not the password the database was initialised with (secret recreated after first start): rotate as in 3.D, or drop the volume if the database is still empty. * **NPM 502 for the pf host** → NPM is not on `net-pf` (`podman network inspect net-pf | jq '.[0].containers | keys'`); check the `Network=net-pf.network` line in the live `nginx-proxy-manager.container` and that NPM was restarted afterwards. Other sites also 502 → host cheat sheet §7 "NPM says 502". * **Certificate request fails** → the DNS name does not point to the FortiGate WAN IP yet, or TCP 80 does not reach NPM; fix DNS, then request the certificate in the NPM UI. * **App container exits at once, log says `PF_SECRET_KEY ... shorter than 32 characters`** → the `pf-app-secret` secret is missing or empty; recreate it (`openssl rand -hex 32 | tr -d '\n' | podman secret create pf-app-secret -` after `podman secret rm pf-app-secret`) and restart `site-pf.service`. Existing login tokens become invalid (30 minutes anyway). * **The first-run page does not appear** → an administrator already exists (`podman exec -it site-pf python -m app.cli list-users`). To get the first-run page again on a system that has a user created by the CLI: `python -m app.cli set-role NAME user` for every administrator — the page appears when **no** administrator exists; the setup code is in `podman logs site-pf | grep FIRST-RUN` (printed at start while no administrator exists, and again when the first-run page is opened without (a new one — grep again after opening the page; on every start, valid until used or until the next start). used). * **The chart for 1D / 3D shows few points** → prices are daily closes; there is no intraday data. 1W and longer are the meaningful periods. * **Login answers 429** → either the per-IP limit (5 attempts per minute, the answer comes back after a minute) or the account lock (5 wrong passwords or codes lock the account for 15 minutes): `podman exec -it site-pf python -m app.cli unlock NAME`. * **Rate limit hits everybody at once** → every request seems to come from one address (the proxy). Check that NPM passes `X-Forwarded-For` (it does by default) and that the app runs with `--proxy-headers` (it does in the `Containerfile`). * **Authenticator code is always refused** → phone clock drift (turn automatic time on), or the code was already used (each 30-second code works once). Lost phone: an administrator uses *Reset 2FA* (or *Set temporary password* with "also remove the old 2FA") in the GUI, or on the host `python -m app.cli reset-2fa NAME`; then the person enrols again. * **A dividend is stuck in `pending_price`** → no price for its pay date yet (pay date in the future, or the data source was down); the daily sync completes it. Force it: `POST /api/v1/dividends/sync`. * **`pg_dump` prints a warning about circular foreign-key constraints** → a notice, not an error: the dump still completes. Prove it with the restore test in 3.C (`pg_restore -l` lists the tables, the row counts match). * **`podman build` prints "Running pip as the 'root' user …"** → harmless; it happens inside the image build, not on the host. * **A hand-entered dividend appears next to the one from the market feed** → the feed row is keyed by ex-date, a hand-entered one under another date (typically the pay date) is a second row. The Dividends page compares them (chart "Dividends you entered vs. market data" and a *differs* tag); *Use market data* replaces the hand-entered row, *Delete* removes it together with its purchase. Feed rows cannot be deleted — the next sync would add them back; switch reinvestment off for the holding instead. * **"Market-suggested basket" says "not available right now"** → Yahoo could not be reached from the app container (or the benchmark has too little history); `podman logs site-pf | grep benchmark`. The rest of the page is unaffected. * **Build fails on a dependency** → no wheel for Python 3.12 (the `--only-binary` guard); pick a version with a wheel, do not install a compiler into the image. --- ## 5. Using the application ### A. Web GUI, roles and first run Open `https://pf.sibernet.com.tr`. Everything the API can do is in the GUI (the API stays for a later mobile app). **First run.** While no administrator exists the page asks for the administrator password. It also needs the **setup code** that the app prints to its log at start (so only someone with host access can claim the system): ```bash podman logs site-pf | grep FIRST-RUN # printed at start while no administrator exists, and when the first-run page code is opened without one; valid until used or the next used; a new one each start ``` The administrator created there is the **Owner** (user name `admin` unless typed otherwise). **Roles.** | | User | Administrator | Owner | |---|---|---|---| | Portfolio, trades, dividends, charts, analysis | yes | **no** (403 on every data endpoint) | no | | Create / deactivate accounts, set temporary passwords, reset 2FA, require 2FA, set e-mail / recovery, unlock | no | yes, for users | yes, also for administrators | | Create other administrators | no | no | yes | | 2FA | **mandatory** (`PF_REQUIRE_2FA=true`); QR code at the first login | optional (can be added later in *My account*; an enforcement per account is possible) | like an administrator | Several users and several administrators are possible. The Owner cannot be deactivated. New accounts get a **temporary password** (the GUI can generate one and shows it once); the person must choose their own at the first login. Password rules: at least 12 characters, upper and lower case, a digit. **2FA login.** At the first login a user who needs 2FA gets a restricted session that can only enrol: the page shows a QR code (and the secret for manual entry); after the 6-digit code is confirmed the session continues. Later logins ask for the code (each code works once; ±30 s clock tolerance). **Lost password / lost authenticator.** | Case | What happens | |---|---| | Lost password | Administrator: *Users → Reset a password* (temporary password, forced change at the next login). | | Lost authenticator (password known) | Administrator: *Reset 2FA*; the person enrols again at the next login. If the account has an e-mail address, a notice is sent and the **password cannot be changed by the person for 24 hours** (`PF_PW_LOCK_HOURS`). | | Lost both | Administrator: *Reset a password* with "also remove the old 2FA" — one step; the forced password change at the next login stays allowed, enrolment is repeated. | | Self-service | Only if an administrator gave the account an **e-mail address** and ticked **allow e-mail recovery** (*Users → E-mail / recovery…*), **and** SMTP is configured. Login page: *Forgot password* / *Lost authenticator* → an e-mail with a single-use link (30 min; it dies with any later password or 2FA change; the link travels in the URL fragment, not in server logs). *Lost authenticator* also asks for the current password, then removes 2FA, sends a notice and starts the 24 h password lock; *Forgot password* is refused while that lock is on. The reply to a request is the same for every account, so nobody can probe which users exist. | | Nothing configured | Without `PF_SMTP_HOST` no mail is sent, the login page shows no recovery links, and the administrator path above is the only one. | On the host (break-glass, no GUI): `podman exec -it site-pf python -m app.cli list-users | set-password NAME | unlock NAME | reset-2fa NAME | set-role NAME user|admin | create-user NAME [--admin]`. The password is typed at a hidden prompt, never given as an argument. **Data isolation.** Every query filters on the logged-in user's id; another user's transactions, dividends and settings answer 404, not 403. Instruments (`/assets`) are a shared catalogue; ownership exists only through transactions. ### B-0. Pages (user) * **Portfolio** — *Total wealth* (large figure, decimals small) in the display currency chosen at the top right of the card (TRY, USD, EUR, GBP, or any code with a Yahoo rate; saved per user), the change over the chosen period as a green or red pill, and a chart: **1D, 3D, 1W, 1M, 3M, 6M, YTD, 1Y, All**. The line is **green** when the portfolio gained over the period and **red** when it lost; "gain" excludes money you paid in or took out during the period (a deposit of money is not a win). The soft fill, the dashed level and line is the dot at the end are drawn the same way on every chart. net money paid in. Prices are daily closes, so 1D and 3D have few points. Below: totals (with *Interest received* once a deposit has interest), totals, positions, **Holdings: market analysis and suggestion**, **Ideal basket vs. your portfolio**, asset mix versus the risk profile, weight by holding, risk. Every symbol in these lists opens that holding's own page. * **Holding** (menu item, or click any symbol) — one holding on one page: quantity, average cost, last price, value, unrealized and realized P/L, total return, dividends, DRIP units; a price chart with the same period buttons and green/red line, with **markers** for your operations (buy ▲, sell ▼, dividend ◆, interest ●; hover for date, quantity, price) and, under it, **every buy, sell, dividend, reinvestment and interest line** of that asset with the quantity held after each. A deposit shows its balance instead of a price. * **Holdings analysis** — one row per holding with weight, unrealized P/L, RSI, trend and a suggestion — **BUY, ACCUMULATE, HOLD, REDUCE, SELL** — and the reasons. It is a score from six rules (RSI over 70 / under 30, MACD against its signal line, price against the 50-day average, 50-day against 200-day average, Bollinger bands): +3 or more BUY, +2 ACCUMULATE, −2 REDUCE, −3 or less SELL, otherwise HOLD. Technical signals only — they know nothing about fundamentals, taxes, your goals or your other assets. Deposits are not analysed. Information, not investment advice. * **Ideal basket vs. your portfolio** — four boxes at the top say what to **add or remove** per class (Deposits, Funds, Stocks, Crypto) to reach your ideal basket; below, the ideal and the real mix as bars and a table. A class within 5 points counts as on target. The **Market-suggested basket** box (B-2) sits inside this card. * **Trades** — new trade; for a **deposit** the form asks for the amount, the **monthly rate**, the **term** (1, 3, 6 months or 1 year) and the **tax withheld on the interest** (B-1); *Deposits and maturities*; *Interest received*; one history of trades, dividends (with per-share), dividend reinvestments and interest, each with a *Delete* where it is allowed. * **Trades**, **Dividends** — reinvestment (reinvestment switch per holding, sync, chart, **Record a dividend by hand**, the chart **Dividends you entered vs. market data**, history. A hand-entered dividend is compared with the market feed (*matches market data* / *differs* with the reason / *cannot verify*); *Use market data* replaces it, *Delete* removes it with its purchase. * chart), **Market** (price chart and indicators per instrument), **Settings** (*Dividends*: reinvest by default; *Ideal basket*: radio buttons **Conservative / Balanced / Growth / My own** — a preset locks the four fields, *My own* unlocks them and needs a total of exactly 100 %; 2FA; (risk profile, reinvestment default, 2FA, password). * Administrators see **Users** and **My account** only. ### B. Dividend reinvestment (DRIP) | Rule | Behaviour | |---|---| | Switch | per holding (`PUT /api/v1/holdings/{asset_id}/reinvest`), else the user default (`PUT /api/v1/auth/profile` `drip_default`), else `PF_DRIP_DEFAULT` (on) | | Entitlement | shares held at the close **before** the ex-date; a purchase on the ex-date earns nothing | | Amount | `shares × per-share × (1 − withholding)` | | Reinvest on | a BUY (`source: drip`, fee 0) is booked at the first close on/after the pay date; quantity = net ÷ price, rounded **down** to 9 decimals (as brokers such as Midas keep them); 6 decimals; the rest is kept as `cash_residual` | | Reinvest off | the net amount is recorded as cash income (`status: paid`) | | No price yet | `status: pending_price`; the daily sync completes it (the later dividends of that asset wait, so the share count stays right) | | Compounding | reinvested shares are normal shares and earn the next dividend | | Idempotent | one row per user, asset and ex-date — syncing twice does nothing | | Manual entry | `POST /api/v1/dividends` for companies the feed does not know; optional `withholding`, `reinvest`, `reinvest_price`. An entry within 30 days of another dividend of the same holding is refused with 409 ("looks like the same payout") unless `?force=true` (the GUI asks). `DELETE /api/v1/dividends/{id}` removes a hand-entered row with its purchase and recomputes the later ones (feed rows: 409). `GET /api/v1/dividends/verify` compares hand-entered rows with the feed (amount within 2 % or equal after rounding to 2–4 decimals; ex-date within 3 days → `ok`; same amount but another date → `date_differs`; otherwise `amount_differs` / `no_data`); `POST /api/v1/dividends/{id}/use-market` replaces the row with the feed's version `reinvest_price` | | Corrections | adding or deleting a manual trade recomputes that asset's dividends; `PUT .../reinvest?apply_to_history=true` and `POST /api/v1/dividends/rebuild/{asset_id}` do it on demand. DRIP purchases cannot be deleted by hand (409) | | Returns | reinvested dividends raise the cost basis but are **not** new money: `money_invested` counts only your own purchases, `total_return = market value + sale proceeds + cash dividends − money invested` | Limits to know: Yahoo reports ex-dates only, so the pay date is `ex-date + PF_DIVIDEND_PAY_LAG_DAYS` (0) — for a company that pays about three weeks after (0); the ex-date (NVIDIA) the reinvestment price is therefore the one around the ex-date, a small difference; the dividend is assumed to be in the asset's own currency; TEFAS funds and crypto have no dividend events; withholding is one global rate unless you pass it per manual entry. ### B-1. Deposits and interest A deposit (savings or time deposit) is an instrument of type `deposit`: no price feed, one unit = one unit of its currency, valued at its balance. | Rule | Behaviour | |---|---| | Pay in | a buy with the amount, the **monthly rate in percent**, the **term** (1–120 months, the GUI offers 1, 3, 6, 12) and the **withholding %** on the interest; the maturity date is the pay-in date plus the term. A pay-in without a rate is refused (422); price is forced to 1 | | Expected interest | `amount × monthly rate × months` (simple, per term), minus withholding = net. Shown in *Deposits and maturities* (`GET /api/v1/deposits`) with the annual equivalent `((1 + r)^12 − 1)` and a status: running, due, booked | | Booking the interest | **by hand**, nothing is booked automatically: at maturity press *Book interest* (pre-filled with the expected net and tax, both editable to what the bank really credited) or use *Interest received* for interest that belongs to no listed term. `POST /api/v1/interest` (`amount` = credited, `withheld`, `trade_date`, optional `deposit_tx_id`); a pay-in takes its interest once (409 after) | | Effect | the interest raises the balance and counts as **gain** (unrealized, realized on withdrawal) — it is not money you paid in, so `money_invested` stays and `total_return` rises. The history line shows the gross (credited + withheld) | | After booking | the maturity row shows the realized monthly yield after tax | | Not for deposits | no dividend reinvestment, no technical analysis, no price chart (the Holding page draws the balance) | Deposit rates change from month to month; the latest pay-in's rate is shown as an information line in the market-suggested basket and does not move the basket. ### B-2. Ideal basket and the market-suggested basket The ideal basket is four whole percentages (Deposits, Funds, Stocks, Crypto) that add up to 100, set in *Settings*. The presets are Conservative 45/30/20/5, Balanced 20/35/40/5, Growth 5/20/65/10 (deposits/funds/stocks/crypto); *My own* is stored in `users.ideal_basket`, a preset is the risk profile itself. The **market-suggested basket** starts from the user's own basket and tilts it with a few explainable rules on benchmark closes (cached 6 hours, never stored): a trend filter (price against its 200-day average) and 6-month momentum for **S&P 500** and **BIST 100 read in US dollars** (index ÷ USD/TRY, so inflation does not make a lira index look bullish), **Bitcoin**, and the **VIX** as a risk-off flag. Stocks move up to ±`PF_SUGGEST_MAX_TILT` points, crypto up to ±2.5 per signal point within the same cap; what risk assets gain or lose is taken from / given to deposits and funds, and the total stays exactly 100. Every tilt has its reason in plain words. **Nothing changes until the user presses "Use this as my ideal basket".** Rule-based reading of a few prices, not advice, and it knows nothing about the user's taxes or goals. ### C. Endpoints (`/api/v1`, bearer token except `login`) | Area | Calls | |---|---| | Setup | `GET setup/status`, `POST setup/admin` (first run, needs the setup code) | | Auth | `POST auth/login`, `auth/2fa/setup` (QR as data URI), `auth/2fa/confirm`, `auth/2fa/disable`, `auth/password`, `auth/recover/request`, `auth/recover/confirm`; `GET auth/me`; `PUT auth/profile` (`risk_profile`, (`risk_profile`: conservative / balanced / growth, `drip_default`, `display_currency`, `ideal_basket` = `{"deposit":20,"fund":35,"stock":40,"crypto":5}` summing to 100, `reset_ideal_basket`) `display_currency`) | | Admin (administrators only) | `GET/POST admin/users`, `POST admin/users/{id}/reset-password` (`remove_2fa`), `reset-2fa`, `require-2fa`, `recovery` (e-mail, self-service), `unlock`, `active` | | Instruments | `POST assets` (stock/crypto = Yahoo symbol such as `THYAO.IS`, `AAPL`, `BTC-USD`; fund = TEFAS code; deposit = any name, no lookup; looked up on creation, `?verify=false` skips it), `GET assets` | | Trades | `POST transactions` (buy/sell, quantity, price, fee, trade_date; selling more than held is refused), `GET transactions` (`asset_id`, `source`, paging), `DELETE transactions/{id}`, `GET activity` (trades, dividends and interest in one list), `GET deposits`, `POST interest` transactions/{id}` | | Dividends | `GET dividends`, `POST dividends` (`?force=true`), `DELETE dividends/{id}` (hand-entered only), `GET dividends/verify`, dividends`, `POST dividends/{id}/use-market`, `POST dividends/sync` (6 per hour), `POST dividends/rebuild/{asset_id}`, `GET/PUT holdings/{asset_id}/reinvest` | | Portfolio | `GET portfolio` (positions, totals in the user's display currency, FX conversion), `currencies`, `portfolio/history?period=1d\|3d\|1w\|1m\|3m\|6m\|ytd\|1y\|all` (points + summary: start/end value, money in, gain, gain %), `portfolio/analysis`, `portfolio/allocation`, `portfolio/rebalance`, `portfolio/suggestion`, `portfolio/risk`, `holdings/{asset_id}/detail` (price or balance series, position, every operation) `portfolio/risk` | | Market | `GET market/{asset_id}/indicators`, `market/{asset_id}/history` | `portfolio/rebalance` compares the asset-class mix (deposits (funds / funds / stocks / crypto) with an illustrative target for the user's ideal basket (a preset of the chosen risk profile or the user's own) (conservative 65/30/5, balanced 40/50/10, growth 20/65/15) and names the amount to add or remove move for any class more than 5 points off; `portfolio/suggestion` returns the market-suggested basket with its reasons (B-2). Indicators, off. Indicators and rebalancing and suggestions are information, not personal investment advice. --- ## Document history | Version | Date | Status | Changes | |---------|------------|------------|---------| | 1.4 1.3 | 2026-10-31 2026-10-29 | Latest | App 1.1.2: **ideal basket** in Settings (presets as radio buttons or "My own") with four classes incl. **deposits**, "add / remove" summary and the **market-suggested basket** (S&P 500, BIST 100 in USD, Bitcoin, VIX; cap `PF_SUGGEST_MAX_TILT`); **deposits** with monthly rate, term and withholding, **interest** booked by hand, *Deposits and maturities*; **Holding pages** with price chart, operation markers and every operation; **dividend check** against market data, delete and duplicate warning for hand-entered dividends; fractional shares to **9 decimals**; **Currencies** page for the administrator (pick from a list or type a code; a code is accepted only when Yahoo has a rate for it; the list feeds the display-currency switch and the currency dropdown of new instruments); trade form with **Total** and **Get market price for this date** (close of that day, or the last trading day before it; you can overtype it, and typing a price yourself always works); (any two of quantity / price / total, the third is calculated); instrument type **fund / ETF on Yahoo** (counts as fund in the allocation, priced and paying dividends like a stock; an existing stock can be switched on its holding page, *Counts as*); a newly bought stock fetches its **dividend history at once**; `BRK.B`-style symbols are found as `BRK-B`; new look for charts (soft fill, dashed start level, end dot, large figures) and the self-hosted Inter font. Migrations: `users.ideal_basket`, five `transactions` columns, enum values `deposit`, `etf` and `interest`, share columns widened to 9 decimals, new table `app_settings` (idempotent). §3.A: upgrade check commands and rollback caveat; §4: new entries; §5 B-1 and B-2; 51 tests. `pf-install.sh` 1.3. | | 1.3 | 2026-10-29 | Superseded | App 1.1.1: **web GUI** (first-run administrator page with setup code, roles User / Administrator / Owner, QR enrolment, mandatory 2FA for users and optional for administrators, forced password change), **total wealth** with currency switch and period chart (green / red), **holdings analysis with BUY / HOLD / SELL suggestions**, admin-assignable **e-mail recovery** (optional SMTP), e-mail notice and 24 h password-change lock after 2FA removal, "reset password and remove 2FA" in one step. Seven added `users` columns (idempotent). `pf-install.sh` 1.2 (image tag 1.1.1, commented SMTP lines, first-run "Next" text). §5 rewritten; §3.A upgrade from the tarball; §4 entries; config variables; 38 tests. | | 1.2 | 2026-10-29 | Superseded | The real application replaces the starter: login with TOTP 2FA (rate limit, lockout, replay protection, no public registration), transactions, **dividend reinvestment (DRIP)**, Yahoo/TEFAS market data on a TimescaleDB hypertable, positions and allocation in the base currency, indicators, risk statistics, class-level rebalancing hints, admin CLI. `pf-install.sh` 1.1: `--app-src` instead of `--init-app`, image tag 1.1.0, longer start period. New §5 (users and 2FA, DRIP rules, endpoint list); §2 file map, config variables, tests; §4 entries for secret key, 429, authenticator codes, `pending_price`. Pins: `greenlet` added (SQLAlchemy async), `PyJWT`/`bcrypt` instead of python-jose/passlib. | | 1.1 | 2026-10-29 | Superseded | Rewritten around `pf-install.sh` (one procedure instead of nine manual steps, with the live-vs-repo NPM `Network=` guard). Two networks (`net-pf-db` is `Internal=true`); DB healthcheck and `Notify=healthy`; explicit `.volume` unit; pinned images (`2.31.0-pg16`, `site-pf:<tag>`) and a fully pinned, wheels-only `requirements.txt`; secrets as hex (URL-safe) and the URL built with `quote`; `/docs` off; Access List option; Timescale-aware backup/restore, password rotation and DB upgrade; troubleshooting; history table. Fixes of the 1.0 draft: wrong `diff` expectation (`+` vs `>`), NPM snippet repeating section headers, unpinned `latest` tags. | | 1.0 | 2026-10-29 | Superseded | Draft (manual nine-step procedure). Never committed. |