v4.7.0 reaches outward: a new server-side integration subsystem polls popular self-hosted software for health and folds it into Alerts, and the Linux agent can now run as a container to monitor a Docker host with no install. Hard-reload once after upgrading (cache remotepower-shell-v4.7.0).

• 26 homelab software integrations. Pi-hole, AdGuard, TrueNAS, UniFi, Home Assistant, Proxmox Backup Server, Kubernetes/k3s, vCenter/ESXi, Unraid, Traefik, Nginx Proxy Manager, Caddy, Netdata, Grafana, Uptime Kuma, Jellyfin, Plex, Nextcloud, the download clients (qBittorrent, Transmission, Deluge, SABnzbd, NZBGet), the *arr suite (one Servarr connector for Sonarr/Radarr/Prowlarr/Lidarr) + Bazarr, and Overseerr/Jellyseerr. Set them up under Settings → Integrations: pick a type, add a URL + token, Test.
• Health → Alerts + dashboard. An unhealthy or unreachable target raises an integration_down alert (auto-resolved on recovery) routed through your channels, plus an Integration health dashboard widget and live status badges.
• Read-only & SSRF-guarded. Every outbound call is blocked from loopback / link-local / cloud-metadata (RFC1918 LAN allowed), re-validated at connect time, no redirects. Credentials are stored server-side and redacted from every response; the raw URL is admin-only.
• Containerized agent. Enroll device → Generate Docker compose, then docker compose up -d on a Docker host. The agent reads the host's facts (shared namespaces, host rootfs mounted read-only), names itself after the host, and persists credentials in a volume. Multi-arch image at ghcr.io/tyxak/remotepower-agent. Standard caps, no --privileged.
• Enterprise declutter. A Show Homelab software switch (default on) is an instance-wide kill switch — off stops polling, hides the section, and removes the widget.

See docs/integrations.md and docs/docker-agent.md. Older releases → CHANGELOG.md.

v4.6.1 is a small patch on top of v4.6.0. It fixes two issues that only surface under the optional SCGI prefork worker, closes a ReDoS and a defence-in-depth XSS finding, and removes a brief UI flash on page reload. No breaking changes, no schema changes. Hard-reload once after upgrading (cache remotepower-shell-v4.6.1).

• SCGI worker: agentless ICMP devices stay online. The persistent worker runs with NoNewPrivileges, which blocked ping's capability elevation, so reachability sweeps reported agentless devices down. The unit now grants CAP_NET_RAW ambiently. Classic fcgiwrap/CGI was never affected.
• SCGI worker: Postgres fork-safety. A forked worker child inherited the parent's psycopg connection, corrupting the protocol stream (consuming input failed: EOF detected). Each connection is now tagged with its owning PID and transparently reconnected in a child — the same guard the SQLite backend already had.
• Security: ReDoS in TLS-host validation. _valid_tls_host used a nested-quantifier regex that backtracked catastrophically on a long letter/hyphen run. It now validates label by label in linear time, same accept/reject set.
• Security: package update count coerced before render. The per-device update badge coerces the upgradable count to a number (or ?) before it reaches innerHTML — a non-numeric value can never become markup. The strict CSP is an independent second barrier.
• UI: no subtitle flash on reload. Each page's descriptive subtitle is folded into a hover info icon at runtime; on a hard reload it briefly painted as raw text before the script ran. The subtitle is now hidden by default in CSS and revealed inline only in the Old UI.

A stability + hardening patch — every install keeps working exactly as before. See docs/v4.6.1.md. Older releases → CHANGELOG.md.

v4.6.0 gives RemotePower a look of its own — an Industrial interface that replaces the generic dark-slate template, while keeping our blue accent — and pairs it with a thorough sweep across the whole project: bug fixes, security hardening, performance, and data-binding polish. A toggle lets anyone keep the old look. Hard-reload once after upgrading (cache remotepower-shell-v4.6.0).

• Industrial "New UI" (default). Warm graphite/steel palette (keeping the RemotePower blue), with the sidebar/nav set in self-hosted IBM Plex Mono and the rest of the UI in the familiar Inter, plus instrument-panel motifs: corner ticks on panels, dashed technical rules, mono uppercase eyebrow labels, tabular figures, sharper corners. A genuine re-skin of the chrome (sidebar, headers, cards, tables, controls) via body[data-ui="industrial"].
• New UI / Old UI toggle. A new Settings → Interface tab (and a control in My Account → Appearance, so non-admins can switch too) flips between the Industrial look and the classic one. Per-browser, default New, applied instantly with no reload.
• Navigation tidy-up. A dedicated Admin group (Links now lives there), and every sidebar group is alphabetically sorted for quicker scanning.
• Bind-it-together & polish. The device drawer now shows CPU model, kernel, total RAM and total disk alongside the live usage; more lists and tables cap at ~15 rows then scroll internally; the disk-forecast table sorts correctly by GB and percent.
• Security hardening. Tighter SSRF guards on appliance / integration targets, resolved-role checks on a couple of read endpoints, and hardened agent credential storage on Windows and macOS. Independently pentested clean (see the security review).
• Performance. Fewer per-tick round-trips on the dashboard, single-row reads on a hot path, and lighter agent heartbeats (cached tool lookups, no blocking uptime probe).
• CSP-safe by construction. Pure CSS plus one data-ui attribute; the toggle is wired through the existing data-action dispatch with no inline scripts, styles or on* handlers.

Switch the look any time under Settings → Interface. See docs/v4.6.0.md. Older releases → CHANGELOG.md.

v4.5.0 makes TLS easy to stand up where a real cert is hard — internal / airgapped / no-public-DNS instances. A real cert is still recommended. Hard-reload once after upgrading (cache remotepower-shell-v4.5.0). Full guide: docs/tls-selfsigned.md.

• Self-signed CA, not a bare cert. make tls-selfsigned HOST=rp.internal (tools/gen-ca.sh) generates a private CA + a server leaf (ECDSA P-256, SAN, serverAuth), installs the leaf for nginx, and prints the CA's SHA-256 fingerprint. Agents trust the CA, so make tls-renew re-issues the server cert without touching any client.
• Fingerprint-verified rollout. install-client.sh --server https://rp.internal --ca-fingerprint <sha256> (and the macOS / Windows installers) fetch /ca.crt over HTTP and refuse to trust it on a fingerprint mismatch — a safe bootstrap with no TLS yet. The CA is wired in via RP_CA_BUNDLE; full verification (hostname + TLS 1.2 floor) is never weakened.
• One nginx routing snippet. The HTTP and HTTPS server blocks now include the same remotepower-locations.conf (and serve /ca.crt) so they can't drift — HTTPS is a clean uncomment.
• Docker opt-in TLS. RP_TLS_SELFSIGNED=1 makes the container generate a CA+leaf into the data volume on first boot and serve HTTPS on :8443, printing the fingerprint to docker logs.
• Self-signed → real is server-only. Agents trust the system roots and the CA simultaneously, so you point nginx at a real (Let's Encrypt) cert and reload — agents keep working with no redeploy.

A TLS-onboarding release — HTTP-only and real-cert installs are untouched. See docs/v4.5.0.md and docs/tls-selfsigned.md. Older releases → CHANGELOG.md.

v4.4.1 resolves the open CodeQL code-scanning alerts and runs a documentation-coverage sweep. No runtime behaviour changed. Hard-reload once after upgrading (cache remotepower-shell-v4.4.1).

• All 13 CodeQL alerts are false positives. Each (1 Critical, 12 High) was read against the actual code path and the feature that governs it — none is an exploitable vulnerability. The full write-up, mapping every alert to the feature/function that already covers it, is in docs/security-review-4.4.1.md.
• XSS / DOM-as-HTML (7). Every flagged innerHTML assignment interpolates only values that pass through the project's escHtml() / escAttr() sanitizers, which CodeQL's default model doesn't recognise; the strict no-inline CSP is an independent second barrier.
• SSRF (1, Critical). fetch('/api' + path) is the same-origin front-end API client — an origin-pinned relative path, never a user-supplied host. Not request forgery.
• Weak hashing (2). One points at the audit-log chain, which is HMAC-SHA256 (strong); the other is a non-security dedup key already marked usedforsecurity=False. The two MD5 cache-key fingerprints now carry usedforsecurity=False too — the signal the scanner honours.
• Clear-text logging / storage (3). The admin-gated diagnostics download (by design, no-store), a non-sensitive stderr diagnostic, and the app's own atomic chmod 0600 data-file write — none is a secret-disclosure path.
• Documentation pass. A sweep across the in-app docs, README, Manual and per-version notes confirmed every shipped feature/function is documented and the headline counts are accurate (65 webhook events, the 66-widget dashboard catalog, 18 MCP tools).

A documentation + triage release — every install keeps working exactly as before. See docs/v4.4.1.md and docs/security-review-4.4.1.md. Older releases → CHANGELOG.md.

Per-release notes for the five most recent versions are kept above. The complete, forensic release history — every version, newest first — lives in CHANGELOG.md at the repository root.

Quick (interactive): Click "Enroll device" in the dashboard. You get a 6-digit PIN, valid for 10 minutes. On the target machine: sudo remotepower-agent enroll, paste the PIN.

Automated (Ansible / cloud-init): Use the API to mint a one-time-use enrollment token (Settings → API or POST /api/enrollment-tokens), pass it via $REMOTEPOWER_ENROLL_TOKEN or /etc/remotepower/enroll-token, then run remotepower-agent enroll-token --server https://<host>. Token is consumed atomically — same one can't enroll twice. Default expiry 24h, max 7 days.

The token resolution order is: --token arg → environment variable → on-disk file (mode 600, auto-deleted on success).

Docker host (no install): Click "Enroll device → Generate Docker compose" for a ready-to-run docker-compose.yml (your server URL + a one-time token pre-filled). On the host: docker compose up -d. The agent runs as a container, monitors the host (shared PID/network namespaces, host rootfs mounted read-only) and is named after the host automatically. Standard capabilities, no --privileged. Compliance scans and systemd/journal collection aren't available from a container — see docs/docker-agent.md.

RemotePower fires metric_warning, metric_critical, and metric_recovered webhooks when a device's resource crosses its configured threshold. Defaults:

Per-device overrides: Devices page → ⋯ menu → "Metric thresholds". The modal shows current sysinfo values for context, then warn/crit fields per metric (empty = use default). Per-mount disk overrides go in the bottom section — useful when /var fills with logs or /backup is meant to fill.

Hysteresis: a metric must drop 5 points below its warn threshold before "recovered" fires. Without this, oscillation around 80% would generate webhook spam every 60s.

Trends over time: Monitor page → Device metrics row → "Trend" button. Same chart as the per-device sysinfo modal.

Click "Web terminal" on a device. Modal asks for SSH user, port, password, plus your RemotePower admin password (re-prompted every time, by design).

Architecture: a separate remotepower-webterm daemon handles the WebSocket+SSH bits because RemotePower's CGI can't hold persistent connections. nginx proxies /api/webterm/connect to it. SSH credentials live only in memory for the duration of the session — never persisted.

Setup (one-time): sudo bash packaging/install-webterm.sh. Auto-detects the CGI user (www-data, nginx, http, etc.), installs Python deps, generates the daemon ↔ CGI shared secret, prints the nginx snippet you need to add. Run with --dry-run first if you want a preview.

Recordings: Every session recorded to /var/lib/remotepower/webterm-sessions/<id>.cast in asciinema v2 format. Output-only by default (set RECORD_INPUT=1 in daemon env to capture keystrokes too — only do this if you've thought about who can read the recordings dir). Replay with asciinema play <file>.

Per-device dropdown menu has the common actions: shutdown, reboot, Wake-on-LAN (if device has a known MAC), agent self-update, upgrade packages, custom command.

Commands queue in cmds.json. The agent picks them up on its next heartbeat (default 60s, configurable per-device). Output comes back on the heartbeat after execution finishes — for apt upgrade that can be a few minutes.

Batch mode: tick checkboxes on multiple devices, the batch bar at the bottom of the page lets you run any of those actions across all selected devices at once.

Custom commands: "Custom command" in the menu, or save reusable ones to the Library (Admin → Library) and pick from a dropdown when you run them.

Settings → Webhooks. RemotePower auto-detects the format from your URL: Discord webhook URLs get embed-style messages; ntfy URLs get priority + emoji tags; everything else gets generic JSON.

Events you can subscribe to: device_offline / device_online, monitor_down / monitor_up, service_down / service_up, patch_alert, cve_found, log_alert, container_stopped / container_restarting, metric_warning / metric_critical / metric_recovered, command_queued / command_executed.

Each event can be toggled independently. The "Send test event" button fires a sample payload so you can verify your endpoint works before relying on it.

Monitor page → "Add target". The server runs these probes itself, not the agents — useful for checking your ISP gateway is up, a web service is responding, or an SSH port is open.

Probes run on a schedule (monitor_interval, default 300s, minimum 60s). The schedule piggybacks on incoming CGI requests, so as long as any agent is heartbeating or anyone's browsing, monitors run on time. State transitions fire monitor_down / monitor_up webhooks.

RemotePower can connect to one Proxmox VE node and manage its guests. Configure it under Settings → Proxmox: the node's host, the node name, and a Proxmox API token (created in Proxmox under Datacenter → Permissions → API Tokens). Use a scoped token — VM.Audit and VM.PowerMgmt are enough — not a full-access token. There's a "Test connection" button to verify the setup.

Once configured, the Virtualization page lists the node's QEMU virtual machines: status, CPU and memory while running, uptime. Each guest has Start and graceful Shutdown actions. The connection is server-to-API — the RemotePower server talks to the Proxmox REST API directly; no agent runs on the Proxmox node.

This is a single-node integration. The Virtualization nav entry is always visible; if Proxmox isn't configured the page simply tells you so.

LXC containers on the Proxmox node appear on the Containers page, in a section below the agent-reported Docker/Podman containers. They carry the same Start and Shutdown actions as QEMU VMs.

The LXC section only appears once Proxmox is configured (Settings → Proxmox). Like the VM list, it's fetched live from the Proxmox API each time you open the page.

Each Proxmox guest — QEMU on the Virtualization page, LXC on the Containers page — has a Snapshots button. It opens a panel listing the guest's snapshots and lets you create, roll back, and delete them.

Create — give the snapshot a name (a letter followed by letters, digits or underscores) and an optional description. Snapshots are disk-only — the VM's RAM state is not captured. A typical workflow: take a snapshot named before_upgrade immediately before a risky change.

Rollback — returns the guest to a snapshot's state. This is destructive: every change made since the snapshot was taken is discarded. Because of that, RemotePower asks you to type the guest's name to confirm — a deliberate speed bump, not just an OK button. After a rollback the guest is at a crash-consistent disk state (no live memory, since snapshots are disk-only).

Delete — removes a snapshot. Irreversible, but it does not affect the running guest — only the saved point-in-time state is gone.

Snapshot operations run asynchronously on Proxmox. RemotePower sends the request and refreshes the list shortly after; a slow operation may not show on the first refresh — reopen the panel. Every snapshot action is recorded in the fleet event log.

Set a default SSH username under Settings → Security → SSH preferences. It's stored per-user.

On the Devices page, a small SSH icon appears next to each device's hostname. Clicking it builds an ssh:// link to that device — using the device's IP if known, otherwise its hostname — with your default username, and also copies the plain ssh user@host command to your clipboard.

Whether the ssh:// link opens a terminal depends on your own machine having an ssh:// handler registered (the OS, PuTTY, a terminal emulator). If it doesn't, the copied command is the reliable path — paste it into any terminal.

A lightweight way to see a mailbox's message count without any IMAP/SMTP setup. Configure it under Settings → Mailbox monitor: pick a device, enter one or more absolute directory paths, and Save. The agent counts the regular files directly inside each directory and reports the numbers in its heartbeat — for a Maildir new/ folder that file count is the unread-message count.

Tick "Show this device's mailbox count on the dashboard" to add an "Unread mail" tile to the Home dashboard, alongside the devices / updates / drift / CVE tiles.

Counts refresh on the agent's schedule — roughly every five minutes, not instantly. The agent needs read access to the directory; if it doesn't have it, the count shows the reason (e.g. "permission denied") rather than a number. No email content is ever read — only files counted.

The agent submits its full package inventory (used for CVE scanning) and the patch/upgradable count only every few hours. Right after you patch a host, use the "Scan packages now" item in the device action menu to get a fresh report sooner.

It sets a one-shot request; the device sends a fresh package list and patch count within a heartbeat or two — typically a minute or two, not instantly. The agent must be running 2.4.5 or newer to act on it.

RemotePower can expose a small machine-readable fleet summary at /api/status for external dashboard tools — Uptime Kuma, Homepage, a Grafana panel.

Generate a status token under Settings → Advanced → Status endpoint. The endpoint then answers at /api/status?token=YOUR_TOKEN — reachable by a polling tool, but not public. It returns a rolled-up health word (ok / warning / critical), device online/offline counts, and attention counts by severity. Rotate or disable the token any time from the same place.

Settings → Security → "Enable two-factor". Scan the QR with any authenticator app (1Password, Authy, Google Authenticator, etc.). After enabling, every login asks for a 6-digit code in addition to your password.

To disable, you need to authenticate with a current TOTP code first — prevents someone with stolen session cookies from removing your second factor.

Every fleet table — Devices, Services, CVE Findings, Containers, Monitor, TLS, Patches, Audit Log, Command History, Schedule, Maintenance, plus admin tables — has a substring filter and clickable column headers.

First click sorts ascending, second descending, third clears. Hold Shift to add a secondary sort key (small superscript shows the priority).

The Devices grid has four density modes: Minimal (table layout, multi-select via checkboxes), Compact, Comfortable, Spacious. All preferences (filter, sort, density) sync per-user across browsers.

All state lives in /var/lib/remotepower/ as JSON files. To back up:

sudo tar czf rp-backup-$(date +%F).tar.gz /var/lib/remotepower/

To restore: stop nginx briefly so no writes interleave, untar, restart.

v1.12.1+ keeps a rolling .bak next to each file. If a file ever ends up corrupted, load() automatically falls back to the .bak with a warning to the nginx error log. The dashboard keeps working with last-known-good data.

Devices show "Offline" but agent is running: agent token doesn't match server. Check journalctl -u remotepower-agent for "Credentials rejected" — re-enroll on the device with sudo remotepower-agent enroll.

Web terminal fails with 404: nginx routes /api/webterm/connect to fcgiwrap instead of the daemon. The exact-match location = /api/webterm/connect block must come BEFORE any location ^~ /api/ in the same server block.

Web terminal fails with 502: nginx is correctly trying the daemon but the daemon isn't running. sudo systemctl status remotepower-webterm and journalctl -u remotepower-webterm.

Per-mount disk thresholds aren't taking effect: Agent must be v1.11.10+ to report per-mount data. Push an agent self-update from the toolbar.

Update history is empty for old agents: Pre-v1.11.7 had a bug where command output never reached the server. Push agent self-updates; the next upgrade will populate history.

Every dashboard action has an underlying API endpoint. Browse the full schema at /swagger.html (also linked from the sidebar as "API Reference").

Auth: two methods. Session tokens (login flow, time-limited) for short-lived scripts. Named API keys (Admin → API Keys → "New key") for CI / cron / Ansible — these don't expire. Pass via X-Token: <token> header.

Common patterns:

# List devices
curl -H "X-Token: $T" https://remote.example.com/api/devices | jq

# Reboot a specific device
curl -X POST -H "X-Token: $T" -H "Content-Type: application/json" \
  -d '{"device_id":"abc123"}' \
  https://remote.example.com/api/reboot

# Set per-device thresholds
curl -X PATCH -H "X-Token: $T" -H "Content-Type: application/json" \
  -d '{"mem_warn_percent": 70, "disk_per_mount": {"/var": {"warn":70,"crit":85}}}' \
  https://remote.example.com/api/devices/abc123/metric-thresholds

Home page. Every enrolled device shown as a card or table row, depending on density. Each card shows: status (online/offline), name, OS icon, last-seen time, group badge, tag pills, key sysinfo (CPU/RAM/disk sparklines if metrics are flowing), pending updates count, open CVEs count.

Density modes: top-right toggle. Minimal = table, one row per device, sortable, multi-select. Compact = small cards. Comfortable = default. Spacious = roomy cards with bigger metric blocks.

Per-device dropdown (⋯ button) covers the common actions: reboot, shutdown, custom command, agent update, upgrade packages, web terminal, metrics chart, metric thresholds, edit notes, edit tags, change group, delete.

Batch actions: tick checkboxes on multiple devices, the batch bar at the bottom lets you run any of the above across the selection.

Filter & sort: top-bar substring filter matches name, hostname, group, tags, IP, OS. Click any column header (in minimal mode) to sort; Shift-click for secondary sort. Filter and sort persist per user.

Per-device structured metadata that the agent doesn't auto-discover: asset ID, server function (web / db / nas / firewall…), hypervisor URL, SSH port, free-form Markdown documents.

Multiple documents per asset (v2.0): attach as many titled Markdown docs as you want — runbook, hardware spec, change log, vendor contacts. Each doc has its own title, body, and timestamp. Docs render as expandable cards in the asset view.

Credentials vault: store SSH passwords, BMC credentials, vendor portal logins encrypted at rest. Encryption is AES-GCM with PBKDF2-SHA256-derived keys; the passphrase is shared across admins (set once at vault setup). Reveal events are audit-logged. Each credential row exposes an ssh:// link button if the asset has an SSH user/port set.

Server functions list: editable in Settings, used as autocomplete suggestions when filling the function field on an asset.

All containers across your fleet, in one table. Auto-detected from each agent: Docker (via socket), Podman (rootless or root), Kubernetes pods (via kubelet, where applicable). Read-only — RemotePower doesn't manipulate containers, just observes them.

Per-row info: device, container name, image, status, restart count, ports, namespace (k8s). Filter by name, image, device, or namespace.

Alerts: three webhook events fire automatically — container_stopped (unexpected stop), container_restarting (restart count climbed), containers_stale (device hasn't reported container info recently — usually means the agent's container detection isn't working).

A read-only, server-side subsystem polls popular self-hosted software for health on a cadence and folds the result into the Alerts inbox and the dashboard. Nothing is installed on the target — each connector only reads a service's own health/status API. Set one up under Settings → Integrations: pick a type, point it at the service on your LAN, add an API token, and Test.

26 connectors: Pi-hole, AdGuard, TrueNAS, Unraid, Kubernetes/k3s, vCenter/ESXi, Proxmox Backup Server, UniFi, Traefik, Nginx Proxy Manager, Caddy, Netdata, Grafana, Uptime Kuma, Jellyfin, Plex, Home Assistant, Nextcloud, the download clients (qBittorrent, Transmission, Deluge, SABnzbd, NZBGet), the *arr suite (one Servarr connector for Sonarr/Radarr/Prowlarr/Lidarr) + Bazarr, and Overseerr/Jellyseerr.

Alerts & widget: an unhealthy or unreachable target raises an integration_down alert (auto-resolved on recovery) routed through your channels, plus an Integration health dashboard widget and live status badges.

Read-only & SSRF-guarded: every outbound call is blocked from loopback / link-local / cloud-metadata (RFC1918 LAN allowed), re-validated at connect time, with no redirects. Credentials are stored server-side and redacted from every response; the raw URL is admin-only. A Show Homelab software switch (default on) is an instance-wide kill switch — off stops polling and hides the section + widget.

The Linux agent can run as a container that monitors its Docker host and reports to the server, with no package install on the host. In the UI choose Enroll device → Generate Docker compose, then run docker compose up -d on the host.

The agent reads the host's facts (it shares the host PID and network namespaces and mounts the host root filesystem read-only), names itself after the host, and persists its credentials in a volume so a recreate doesn't re-enroll. Published multi-arch at ghcr.io/tyxak/remotepower-agent.

It runs with standard capabilities and no --privileged: SMART/DMI is an opt-in profile, and container inventory via the Docker socket is opt-in (with a host-root warning). Host package inventory and CVE scanning read the host package database directly; host compliance scans and systemd/journal collection are honestly reported as unavailable from inside a container.

The Monitoring → GPUs page shows every GPU across the fleet in one rich view — NVIDIA and AMD — with utilisation and VRAM meters, temperature, power draw and fan speed, ordered hottest-busiest first, plus a fleet summary (count, per-vendor, total power). Hosts report via nvidia-smi / rocm-smi, with a tooling-free amdgpu sysfs fallback for AMD hosts that have no ROCm tooling.

Each GPU card carries temperature + utilisation trend sparklines (about the last four hours), so you can see a card heating up or a job ramping without opening the host.

Thermal alerting: a GPU at or above the temperature threshold (default 85 °C, configurable) raises the standard high-temperature alert and auto-resolves when it cools — GPU sensors participate alongside the existing CPU/board sensors, so it reuses the existing hardware-temperature alert with no new alert type.

Unmonitored hosts show their data too. Telemetry and inventory views — thermal, power, storage, exposure, predictive-health/SMART, patches, listening ports, processes and this GPU page — now display unmonitored devices, flagged in the UI; only alerting stays suppressed for them.

Manual topology map drawn from each device's connected_to field. RemotePower doesn't auto-discover topology — set the upstream switch / router for each device, and the graph renders.

Switches, APs, and other agentless devices live as full-fledged records in the device list (with their own CMDB / vault / SSH-link as agented devices). Use Devices → Add agentless device.

The graph is positional — drag nodes around, the layout persists. Useful for finding the "who's downstream of this switch" answer at a glance.

The Monitoring sidebar group has six items that each deep-link to a section of this page and smooth-scroll to it on arrival:

• Targets — ICMP ping, TCP port, HTTP HEAD probes the server runs against external targets. State transitions fire monitor_down / monitor_up webhooks.
• Device Metrics (v1.12.0) — fleet-wide memory / swap / CPU / disk view, color-coded by alert level. Filter by name, group, mount path.
• Listening Ports (v2.8.1) — all open ports across monitored devices, grouped by port number with process name and device list. Live filter by port, process, or device.
• Custom Scripts — fleet-wide results table for your bash health-check scripts. Filter by name or status.
• Services — systemd unit health matrix for watched services.
• Logs — fleet log tail with regex pattern-alert rules.

Trend button (v2.0): next to each device in Device Metrics, opens the time-series chart modal for that device.

Server-side probes that warn before things expire. Add a hostname (or hostname:port for non-443) and the type — TLS for cert expiry, DNS for record-set health. Probes run on schedule alongside the monitor probes.

TLS: connects, fetches the leaf cert, reports expiry days remaining + issuer + SAN list. Alerts at <30 days, critical at <7. Self-signed and otherwise-invalid certs flagged with reason.

DNS: resolves A/AAAA/MX/TXT (whichever exist), reports the TTL of the soonest-expiring record. Useful for catching domain registrar issues before the domain itself lapses.

Both fire alerts via the existing webhook system — no separate config needed.

Aggregates pending-update counts from every agent. Linux agents run apt list --upgradable / dnf check-update / pacman -Qu / apk list --upgradable on a schedule (default every 3 hours) and report the count + package list back.

Per-device row: total upgradable, security-only count (where the package manager exposes that), oldest pending update age. Click into a device for the full list.

Patch alert webhook: Settings → Webhooks → Patch alert. Configurable threshold; fires patch_alert when a device has more than N pending updates.

Run upgrade: per-device dropdown → "Upgrade packages" runs the appropriate package manager non-interactively and returns the output as Update history. Batch select to upgrade many at once.

Define arbitrary bash scripts server-side and assign them to any set of enrolled devices. The agent runs each assigned script every 5 minutes with a 30-second timeout. Exit code 0 = OK; anything else = FAIL.

Create a script: Custom Scripts → New script. Paste a body directly, or type a description and click AI Generate to have the AI draft one. Assign the script to one or more devices using the device picker. Save.

Results: The Custom Scripts page shows a fleet-wide table — script name, device, status badge, last output snippet, when it last ran, and how long it took. Click any output snippet to see the full stdout/stderr.

Alerts: Status changes fire webhooks — custom_script_fail when a script flips from OK to FAIL (includes the first line of output), custom_script_recover when it returns to OK. Both are edge-triggered: they fire once on the transition, not on every failing run.

Script execution: Scripts run as the agent user (root on most setups), with stdout and stderr merged, capped at 4 KB. The script body is written to a private temp file (chmod 700), executed by /bin/bash, then deleted. Scripts are pushed from the server — no SSH needed.

Examples: check a web endpoint returns 200, verify a backup file was recently modified, test a database port is accepting connections, confirm a cron job's sentinel file exists.

Define the desired state of each Linux host server-side. The agent applies it on the next heartbeat (~60 s) and reports current state every 15 minutes so the server can detect drift.

Open the editor: Devices page → device dropdown → Host Config. A modal opens with one tab per section.

Sections managed:

• Repos — full content of /etc/apt/sources.list or /etc/yum.repos.d/remotepower.repo
• Netplan — written to /etc/netplan/01-remotepower.yaml, then netplan apply
• nmcli — connection file at /etc/NetworkManager/system-connections/remotepower-managed.nmconnection
• resolv.conf — DNS resolver config; resolves symlinks on systemd-resolved hosts
• /etc/hosts — static host entries
• Services — list of systemd units that should be enabled; agent runs systemctl enable --now
• Users — ensure local users exist with correct shell, groups, and SSH authorized_keys (no passwords)
• Groups — ensure local groups exist
• Sudoers — written to /etc/sudoers.d/remotepower, validated with visudo -c before applying
• MOTD — login banner written to /etc/motd

Fetch current: Each tab has a "Fetch current" button that loads the live state last reported by the agent. Use it to pre-fill a section before editing, or to inspect what's actually running.

Drift detection: The agent reports current state every 15 minutes. The server compares it against the desired config. If any section diverges, an amber banner appears on the modal and a config_drift webhook fires — once on first detection, not on every heartbeat. Drift is audit-only; the agent does not auto-remediate.

Security: Only admins can write host config. Sudoers content is syntax-checked before writing. authorized_keys are written with mode 0600 and correct ownership. No passwords are ever stored — SSH keys only.

Cross-references each agent's installed-package list against OSV.dev on a schedule (default daily). Findings are severity-ranked and grouped by device.

Each finding shows: CVE ID, severity (CRITICAL / HIGH / MEDIUM / LOW), affected package + installed version, fixed version (if any), summary, references. Click for the full OSV record.

Ignore list: known false positives or accepted risk can be ignored per CVE-ID + per-package combo. Ignored findings disappear from the active view but stay in audit history. The "Show ignored" toggle reveals them.

cve_found webhook: fires when new CVEs appear that aren't on the ignore list. Useful for plugging into the team's incident channel.

Per-device watched-services view. Define which units to watch in Settings → Service watch (or via API), and the agent reports their state on every heartbeat: active, inactive, failed, activating, etc.

Matrix view: rows are devices, columns are services; cell color reflects state. Filter by service name, device, or group; sort by failure count.

Webhooks: service_down when a watched unit transitions to inactive/failed; service_up on recovery. Maintenance windows suppress these without losing the audit trail.

Per-device journalctl output for watched units. Agents submit log lines on every heartbeat; server keeps a rolling 6-hour buffer.

Search: regex over the buffered logs. Filter by device, unit, time range. Useful for "did this error happen anywhere in the last hour?"

Pattern alerts: Settings → Log rules. Define regex patterns; matches fire log_alert webhook with the matched line, device, and unit. Useful for kernel oops, OOM kills, fail2ban bans, anything you want a notification on.

Per-device or fleet-global rules: rules can be scoped to a device (only matches there fire) or fleet-global (matches on any device fire).

Schedule reboots, shutdowns, package upgrades, or arbitrary commands. Two modes:

One-shot: pick a date+time, RemotePower fires the command once then discards the job.

Recurring: choose a friendly preset (every hour, every N hours, daily at HH:MM, weekly on day+time, monthly on day+time) or enter a custom 5-field cron expression. DOW follows cron convention (0 = Sunday).

Run a script: select "Run script…" in the command picker to choose any saved script from the Scripts library. The body is resolved at fire time so editing the script updates all pending recurring jobs.

Per-device, per-group, or fleet-global. Two optional checkboxes on create:

• Maintenance window: auto-checked for disruptive commands (reboot, upgrade); suppresses all suppressible alerts for 1 hour around the scheduled time.
• Add to calendar: creates a calendar entry at the next computed occurrence. For recurring schedules (daily, weekly, monthly), the calendar event inherits the same recurrence so all future runs appear in the calendar automatically.

Month-view calendar of upcoming scheduled commands and maintenance windows. Useful for "what's planned this week?" and avoiding accidentally double-booking maintenance.

Create events: click any day cell to create a one-off event. The event modal has a Recurrence dropdown: None, Daily, Weekly, Monthly, or Yearly. Recurring events are stored once on the server and expanded into occurrences within the displayed window (cap: 500). They appear with a refresh icon in the grid.

Delete recurring events: opening a recurring event shows a "Delete all occurrences" confirmation so a single click removes the entire series.

Schedule integration: the "Add to calendar" checkbox on the Schedule form creates a recurring event automatically when the schedule itself is recurring — daily/weekly/monthly schedules produce matching calendar recurrence.

Free-form per-fleet task list. Write down what you need to do, check items off, leave notes. Doesn't drive any automation — just a place to keep track of "rebuild X next maintenance window" or "investigate why Y is restarting".

Tasks have title, description, optional due date, status (open / in-progress / done). Filter by status; sort by due date.

Schedule windows during which webhook alerts are suppressed for specific devices, groups, or the whole fleet. Useful for planned downtime where you don't want to be paged for an expected outage.

Two modes: one-shot (start time + end time) or recurring (cron pattern + duration). Scope: device, group, or all.

Suppressed alerts are still logged — you can see exactly what would have fired during the window via the audit log. So if something unexpected happens during maintenance, you can find it after the fact.

Every command run via RemotePower, with actor (which user), target device, command, exit code, output preview, timestamp.

Used together with the Audit log (which covers admin actions like creating users, changing settings) to reconstruct what happened during an incident.

Rolling buffer of last 200 commands. For longer-term retention, configure log forwarding to your central logging system.

Tabs:

• Account — change password, enable/disable TOTP 2FA, view active sessions, revoke sessions.
• Webhooks — add destination URLs (Discord, ntfy, Slack, generic JSON), per-event toggles, "send test event" buttons.
• SMTP — outbound email settings for the digest endpoint and password resets.
• LDAP — bind URL, search base, attribute mappings; "Test connection" button.
• Service watch — list of systemd units the agents should monitor, applied to all devices unless overridden.
• Log rules — regex patterns for log_alert events.
• Server functions — autocomplete list for CMDB asset function field.
• Backup — one-click ZIP download of all state JSON files.

Manage local user accounts. Each user has a username, bcrypt password, role (admin or viewer), optional TOTP secret, optional email for password reset.

Roles: admin can do everything. Viewer is read-only — can browse all pages but every write endpoint returns 403.

LDAP users auto-create on first successful bind; their role defaults to viewer until an admin promotes them.

Named API keys that don't expire (unlike session tokens). Use for CI scripts, cron jobs, Ansible, Grafana scraping /api/metrics.

Keys are bound to a user account — when you create a key, it inherits your role. To make a "ci-bot" key with limited access, create a viewer-role user first and create the key while logged in as that user.

Pass via X-Token: <key> header (same as session tokens). Keys can be revoked instantly from this page; revoked keys 401 immediately.

Reusable shell-command bookmarks. Save common diagnostics ("show disk usage by mount"), restart routines ("kick the failing service"), or chained operations as named entries.

When running a custom command on a device, the modal has a dropdown that auto-populates from this library — no need to remember the exact find incantation.

Every administrative action: user created/modified/deleted, password changed, settings changed, API key minted/revoked, enrollment token created, vault unlocked, credentials revealed.

Each entry: timestamp, actor (user), action, detail string, source IP. Filter by actor, action, or substring; sort by any column.

For command execution, see the History page (separate). The Audit log is for "who changed what configuration" rather than "who ran what command". Now in the Security sidebar group alongside TLS/DNS, Patches, CVEs, and Drift.

Card grid grouped by category. Click any card to open the link in a new tab. Group links by category — useful for "Monitoring", "Infrastructure", "Vendor portals", etc.

Scope: mark a link as Internal (LAN-only, behind VPN, etc.) or External. Internal links get an amber dashed border; external links get an accent (blue) solid border — so it's visually obvious at a glance which links require VPN.

Dashboard widget: once at least one link is saved, a "Quick links" card automatically appears on the Home dashboard showing all links as a compact grid. Click "Manage →" to jump to the full Links page.

Display-only — RemotePower doesn't proxy or health-check these URLs.

Navigation: Links is a standalone top-level item in the sidebar, directly below Home — accessible to all users regardless of role.

Like the command library but for full bash scripts — multi-line, with `set -euo pipefail`, error handling, the works. Stored in scripts.json; created and edited via the Scripts page.

Save flow: paste a script → editor runs bash -n (syntax check) and a regex-based dangerous-pattern detector (rm -rf /, fork bomb, dd if=/dev/zero of=/dev/sda, curl|bash, chmod 777 /, etc.). Both must pass before save.

Run on a device: Devices page → ⋯ menu → "Run script…", pick from the library, confirm. Output comes back on the next heartbeat after execution finishes.

Batch run: tick checkboxes on multiple devices, batch bar → "Run script", pick from the library. The batch tracker shows per-device progress; output collects under the batch job ID for ~1h.

Tagged runs: instead of ticking devices, target a tag or group via the API: POST /api/exec/batch with tag, group, or device_ids.

Full reference: docs/scripts.md.

Disabled by default. Enable in Settings → AI assistant. Five providers: Anthropic (Claude), OpenAI (ChatGPT), DeepSeek, Ollama (local), LocalAI (local). Pure stdlib — no extra pip deps. Pick a local provider (Ollama / LocalAI) if you don't want data leaving the building.

Where the AI buttons appear:

• Device dropdown (⋯ menu) → Investigate, Generate runbook.
• Device detail modal → Explain on each command output, Find the problem on the journal panel.
• Services page → service detail → Diagnose on failed/inactive units.
• TLS page → table row → Triage on warning/critical/error certs.
• Patches page → table row → Prioritise on devices with pending updates.
• CVE Findings → row → Triage.
• Scripts editor → Generate from prompt / Explain / Audit for risks.
• Notifications → webhook log row → Explain.
• Help → AI Assistant → standalone chat with model picker and Ollama server stats.

Privacy: hostnames and IPs are redacted before leaving the building unless you toggle "Send hostnames" / "Send IP addresses" in Settings. Bearer tokens, AWS keys, and long hex strings are always redacted regardless. AI-generated scripts go through the same dry-run + dangerous-pattern detection as anything else — no AI-trusted bypass.

Slow local models (smallthinker, qwq, deepseek-r1) routinely take 60–180 s per response. The HTTP timeout is 5 min on the Python side, but nginx's fastcgi_read_timeout defaults to 60 s and will cut you off first — add a location /api/ai/ block with fastcgi_read_timeout 300s;. See docs/ai.md for the snippet and the full reference.

Rate limits: per-user per-day cap in Settings (default 100; 0 = unlimited). Per-button max_tokens tuned client-side so a short Explain doesn't sit waiting for 4000 tokens to generate.

What it is: a structured operations document for each device, generated by the AI from the device's current state (sysinfo, watched services, containers, recent commands, journal, CVE findings, patch status). Saved per-device; regenerable any time.

How to generate: Devices page → ⋯ menu → Inspect → Generate runbook. The AI gets the live snapshot of the device plus the fleet context (so it can say "this is your mail server" rather than just "this is a Linux box"). Takes 15–90 s depending on provider and model.

What's in it: sections for purpose / role, installed stack, running services, listening ports, scheduled jobs, recent activity, "things to know," and known risks. Format is Markdown — rendered inline on the device detail modal under a Runbook section.

When it's worth re-running: after a major change to the device, before handover, when documenting a system for a colleague, after the OS upgrade. The data is current as of when it ran — the timestamp is shown alongside the runbook.

Privacy: runbooks contain hostnames and IPs by design (a redacted runbook is useless). The redaction toggles in Settings → AI assistant apply, but you'll want "Send hostnames" + "Send IP addresses" on for runbook generation specifically. Storing hostnames in a runbook that lives in your own JSON file isn't a privacy concern; what matters is whether they cross over to a cloud provider on generate.

What it is: the agent computes SHA-256 hashes of a list of watched config files every few heartbeats and ships them in the heartbeat payload. The server stores baselines (first-seen hash per file per device) and fires a drift_detected webhook when a current hash diverges from the baseline.

Hash-only by design. The contents of /etc/sudoers, /etc/ssh/sshd_config, etc. never cross the wire on routine polling. To see what actually changed, an operator action queues a cat command through the existing exec mechanism (subject to the usual audit + permission checks).

Where to find it: sidebar → Security → Drift. Fleet-wide table shows devices sorted with drifted ones at the top; click Detail for per-file status, drift count, history of the last 20 changes, and the Accept as baseline button.

Default watched list: /etc/ssh/sshd_config, /etc/sudoers, /etc/fstab, /etc/crontab, /etc/hosts, /etc/resolv.conf, /etc/nsswitch.conf, /etc/pam.d/sshd. Customisable globally (cfg['drift']['default_watched_files']) and per-device (device.watched_files).

Webhook: drift_detected fires once per change (debounced — not on every poll that reports the same new hash). Route it to a channel you check, especially for after-hours alerts.

Compliance: covers configuration management controls in SOC 2 (CC6.1, CC6.6), ISO 27001 (A.12.4.3, A.14.2.4), HIPAA (164.312(c)), PCI DSS (11.5), FedRAMP. Baseline-acceptance audit-log entries are designed to be readable as evidence.

Agent version: requires v2.2.0+ for the agent-side hash reporting. Older agents work normally otherwise but show "no drift data" on the Drift page.

Full reference: docs/drift.md.

What it is: RemotePower ships an MCP (Model Context Protocol) server at mcp/remotepower-mcp.py. Connect it to an MCP-compatible AI host and you can ask things like "Which devices have pending security updates?" or "Show me the journal for mail01 from the last hour" in plain English.

Architecture: the MCP server runs on the operator's laptop, not on the RemotePower server. The AI host (Claude Desktop, Cursor, etc.) spawns it as a stdio subprocess. The MCP server makes HTTPS calls to RemotePower's REST API on behalf of the AI using an API token you provision. Credentials live in the host's config file and never reach the AI provider.

18 tools: 14 read + 4 guarded write. Read: list_devices, get_device, get_journal, get_services, get_containers, get_cves, get_drift, get_recent_commands, get_runbook, get_patches, get_tls, get_snmp_data, search_devices, search_fleet. Write — each gated behind a server-side allow-list, per-MCP-token role, and an explicit confirmation flag: reboot_device, run_saved_script (saved, vetted scripts only — no free-form commands), force_package_scan, force_acme_rescan. Every write is audit-logged with the AI host and the prompt that triggered it.

Setup (Claude Desktop):

1. Settings → API keys → Generate a viewer token; copy the rpk_... value.
2. Copy remotepower-mcp.py to your laptop (scp works).
3. Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows) — add an mcpServers.remotepower entry with the command, args, and env (URL + token).
4. Restart Claude Desktop. The tool indicator should show remotepower.

Device-name resolution: exact → prefix → substring → ambiguity error. You can pass web01 and the server finds web01.example.com automatically (as long as there's exactly one match).

Pure stdlib Python. No pip install. A single ~900-line file.

Full reference: docs/mcp.md.

RemotePower has webhook (Discord / Slack / ntfy / generic JSON) and SMTP email channels. Pick whichever you actually check.

Recommended baseline (most operators want):

1. One webhook for page-me-now events: device_offline, service_down, monitor_down, metric_critical, cve_found. Point at ntfy / Pushover / your phone.
2. One webhook for FYI-during-business-hours events: metric_warning, patch_alert, log_alert, container_stopped. Point at a Discord/Slack channel.
3. Email digest for the slow stuff (weekly CVE summary, patches pending) once cron_email_digest is enabled.

Per-event toggles: each webhook URL has independent toggles for every event type. The "Send test event" button fires a sample payload so you can verify the integration works before relying on it.

Maintenance windows suppress alerts during planned downtime. Suppressed alerts are still logged — you can see what would have fired in the audit log. So if something unexpected happens during maintenance, you can find it after.

Explain on alerts: each webhook log row has an "Explain" button that asks the AI to rewrite the raw alert into a single short paragraph ("postfix on mail01 stopped at 14:32 after matching connection refused 3 times — likely upstream MX is unreachable; check DNS and try a manual delivery"). Useful during on-call when you're seeing a wall of cryptic alerts at 3am.

Clicking a device name (Devices page or Dashboard fleet roster) opens a full-screen drawer with two tabs:

Actions & Settings tab — all actions for that host in one grid: Run command · Reboot · Shut down · Wake on LAN · Upgrade packages · Scan packages · Web terminal · Run script · Update agent · Docker compose · Host config · AI Investigate · CMDB · Runbook · Maintenance · Adjust poll · Remove device. Below the grid: inline settings form (group, tags, monitored toggle, poll interval, watched services, log rules, drift files, command allowlist) with a Save button.

Audit tab — 11 collapsible sections, all lazy-loaded on first open: System info (uptime, IPs, mounts, AI "Find the problem") · Listening ports (searchable) · Packages · Logs (unit selector) · Command history (last 5, expand per entry) · Fleet events · Drift state · CVE summary · Containers · Metrics · Host config.

The ⋮ button on the Devices page also opens the drawer directly on the Actions & Settings tab.

Settings → Dashboard tab controls what appears on the home dashboard for all users:

Brute-force detection — enable/disable, set threshold (default 20 failed attempts from same IP in a 5-minute window), and window length.

Backup file monitoring — add file paths and max-age thresholds. The agent checks mtime every heartbeat; backup_stale webhook fires edge-triggered and a Needs Attention item appears.

Process thresholds — watch a process by name and fire process_alert (edge-triggered, with a process_recovered follow-up) when its CPU or memory percentage crosses the limit. Evaluated server-side against each heartbeat's top processes.

Log ignore patterns — regex patterns (case-insensitive) that silently drop matching log lines before storage and alerting. Useful for noisy kernel notes like Note: After setting.*old kernels.

Needs Attention kinds — one toggle per alert type. Toggling a kind off hides it from both the Needs Attention panel and the Recent Activity feed simultaneously.

Recent Activity events — toggle the remaining informational events (device online, command queued/executed) independently.

Fleet-wide view: Monitor page → Listening Ports section. Shows all open ports across all online monitored devices, grouped by port number, with process name and which devices are listening. Shows 10 rows by default; expand button for the rest.

Per-device view: Device drawer → Audit tab → Listening Ports section. Searchable table filtered by port number or process name.

Port data is collected by the agent as part of sysinfo (every ~10 minutes) and persisted in devices.json. New ports trigger a new_port_detected webhook compared against the port baseline.

RemotePower checks installed packages against OSV.dev. The dashboard tile always shows Critical CVEs first with the count, and high/med/low counts in the subtitle.

Scan all devices — runs OSV lookups for every device that has submitted a package list. Can take a minute for large fleets. Button shows progress and result.

Send list (per-device) — asks the agent to send its full installed package list on the next heartbeat (~60 seconds). The CVE scanner then runs automatically. Use this after installing new packages before the agent's next scheduled scan.

Scan (per-device) — re-runs the OSV lookup using the already-stored package list. No need to wait for the agent.

The dashboard tile shows 0 for critical even when there are only high/medium/low findings — click it to see the full breakdown per device.

The IaC Generator page (left nav) produces Terraform, Ansible, Pulumi, or Cloud-init code that describes a managed device's current state. Useful for codifying a host you've configured by hand, or for migrating a host to a new platform.

Flow:

1. Select a device, the categories of state you care about, and an output format.
2. Click Generate IaC (full LLM flow) or Gather RAW JSON (collect only, download masked state).
3. On the agent's next heartbeat (~60 seconds), it collects the requested categories and sends raw state back.
4. The server calls your configured AI provider (Settings → AI) with the raw state, and returns the resulting code.

Categories (18): OS & identity, installed packages, systemd services (enabled), local users (uid≥1000), groups (gid≥1000), SSH authorized_keys (LLM converts to variables), network configuration, mounts (fstab), containers (Docker/Podman), custom repos, firewall, cron jobs (incl. RemotePower scheduled), TLS certificates (paths only), system environment (non-default), snaps (Ubuntu), kernel modules (persistent), sysctl parameters (non-default), RemotePower-specific (tags, group, custom scripts, host-config desired state).

Output formats: Terraform (HCL), Ansible (YAML), Pulumi (Python), Pulumi (TypeScript), Cloud-init (YAML).

Output tabs: Generated Code | AI Conversation. The Conversation tab shows the full system+user prompt and the raw LLM response, so you can see exactly what the model produced before fence-stripping.

Re-run AI / JSON download: After a successful generation, the Re-run AI button re-prompts the LLM using the already-collected data (no second agent wait). The { } JSON button downloads the masked state as a JSON file.

Extra instructions: A textarea below the format dropdown lets you append custom guidance ("use Terraform 1.5+ syntax", "wrap in a reusable module"). Persisted in localStorage.

Reasoning-model safety net: The prompt instructs the LLM to wrap output between <<<BEGIN_IAC>>> and <<<END_IAC>>> markers. The stripper extracts only what's between them — so DeepSeek-R1-style reasoning prose gets discarded automatically.

Security: Before the payload leaves your server for the LLM, env vars whose name matches PASSWORD|SECRET|TOKEN|KEY|PASS|AUTH|CRED|PRIVATE are masked with <REDACTED_BY_REMOTEPOWER>. TLS certificate paths are sent but never the certificate contents. SSH authorized_keys are sent raw — the LLM is instructed to convert them to variables.

Endpoints: POST /api/iac/request · GET /api/iac/status/<id> · POST /api/iac/generate · GET /api/iac/payload/<id> (v3.0.0)

Each Needs Attention card and each stale Containers row now has a × button that hides that specific entry from view. Ignores are per-item (one specific CVE on one specific device, not the whole category).

Restore: Settings → AI Assistant → Ignored items lists every hidden entry with a Restore button. Three categories are tracked separately: Needs Attention, Stale containers, Devices.

Stability: Needs Attention items are keyed by a stable SHA1 of kind+device+summary, so the same alert firing in a later poll stays hidden until you restore it. New variants (e.g. the same CVE but a new package version) get a fresh key and re-appear.

Storage: /var/lib/remotepower/ignored_items.json

Endpoints: GET /api/ignored · POST /api/ignored · POST /api/ignored/remove

The Logs page ingestion path was rearchitected to handle re-submitted file-based logs (apt.history, nginx access, syslog) correctly.

Previous bug: agents resubmit log files on every poll. The server used to stamp every line with now, so apt.history entries from days ago appeared as "new" and the 6-hour TTL never evicted them. Worse, the resulting bloat pushed nginx and brute-force lines past the 2 MB byte cap (silently dropped).

Fix (v3.0.1):

1. Each incoming line is hashed (sha1[:16]) and compared against the buffer. Duplicates are dropped at ingestion.
2. The line's own embedded timestamp is used when present. Supported formats: apt.history Start-Date: YYYY-MM-DD HH:MM:SS, nginx access [DD/MMM/YYYY:HH:MM:SS +TZ], syslog Mon DD HH:MM:SS, ISO YYYY-MM-DDTHH:MM:SS.
3. The 2 MB byte cap was removed. With dedupe + TTL, the buffer can't grow unboundedly on idle units anymore.

Impact: If you upgraded from a buffer that had 11 000+ apt.history lines, give it one full TTL window (6 hours) for old entries to drop out. Nginx and brute-force events should re-appear immediately after restart since they're no longer being crowded out.

Log alert rules (per-device and fleet-wide) carry a severity field — OK, WARN, or CRIT — modelled after CheckMK's classification.

Semantics:

• WARN (default) — fires the log_alert webhook with severity: "WARN".
• CRIT — same as WARN but with severity: "CRIT" in the payload, so your alerting routes accordingly.
• OK — silent: the rule fires (the match is recorded internally) but no webhook is sent. Use OK rules as noise-suppressors that confirm an expected pattern is still present, without alert spam.

Set the severity when creating or editing a rule in the Logs page rule modal.

Each AI feature (IaC Generator, Investigate, Diagnose Service, Runbook, etc.) uses a system prompt to set tone and constraints. Defaults are tuned for general-purpose models but may need adjusting per-model — a DeepSeek-R1 prompt doesn't necessarily fit a small local Llama.

Where: Settings → AI Assistant → Prompt customization. One card per AI feature (15 cards total).

Per card:

• System prompt textarea — edit; click Save prompt to persist; clear and save to revert to default.
• ● customized badge appears when overridden.
• Default button restores the hardcoded default for that feature.
• Fine-tuning (collapsible) — per-call temperature (0.0–2.0), top_p (0.0–1.0), max_tokens (1–16000), num_ctx (512–131072, Ollama/LocalAI only). Empty fields fall back to provider defaults.
• ● tuned badge appears when any fine-tuning value is set.

Storage: config.json under ai_prompt_overrides and ai_param_overrides.

Endpoints: GET /api/ai/prompts · POST /api/ai/prompts · GET /api/ai/params · POST /api/ai/params

The Device Drawer's Force-upgrade button pushes the currently-bundled agent binary to a device, even if the agent already reports the same version.

Use cases: recovery after a corrupt or truncated update, pushing a rebuilt binary at the same version, testing the self-update path.

How it works: the server sets force_agent_upgrade: true on the device record. The next heartbeat (within online_ttl seconds) delivers the flag and the server clears it. The agent calls check_for_update(force=True), which skips the version comparison and re-downloads + replaces its binary. Systemd then respawns the agent with the new binary.

Endpoint: POST /api/devices/<id>/agent/force-upgrade (admin only)

Confirmed in the audit log as agent_force_upgrade.

The "RemotePower v<X> is available" banner has a Snooze 30d button. Click it to hide the banner for that specific version for 30 days.

Snooze is per-version, so a newer release re-shows the banner. State lives in localStorage under rp_version_snooze_<version>.

Custom scripts let you ship arbitrary monitoring logic to selected devices, run on a schedule, and surface output in the UI.

Library: Scripts page → New script. Each script has a name, description, runtime (bash or python3), assignment selector (devices/tags/groups), schedule (interval in minutes), and timeout. Server pushes the body and metadata in the heartbeat response; agent stores them locally and runs them on schedule.

Output: per-device → Scripts page → Click a device row, or via the Device Drawer → Custom scripts chip. Each run records rc, output (capped at 8 KB), and ts.

Security: Scripts are pushed by the server, signed implicitly via the device token, and run as the agent user (root by default). Treat the script library as you would a CI/CD pipeline.

Per-device Host Configuration captures a desired state for a small set of system attributes; the agent reconciles on every poll.

Where: Device Drawer → Host config.

Tabs:

• Packages / Sysctl / Units / Modules — structured fields. Each reconciled by the agent on every poll.
• Logrotate — free-text content for /etc/logrotate.d/remotepower. The agent writes this file verbatim; useful for adding custom log rotation without SSH.
• Cron (root) — free-text content for root's crontab (crontab -u root). The agent applies via a temp file on each poll; useful for device-specific jobs you want managed centrally.

Reconciliation: agent reports the current state alongside the desired state. A diff is visible in the UI. Drift events are surfaced under the device's Drift tab.

Files: server keeps host_config_current/<dev_id>.json for each device's reported state.

The UI ships an in-page debug logger that captures api() calls, toast() events, click handlers, and other instrumented events.

Enable: Settings → Advanced → Enable debug logging. State persists in localStorage under rp_debug, so a fresh page load picks it up immediately.

Output: debounced (250 ms) flush to the server, surfaced in the audit log and (when enabled) a developer console panel.

Use this when something looks broken but the server logs are quiet. Most "the button doesn't fire" bugs show up clearly here.

The Documentation page header and the About page both have a Report an issue button. It opens a prefilled GitHub bug report in a new tab.

What it includes: the RemotePower version, your browser/environment, the current page, and any client-side errors captured this session (automatically scrubbed of token/secret-shaped values).

What it never includes: credentials, session tokens, or fleet/device data. The report is a public GitHub issue — review it before submitting. Security vulnerabilities should go through SECURITY.md (private), not the public tracker.

Several security improvements accumulated through the 2.x and 3.0 line:

• Password storage: PBKDF2-HMAC-SHA256 at 600 000 iterations with per-user salt. The previous bare-SHA-256 fallback was removed.
• Default admin: the first admin/admin account has must_change_password set. A persistent banner in the UI nags until the password is changed.
• Backup export: the /api/backup export now redacts Proxmox tokens, SMTP passwords, LDAP bind passwords, and similar secrets. The backup is otherwise full-fidelity.
• IaC payload masking: env vars whose name matches PASSWORD|SECRET|TOKEN|KEY|PASS|AUTH|CRED|PRIVATE are masked before being sent to the LLM. TLS cert paths are sent, never the contents.
• Recursion depth guard (v3.0.1) on the IaC mask walker so a pathological JSON payload can't trigger a RecursionError.

Lives under Security → TLS / DNS expiry, in the "ACME certificates" section below the existing TLS watchlist.

How it works: the agent walks ~/.acme.sh/ on each device (root, $HOME, or /etc/acme.sh/ — first match wins), parses every <domain>/<domain>.conf, and reports state to the server. No credentials leave the device — the agent uses whatever's already in ~/.acme.sh/account.conf or env. acme.sh's own cron handles renewals; RemotePower visualises and provides force-renew/revoke/cancel.

DNS-01 only. v1 supports dns_cf (Cloudflare) explicitly with credential-location hints; the provider dropdown lists Route53, DigitalOcean, deSEC, Hetzner, Porkbun, etc. RemotePower never touches nginx/apache/HTTP-01 plumbing.

Wildcards are supported via the Issue wizard's checkbox. Must-staple is intentionally not exposed — Let's Encrypt is sunsetting OCSP (August 2025), so OCSP-stapled certs would fail.

Cancel pending actions: any queued renew/issue/revoke with no rc yet shows a Cancel button. If the action is still in the queue it's removed cleanly; if the agent has already grabbed it, RemotePower stops polling but the agent may still complete the operation (last-write wins on the meta).

Per-domain action logs (captured stdout from acme.sh, 256 KB cap) live in /var/lib/remotepower/acme_logs/.

Every Needs Attention card with a supported alert kind (patches, disk, drift, service_down, reboot, brute_force) shows an Investigate button. Three-tab modal:

1. Diagnostic — server queues a hardcoded read-only command on the agent (e.g. for disk: df -h + top dirs + files >500MB + journal disk usage). Live-polls every 2s, up to 3 min.
2. AI Analysis — when diagnostic completes, the AI runs automatically with a playbook-specific system prompt (5 new keys in Settings → AI Assistant: mitigate_cpu, _memory, _disk, _service, _patches). It outputs a root cause + one specific fix command between BEGIN_FIX / END_FIX markers.
3. Apply Fix — choose pre-approved playbook fix, AI-suggested fix, or paste your own. Safety classifier shows red banner for denylisted commands (rm -rf /, dd of=/dev/sd*, fork bombs, etc. — refused outright) or amber for sensitive ones (reboot, kill -9, systemctl stop, apt purge, curl | bash — requires typing RUN).

All investigations and fixes go to the audit log. Output captured per-action in /var/lib/remotepower/mitigate_logs/.

Diagnostic commands are server-defined — user input never flows into the shell. Service unit names go through a strict regex (alphanumeric + ._@-) before any template substitution; shell metachars and path traversal are rejected.

Needs Attention = "fix this NOW". Only items whose underlying state is currently broken: a service that's down right now, a probe that's failing right now, pending patches that still need to be applied, etc.

Recent Activity = event log of things that happened in the past — transitions, dispatches, ACK'd alerts. A service that went down at 14:30 and came back up at 14:35 shows in Recent Activity but disappears from Needs Attention once it's healthy again. Click the ✕ Clear button in the Recent Activity header to hide all current entries; this lasts for the browser session (persisted via sessionStorage) and resets when you close the tab.

v3.0.1 audit (attention coverage): these state-derived kinds now appear in NA whenever the condition is active:

• service_down — any watched systemd unit in failed (critical) or inactive/deactivating (warning). Carries the unit name as target so the Investigate button can run systemctl status + journalctl -u <unit> automatically.
• monitor_down — any monitor target whose latest probe came back ok: false.
• custom_script_fail — any custom monitoring script reporting non-zero rc in its latest result.

Already covered: offline, patches, cve, drift, mailbox, brute_force, snapshot, backup, disk, tls, reboot, agent_version.

Per-device and fleet-wide log rules live in Settings → AI Assistant → Log alerts. Each rule:

• Source type: systemd unit (via journalctl) or arbitrary file path (agent tails the file directly).
• Pattern: Python regex against each new line.
• Threshold: minimum matches before firing.
• Severity: OK (silent on match — used to suppress noise), WARN, CRIT.

File-path rules: agent tracks inode + byte position in /var/lib/remotepower/file-log-state.json. Rotation (inode change) resets to position 0; truncation also resets. First-time setup skips existing content — only new lines from then on are sent, so a freshly-configured rule doesn't dump the entire historic file.

Submitted as synthetic unit file:<path>. Caps: 200 lines and 256 KB per poll, per file.

Per-device drawer → Force upgrade button. Sets a one-shot flag on the device record. On the next heartbeat (within 60s), the server includes the flag in the response; agent calls check_for_update(force=True), which bypasses the version-compare check and re-downloads the binary regardless of "we're already at version X". Flag is cleared atomically inside the heartbeat lock so it fires exactly once.

v3.0.1 fix: the flag had been silently dropped (copied to the wrong scope inside the heartbeat handler) so the operator got a success toast but nothing happened. Now wired through correctly, with a regression test. Use for re-deploys, recovery from corrupted binaries, or rolling out same-version rebuilds.

Grouped layout: the sidebar organises all pages into five collapsible sections. Each group opens and closes independently; state persists in localStorage. Fleet and Monitoring are expanded by default; Admin is collapsed.

Monitoring deep-links: each item in the Monitoring group navigates to the Monitoring page and smooth-scrolls to its section — so clicking "Listening Ports" skips straight to that section without manual scrolling.

Narrow mode: click ◀ at the top to shrink the sidebar to a 56-px icon strip. Click ▶ to expand. Preference saved in localStorage and applied before first paint. Mobile ignores this — the slide-in drawer is unchanged.

Standalone items: Home and Links sit above the groups as always-visible direct links (no group required to reach them).

Patch status detection works across all four: apt (apt list --upgradable), dnf (dnf check-update), yum (yum check-update — RHEL 7 / older CentOS, reports as manager: dnf so it shares the CVE path), pacman (pacman -Sy + pacman -Qu).

pacman 7+ sandbox: the new download sandbox user (alpm) fails on hosts where that user isn't usable — CachyOS in particular. v3.0.1 probes pacman --help for --disable-sandbox support and passes it when available, in both the agent's status check and the server's _UPGRADE_CMD. On real failure, upgradable is reported as None (UI shows "unknown") instead of 0 ("fully patched") — the previous silent-0 was the reason CachyOS devices appeared up-to-date when they weren't.

CVE scanning: uses OSV.dev. Mapping: Debian/Ubuntu by codename, Rocky/AlmaLinux/RedHat for rpm-based, no Fedora (no OSV feed). Arch / CachyOS intentionally unsupported — Arch packages aren't in OSV.

Every Needs Attention card and Containers row has an × button. Click it to hide the item; restore from Settings → Ignored items. Three categories: Needs Attention (per-alert), Stale containers (per-container), Devices (whole-device hide on Containers page).

Stored in /var/lib/remotepower/ignored_items.json. Ignores survive backups and aren't redacted in the export.

Up to 20 webhook destinations, each with its own format adapter (Discord, Slack, Pushover, Teams, ntfy, generic JSON). Per-destination filters: limit by event name or minimum priority. Configure in Settings → Notifications → Webhook destinations.

Pushover uses form-encoded POST with token + user + message + priority. Internal critical (priority=2) maps to Pushover priority=1 (high), not 2 (emergency tier) — that requires retry+expire and forced acknowledgment, must be operator-explicit. Credentials are write-once-and-redacted (UI shows ••••• (set)) and redacted from the backup export.

The legacy single webhook_url field still works for backward compatibility. New setups should use the multi-webhook editor.

Full doc at docs/webhooks.md.

New sidebar entry. Reports server version + memory, /var/lib/remotepower disk usage with the top 20 largest files, fleet device freshness, webhook delivery success rate (24h + 7d), audit log size, and scheduled backup state.

Closes the "who monitors the monitor?" gap — if RemotePower itself starts misbehaving (webhooks failing silently, disk filling, agents stuck), the page surfaces it. GET /api/self/status works for external monitoring (Uptime Kuma, Grafana, Homepage).

Full doc at docs/self-monitoring.md.

Daily gzipped tarball, 14-day retention (both configurable in Settings → Advanced → Scheduled backup). Triggered via the heartbeat hook with a 24h sentinel and stale-lock recovery.

Manual triggers (Server status page):

• Run backup now — immediately snapshots /var/lib/remotepower regardless of the 24h cadence.
• Export backup — also triggers a snapshot before streaming the ZIP download.
• Clear backup archives — deletes all remotepower_data_*.tar.gz from the backup directory and resets the backup state. Useful before a migration or to reclaim disk space. Requires confirmation.

Excluded from tarball: the backups directory itself, in-flight .tmp.* files, existing .gz archives. Owner/group stripped so restoring on a different host doesn't fail.

No in-UI restore — backups are tarballs you extract yourself. Procedure in docs/self-monitoring.md.

From the command palette (/ → "Bulk actions") or Settings → Advanced → Bulk actions. Filter by all monitored / by group / by tag, then pick an action: upgrade packages, reboot, shutdown, force package scan, force ACME rescan.

Destructive actions (reboot, shutdown) require typing RUN to confirm. Every queued command is audit-logged. No undo once a command is queued — operator must reach the agent another way (SSH) to cancel before the next heartbeat picks it up.

Full doc at docs/bulk-operations.md.

Press / or Ctrl-K to open the global search. Indexes all pages, every device (from the cached list), and quick actions. Arrow keys to navigate, Enter to activate.

? shows the cheat sheet. g-prefix shortcuts: press g, then h/d/l/s/c/m/a/v within 1.5 seconds to jump to Home/Devices/Logs/Settings/CVE/Monitor/Audit/serVer-status.

All shortcuts disabled when an input field has focus. Full doc at docs/keyboard-shortcuts.md.

When you've issued or renewed a cert via the acme.sh CLI on the host and don't want to wait up to an hour for RemotePower to catch up: from the ACME table, click Rescan on the device row.

Sets a one-shot force_acme_rescan flag on the device. Next heartbeat (within 60s by default) carries the signal, agent re-walks ~/.acme.sh/ regardless of the ACME_CHECK_EVERY cadence. Same flag-on-heartbeat-lock pattern as force-upgrade and force-package-scan.

CGI gives a fresh interpreter per request, so the cache lives only as long as one handler. Within that handler though, CONFIG_FILE was being parsed up to 4× per heartbeat and LONGPOLL_FILE 3× in handle_longpoll_exec. The cache deduplicates redundant reads.

Two safety properties: deep-copy on every load() so caller mutations don't corrupt the cache, and explicit invalidation in _LockedUpdate.__exit__ when the save aborts due to exception — otherwise the next load would return uncommitted in-flight changes.