Create a rasberrypi powered TV box to connect to various services through a tailnet
  • Shell 86.7%
  • Python 10.7%
  • Go Template 2.6%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Kurt Anderson 26fd3abc86 Root-own /opt/kiosk, closing a kiosk->root escalation
provision.sh chowned all of /opt/kiosk to kiosk, including
kiosk-wifi-reset.sh, which kiosk may run as root via NOPASSWD sudo, and
the root-run firstboot/firewall scripts. Anything running in the Kodi
session could rewrite one and get root. Now root owns the directory and
its scripts. Only the two files kiosk writes stay kiosk-owned: the
multistream selection and its Dispatcharr login, now mode 600.

This also fixes the setup hotspot's firewall exception: comitup runs
external_callback as the script file's owner, so as a kiosk-owned file
its `nft add element` silently failed and phones never got a DHCP
lease.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 23:51:37 -07:00
files Fix setup hotspot's captive portal under exit-node routing 2026-09-26 23:51:37 -07:00
fleet
zabbix
.env.example
.gitignore
build.sh
check.sh
KODI-MIGRATION-PLAN.md
provision.sh Root-own /opt/kiosk, closing a kiosk->root escalation 2026-09-26 23:51:37 -07:00
README.md

pi-stick-kiosk

Turn a Raspberry Pi 4/5 into a reproducible, open, HDMI-stick-style media client — Jellyfin, Plex, and Live TV (via Dispatcharr) all inside Kodi, no vendor lock-in, boots straight into Kodi's home screen. A remote/keyboard is needed once for the golden device's interactive addon setup (see KODI-MIGRATION-PLAN.md Phase 1); after that, a properly-paired unit needs no keyboard/mouse day to day.

Design goals:

  • One golden image, cloned onto every stick. You configure a device once — including signing into Jellyfin/Plex/Live TV addons inside Kodi — shrink the SD card into a .img, and every future stick is just "flash this file." No re-running install or sign-in steps per unit.
  • Per-device config lives on the boot partition, as a plain text file editable from Windows/Mac/Linux without booting Linux at all — hostname, Wi-Fi, and Tailscale/Zabbix settings, specifically. (Jellyfin/Plex/Live TV access is signed into Kodi once on the golden device and cloned along with everything else, not templated per-stick — see "Managing sticks" below for what that trades away.)
  • Safe to clone. SD card cloning has three classic footguns — stale SSH host keys, duplicate machine-id, unexpanded filesystem. A first-boot service fixes all three automatically, once, per device.
  • Survives being plugged into a random TV. HDMI hotplug/EDID quirks are handled in config.txt so it doesn't come up blank on a TV it wasn't tested against.
  • Idempotent provisioning script. provision.sh can be re-run safely if you want to update the golden image later — it doesn't double-install or duplicate config.

How the pieces fit together

sillygoose/
├── .env.example                  # kept dormant for a future per-device secret
├── build.sh                      # renders .env values into local/ (gitignored) — currently unused
├── provision.sh                  # run ONCE on the golden device (as root)
├── KODI-MIGRATION-PLAN.md        # phase-tracker for the current Kodi-based setup
├── files/
│   ├── kiosk.env.example         # template for /boot/firmware/kiosk-config.txt
│   ├── kiosk-firstboot.service   # runs once per cloned SD card
│   ├── kiosk-firstboot.sh
│   ├── kiosk-session.service     # the actual kiosk session — launches Kodi
│   ├── kiosk-heartbeat.service   # periodic health status -> journal
│   ├── kiosk-heartbeat.timer
│   ├── kiosk-heartbeat.sh
│   ├── kiosk-logs-bundle.service # nightly log bundle for remote pull
│   ├── kiosk-logs-bundle.timer
│   ├── kiosk-logs-bundle.sh
│   ├── kiosk-status.sh           # quick manual status check over SSH
│   ├── kiosk-pair-remote.sh      # pairs this unit's own Bluetooth remote (per-clone step)
│   ├── kiosk-display-mode.sh     # X11/labwc-era cinema-rate script — not currently
│   │                             #   deployed by provision.sh; kept pending on-device
│   │                             #   confirmation that Kodi's own refresh-rate-matching
│   │                             #   setting covers the same job (KODI-MIGRATION-PLAN.md
│   │                             #   Phase 1)
│   ├── journald-kiosk.conf       # persistent, size-capped journal config
│   ├── logrotate-kiosk-heartbeat # rotation for the flat heartbeat log
│   └── config.txt.append         # HDMI/GPU tuning appended to boot config
├── fleet/
│   └── prep-sd-card.sh           # helper: writes kiosk-config.txt onto a
│                                  # freshly-flashed clone from your laptop
└── zabbix/
    ├── zabbix_agent2.conf.tmpl        # active-checks-only, identical on every device
    ├── zabbix_agent2_site.conf.tmpl   # per-device Hostname/ServerActive drop-in
    ├── pi5_thermal.sh                 # optional Pi 5 thermal/throttle add-on
    ├── userparameter_pi5.conf         # (see README section below)
    └── template_pi5_thermal.yaml      # .yaml = manual one-time server import

Personalizing your fork

Jellyfin/Plex/Live TV access isn't tracked-file config anymore — it's signed into Kodi's own addons interactively on the golden device (Step 1, Phase 1) and cloned along with everything else, so there's nothing to templated-URL-substitute for those. .env.example/build.sh/local/ still exist, dormant, in case a future per-device secret needs the same "never commit the real value" treatment __PLACEHOLDER__ tokens used to give Jellyfin/Dispatcharr/extra-tile URLs — not currently wired to anything.


Step 1 — Build the golden device

  1. On your laptop: flash Raspberry Pi OS Lite (64-bit, Bookworm) to an SD card with Raspberry Pi Imager. In the advanced options (gear icon), set a temporary hostname (e.g. kiosk-golden), enable SSH with your public key, and connect it to your network — Wi-Fi or Ethernet.

    Same image works for Pi 4 and Pi 5, so you can build one golden card and clone it onto mixed hardware.

  2. Put the card in the Pi and power it on. Then, on the Pi (over SSH from your laptop):

    ssh pi@kiosk-golden.local
    sudo apt-get update && sudo apt-get install -y git   # not preinstalled on Lite
    git clone https://your-forgejo/you/pi-stick-kiosk.git   # or scp it over
    cd pi-stick-kiosk
    sudo ./provision.sh
    
  3. provision.sh, running on the Pi, installs Kodi + its PVR/Plex dependencies, sets up autologin into a kiosk session running Kodi directly (no X11, no Wayland compositor — Kodi's own GBM/DRM backend talks to the display directly), tunes HDMI/GPU settings, and installs the first-boot re-individualization service. That service fires on any boot where its "already run" flag is missing — which is true right now, since this card has never booted before, so it'll also fire once on the golden device itself. That's fine; see step 5.

  4. On the Pi: reboot. Expect a brief extra, automatic reboot partway through — that's the first-boot service regenerating SSH host keys and machine-id, and applying whatever HOSTNAME is set in kiosk-config.txt (still the default stick-changeme at this point, so it'll just log a warning and leave it). The boot after that one lands in Kodi's home screen.

4b. On the Pi, with a keyboard/remote physically attached (this is the one part of setup that's genuinely interactive — see KODI-MIGRATION-PLAN.md Phase 1 for the full checklist): set Kodi's audio output device to the PipeWire/Pulse sink, confirm the "adjust display refresh rate" setting matches content cleanly on your TV, install and sign into the Jellyfin ("Add-on mode", not "Native" — avoids a database conflict with Plex's addon), PlexKodiConnect, and PVR IPTV Simple Client (pointed at your Dispatcharr instance) addons. This step is what actually makes the device useful, and it's the one part that gets baked into every future clone — do it once, carefully, here.

  1. On the Pi, before you shut down for shrinking: re-clear the first-boot flag so this run doesn't get baked into the golden image — otherwise every future clone would inherit sillygoose's own regenerated SSH host keys and machine-id instead of getting fresh ones of their own, which is exactly the footgun this service exists to prevent.

    sudo rm -f /var/lib/kiosk-firstboot-done
    
  2. Once you're happy with it, sudo poweroff (on the Pi), then physically pull the SD card back out and take it to your Linux box for step 2.

Step 2 — Shrink it into a distributable image

On a Linux box (or WSL), with the golden SD card in a reader — using PiShrink:

sudo dd if=/dev/sdX of=golden-raw.img bs=4M status=progress conv=fsync
sudo pishrink.sh -z golden-raw.img stick-image.img

-z gzips it too. stick-image.img.gz is now your "throw it on every stick" artifact. Keep it in your Forgejo/NAS as the versioned build output — treat it like any other image build, bump a version tag in kiosk-config.txt's comments when you rebuild.

Step 3 — Deploy a new stick

On your laptop (or Linux box/WSL), with a new, blank SD card in the reader — flash the shrunk image onto it:

gunzip -k stick-image.img.gz
sudo dd if=stick-image.img of=/dev/sdX bs=4M status=progress conv=fsync

(or use Raspberry Pi Imager → "Use custom" → point at stick-image.img, which also lets you re-set Wi-Fi credentials per device without touching the boot partition manually.)

Before first boot — while that same card is still plugged into your computer, not in a Pi — edit its config. The dd/Imager write above recreated the card's boot partition as plain FAT32, so your OS mounts it just like a USB drive: /Volumes/bootfs on macOS, a drive letter on Windows, or wherever you mount it on Linux. Open kiosk-config.txt there in any text editor:

HOSTNAME=stick-livingroom
IMAGE_VERSION=2026.08
WIFI_SSID=
WIFI_PSK=

Don't know the destination Wi-Fi yet, or the unit might move later? Leave WIFI_SSID/WIFI_PSK blank — see "Wi-Fi self-service" below for what happens instead.

Or, still on your laptop, run fleet/prep-sd-card.sh to write that file for you instead of hand-editing. Note it only writes the file — it expects the partition to already be mounted at the path you give it (on macOS that's automatic on insert; on Linux you may need to mount it yourself first, see the script's header comment for the exact command):

./fleet/prep-sd-card.sh /Volumes/bootfs stick-livingroom

To also join Tailscale/Headscale on first boot, set TAILSCALE_AUTHKEY (and TAILSCALE_LOGIN_SERVER, if using a self-hosted Headscale instance) in the environment before invoking it — see the "Tailscale, dormant by default" section below:

TAILSCALE_AUTHKEY=<preauthkey> TAILSCALE_LOGIN_SERVER=https://headscale.example.com \
  ./fleet/prep-sd-card.sh /Volumes/bootfs stick-livingroom

Now eject the card, move it to the Pi, and power it on. Everything from here happens on the Pi itself — no laptop, no SSH. On this first boot the kiosk-firstboot service:

  1. Expands the root filesystem to fill the SD card (PiShrink shrank it down to minimum size).
  2. Regenerates SSH host keys and the systemd machine-id, so ten cloned sticks don't all claim to be the same host on the network.
  3. Sets the hostname from kiosk-config.txt.
  4. Reboots once, then never runs again (flag file at /var/lib/kiosk-firstboot-done).

After that reboot it comes up straight into Kodi, already signed into Jellyfin/Plex/Live TV from the golden device's setup. One thing left, specific to this physical unit — its Bluetooth remote is a 1:1 accessory, not something the golden image can carry: put the remote into pairing mode, then, from your laptop (over the same network, before handing the stick off):

ssh stick-livingroom sudo /opt/kiosk/kiosk-pair-remote.sh

Hand it to whoever it's for.


Hardware: case, not bare dongle

This build targets a Pi 4/5 in a small case with a short HDMI cable, sitting next to the TV — not a bare board jammed straight into the HDMI port. From the recipient's perspective it's the same "plug it in and it works" experience, but you get better cooling, a cable that isn't cantilevering the whole board off a TV's HDMI socket, and one less way for the connection to be marginal on a random TV. config.txt.append reflects this — it keeps hdmi_force_hotplug (still useful for TVs with flaky EDID reporting) but drops the HDMI signal-boost setting that dongle-in-port builds need to compensate for a shorter, weaker connector path.

HDMI audio

Audio goes out over the same HDMI cable, through PipeWire (+ WirePlumber + pipewire-pulse) rather than raw ALSA. That's not the usual "just use PulseAudio" default-following — it's load-bearing here: the vc4-hdmi driver only exposes its hardware PCM as a single exclusive-access device in a format (IEC958_SUBFRAME_LE) that ALSA's own dmix plugin can't mix. Without a real mixing layer, two things racing for that one handle means whichever loses gets "Device or resource busy" with no retry, leaving video playing with silent audio for the rest of that session. PipeWire does its own mixing before encoding to whatever format the hardware actually needs, so it isn't limited by dmix's format restrictions the way a plain /etc/asound.conf fix would be. Kodi's own Settings → System → Audio needs its output device pointed at the PipeWire/Pulse sink — this doesn't happen automatically the way Chromium's autodetection used to, it's part of the golden-device setup checklist (KODI-MIGRATION-PLAN.md Phase 1).

Overall loudness is the PipeWire sink volume, layered above whatever volume Kodi's own player is set to — maxing the in-player slider doesn't help if the sink itself is still attenuating. WirePlumber's own default for a brand new sink is quiet regardless (device.routes.default-sink-volume, ~40%) — hit live pairing a Bluetooth speaker, where neither Kodi's volume nor the speaker's own physical controls seemed to do anything until this was found; every golden image now seeds files/kiosk-wireplumber-sm-settings (100%) to ~kiosk/.local/state/wireplumber/sm-settings so this doesn't need rediscovering per device. A keyboard's dedicated volume/mute keys should work via Kodi's own default keymap (it binds the common XF86Audio* keysyms out of the box) — worth confirming during Phase 1's audio check rather than assuming, since this repo no longer carries a custom keybind config the way files/labwc-rc.xml used to.

Bluetooth audio

A "Pair Bluetooth Speaker" entry in Kodi's Favourites opens a device picker (a small Kodi addon, script.kiosk.btpair) — Kodi has no Bluetooth pairing UI of its own on Linux, so this drives bluetoothctl directly instead of requiring a terminal. It scans for a few seconds and lists newly-seen (not-already-paired) devices by name — filtered to just audio devices via BlueZ's own Icon property (populated from the Bluetooth inquiry response, available pre-pairing), since a real home turns up a dozen-plus irrelevant nearby devices (phones, lightbulbs, fitness bands) that would otherwise drown out the one speaker you actually want. Picking one pairs + trusts + connects it. libspa-0.2-bluetooth (the PipeWire↔BlueZ bridge that turns a connected device into a real audio sink) is part of every golden image's package list alongside the rest of the PipeWire/WirePlumber stack from HDMI audio above; once paired, WirePlumber promotes the speaker to the default sink on its own, and Kodi's audio output (already "Default Output Device (PIPEWIRE)") just follows it — no separate Kodi setting to change. No sudoers grant is needed for this one, unlike Wi-Fi/multistream's triggers — BlueZ's default D-Bus policy already lets the kiosk user scan/pair/connect directly. A speaker's own firmware sometimes reconnects to a known host on its own power-on, but not reliably enough to depend on — confirmed live that one didn't reconnect after a sillygoose reboot despite never having been powered off — so kiosk-bt-reconnect.service also has the host side try, once at every boot, against any already-trusted audio device. The Favourite is only needed for a device's first pairing.

Tailscale, dormant by default

Every golden image ships with Tailscale installed but logged out — it costs nothing to have on standby, and since these devices leave your LAN the moment you hand them out, you'll likely want it eventually even if you don't have a concrete plan today. To activate it for a given stick, generate a one-off auth key at login.tailscale.com/admin/settings/keys and drop it into that device's kiosk-config.txt as TAILSCALE_AUTHKEY before first boot (or add it later over a LAN SSH session and re-run kiosk-firstboot.sh's Tailscale block by hand). --ssh is passed automatically, so once a stick is on your tailnet you can tailscale ssh into it without separately managing keys per device.

Running your own Headscale instead of the public coordination server? Set TAILSCALE_LOGIN_SERVER in kiosk-config.txt alongside TAILSCALE_AUTHKEY (generate the key from your Headscale instance, not tailscale.com). Leave it blank to join official Tailscale, unchanged from the default above.

Host firewall: Tailscale-only network access

The moment TAILSCALE_AUTHKEY is set, kiosk-firstboot.sh also enables a host firewall (nftables, ruleset in files/kiosk-fw.nft) that restricts inbound traffic to loopback, established connections, and Tailscale — nothing else. SSH becomes Tailscale-only: LAN/mDNS SSH (e.g. stick.local) stops working, and only tailscale ssh <hostname> (or plain ssh <hostname>.<your-tailnet>, since --ssh is already always passed) reaches the device from then on. Outbound traffic is never restricted by this firewall — that would risk the device being unable to reach Tailscale's own servers to join the tailnet in the first place.

The one deliberate exception: comitup's open setup hotspot (see "Wi-Fi self-service" below) still works normally — kiosk-wifi-hotspot-notify.sh opens a narrow exception (DHCP/DNS/HTTP only) on wlan0 for exactly as long as the hotspot is up.

Applying this is guarded by a timed auto-revert (files/kiosk-fw-safe-apply.sh) so a bad rule self-heals in minutes instead of requiring a keyboard/monitor on the device — confirm it with sudo /opt/kiosk/kiosk-fw-confirm.sh from a fresh Tailscale SSH session opened after the change. If the firewall and Tailscale ever end up disagreeing with each other and the timer isn't enough, set FIREWALL_DISABLE=1 in kiosk-config.txt — pull the SD card, edit that one line (plain FAT32, no Linux needed), and the firewall won't load on the next boot.

Optionally, set TAILSCALE_EXIT_NODE in kiosk-config.txt to also route all of this stick's non-tailnet-internal outbound traffic (NTP, apt, Kodi addon repos — anything not going to another *.ts.kurtnet node) through a Tailscale exit node, instead of leaving directly over whatever LAN/customer Wi-Fi the stick is sitting on. This needs a one-time setup on your Headscale server, outside this repo:

  1. Pick an already-Headscale-joined machine with real internet egress, enable IP forwarding on it, and run tailscale up --advertise-exit-node --advertise-tags=tag:exit-node.
  2. Approve its two auto-advertised routes (0.0.0.0/0, ::/0) — via headscale routes enable -r <id> (or headscale nodes approve-routes) or Headplane's Routes page. This is a one-time, per-exit-node approval, not something you repeat per stick.
  3. Recommended: add autoApprovers: { exitNode: ["tag:exit-node"] } to your Headscale ACL policy so a replacement/second exit node self-approves on join.

Traffic to other *.ts.kurtnet nodes (Jellyfin, Dispatcharr, Plex) is unaffected either way — Tailscale always routes tailnet-internal traffic over its own mesh, regardless of exit-node selection.

Wi-Fi self-service

Every golden image ships with comitup (an official Debian package, apt install comitup — no extra repo needed) managing Wi-Fi fallback on top of NetworkManager. It's the answer to "what if I don't know the destination Wi-Fi yet, or the unit moves later":

  • No known network reachable (nothing set in kiosk-config.txt, wrong credentials, or the unit relocated somewhere new) → the stick broadcasts its own open hotspot, kiosk-setup-<hostname>, with a captive-portal page for entering real Wi-Fi credentials on site. No SSH, no laptop, no prior knowledge of the network required — this is meant for an end user, not just you.
  • Kodi shows setup instructions on screen while the hotspot is up (the SSID name and what to do), and clears them once connected — see files/kiosk-wifi-hotspot-notify.sh.
  • Remembers every network it's ever been given, and cycles through them automatically — a unit that's moved between two known locations doesn't need to be told which one it's at each time.
  • Manual trigger, no SSH needed: a "Reset Wi-Fi" entry in Kodi's Favourites (navigable with the unit's own remote) drops the current connection and re-opens the setup hotspot — for when a unit still has some connectivity (e.g. a stale network still in range) but needs to be pointed at a different one.
  • Remote support path still works too: comitup-cli d over SSH/Tailscale does the same thing as the Kodi menu entry, for when you'd rather fix it yourself than walk someone through the captive portal.
  • Manual on/off, independent of Ethernet: a second Kodi Favourites entry, "Toggle Wi-Fi", turns the Wi-Fi radio fully on or off on demand — useful if you'd rather not be reminded about Wi-Fi setup at all on a unit that's happily wired. Turning it off stops comitup's setup hotspot (and its on-screen nagging) as a side effect; turning it back on resumes normal Wi-Fi self-service. This is independent of Ethernet — nothing here happens automatically based on whether a wired connection exists.

WIFI_SSID/WIFI_PSK in kiosk-config.txt are still worth setting when you do know the destination network in advance — first boot just connects directly and the hotspot never appears. Leaving them blank is a supported, first-class path now, not a gap.

Multistream viewer

A "Multistream" entry in Kodi's Favourites opens a channel/layout picker (a small Kodi addon, script.kiosk.multistream) and hands the screen to a separate, on-demand mpv-based viewer for watching several Dispatcharr live channels at once — picture-in-picture, one-big-two-small, or a 2×2 grid — without going back to Chromium (removed earlier specifically for the tearing its own compositing path caused). Live-tested end to end: picking channels, playback, audio-pane cycling/mute/volume via the T1-Remote, and returning to Kodi. See KODI-MIGRATION-PLAN.md's "Phase 7" and "Phase 8" for the full design and findings. Kodi itself is unaffected either way — it's still the default, boots the same as always, and a crash in the multistream viewer falls straight back to Kodi automatically.

Managing sticks once they leave your network

  • Tailscale (above) gets you back in without port-forwarding or static IPs, wherever the stick physically ends up.
  • Update policy is already locked in on the golden image — provision.sh narrows unattended-upgrades to security-only patches with automatic reboot disabled (see Step 1), baked in before the image is ever shrunk and cloned. Nothing to remember per stick: an update that breaks Kodi or the display driver on a device sitting in someone else's living room is a much worse day than on a server you can console into, and by the time it's out there you can't SSH in to fix it easily.
  • Hostname/Tailscale/Zabbix can still be updated remotely, no re-imaging needed — edit /boot/firmware/kiosk-config.txt over SSH and apply the relevant piece by hand (hostnamectl, or re-running kiosk-firstboot.sh's Tailscale block, per the Tailscale section above). Wi-Fi no longer needs SSH at all — see "Wi-Fi self-service" above for the on-device setup hotspot and Kodi-menu reset trigger.
  • Jellyfin/Plex/Live TV endpoints are a real regression here, not something to gloss over: they're signed into Kodi's addons on the golden device, not templated per-stick, so a URL change means redoing that sign-in and re-cloning already-deployed units, not a live SSH edit. This is currently being hit for real: moving the addons from LAN URLs to Headscale-routed ts.kurtnet hostnames is in progress (see KODI-MIGRATION-PLAN.md Phase 2) — deferred to the next golden-image prep session rather than edited in place on an already-connected device. Not solved yet — see KODI-MIGRATION-PLAN.md Phase 5 if this ends up mattering enough to build a remote-editable path.

Logging and pulling logs remotely

Every stick logs to the systemd journal, made persistent across reboots (RPi OS Lite defaults to RAM-only logs, which throw everything away on every reboot — not acceptable for a device you can't walk over and look at). Journal storage is capped at 150MB total with 60-day retention, so a forgotten stick doesn't slowly fill its SD card.

What gets logged:

  • kiosk-firstboot — the one-time re-individualization run (hostname set, SSH keys/machine-id regenerated, Wi-Fi/Tailscale join attempts). This is exactly the record you want later when a stick is misbehaving and you're trying to tell "bad setup at individualization time" apart from "something broke after the fact" — so besides the journal, it's also appended to /var/log/kiosk-heartbeat.log's sibling, /var/log/kiosk-firstboot.log, a small flat file that is not subject to the 150MB/60-day journal cap below. It's a handful of lines per device, so it's never rotated — it just persists for the life of the SD card.
  • kiosk-session.service — Kodi itself starting/stopping/crashing (journalctl -u kiosk-session.service)
  • kiosk-heartbeat — a structured status line every 15 minutes: uptime, load, disk/memory free, CPU temp, whether the kiosk session and Kodi are actually running, network reachability, and Tailscale state. Example line: journalctl -t kiosk-heartbeat -n 1
    host=stick-livingroom version=2026.08 uptime_s=48213 load1=0.12
    disk_used=41% disk_avail=15G mem_avail_mb=3102 cpu_temp_c=52.1
    kiosk_session=active kodi=yes net_route=yes net_reachable=yes
    tailscale=Running
    
    This is grep/parse-friendly on purpose — if you ever want to pipe it into Zabbix as a log-monitoring item instead of just eyeballing it over SSH, the format is ready for that.
  • system-errors at severity err or worse, system-wide — catches issues outside the kiosk's own units too (display driver, kernel, etc.)

Pulling logs remotely: once a stick is on Tailscale (see above), a fresh log bundle is always sitting at a fixed path, regenerated nightly at 4:30am plus on demand:

# one command, from anywhere on your tailnet
ssh stick-livingroom 'sudo cat /var/log/kiosk-bundle-latest.tar.gz' \
  > stick-livingroom-logs.tar.gz

# or generate a fresh one first if you don't want to wait for the nightly run
ssh stick-livingroom 'sudo /opt/kiosk/kiosk-logs-bundle.sh'

The bundle contains the full, never-rotated kiosk-firstboot.log, 14 days of kiosk-tagged journal output, system-wide errors, the full current boot's log, a point-in-time snapshot (disk, memory, temp, kiosk-session status, Tailscale status, last 20 heartbeats), and a copy of kiosk-config.txt — with WIFI_PSK and TAILSCALE_AUTHKEY redacted, since there's no reason secrets should ride along in a file you might copy around further.

For a faster look without pulling the whole bundle:

ssh stick-livingroom 'sudo /opt/kiosk/kiosk-status.sh'

which prints uptime, disk, the last 5 heartbeats, and current kiosk-session/Tailscale state in one screenful.

Zabbix, over Tailscale only

Zabbix agent2 is installed on every stick, same dormant-until-configured pattern as Tailscale: it stays disabled until a device's kiosk-config.txt sets ZABBIX_SERVER_ACTIVE.

This is deliberately active-checks-only — the agent calls out to your server on a schedule and pulls its own item config, rather than your server reaching in. That's the right shape for devices that leave your LAN: it means nothing needs to listen for inbound connections from anywhere, on any network these sticks end up on. Passive checks are pinned off entirely (Server=127.0.0.1, ListenIP=127.0.0.1) rather than just "not used" — even over Tailscale, nothing but loopback can query the agent directly. DenyKey=system.run[*] blocks remote command execution regardless of what gets configured server-side later.

Hostname/HostMetadata/ServerActive themselves live in their own drop-in (zabbix_agent2_site.conf.tmpl -> /etc/zabbix/zabbix_agent2.d/site.conf), not in the main zabbix_agent2.conf.tmpl — provision.sh seeds that drop-in only if it's missing, so re-running provision.sh to update the golden image (or an already-provisioned device) can never reset a device's identity back to placeholders the way overwriting the main conf would.

Point ZABBIX_SERVER_ACTIVE at your Zabbix server or proxy's Tailscale hostname/IP, never a public address:

ZABBIX_SERVER_ACTIVE=zabbix-server.your-tailnet.ts.net:10051
ZABBIX_HOST_METADATA=kiosk-stick

ZABBIX_HOST_METADATA tags the host so a Zabbix autoregistration action can auto-add each stick under the right template as it checks in for the first time, instead of creating hosts by hand for every unit you hand out — set one up under Configuration → Actions → Autoregistration actions, matching on host metadata, before you ship the first stick.

The heartbeat data (uptime, disk, temp, kiosk-session/Kodi liveness, network reachability) is written both to the journal and to a flat file at /var/log/kiosk-heartbeat.log (rotated daily, 7 kept) specifically so it's usable as a Zabbix log[] item source — journald isn't a format Zabbix reads natively, so the flat file exists purely as a bridge for this. Everything else about the logging setup above still applies unchanged.

If you provision a stick without a working Tailscale connection but with ZABBIX_SERVER_ACTIVE set, the agent still starts — it just retries until connectivity exists rather than failing hard, so ordering between the two doesn't need to be perfect.

Optional: Pi 5 thermal/throttle template

zabbix/ adds richer, Zabbix-native monitoring than the heartbeat's plain CPU-temp field: ARM clock speed and the full decoded vcgencmd get_throttled bitmask (under-voltage, frequency capping, throttling, and the soft temperature limit — both "happening right now" and "happened at some point since boot"), as proper typed items with history/trends and threshold-based alerting, polled every 60s.

  • zabbix/pi5_thermal.sh and zabbix/userparameter_pi5.conf are deployed to every device automatically by provision.sh — same dormant-until-configured posture as the rest of Zabbix, nothing to do per device.
  • zabbix/template_pi5_thermal.yaml is not deployed to the Pi at all — it's a one-time, server-side import. On your Zabbix server: Data collection → Templates → Import, then link the "Pi5 Thermal Monitoring" template to the relevant host(s), or add a "Link template" operation to the same autoregistration action described above so every stick picks it up automatically as it checks in.

If you outgrow golden-image cloning

Once you're iterating a lot, look at pi-gen — the actual tool Raspberry Pi OS itself is built with. You'd turn provision.sh into a custom pi-gen stage and get a fully from-scratch, CI-buildable .img with no golden-device step at all. Not included here because it's a bigger lift (Docker-based build environment, stage/substage plumbing) and the golden-image approach gets you the same reproducibility with much less ceremony for a homelab-scale batch of sticks.