trying to work out oauth for Bungie API
  • JavaScript 58.5%
  • PHP 23.5%
  • CSS 8.7%
  • Python 7.8%
  • HTML 1.1%
  • Other 0.4%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Kurt Anderson 83a8695751 Fix permanent PGCR-collection gaps for prolific guardians
collect_pgcrs.py stopped paging a character's activity history as soon
as it hit an already-cached instance, assuming everything older was
cached too. That only holds if a run can page far enough to verify it,
but modes=4,82 is a confirmed no-op (Bungie returns every activity type
regardless), so a busy guardian's raid/dungeon entries share page budget
with all their other activity. Once MAX_PAGES_PER_RUN ran out before
reaching a real known boundary, the gap was permanent: every later run
restarted at page 0, found the now-fresh top of history "known", and
never paged deep enough to notice.

collect_for_member() now splits each run into an always-fresh front pass
plus a backfill pass that resumes from a persisted per-character cursor
(collect_cursor.json) instead of restarting at page 0. New
--catch-up-pages flag does a one-off deep scan for manual repair,
deliberately ignoring the stop-at-first-known shortcut since coverage
under the old bug is spotty rather than a clean wall.

Ran once against production, recovering 11,595 previously-missing PGCRs
across all 7 opted-in guardians.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-06 08:51:19 -07:00
app Add click-to-select fireteam roster popover to friends app 2026-08-06 08:51:06 -07:00
docs Configure log rotation for armor/fireteam and deploy armor logging 2026-07-20 21:46:28 -07:00
nginx Add friends app: no-login PGCR lookup/compare tool 2026-08-06 00:05:26 -07:00
sql Add friends app: no-login PGCR lookup/compare tool 2026-08-06 00:05:26 -07:00
tools Fix permanent PGCR-collection gaps for prolific guardians 2026-08-06 08:51:19 -07:00
.gitignore Add Guardian Inspector: 3D equipped-gear viewer with per-character switching 2026-07-20 03:53:42 -07:00
CHANGELOG.md Fix permanent PGCR-collection gaps for prolific guardians 2026-08-06 08:51:19 -07:00
deploy-armor.sh Rename deploy.sh to deploy-armor.sh, fix .armor.env references 2026-07-15 10:33:20 -07:00
deploy-fireteam.sh Refactor repo to handle multiple web apps being built and deployed 2026-07-15 11:05:03 -07:00
deploy-friends.sh Add friends app: no-login PGCR lookup/compare tool 2026-08-06 00:05:26 -07:00
deploy-inspector.sh Add Guardian Inspector: 3D equipped-gear viewer with per-character switching 2026-07-20 03:53:42 -07:00
env.example Add v1: Bungie OAuth login and armor item viewer 2026-07-12 01:40:32 -07:00
LICENSE Add v1: Bungie OAuth login and armor item viewer 2026-07-12 01:40:32 -07:00
README.md Add friends app: no-login PGCR lookup/compare tool 2026-08-06 00:05:26 -07:00

D2 Tools

Destiny 2 web tools, sharing one nginx host, one PHP/PDO/MySQL deploy pattern, and one Bungie.net OAuth approach — each app gets its own DB tables and its own deploy script, so nothing crosses between them.

Apps

  • armor — sign in with Bungie.net and see your armor (helmet, gauntlets, chest, legs, class item) across all characters and the vault, with per-piece archetype/set-bonus/tier detail and drag-to-equip.
  • fireteam — leave-the-tab-open live fireteam dashboard: one card per guardian (1–12), gear rail, an activity bar (3-month or last-100-activities view, toggled by clicking your own card's bar), and a stat panel that switches based on what the fireteam is playing. Opt-in PGCR collection per guardian for deeper stats later.
  • friends — no-login, two-column PGCR lookup/compare tool built on that opt-in collection: pick a guardian on the left to see their activity across every activity type (checkboxes filter by Vanguard/Crucible/Gambit/Raids/Dungeons/World — fireteam's own bucket bar groups the last two into one purple "Raids & Dungeons" category, friends filters them separately), pick a second guardian on the right to dim out any left-column activity they weren't actually in.

How this repo works

This repo is the dev/codebase — it is not served directly. The live site is served by nginx from /var/www/d2stats.bjorna-3.net/. All app code lives under app/, mirroring the deploy target, with one .{app}.env and one deploy-{app}.sh per app:

app/
  .armor.env      <- secrets/config (gitignored, never commit)
  armor/          <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/armor/
  .fireteam.env   <- secrets/config (gitignored, never commit)
  fireteam/       <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/fireteam/
  .friends.env    <- secrets/config (gitignored, never commit)
  friends/        <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/friends/
tools/            <- cron-invoked scripts (manifest refresh, PGCR collection), run from this
                     checkout directly — not part of any deployed tree

Run ./deploy-armor.sh, ./deploy-fireteam.sh, or ./deploy-friends.sh to rsync that app's app/{app}/ to its docroot and its .{app}.env to /var/www/d2stats.bjorna-3.net/{app}.env (kept one level above the docroot so it's never web-reachable).

Each app's .env is checked server-side only; Bungie API calls happen directly from the browser using the API_KEY/OAUTH_CLIENT_ID that config.php hands it — PHP only ever handles the OAuth secret exchange/refresh and session storage. See docs/architecture.md (armor) and docs/AGENT_BRIEF-fireteam.md (fireteam) for the full design. friends is the exception: no login, no OAuth, and nothing in it — not even its cron script — talks to Bungie at all. It's a read-only view over friends_pgcr_index/friends_pgcr_participants, populated by indexing the PGCR files fireteam's own hourly collect_pgcrs.py already fetches (see the cron section below for why that already covers every activity type, not just raid/dungeon). Its .env's API_KEY/OAuth fields are consequently unused by anything in this repo — only DB_* is actually read.

Setup

  1. Copy env.example to app/.armor.env / app/.fireteam.env / app/.friends.env and fill in real values (Bungie app credentials — see docs/app-setup-guide.md; each app needs its own Bungie Application Portal registration and its own MySQL credentials or a shared svc_site account against the d2stats database).
  2. Run the relevant sql/NNN_*.sql files against d2stats (001 armor sessions, 002 fireteam sessions, 003 fireteam PGCR-collection opt-in table, 005/006 friends' PGCR index/participants tables, 007 adds friends' started-fresh/completed columns). These are DDL, so they need a DB user with ALTER/CREATE — svc_site (the app's runtime user) only has SELECT/INSERT/UPDATE/DELETE on d2stats and can't run them itself. After applying 007 for the first time, backfill the columns on already-indexed rows with tools/.venv/bin/python3 tools/index_friends_pgcrs.py --reindex-all --history-dir ... --manifest-dir ... --env-file ... (safe to re-run; it's an upsert over every cached file, not just new ones).
  3. Add the app's location /{app}/ nginx block (see nginx/d2stats.bjorna-3.net.example), then sudo nginx -t && sudo systemctl reload nginx.
  4. Run ./deploy-armor.sh, ./deploy-fireteam.sh, and/or ./deploy-friends.sh.
  5. fireteam only — create tools/.venv (python3 -m venv tools/.venv && tools/.venv/bin/pip install pymysql), a writable fireteam-manifest-cache//fireteam-history-cache/ pair one level above the docroot, and a /var/log/d2-tools/fireteam/ log directory (kurt:www-data, 775). Then add the two cron entries below.
  6. friends only — reuses fireteam's tools/.venv, manifest cache, and history cache (no new directories needed) plus a /var/log/d2-tools/friends/ log directory (kurt:www-data, 775). Then add the cron entry below — it only indexes files fireteam's own collector already fetched, no API_KEY or Bungie call of its own.

fireteam cron jobs

# manifest refresh (activity/mode/season tables, hash-families, item icons) — daily
0 6 * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/update_manifest.py --manifest-dir /var/www/d2stats.bjorna-3.net/fireteam-manifest-cache --env-file /var/www/d2stats.bjorna-3.net/fireteam.env >> /var/log/d2-tools/fireteam/manifest.log 2>&1

# opt-in PGCR collection for checked guardians — hourly. Despite the `modes=4,82` param and
# the "raid/dungeon only" framing below, GetActivityHistory's `modes` filter is a confirmed
# no-op (see app/fireteam/lib/bungie_history.php:14), so this already fetches full PGCRs for
# every activity type those guardians play, not just raids/dungeons — which is exactly what
# lets the friends app's indexer (below) cover all five categories with no collector of its own.
0 * * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/collect_pgcrs.py --history-dir /var/www/d2stats.bjorna-3.net/fireteam-history-cache --env-file /var/www/d2stats.bjorna-3.net/fireteam.env >> /var/log/d2-tools/fireteam/pgcr-collect.log 2>&1

friends cron job

# indexes whatever collect_pgcrs.py has already fetched into friends_pgcr_index/participants
# — no Bungie call of its own, so this can run often and cheaply
*/15 * * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/index_friends_pgcrs.py --history-dir /var/www/d2stats.bjorna-3.net/fireteam-history-cache --manifest-dir /var/www/d2stats.bjorna-3.net/fireteam-manifest-cache --env-file /var/www/d2stats.bjorna-3.net/friends.env >> /var/log/d2-tools/friends/index.log 2>&1

Docs

License

GPLv3 — see LICENSE.