- JavaScript 58.5%
- PHP 23.5%
- CSS 8.7%
- Python 7.8%
- HTML 1.1%
- Other 0.4%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
collect_pgcrs.py stopped paging a character's activity history as soon as it hit an already-cached instance, assuming everything older was cached too. That only holds if a run can page far enough to verify it, but modes=4,82 is a confirmed no-op (Bungie returns every activity type regardless), so a busy guardian's raid/dungeon entries share page budget with all their other activity. Once MAX_PAGES_PER_RUN ran out before reaching a real known boundary, the gap was permanent: every later run restarted at page 0, found the now-fresh top of history "known", and never paged deep enough to notice. collect_for_member() now splits each run into an always-fresh front pass plus a backfill pass that resumes from a persisted per-character cursor (collect_cursor.json) instead of restarting at page 0. New --catch-up-pages flag does a one-off deep scan for manual repair, deliberately ignoring the stop-at-first-known shortcut since coverage under the old bug is spotty rather than a clean wall. Ran once against production, recovering 11,595 previously-missing PGCRs across all 7 opted-in guardians. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
| app | ||
| docs | ||
| nginx | ||
| sql | ||
| tools | ||
| .gitignore | ||
| CHANGELOG.md | ||
| deploy-armor.sh | ||
| deploy-fireteam.sh | ||
| deploy-friends.sh | ||
| deploy-inspector.sh | ||
| env.example | ||
| LICENSE | ||
| README.md | ||
D2 Tools
Destiny 2 web tools, sharing one nginx host, one PHP/PDO/MySQL deploy pattern, and one Bungie.net OAuth approach — each app gets its own DB tables and its own deploy script, so nothing crosses between them.
Apps
- armor — sign in with Bungie.net and see your armor (helmet, gauntlets, chest, legs, class item) across all characters and the vault, with per-piece archetype/set-bonus/tier detail and drag-to-equip.
- fireteam — leave-the-tab-open live fireteam dashboard: one card per guardian (1–12), gear rail, an activity bar (3-month or last-100-activities view, toggled by clicking your own card's bar), and a stat panel that switches based on what the fireteam is playing. Opt-in PGCR collection per guardian for deeper stats later.
- friends — no-login, two-column PGCR lookup/compare tool built on that opt-in collection: pick a guardian on the left to see their activity across every activity type (checkboxes filter by Vanguard/Crucible/Gambit/Raids/Dungeons/World — fireteam's own bucket bar groups the last two into one purple "Raids & Dungeons" category, friends filters them separately), pick a second guardian on the right to dim out any left-column activity they weren't actually in.
How this repo works
This repo is the dev/codebase — it is not served directly. The live site is served by nginx from /var/www/d2stats.bjorna-3.net/. All app code lives under app/, mirroring the deploy target, with one .{app}.env and one deploy-{app}.sh per app:
app/
.armor.env <- secrets/config (gitignored, never commit)
armor/ <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/armor/
.fireteam.env <- secrets/config (gitignored, never commit)
fireteam/ <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/fireteam/
.friends.env <- secrets/config (gitignored, never commit)
friends/ <- deployed 1:1 to /var/www/d2stats.bjorna-3.net/friends/
tools/ <- cron-invoked scripts (manifest refresh, PGCR collection), run from this
checkout directly — not part of any deployed tree
Run ./deploy-armor.sh, ./deploy-fireteam.sh, or ./deploy-friends.sh to rsync that app's app/{app}/ to its docroot and its .{app}.env to /var/www/d2stats.bjorna-3.net/{app}.env (kept one level above the docroot so it's never web-reachable).
Each app's .env is checked server-side only; Bungie API calls happen directly from the browser using the API_KEY/OAUTH_CLIENT_ID that config.php hands it — PHP only ever handles the OAuth secret exchange/refresh and session storage. See docs/architecture.md (armor) and docs/AGENT_BRIEF-fireteam.md (fireteam) for the full design. friends is the exception: no login, no OAuth, and nothing in it — not even its cron script — talks to Bungie at all. It's a read-only view over friends_pgcr_index/friends_pgcr_participants, populated by indexing the PGCR files fireteam's own hourly collect_pgcrs.py already fetches (see the cron section below for why that already covers every activity type, not just raid/dungeon). Its .env's API_KEY/OAuth fields are consequently unused by anything in this repo — only DB_* is actually read.
Setup
- Copy
env.exampletoapp/.armor.env/app/.fireteam.env/app/.friends.envand fill in real values (Bungie app credentials — seedocs/app-setup-guide.md; each app needs its own Bungie Application Portal registration and its own MySQL credentials or a sharedsvc_siteaccount against thed2statsdatabase). - Run the relevant
sql/NNN_*.sqlfiles againstd2stats(001armor sessions,002fireteam sessions,003fireteam PGCR-collection opt-in table,005/006friends' PGCR index/participants tables,007adds friends' started-fresh/completed columns). These are DDL, so they need a DB user withALTER/CREATE—svc_site(the app's runtime user) only hasSELECT/INSERT/UPDATE/DELETEond2statsand can't run them itself. After applying007for the first time, backfill the columns on already-indexed rows withtools/.venv/bin/python3 tools/index_friends_pgcrs.py --reindex-all --history-dir ... --manifest-dir ... --env-file ...(safe to re-run; it's an upsert over every cached file, not just new ones). - Add the app's
location /{app}/nginx block (seenginx/d2stats.bjorna-3.net.example), thensudo nginx -t && sudo systemctl reload nginx. - Run
./deploy-armor.sh,./deploy-fireteam.sh, and/or./deploy-friends.sh. - fireteam only — create
tools/.venv(python3 -m venv tools/.venv && tools/.venv/bin/pip install pymysql), a writablefireteam-manifest-cache//fireteam-history-cache/pair one level above the docroot, and a/var/log/d2-tools/fireteam/log directory (kurt:www-data,775). Then add the two cron entries below. - friends only — reuses fireteam's
tools/.venv, manifest cache, and history cache (no new directories needed) plus a/var/log/d2-tools/friends/log directory (kurt:www-data,775). Then add the cron entry below — it only indexes files fireteam's own collector already fetched, noAPI_KEYor Bungie call of its own.
fireteam cron jobs
# manifest refresh (activity/mode/season tables, hash-families, item icons) — daily
0 6 * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/update_manifest.py --manifest-dir /var/www/d2stats.bjorna-3.net/fireteam-manifest-cache --env-file /var/www/d2stats.bjorna-3.net/fireteam.env >> /var/log/d2-tools/fireteam/manifest.log 2>&1
# opt-in PGCR collection for checked guardians — hourly. Despite the `modes=4,82` param and
# the "raid/dungeon only" framing below, GetActivityHistory's `modes` filter is a confirmed
# no-op (see app/fireteam/lib/bungie_history.php:14), so this already fetches full PGCRs for
# every activity type those guardians play, not just raids/dungeons — which is exactly what
# lets the friends app's indexer (below) cover all five categories with no collector of its own.
0 * * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/collect_pgcrs.py --history-dir /var/www/d2stats.bjorna-3.net/fireteam-history-cache --env-file /var/www/d2stats.bjorna-3.net/fireteam.env >> /var/log/d2-tools/fireteam/pgcr-collect.log 2>&1
friends cron job
# indexes whatever collect_pgcrs.py has already fetched into friends_pgcr_index/participants
# — no Bungie call of its own, so this can run often and cheaply
*/15 * * * * cd /home/kurt/git/d2-tools && tools/.venv/bin/python3 tools/index_friends_pgcrs.py --history-dir /var/www/d2stats.bjorna-3.net/fireteam-history-cache --manifest-dir /var/www/d2stats.bjorna-3.net/fireteam-manifest-cache --env-file /var/www/d2stats.bjorna-3.net/friends.env >> /var/log/d2-tools/friends/index.log 2>&1
Docs
- docs/architecture.md — armor's OAuth + session design (fireteam's is the same pattern, see the brief below)
- docs/app-setup-guide.md — Bungie app registration reference
- docs/plan-v1.md through plan-v5.md — armor's versioned build checklists
- docs/AGENT_BRIEF-fireteam.md — fireteam's hard rules, API landmines, and bucket/panel design (start here for fireteam)
- docs/fireteam-plan-v1.md — fireteam's approved v1 build plan
- docs/oauth-documentation.txt, docs/about-the-api.txt — Bungie's own API/OAuth reference docs
License
GPLv3 — see LICENSE.