# Claude Code Audit — Balance of Power Experimental Chamber Forecast

**Date:** 2026-10-04 (audit performed ~16:00–17:00 ET)
**Auditor:** Claude Code (independent, read-only)
**Subject:** `docs/research/2026-10-04/CHAMBER_FORECAST_HANDOFF.md`; runtime commit `a9cf06177` (`src/balance_of_power/chamber.py`, `chamber_sources.py`, `config/balance_of_power/chamber-ballot-review-2026.json`, tests); root-site commits `db7b9f34a`, `64ea95c6e`, `48d6c17ff` (`election_chamber.py`, `templates/fin/chamber.html`, `static/fin/chamber.js`, `election_model_alerts.py`).
**Constraints honoured:** no runner actions against production, no ledger writes, no paid calls, no timer changes. Reproductions ran against read-only copies of the ledger and the pinned artifact; unit tests ran in temporary folders only.

---

## 1. Summary

The build does what the handoff says it does, and the handoff is unusually candid about what it does not establish. Every number I could recompute matched exactly: all 18 published scenarios, all 42 historical-check rows, the seat accounting, the Senate roster facts, and the nine California same-party ballots against the official certified list. The public page is live, fresh, pinned to the registered policy, and carries no chamber-control probability.

Two findings matter for the forecast itself:

1. **The incumbency/experience feature is systematically under-credited in live use.** 89 of the 336 modeled races where the roster incumbent is on the ballot receive no prior-service credit; at least 56 of those are name-format mismatches (`David Valadao` vs. archive `DAVID G. VALADAO`). The coefficient is 9.8 margin points. Three seats — CA-22, PA-01, VA-02, all Republican incumbents — are shown on the public seat grid as Democratic-leaning only because of this. The aggregate seat effect is small (≈ −1.2 expected Democratic House seats) because the misses fall on both parties, but the per-seat errors are large.
2. **The historical count check shows a directional bias the handoff does not report.** The House model over-predicted Democratic seats in six of seven cycles (mean signed error +4.9 seats; +7 to +15 in 2012–2022 non-wave years) and under-predicted by 19 in the 2018 Democratic wave. Errors track the national environment. The correlated-error settings widen the ranges but, as the handoff itself notes, do not correct the mean.

Neither invalidates the "conditional scenarios" framing, which is honest. Both argue for the handoff's own closing advice: better source coverage and a simple national adjustment before any more simulation machinery.

---

## 2. What I verified (all passed)

| Claim | Check | Result |
|---|---|---|
| Published scenario numbers | Re-ran `simulate()`/`summarize()` on the 432 modeled rows from ledger run `572aa7d8…` for all 3 dependence × 3 shift settings, both chambers | **All 18 scenarios identical** (expected bounds and envelopes) |
| Copula preserves marginals | Per-seat simulated win frequency vs. analytic Student-t probability, independent and stronger-shared | Max abs. difference 0.014 (House), 0.009 (Senate): Monte Carlo noise only |
| Shift monotonic | E[D House] at −4 / 0 / +4 (shared): 185.1 / 198.5 / 212.5 | Monotone, same seed |
| Seat accounting closes | House 405 modeled + 21 unresolved + 9 ballot = 435; Senate 27 + 8 + 65 = 100 | Closes |
| Senate roster facts | Class II set = 33 states; + FL, OH specials = 35 elections; continuing 32 D / 31 R / 2 I (ME, VT) | Consistent with the public Senate composition (53 R, 45 D, 2 I; 22 R and 13 D seats up) |
| Historical check | Re-ran `historical_check()` on pinned artifact `2eca7299…` | **All 42 rows identical** to `CHAMBER_SCENARIO_EVALUATION.json` |
| California same-party ballots | Parsed all 52 House districts in the retained official PDF (`cert-list-candidates.pdf`, 27 Aug 2026) | **Exactly nine** same-party districts: 4, 7, 11, 12, 14, 29, 34, 37 (D) and 40 (R), matching the review file; party slates match the roster in all 52 districts; CA-06 correctly unresolved (Kevin Kiley listed as No Party Preference) |
| Non-model fields unused | `chamber_sources.normalize` reads only state, seat status, district/seat number, candidates, seat party, special flag, incumbent name | Ratings, Kalshi prices, `margin_president`, `is_safe` never read |
| Registration integrity | v1 (`33624d80…`, 30 unresolved) and v2 (`fc3f9aaf…`, 21 unresolved + 9 ballot) both present, append-only; `CHAMBER_SCENARIO_REGISTRATION.json` id and policy SHA match the ledger | Matches handoff |
| Tests | `tests/balance_of_power/test_chamber.py` (runtime), `tests/test_election_chamber.py` + `tests/test_election_model_alerts.py` (root) | 10/10 and 11/11 pass; all use `tmp_path` |
| Public page | `https://vibe-infoveil.lampbotics.com/elections#chamber-forecast` and `/models` link | Live; embedded `as_of` = `2026-10-04T19:43:12Z` (the v2 run); "awaiting independent audit" wording present; `majority_probability: null`; view refuses stale (>26 h), unpinned, wrong-version or invalid runs |
| Daily reminder | `election_model_alerts.checks` | Flags chamber run missing/stale/failed and prompts review when Tide's national polling becomes eligible; does not auto-change the model |
| systemd | `intuitionist-balance-of-power.service` runs `collect` from the `autolab-runtime` worktree with `ProtectHome=read-only`; `chamber_publish` only reads the committed review JSON and writes to the ledger | Path resolves; **first scheduled execution of the chamber code is 19:45 ET today and has not yet happened** (the 07:45 run predates the commit) |

---

## 3. Findings

### F1 — Prior-service feature: live matching fails for ~26% of running incumbents (High)

`forecast_data.prior_service` credits a candidate only if `name_key(name)` (uppercase, alphanumerics only) exactly equals a federal winner's name in the state within the previous eight years. In training, candidate names and winner names come from the same results archive, so matching is near-perfect. In live use, names come from 270toWin's listing, which drops middle initials and uses short first names.

Measured on the v2 run:
- 432 modeled races; in 336 the roster incumbent is among the candidates.
- **89 of those 336 receive no prior-service credit** for the incumbent.
- **56 are recoverable** by a surname + first-initial match against the archive (e.g. `BRIAN K. FITZPATRICK`, `JENNIFER A. KIGGANS`, `DAVID G. VALADAO`, `ROSA L. DELAURO`, `JOHN W. HICKENLOOPER`, `CHRISTOPHER A. COONS`). The remainder are mostly archive-coverage cases (2025 special-election winners; races with no major-party opponent).
- The fundamentals coefficient on `service_advantage` is **9.84 margin points** (House-28 fit; β = [2.05, 8.34, 9.84] on intercept, lean/10, service). Seat scales are 5.9–7.2 points, so an uncredited incumbent is a ~1.4σ error at the seat level.
- Crediting the 56 recoverable incumbents moves expected Democratic seats by **−1.15 (House) and +0.39 (Senate)** — small in aggregate — but **flips the favoured party in CA-22 (p 0.59→0.15), PA-01 (0.68→0.21) and VA-02 (0.66→0.19)**. The public seat grid currently colours these three Republican-held seats blue.

This is a train/serve skew, not a tuning question. It is the concrete form of the handoff's Question 1.

**Fix (no refit needed):** normalise names before matching (strip middle initials/suffixes/nicknames, or match on surname + first initial within state and chamber), record the matching rule in the policy, and re-register as v3. Because the feature definition changes, the historical check should be re-run with the same normalisation to confirm it does not alter the training-era matches.

### F2 — Historical count check: unreported directional bias (Medium)

From the reproduced evaluation (eligible subsets, 28-day lead, fundamentals where no polls):

| Cycle | House eligible | Actual D | Model E[D] | Signed error |
|---|---:|---:|---:|---:|
| 2012 | 326 | 143 | 152.5 | +9.5 |
| 2014 | 337 | 142 | 156.8 | +14.8 |
| 2016 | 327 | 146 | 154.7 | +8.7 |
| 2018 | 350 | 183 | 163.8 | **−19.2** |
| 2020 | 381 | 194 | 206.1 | +12.1 |
| 2022 | 372 | 191 | 198.4 | +7.4 |
| 2024 | 341 | 167 | 167.7 | +0.7 |

Mean signed error **+4.9 seats** (Senate +0.9). The pattern is the national environment: the model over-predicts Democrats in Republican-leaning or neutral years and badly under-predicts them in the 2018 Democratic wave. A copula with a fixed variance share spreads this error; it cannot centre it. The handoff says "the correlated versions mainly widen ranges; they do not repair mean errors", which is correct, but the table reports only absolute errors, so the direction and the midterm pattern are invisible to a reader. The "shared" 7/7 coverage comes from 10–90% ranges averaging **36 seats wide** (47 for stronger-shared); "independent" ranges average 10 seats and cover 1/7.

Implication for 2026: the current House centre (D 206.5–227.5) has no national-environment term at all. The handoff's final paragraph — a simpler calibrated national adjustment before more simulation — is supported by its own evaluation. Tide is the natural source once it clears its 3-firm minimum (currently 1 eligible firm).

### F3 — Silent fallback to background for three watchlist races (Medium)

`race_inputs` uses local polls only if the registry's `dem`/`rep` names equal the roster names exactly. Three of the 26 registry races can never match until names are reconciled:

| Race | Registry | 270toWin roster |
|---|---|---|
| CA-48 | Marni von Wilpert | Marni **V**on Wilpert |
| FL-27 | Maria **Elvira** Salazar | Maria Salazar |
| NM-02 | **Gabe** Vasquez / **Greg** Cunningham | **Gabriel** Vasquez / **Gregory** Cunningham |

Today this changed nothing: their latest polls (fieldwork 1–13 Sep) fall outside the 21-day window, as do OH-01 and NE-01, which is why only 19 of 26 watchlist races are poll-informed (plus ME-SEN and MO-02 unresolved). But when fresh polls arrive, these three races will stay "background only" with no indication that a matched poll record existed. The handoff's statement that a mismatch "cannot silently import the wrong race's polling" is true; the mismatch does, however, silently discard the right race's polling. Add a run-level flag (`poll_record_present_but_unmatched`) and a casefolded/normalised comparison.

### F4 — OH-SEN treated inconsistently between Compass and the chamber layer (Medium)

`two_chamber.py` withholds OH-SEN and ME-SEN because "special-election or ranked-choice ballot provenance requires a separately tested extension." The chamber layer models OH-SEN-1 in **service** mode with the same polls. Either Compass's exclusion is over-cautious or the chamber layer inherits an untested case; the two should agree, and the handoff should say which.

### F5 — Five "unresolved" House seats are labelled unopposed by the source (Low; resolvable)

FL-10, MA-02, MA-05, MA-07 and WI-02 each list a single incumbent with `general_election_active = 3`; the same page renders "Incumbent is unopposed" exactly five times. The code keeps only `general_election_active == '2'`, so the candidate arrays become empty and the seats go unresolved. The handoff's sentence "Empty candidate arrays are **not** treated as proof of an uncontested election" is slightly misleading: the source arrays are not empty; the filter empties them. Keeping these unresolved is defensible (aggregator, not official), but the same official-list procedure used for California would resolve all five from the Massachusetts, Florida and Wisconsin certified ballots.

Also worth checking: Louisiana moved congressional races to closed party primaries starting in 2026, so LA-02 and LA-SEN may be ordinary two-party November contests rather than runoff-scope cases. I have not verified this against the Secretary of State and am flagging it as a question, not a finding.

### F6 — Source data quality in the roster (Low)

AK-SEN lists Dan Sullivan twice (`Daniel J. Sullivan` and `Dan Sullivan`, both R, both active) plus a second Republican. Alaska is unresolved anyway, but the duplicate sits inside the roster hash: when 270toWin cleans it, the chamber run will withhold until re-registration (see F7).

### F7 — Operational fragility by design (Low; acknowledge)

- Any change to candidate text in either 270toWin listing changes `roster_sha256` and withholds the public section until a human re-registers a new version. Fail-closed is the right default; expect it to trigger often in the final month.
- `election_chamber.view` hard-codes `bop-chamber-scenarios-v2`; each re-registration needs a site change too.
- `chamber.html` hard-codes "Nine California contests … eight Democratic and one Republican"; this text will not follow the review file.
- The first sandboxed execution of the chamber code (19:45 ET today) has not been observed. The earlier `mode=ro` WAL failure under `ProtectHome=read-only` was fixed for other components; the chamber path reads only committed files and the ledger, so I expect it to pass, but it is unverified.

### F8 — Documentation nits

- "Its text, raw PDF and source receipt remain local" — only the PDF and receipt are present; no extracted text file.
- Handoff lists "ID-SEN, MT, NE, SD" as unresolved for independents: confirmed (Achilles, Bodnar, Osborn, Bengs).

---

## 4. Responses to the handoff's questions for the auditor

1. **Marginals for unpolled seats.** Not yet suitable as published: see F1 (feature under-credit, three visible flips) and F2 (no national term). Fix F1 first; it needs no new coefficient.
2. **Estimating dependence from earlier cycles.** With seven cycles the shared variance share cannot be estimated with any precision; the reproduced evaluation shows the independent setting is clearly under-dispersed (sd ≈ 4 seats vs. realised |error| ≈ 10) and the shared setting is roughly consistent (sd 14.7 vs. MAE 10.3, ratio 0.70 against 0.80 for a normal). A defensible interim: fit a single national error term as the year-level residual mean from the fundamentals folds (that is the +4.9 / −19.2 pattern above) and treat its variance as the national share. Report it as estimated-from-seven-points.
3. **Official ballots.** Yes for the five unopposed seats (F5) using state certified lists; possibly Louisiana (F5 question). Independents (CA-06, ID, MT, NE, SD, AK, ME) should stay unresolved; the model has no basis for them.
4. **District-vintage inputs.** Not re-audited here beyond confirming all five at-large seats match current features and Missouri is withheld. A state-by-state geometry audit remains open from earlier audits.
5. **Envelope usefulness.** The display is honest. The widest unresolved-seat allocation (all 21 to one party) is not a plausible scenario, and users will read the pale bar as a forecast range anyway. Resolving F5 removes five seats from the gap at no modelling cost.
6. **Historical slates.** Agree; the eligible subsets (326–381 of 435) exclude up to a quarter of the chamber in some years, so no chamber-level validation is possible from this check.
7. **Generic-ballot signal.** The evaluation already shows where it would help (F2). Design: one national term, estimated from strictly earlier cycles, added to the fundamentals mean, with local polls left as they are (they already embed the environment, so apply the term to background-only seats or shrink it where polls exist).
8. **Simulator marginals.** Verified numerically across all 432 seats (max deviation 0.014). Negative or regional correlations were not tested; with one contested seat per state the Senate state term is inert, as documented.
9. **Party vs. caucus.** Handled correctly in code and copy (independents separate, no 50–50 assignment, no caucus assumed for new independents).

---

## 5. Recommendations, in order

1. Normalise candidate names in `prior_service` and in registry/roster poll matching (F1, F3); re-register as v3; re-run the historical check to confirm training-era matches are unchanged.
2. Report signed errors by cycle in the evaluation and on the page (F2).
3. Decide OH-SEN consistently across Compass and the chamber layer (F4).
4. Resolve the five unopposed seats from official state lists (F5); check Louisiana's 2026 rules.
5. Add a run-level flag for "poll record present but unmatched" (F3) and drive the California sentence in the template from the review file (F7).
6. Confirm the 19:45 ET run writes `experimental_scenarios` under the sandbox; if it does not, the alert path (verified) will say so at 08:45 tomorrow.

## 6. Pending / not done

- First scheduled sandboxed chamber run (19:45 ET, 2026-10-04) not yet observed.
- Louisiana ballot rules not checked against an official source.
- No new geometry audit of district vintages.
- No test of regional or negative correlations.