# Balance of Power external audit report — Claude Code

**Auditor:** Claude Code (Fable 5.1), the project's assigned independent auditor. Not a human reviewer and not external to the project; see the independence statement at the end.

**Audit date:** October 3, 2026, about 10:20–12:00 Eastern.

**Reviewed:** the external audit package `BALANCE_OF_POWER_EXTERNAL_AUDIT_PACKAGE.zip` (31 entries; all 30 manifest hashes match the zip and the unpacked directory); runtime `670d74661` (`codex/autolab`); paper, website and derived results at root `2e83ae8b5`, which contains `2abcc8dad`; the live site `https://intuitionist-fin.lampbotics.com`.

**Access:** full local access to code, raw sources, model and evaluation artifacts, the election ledger (read-only), the budget ledger (read-only), service journals and traces. Nothing was blocked by access. I could not check whether port 8009 is reachable from outside this machine.

**Live actions taken:** none that change state. No model calls, no ledger or schedule writes, no POST requests. Read-only GETs to the public site and to the local port. One previously recorded live digest trace was replayed through the validator offline.

## Independent verdict

**Continue, but simplify.** The software is correct and the evaluation reproduces to the byte. The honest headline in the paper, that learned combination weights have not earned their complexity, is confirmed and is in fact stronger than stated: the fitted gate is nearly constant, so "learned weighting" is, in practice, a fixed blend of about 90% polls and 10% background that happened not to be one of the references. The defensible model is the three-covariate experience regression, or a fixed blend of it with background, with uncertainty calibrated on earlier cycles. Drop the gate from the primary comparison.

**Two things need attention before the next scheduled run.** Today's daily digest failed after three paid completions because the new output contract rejects answers that follow the new instructions, and the service is now in a failed state. The web application listens on all interfaces, so the public-host allowlist can be bypassed by requesting the machine's IP address directly.

**Strongest evidence:** every number in the package reproduces (artifact hashes, per-case scores, mixture quantiles, all six counted comparisons, the House two-week bootstrap). **Biggest uncertainty:** whether any of this transfers to the 2026 watchlist, whose House races rest on single and often sponsored polls, and which has no external forecasts recorded to compare against with the first deadline three days away.

| Dimension | Verdict | Supporting evidence | What could change the judgment |
|---|---|---|---|
| Implementation correctness | Good, with two live defects | 79 runtime and 10 site tests pass; three artifact hashes reproduce; independent re-scoring of 7,629 mixture cases matches exactly; chronology check finds no future cycle in any fit. Defects: digest contract (F1), direct-IP bypass (F2), withdrawn margins still embedded in page data (F5) | A failed reproduction on another machine |
| Data quality and historical availability | Acceptable for development only | Three-day publication assumption, retrospective top-two candidates, 2026 exports of historical geography; 42 of 394 polled 28-day cases carry an archive/outcome disagreement flag and are retained. Current House watchlist: one poll per race, seven sponsored | Recovered publication timestamps; a second poll source |
| Statistical and political validity | Modest and honestly described | Experience covariate gives most of the gain (better than ordinary in 6/6 groups); gate weights 0.75–0.98 with narrow spread within any fold; fixed 75/25 beats learned in 4/6 and in 6/6 gate-active groups (reproduced) | Prospective 2026 results; a model with a national environment term |
| Calibration and useful uncertainty | Mixed | Learned 80% ranges cover 75–83%; experience regression 72–80%; fixed 75/25 covers 88–93% because the background component is wide; polls-alone six-point covers only 60–62% in the House | Calibrating the fixed blend's scale the same way the experts are calibrated |
| Incremental value over simpler alternatives | Small | Learned minus fixed-75 Brier: within ±0.0007 in the Senate; fixed better by 0.0026–0.0072 in the House. Learned minus experience regression: within 0.0003 in four of six groups, and 0.003 and 0.002 in the House 28- and 14-day groups | A gate that varies with inputs and improves gate-active cycles |
| Agent research and learning claims | No learning claims made, correctly. Document arm promising; first live run failed on the output contract | Two originals fetched today with exact quoted fields; one verbatim quote reproduces a source inconsistency ("in Texas" in a North Carolina release) that the assistant did not flag | A completed week of digests with spot-checked extractions |
| Intellectual contribution | Applied evaluation with an informative negative result; no new estimator | Paper says so itself; claims trace to artifacts | Prospective evidence or more cycles |
| Public communication and readiness | Much improved; not ready as a forecast product, and correctly publishes no probabilities | Footer and about page fixed; probabilities withheld; three residual issues (F5) | Fix F5 and keep probabilities withheld |
| Political science publication readiness | Working paper after corrections; not journal-ready | One citation error; weak baselines; six cycles; development comparison | A prospectively evaluated cycle plus a replication of the cited Senate model |
| Practical or commercial value | Source verification plausible; forecasting leadership not | Today's extraction records are the first product-like output | Factual-quality measurements over several weeks |

## Owner goals and alternative designs

| Owner goal | Status | Evidence |
|---|---|---|
| Scientifically grounded forecasts for selected Senate races, extended to the House | Partial | Both chambers modelled with map-specific geography; 24 of 26 races receive shadow forecasts; probabilities withheld from the public, which is the right call |
| A novel model or workflow that could outperform established forecasters | Untested | No external forecast has been captured; one 270toWin polling-average margin was entered by hand. The first 28-day deadline is October 6 at 14:00 UTC |
| Polls plus original reports, questionnaires, datasets, markets, reporting | Partial | Originals are now read on demand from nine publisher hosts; two were read today. Markets are collected but now unusable for context pending contract verification |
| Agents that form expectations, compare with outcomes, retrieve cases | Absent for elections | Election workers are isolated with learning disabled, as the package says |
| Affordability | Delivered | Comparisons are CPU-only; the digest is one bounded paid turn a day (about 12,600 tokens today) |
| Separation from stock decisions | Delivered | Separate store, homes, budget scope; no imports |
| Plain-English website that does not imply validated chances | Mostly delivered | See F5 |
| Paper after methodological review | Delivered as a draft | See the paper review |

**Alternatives, judged against those goals.**

- *Keep the gate as the research arm (Codex's current proposal).* Costs little, but the gate's fitted weights barely move with its inputs, so the prospective "learned versus fixed 75/25" contrast will mostly measure 0.9 against 0.75. Low information.
- *Experience regression with earlier-cycle calibration as the single statistical arm, fixed 75/25 as the reference.* Same inputs, two fewer moving parts, interval scores within 1 point of the learned method in every group. **Recommended.**
- *Replicate Chen, Garnett and Montgomery with their public archive.* The one cited model with a peer-reviewed specification and a real prospective test. Several days of work; it would also give a named external comparator, which nothing here yet has.
- *A source-verification workflow with no headline probability.* The document arm already produces structured, quoted extraction records. Its value can be measured in weeks by a person spot-checking, and it is the one owner request for which this project has an evident advantage.

**Keep:** collection, ledger integrity checks, the two-chamber data pipeline, matched-case evaluation, interval scoring, withheld public probabilities, the document arm.
**Remove:** the learned gate from the primary prospective contrast; `expected_margin` from public page data.
**Restore:** a working daily digest (F1); external benchmark entries before October 6.
**Revise:** calibrate the fixed blend's scale; use a fitted-SD polls-only reference; bind the server to localhost.

## Findings

### F1. The new digest contract rejects answers that follow the new instructions
- **Importance:** high. The daily digest, the project's only live agent output, failed today and will fail again tomorrow.
- **Type:** confirmed defect. **Evidence status:** independently reproduced.
- **Location:** runtime `src/balance_of_power/daily.py:17` instructs "Cite only document IDs returned by document"; `daily.py:74` requires `evidence_ids` to equal exactly the packet ID; `daily.py:77` rejects any digit followed by `%` anywhere in the prose.
- **Reproduction:** the October 3 08:15 run (attempt `4f2cf60a…`) made three completions, read the packet and two documents, returned a well-formed answer with `evidence_ids` = [packet, document, document] and two complete `document_checks`. Replaying that stored answer through `validate()` gives "Must cite and read daily evidence". With `evidence_ids` reduced to the packet alone it still fails on "No numerical probability…" because the prose contains "8%". Journal: `status=1/FAILURE`; `systemctl --failed` lists the unit.
- **Consequence:** about 12,600 paid tokens discarded; no digest since October 2; the first live test of the document workflow is recorded as a failure although the extraction itself worked.
- **Remedy:** accept the packet ID plus any document IDs actually read; restrict the prose guard to probability language and leave poll percentages to the quoted fields or strip the `%` check; add a validator test built from this real answer.
- **Verify:** replay the stored answer; a passing validator and a completed run tomorrow.

### F2. The application is reachable by IP address, bypassing the public-host allowlist
- **Importance:** high for operations; outside the forecasting science.
- **Type:** confirmed defect. **Evidence status:** reproduced from this machine; external reachability untested.
- **Location:** root `public_boundary.py:22-23` applies the reading-page allowlist only when the Host header is exactly `intuitionist-fin.lampbotics.com`; the server binds `0.0.0.0:8009` (`ss -ltnp`); no local firewall rules.
- **Reproduction:** `GET http://194.233.91.49:8009/autolab`, `/research` and `/archive/desk` return 200 from this box. The same paths return 403 through the public hostname. `/api/*`, `/docs` and `/openapi.json` are blocked on all hosts, and no operator token is configured, so those stay closed.
- **Consequence:** if the cloud firewall does not block 8009, the legacy dashboard pages are public by IP.
- **Remedy:** bind to 127.0.0.1 and let the reverse proxy be the only entry, or make the allowlist the default for every host.
- **Verify:** `ss -ltnp` shows 127.0.0.1:8009; the IP request fails to connect.

### F3. The learned gate is nearly a constant
- **Importance:** high for interpretation; it reframes the paper's central comparison.
- **Type:** design weakness. **Evidence status:** independently reproduced from the development artifact.
- **Evidence:** fitted polling weights across all 18 gate-active folds range from 0.75 to 0.98, with medians 0.81–0.96; in any single fold the spread is at most 0.20. The gate intercept dominates (1.0–1.5 on the logit scale) and the three input coefficients are small. "Learned weighting" therefore behaves like a fixed blend near 90/10, which is why it tracks the experience regression (weight 1.0) within 0.0003 Brier in four of six groups and within 0.003 in the other two.
- **Consequence:** the prospective primary contrast, learned versus fixed 75/25, mostly compares two constants. It cannot show whether polling characteristics should change the weight.
- **Remedy:** make the experience regression, or a fixed blend, the statistical arm. If the gate is kept as research, report its fitted weights alongside its scores so readers can see it is not adapting.

### F4. The interval-score advantage of the learned method is a calibration artifact of the fixed blends
- **Type:** design weakness. **Evidence status:** reproduced.
- **Evidence:** fixed 75/25 covers 88–93% with widths 21–24 points; fixed 50/50 covers 87–95% with widths 26–28; the learned method covers 75–83% with widths 14–20. The experts are scale-calibrated on earlier cycles (factors 1.02–1.37 for polling experts, 0.72–0.94 for background) but the fixed mixtures of them are not recalibrated.
- **Consequence:** "learned beats fixed on interval score in 6/6 groups" is true and uninformative: a too-wide range loses an interval score to a near-nominal one by construction. The paper's statement that different objectives rank the methods differently is accurate, but the reason is width, not a genuine trade-off.
- **Remedy:** apply the same earlier-cycle scale calibration to the fixed blends before comparing interval scores. **Verify:** fixed-blend coverage near 80% after calibration.

### F5. Residual public-presentation issues
- **Type:** misleading presentation. **Evidence status:** code-confirmed and rendered page checked.
- **Evidence:** (a) the `/elections` page's `#map-data` JSON still carries `expected_margin` and `model_date` from the withdrawn statistical model for 20 races, although probabilities are `null` (`intuitionist_site.py:88`); (b) `/election-audit` lists only the October 2 review, does not link either report, and describes finding F1 as "an unreliable adjustment for party-linked polls" when the finding was that sponsored polls are not adjusted at all; (c) the sentence "a simpler, fixed blend did better in four of the six comparisons above" refers to a table that has no fixed-blend column; (d) `/models` says "The early results look promising" about unreviewed work.
- **What is fixed and verified:** footer and about-page wording; per-race poll details with sponsor; map colours and counts; stale, missing and corrupt report notices; invented-only slider examples; the direct caveat about losing five of six comparisons and 69–75% coverage; operational routes blocked on the public host; POST blocked everywhere.
- **Remedy:** drop the two fields from the page data; correct the audit page and link both reports; add the fixed-blend column or change "above".

### F6. The polls-alone reference is too narrow for the House
- **Type:** design weakness. **Evidence status:** reproduced.
- **Evidence:** the six-point normal polls-alone reference covers 60–62% of House outcomes at a nominal 80%. A polls-alone model with its scale fitted on earlier cycles would be the fair reference; the archive's House poll error has an RMSE near 8 points.
- **Consequence:** Table 1's "Polls alone" column overstates the gain from adding background.

### F7. The document arm works, and shows why extraction still needs a reader
- **Type:** useful improvement with a caution. **Evidence status:** reproduced from today's trace and ledger.
- **Evidence:** the assistant fetched the Big Data Poll North Carolina release and the UNH Survey Center page, and returned fields with exact quotes that `validate_checks` accepts. One quoted "fieldwork" excerpt reads "interviewed 698 registered voters and 642 likely voters in Texas from September 27 – 29, 2026", on a page titled for North Carolina; the sample sizes match the ledger's North Carolina poll. The inconsistency is in the publisher's own methodology paragraph. The assistant quoted it faithfully and did not flag it.
- **Consequence:** verbatim-quote checks establish provenance, not correctness, exactly as the package says. This is also the kind of source inconsistency the arm exists to surface; it needs a prompt step that asks the assistant to compare quoted fields with the feed and to name disagreements.

### F8. No external benchmark exists for the first fixed deadline
- **Type:** missing evidence. **Evidence status:** ledger read.
- **Evidence:** the ledger holds one benchmark record, a 270toWin Michigan polling-average margin entered October 2. The evaluation policy's first deadline is October 6, 14:00 UTC. Benchmarks must be entered before a deadline to count.
- **Consequence:** the question the owner cares most about cannot be scored for the 28-day horizon unless probability or margin benchmarks for the watchlist races are entered in the next three days, within each source's reuse terms.

### F9. Smaller items
- The old historical-error publisher still writes 26 `statistical_forecast` records a day (status "withdrawn… retained as research comparison"); fine, but the public page should not read them (F5a).
- Current `challenger_forecast` records say "nearest fixed-lead fit; exact anchor on scoring dates" — the 28-day fit is used from lead 40 down to 21. Acceptable if disclosed in the paper's prospective section; it is not.
- CA-48's only poll aged out of the 21-day window, so today's challenger forecast for it is background-only (p = 0.78). The daily digest said "no polls in this snapshot" for NE-01 and NM-02 for the same reason; correct but worth a line on the page.
- Paper reference: Dawid (1984) is pages 278–292, not 278–290. The Chen, Garnett and Montgomery page range (113–133) was not independently verified.

## Reproduction record

| Check or command | Expected builder result | My result | Interpretation |
|---|---|---|---|
| Runtime election tests | 79 passed | 79 passed in 22 s | Software rerun |
| Parent website and boundary tests | 9 passed | 10 passed (site plus boundary suites) | Software rerun, delegated and reported |
| Main comparison artifact | `e642b0de…6088a7` | `e642b0de…6088a7` | Exact rerun |
| Sensitivity comparison artifact | `84661950…6cd9ea` | `84661950…6cd9ea` | Exact rerun |
| Main full model rebuild | `2eca7299…596b8` | `2eca7299…596b8` (about 15 min, CPU) | Exact rerun from raw sources |
| Independent fixed-mixture calculation | Same probabilities and quantiles | 7,629 mixture cases (adaptive, 50/50, 75/25) recomputed from component t-distributions by my own bisection; max difference 0 | Independent recomputation |
| Independent interval score | Width plus outside-miss penalty | Recomputed per case for every method; identical to stored values | Independent recomputation |
| Equal-cycle aggregation and matched cases | Same samples within each contrast | Case counts 134/130/149 and 260/288/319 match; all four counted claims reproduce: learned better than ordinary 6/6; fixed 75/25 better on Brier 4/6 and 6/6 gate-active; learned better on interval score 6/6 | Independent recomputation |
| House 14-day learned minus fixed 75/25 | +0.00723, interval about +0.0015 to +0.0140 | +0.00723; my bootstrap +0.0016 to +0.0142 (different seed) | Independent recomputation |
| Earlier-only training and gate-active sample | No held-out outcome in fits; 2018–2022 active | 0 violations across every expert's training and calibration cycles and every gate; active folds are 2018, 2020, 2022 | Independent check of artifact metadata |
| Paper table values | Tables 1–4 and in-text figures | All checked values match (including 0.1185, 0.1561, 0.1117/0.1116, coverage 74.7–82.8 and 77.7–81.5, 158 and 421 distinct races) | Independent recomputation |
| Package integrity | Manifest hashes | 30 files match the zip and the directory | Verified |
| Public claim and data boundary | Honest scores, no live chance promotion | No percentages for 2026 races rendered; withdrawn margins still embedded (F5a); allowlist bypassable by IP (F2) | Production read-only check |

**Environment:** Python 3.12 project venv at `/root/vibe-infoveillance/venv`, `OPENBLAS_NUM_THREADS=1`, outputs under the session scratch directory, nothing written under the project data or docs trees except this report.

## Stronger comparisons and unfavorable cases

- **By cycle, learned versus fixed 75/25 (Brier).** House 14 days: learned worse in 2012, 2014, 2016, 2020 and 2022, better only in 2018. Senate: differences within ±0.001 at every lead except that the fixed blend is clearly better in 2020 at all three leads (for example 0.0439 versus 0.0577 at 28 days) and clearly worse in 2012 and 2018.
- **Gate-active cycles only (2018–2022).** The learned method is worse than the fixed blend in all six groups and also worse than its own polling expert in 2018 at every lead in both chambers. Its gain over the experience regression comes almost entirely from 2020 and 2022 in the House, where, on my reading of the cases, the background component pulled polled margins toward the Republican side.
- **Competitive subgroup (|lean| ≤ 10).** Learned versus ordinary is indistinguishable in the Senate (−0.0008, +0.0001, −0.0009) and modest in the House. Fixed 75/25 beats learned in all three House horizons and loses in two Senate horizons.
- **Unavailable races.** Two Senate races withheld for ballot format; Missouri polling-only; CA-48 background-only today. These are reported, not hidden.
- **If I were proposing a model:** the experience regression already in the artifact, with its calibrated scale. Its appearance in this list is another development result, not validation; it is defensible because it is the simplest member of a family that was fixed before the comparison.

## Paper review

**Question and answer.** The paper asks whether learning combination weights helps with sparse congressional polling and answers no, not yet. That is a real question and the evidence supports the answer. F3 strengthens it: the learned gate did not learn to adapt.

**Literature.** The precedents are the right ones and are not over-claimed. One citation error (Dawid's page range). The Chen, Garnett and Montgomery comparison is correctly described as a closer precedent that is not replicated here.

**Novelty.** None claimed, correctly. The paper's framing as "an auditable comparison workflow and a specific negative result" is the strongest defensible one.

**Methods and claims.** Every number I checked traces to the artifacts. Two presentational points: the "Polls alone" column is a weak reference for the House (F6), and the interval-score result should be reported with the fixed blends' coverage so readers see why they lose (F4).

**Authorship and disclosure.** Accurate in the Markdown: Codex drafted, human authorship unresolved, my prior audit did not verify the new comparisons. This report is that verification of the numbers; it is not peer review or an endorsement of the conclusions.

**Recommendation.** Circulate as a working paper after the corrections above. A journal submission would need either a prospective cycle or a replication of the cited Senate model as the external comparator, and should probably be framed as a research note: "Do learned combination weights help sparse-polling congressional forecasts? A chronological evaluation" fits the evidence better than the current title, which promises more than the study delivers.

## Minimum informative next study

- **Objective:** does a calibrated three-covariate regression, blended with background at a fixed weight, forecast 2026 polled congressional races at least as well as polls alone and as well as public aggregator numbers for the same target.
- **Race population:** every 2026 Senate race with at least one qualifying poll, not only the six on the watchlist, plus the 20 House districts. Register the list before October 6.
- **Primary comparison:** experience regression versus fixed 75/25 on Brier at 28, 14 and 7 days. Secondary: learned gate, polls-alone with fitted scale, interval score with recalibrated blends.
- **Outcome target:** the general-ballot Democratic-minus-Republican margin from certified results, as the evaluation policy already defines.
- **Information cutoff:** 14:00 UTC at each lead, latest record in the preceding 24 hours, as implemented.
- **Frozen versions:** the pinned artifact hash; any revision becomes a separate arm.
- **Scoring:** Brier, log loss, margin error, interval width and interval score, by chamber and lead; abstentions and unavailable races counted in the denominator.
- **External benchmark:** at least one public margin or probability per race entered before each deadline, within reuse terms, or the outperformance question is declared unanswerable for that lead.
- **Evidence threshold:** one cycle cannot establish superiority. Success means the pipeline completes with no missing forecasts and the statistical arm is not worse than polls alone; failure means missing forecasts, deadline misses, or the arm losing to polls alone on both Brier and interval score.
- **Agent arm, separately:** continue the document digest daily; a person spot-checks ten extraction records a week against the originals; report accuracy, disagreements found with the feed, abstentions and cost. Do not combine this with the statistical score.
- **Cost:** CPU-only statistics; one bounded paid turn a day.

## Prioritized action list

| Priority | Action | Disposition | Expected benefit | Evidence needed | Verification |
|---|---|---|---|---|---|
| 1 | Fix the digest output contract (F1) before tomorrow's 08:15 run | Repair | Restores the only live agent output; stops paid failures | Replay of today's answer | Validator accepts it; a completed run |
| 2 | Bind the application to localhost or default-deny unknown hosts (F2) | Repair | Closes the IP bypass | `ss -ltnp` | IP request refused |
| 3 | Enter external benchmarks for watchlist races before October 6, 14:00 UTC (F8) | Restore | Makes the outperformance question scorable at 28 days | Source terms checked | Benchmark records dated before the deadline |
| 4 | Remove `expected_margin`/`model_date` from page data; fix the audit page; add the fixed-blend column (F5) | Repair | Public page matches what is withheld and what was found | — | Rendered JSON and page text |
| 5 | Make experience regression (or fixed blend) the primary statistical arm; report gate weights (F3) | Revise | Prospective test measures something informative | Already in artifact | Amended policy record |
| 6 | Calibrate fixed blends' scale on earlier cycles; add fitted-SD polls-alone (F4, F6) | Revise | Fair interval and polls-alone comparisons | Development rerun, labelled as such | Coverage near 80% |
| 7 | Paper: Dawid pages; coverage beside interval scores; retitle as a research note | Repair | Accurate working paper | — | Updated PDF/MD |
| 8 | Add a comparison step to the document prompt: quoted fields versus feed values, disagreements named (F7) | Revise | Turns extraction into verification | A week of digests | Spot-check log |
| 9 | Chamber simulation, national environment term, Senate-model replication | Defer | Not supportable with current data and time | More cycles / prospective evidence | — |

## Scope and independence statement

I audited the October 1 pilot (three rounds) and the October 2 build, and several recommendations from those audits shaped this extension: the fixed-blend references, the ordinary-regression comparator, withholding probabilities, the document arm. I am therefore not independent of the comparison design, and this report should not be read as external peer review. I had no part in writing the code, the paper or the package.

The website portion was delegated to a reviewer agent; I verified its load-bearing claims myself (the IP bypass, the footer text, the embedded margins, the public-route status codes) and report the remaining detail as it was reported to me.

Untested or blocked: external reachability of port 8009; reuse terms of any source; the Chen, Garnett and Montgomery page range; the PDF and Word copies of the paper beyond spot comparison with the Markdown; screen-reader behaviour.

Changes made to sources, ledgers, models, schedules or providers: none. This audit certifies that the reported numbers reproduce and that the code does what the documents say. It does not certify forecast accuracy, calibration on 2026 races, or fitness as a public forecasting product.