# Independent methods and fact-check memo: University News pipeline

**Review date:** 2026-08-10  
**Scope:** Frozen addendum, automated script/results/change log, automated CSV schemas and records, and the manual-sample design/codebook. I did not inspect either manual coder's output.

## Bottom line

The corpus reconciliation and the arithmetic in the saved automated results are reproducible, but the semantic results are **not ready to be published as validated population findings**. Two parser defects affect early-text measures, the planned manual instrument cannot validate several primary/secondary automated measures, the actual manual assignment sample is a strongly purposive diversity sample rather than a representative 15% sample, and the strict-versus-broad framing result reverses direction. These issues require correction, validation, or suppression; they do not justify a claim that bias has been established.

## Checks that passed

- The frozen addendum hash is `26bd2ae9dce549a6d488dd3946051841cba8ed3381b374e92f6da7ae03e72e9b`, exactly the value enforced by the analysis script.
- The private-corpus hash is `ec95115b69c7490ce53340c92076eef9bb9c713d7f28a0da07ca638fde2bd797`. The index and corpus each contain 1,527 unique URLs; their URL sets are identical; all 1,527 extraction statuses are `success`; no body is empty. Counts independently reproduce as Administration 671, Academics 266, and Student Life 590, with URL dates 2023-10-02 through 2026-08-05.
- The current script hash (`f62cdec3a149f9568585212801aa6a1c29823f326eead8115ad556ee6f0c1855`), results hash (`af3433ae33e9e37437f6ea9609fe88e1857bbe9cec27ce9222c1d58c4bc5281a`), and preserved first-run results hash (`ba67f122a395aa26060c858835960421d65020845109237bd9d88ae50ddd81bb`) match the recorded values. The topic-proxy correction is disclosed and the adjusted models are labeled post-outcome exploratory.
- CSV parsing yields exactly 1,527 article records and 6,643 actor records, with no duplicate article ID, no actor row pointing to an unknown article, and no body-hash mismatch.
- Direct recomputation from the CSVs reproduces the four saved point estimates and sample sizes. Sorting by article ID and using the frozen NumPy seeds reproduces all four bootstrap intervals exactly.
- Sign conventions are internally correct: negative scrutiny/framing gaps are defined as consistent with less scrutiny/marking of administrators; positive first-quote/quote-share gaps mean greater administrative quotation priority. Downstream prose must retain those conventions.
- The manual assignment manifest is reproducible. It contains 230 unique articles (15.062% of 1,527), coder assignments of 140 each, and exactly 50 overlapping articles. All three recorded SHA-256 hashes match.
- The automated public CSVs contain metadata/coding rather than full bodies. The largest individual evidence excerpt is 25 words; no email address or phone-number pattern was detected in those excerpts.

## Blocking corrections before outcome recalculation

### 1. Early-text parser defects

All 273 records recovered by `authorized_http_crawler.py` have `description` equal to the archive subdesk label (`Administration`, `Academics`, or `Student Life`) rather than the visible article dek. This is 17.9% of the census. The script then treats that value as reporter-written headline/description text for presence, framing, and topic coding.

In addition, 69 of those 273 recovered records have no double-newline paragraph boundaries. `build_rows()` therefore treats the entire article as the “first two paragraphs.” These 69 records are not negligible: 67 enter the strict paired analysis. Their automated framing gap is -0.124, compared with -0.029 for the remaining paired records; this does not show the parser caused the difference, but it shows that the affected records are influential. Official-origin detection fires in 42/69 one-paragraph records versus 15/204 other HTTP-recovered records.

The dek and paragraph structure should be re-extracted from the visible page or the correct structured fields, hashes updated, and every early-text-derived variable recalculated. The addendum expressly permits a documented parser-defect correction. Until then, headline/dek/lede frames, primary topics, official origin, release reliance, and any model using topic should not be finalized.

### 2. Frozen official-origin and lede definitions are not implemented

The addendum defines official origin over the headline, description, or first two body paragraphs. The code tests only the body lede. Adding the currently stored title/description detects three additional cases, but that is not a valid final correction while 273 deks are wrong. The manual codebook also defines the lede as description plus the first two body paragraphs, whereas automated `lede_frame` excludes the description. Align both implementations after the parser repair.

### 3. The two manual samples conflict

`university-automated-results.json` points to `work/university-manual-validation-sample.csv`, generated with seed `university-news-validation-2026-08-10-v1`. The coder assignments instead use `work/university-manual-sample.jsonl`, generated with seed `university-news-manual-audit-2026-08-10-v1`. Both contain 230 rows, but only 29 article IDs overlap. The final package must designate one as the actual audit sample, preserve the other only as an abandoned/preliminary artifact if needed, and make the results/manifest/methodology agree.

### 4. The assignment sample is not representative

The assignment script allocates proportionally only by subdesk-quarter, then cycles through screen signatures starting with the rarest. In practice, the 230 rows come from 230 distinct combined strata. Of 413 population subdesk-quarter/signature strata, 183 strata containing 924 articles receive no article, while 127 selected strata are fully enumerated. Marginal inclusion rates range from 8.9% for long stories to 50.3% for short stories; from 12.2% for “both actor” stories to 71.0% for admin-only stories; and from 12.3% for relevant-screen stories to 68.4% for screened-irrelevant stories.

This is defensible as a purposive stress-test sample for reliability, but unweighted manual percentages or treatment gaps cannot be generalized to the census. Either add a separately identified probability/representative validation sample, or restrict this sample to error analysis and use a prespecified calibration/standardization method whose unsupported strata and assumptions are reported.

## Measures that require suppression or redesign pending validation

- Quote attribution is weak on its own diagnostics: among the 1,295 strict paired articles, mean attribution confidence is 0.629 and 513 articles are below 0.60. Treat first-quote, quote-share, and attribution-verb results as screening outputs until manually validated.
- The current manual codebook cannot validate direct-quote counts, quoted-word share, attribution-verb class, category-specific challenger estimates, source counts/types, or correction/update detection because it contains no corresponding fields. It also has no documented crosswalk from human tone/frame codes to automated `marked_strict`. Add those fields and freeze the crosswalk before examining coder outcomes, or suppress the affected automated population estimates.
- `affected_people_consulted_auto` is actually “any challenger-category quote detected,” not consultation of an affected person beyond the named parties. It must be renamed to its observable proxy or suppressed as a measure of affected-person sourcing.
- `source_count_proxy` counts actor categories plus institutional/document term types, not people; `source_types_proxy` scans mentions anywhere in the body rather than verified sources. These may remain explicitly labeled screening proxies but cannot support claims about unique-source diversity without manual validation.
- The scrutiny score applies an article-wide document flag to actor windows and awards one point whenever the opposing side has a claim. It does not establish that a document or test bears on the focal actor's assertion. Treat it as a heuristic, not verified assertion testing.
- The high-confidence robustness filter uses an article-level average of actor-presence confidence, not confidence in framing, scrutiny, or quotation attribution. Rename it or construct outcome-specific confidence filters.
- Eligibility is not enforced in the automated run: every archive row is classified into a news-type proxy and defaults to `straight_news` if no rule matches. Quantitative results must be recomputed after any manual `eligible_news=0` exclusions so opinion/sponsored/non-news items do not remain in the denominator.

## Robustness and model issues

- The primary strict framing estimate is -0.034 (95% bootstrap interval -0.052 to -0.016), but the broad-dictionary check is +0.110 (0.093 to 0.127). This sign reversal fails the addendum's requirement that direction be stable across substantively related robustness checks. Automated framing cannot be summarized as stable evidence of administrative deference.
- Category-specific strict framing is heterogeneous: students/organizations -0.068, protesters -0.013, workers/unions -0.004, and faculty critics +0.081. The latter is contrary in direction. Pooled framing must not be generalized to every challenger category.
- The adjusted coefficients are numerically identical to the unadjusted gaps because each article contributes an administration row and a challenger row with identical article-level covariates and the model includes no actor-by-covariate interactions. Main-effect controls cannot adjust a within-article actor gap. Use an explicitly paired-gap model and/or prespecified actor-by-topic/time/type interactions, or describe the current model as a clustered-SE re-expression rather than confounder adjustment.
- The robustness suite is implemented almost entirely for framing. It does not apply the declared exclusions/thresholds/category separations to scrutiny and quotation outcomes; it does not provide the planned all-actor-present versus paired comparison, author-fixed-effect sensitivity, or article-clustered sensitivity. The check named `exclude_breaking_obituary_event_recap_under_300` only excludes proxy `breaking_brief` and stories under 300 words; the automated type function never emits `obituary` or `event_recap`.
- Bootstrap intervals are valid article-resampling summaries under the stated procedure, but this is a complete fixed-period census. They quantify sensitivity to resampling articles as a notional superpopulation, not archive-sampling error and not automated-coding error. Final reporting should say so.
- The four unvalidated primary directions are mixed: scrutiny and strict framing are in the hypothesized direction; first-quote order favors administrators; direct-quote word share favors challengers. Secondary first-actor order also favors challengers. None should be converted into a one-sided “bias exists” conclusion before validation.

## Copyright and privacy

No full body appears in the two automated public CSVs. However, 920 actor-level excerpts occur across 520 articles; aggregating by article reaches as many as 124 excerpt words, and 278 articles exceed 25 aggregate excerpt words. Bulk automated excerpts are not necessary to demonstrate specific findings and create avoidable cumulative copyright/privacy exposure. Remove them from the public actor census, or retain only manually selected evidence excerpts capped at 25 words per article. Keep the full-text corpus and coder-assignment JSONL private. No obvious email or phone pattern was found in the current excerpts, but that scan is not a substitute for human harm review.

## Release decision

**Pass for internal reproducibility:** corpus URL reconciliation, hashes, CSV integrity, point estimates, bootstrap arithmetic, and change-log preservation.  
**Do not release as validated findings yet:** all semantic population estimates. Repair the 273 dek records and 69 paragraph structures, reconcile the sample identity, revise or supplement the manual instrument, compute reliability with uncertainty, apply the preregistered suppression threshold outcome by outcome, and present null/contrary results and strong-reporting cases alongside any surviving imbalance.
