# University News corpus README

## Scope and cutoff

This package indexes every unique article linked from Columbia Daily Spectator's official `News | Administration`, `News | Academics`, and `News | Student Life` archives with a URL date from 2023-10-01 through the collection cutoff, 2026-08-10. The archive contained no October 1 article and no article dated August 6–10, so the observed publication-date range is 2023-10-02 through 2026-08-05.

The reconciled frame contains 1,527 articles: 671 Administration, 266 Academics, and 590 Student Life. All 1,527 pages produced a nonempty record. Opinion, City News, Sports, Spectrum, and other sections are outside this University News addendum.

## Authorization and collection

The first audit attempt used a third-party search index and did not crawl Spectator because the site's published Terms restrict automated extraction. The user subsequently stated that they had Spectator authorization to crawl and download the entire University News archive and expressly instructed the research team to proceed. This package records that statement; it does not independently authenticate or transfer the permission.

Collection followed visible archive pagination and ordinary public article URLs. Request starts were separated by at least one second, including retries. No login wall, paywall, CAPTCHA, access control, undocumented API, or robots restriction was bypassed. A structured `Fusion.globalContent` object embedded in each public article page supplied canonical metadata and story elements. The crawl reconciled its URL set against the archive before analysis.

## Public and private data

The repaired private working corpus is `work/private-university-corpus-final.jsonl`, SHA-256 `b697367b2f86dbd04d11c5a25a0d4113a4f94606febf18b2979e6bc8d628f37b`. It contains 1,539,459 normalized article-body words and is **not** in the evidence bundle. The pre-repair version is preserved privately; `corpus-parser-repair-log.md` documents the 70 re-requested pages and description repair.

The public package contains:

- article URLs and published metadata;
- hashes of each normalized body, allowing an authorized holder to verify the analyzed version;
- automated and manual coding decisions;
- evidence pointers and excerpts capped at 25 words;
- corpus and script hashes; and
- scripts that reproduce analysis when supplied with the private input at the documented path.

It does not contain full articles, page HTML, photographs, video, or the raw DOCX inputs. Public subtitles/descriptions are published metadata, not reconstructed article text.

## Files

- `articles.csv`: one row per archive article, including stable ID, title, subtitle, description, author, date, section/subdesk, URL, retrieval date, topics, leadership period, content hash, word count, and coding status.
- `article-coding.csv`: one automated census row per article. Semantic fields remain screening variables unless their manual-validation gate passes.
- `actor-level-analysis.csv`: one row per actor category detected by the strict or broad rules. Absent categories are not emitted; use the article denominator and presence fields rather than treating a missing row as a scored zero.
- `university-manual-audit.csv`: the 230-article manual audit after adjudication, with short excerpts only.
- `university-manual-intercoder-reliability.csv` and `university-automation-validation.csv`: reliability and validity checks.
- `corpus-manifest.json`: reconciliation counts and hashes.

## Stable IDs and versioning

`article_id` is `spec-` plus the first 16 hexadecimal characters of SHA-256 of the canonical URL. `content_hash` is SHA-256 of the normalized body extracted on 2026-08-10. The analysis scores the latest version visible at that cutoff and records visible update/correction notes; it does not infer silent edits or reconstruct earlier versions.

## Reproduction limits

The package can reproduce all post-collection calculations. Exact re-extraction requires continued lawful access to Spectator and may differ if pages change. Hash mismatches should be treated as version differences, not silently accepted. The public bundle intentionally cannot reproduce body-dependent scoring without an authorized local corpus because redistributing full copyrighted articles would be inappropriate.
