# METHODOLOGY - FMCSA National Motor Carrier Dataset (frozen v1.0, 2026-10-11)
Reproducible transformation of official FMCSA open data into a dated edition. Implementation: `scripts/fmcsa-pipeline/fmcsa_build.py` (to be written against this document; a deviation is a defect in the code or a discrepancy record, not a silent change).

## 1. Acquisition
* Sources: official FMCSA datasets on `data.transportation.gov` (Socrata SODA CSV, `$order=:id`, 100,000-row pages, declared User-Agent, at most 3 requests/s). Dataset ids, expected row counts and column maps: `pilot/SOURCE_MANIFEST.csv`, `SOURCE_SCHEMA_DISCOVERY.md`.
* Each edition records, per dataset: retrieval start/end UTC, source-reported total rows (`count(*)`), rows received, SHA-256 of the raw CSV, Socrata `rowsUpdatedAt`. **Row count received must equal the source `count(*)` taken immediately after acquisition**, else the build stops (tolerance 0; if the source changed mid-pull the dataset is re-pulled).
* Raw files stay on the build runner and in the private release artifact; they are not committed to git.
* Network: the agent sandbox cannot reach FMCSA hosts, so acquisition runs on a GitHub-hosted runner (`.github/workflows/fmcsa-pilot89.yml`, manual push-triggered mode).

## 2. Source dates (binding)
* `census_as_of`, `motus_as_of`: Socrata `rowsUpdatedAt` date of the pulled Census / Motus tables (both refresh daily).
* `legacy_as_of` = **2026-05-14, a constant**. FMCSA catalog text on all six legacy tables reads "This dataset was last refreshed on 05/14/2026 and will no longer be updated". Socrata `rowsUpdatedAt` for legacy tables keeps moving and **must never be used**. Each build also computes the maximum parsed date of the "recorded when" columns (`qh9u-swkp.trans_date`; `9mw4-x3tu.orig_served_date`/`disp_served_date`; `6sqe-dvqs`/`sa6p-acbp` serve and effective dates not in the future) and writes them to the manifest as corroborating evidence; a legacy "recorded" date after 2026-05-14 fails the build (future-dated effective/cancellation dates are scheduled events, not updates).

## 3. Normalisation
DOT (strip zero padding; reject `00000000`), docket (prefix + number, zeros after the prefix removed, upper case), dates (three source formats to ISO), amounts (Motus dollars as stored; legacy thousands x 1000; legacy 0 -> blank with `amount_note`), phone digits as in source, strings trimmed only. Details and traps: `EDGE_CASES.md`.

## 4. Per-docket merge (authority and insurance)
For every docket seen in Census, Motus Carrier or legacy Carrier:
1. If the docket has at least one row in Motus Carrier (`inys-ebih`), the Motus rows are the authority record (one row per operating authority) and Motus Insur supplies its active/pending filings. `source_table='motus'`.
2. Otherwise the frozen legacy rows apply (Carrier `6eyk-hxee`: Common/Contract/Broker status; ActPendInsur `qh9u-swkp`). `source_table='legacy_2026-05-14'`.
3. A Motus row never supplements a legacy row of the same docket field-by-field; the docket is taken whole from one source.
4. Census docket status is carried as a separate column; disagreement is flagged, never resolved by choosing Census.
Evidence for the rule: SAFER comparison 86/89 for the merge vs 77 legacy-only, 65 Motus-only, 83 Census; portal comparison 55/57 dockets (see `pilot/INVESTIGATION_REPORT.md`).

## 5. Identity rule N1 and phone rule R-PH1
* N1 as in `SCHEMA.md`, implemented by `scripts/fmcsa-pipeline/n1.py` (7 unit tests) and verified over every output row (`withheld set == Census INDIVIDUAL set`; all other fields byte-identical pre/post).
* R-PH1: `phone` = Census `phone`; only if blank, Motus `bus_telno` (matched by USDOT) with `phone_source='motus_fallback'`. Conflicts (both present and different digits after dropping a leading 1) are counted in QA, not exported. N1 applies to whichever source supplied the phone.

## 6. Roll-ups and aggregates
Authority/insurance roll-ups are computed only from the output child tables. Inspection and crash aggregates are plain sums/counts/min/max of the official columns by `dot_number`; no weighting, scoring, percentiles or severity logic. Dates are parsed once. Rows whose DOT is not in Census are listed in the exception report and not added to `carriers` (they remain in child tables only if their USDOT is in Census).
Cargo name map (Census `crgo_*` flag equal to `X`): genfreight General Freight; household Household Goods; metalsheet Metal: Sheets, Coils, Rolls; motoveh Motor Vehicles; drivetow Drive Away/Tow Away; logpole Logs, Poles, Beams, Lumber; bldgmat Building Materials; mobilehome Mobile Homes; machlrg Machinery, Large Objects; produce Fresh Produce; liqgas Liquids/Gases; intermodal Intermodal Containers; passengers Passengers; oilfield Oilfield Equipment; livestock Livestock; grainfeed Grain, Feed, Hay; coalcoke Coal/Coke; meat Meat; garbage Garbage, Refuse, Trash; usmail US Mail; chem Chemicals; drybulk Commodities Dry Bulk; coldfood Refrigerated Food; beverages Beverages; paperprod Paper Products; utility Utilities; farmsupp Farm Supplies; construct Construction; waterwell Water Well; cargoothr Other (text in `cargo_other_desc`). Flag values other than `X`/blank are counted and listed.

## 7. Outputs
CSV (UTF-8, RFC 4180 quoting, LF, header row, sorted by `usdot_number` then docket/effective date) and Parquet (zstd) for each table and edition view; manifest with SHA-256, row counts, source dates; data dictionary; limitations; changelog; certification report. Two builds from the same raw files must give identical SHA-256.

## 8. Release QA
`ACCEPTANCE_CRITERIA.md`. Independent factual validation (portal/SAFER, fidelity re-fetch by a separate code path, known discrepancies re-checked) and the independent external-AI audit gate are governed by `docs/DATASET_MANUFACTURING_AND_RELEASE_CERTIFICATION_STANDARD.md`.

## 9. What is not done
No imputation, entity-class inference, address standardisation, geocoding, deduplication of carriers, name matching, scoring or "safety" ranking. No live lookups are bundled.

## 10. External-dependency and commercial-source checklist (Process Lessons 2026-09-28)
| Input | Availability | Provenance | Commercial-use rights | Cost | Retention/reproducibility | Derived-use |
|---|---|---|---|---|---|---|
| FMCSA Census, Motus, legacy authority/insurance, inspection, crash tables (data.transportation.gov) | public, daily (Census/Motus), frozen 2026-05-14 (legacy) | official dataset ids and `rowsUpdatedAt` recorded | 17 USC 105 (federal work), DOT Order 1351.34A "open by default", FMCSA data page "no charge"; site is silent on resale; no clearance claimed; Privacy Act systems of records, state privacy and telemarketing laws unassessed (`pilot/SOURCE_RIGHTS_MATRIX.md`) | $0 (runner minutes only) | raw files retained in the private release artifact | no derived-use restriction found |
| MOTUS public portal and SAFER | public pages | observed by script | validation use only; **no portal data is redistributed** | $0 | counts, ids and booleans only in git | n/a |
| Any other external input | none | | | | | |