Merge Pipeline, Test Plan

Status: Draft Last modified: 2026-05-30 06:55 EDT

This page is the test-coverage roadmap for the new merge pipeline (chatter speaker-id + chatter merge + chatter adjudicate + the override-file format + the underlying talkbank-model::merge types). It exists because, per this repo’s root CLAUDE.md red/green TDD rule, every new feature starts with failing tests at the highest level the feature lives at, and we want to enumerate those tests before writing the implementation, so coverage is designed, not discovered.

This is a plan, not yet code. When the implementation work begins, every test case below becomes a real test; the doc then flips to a coverage matrix that gets kept honest by CI.

TDD discipline, what “strict red/green” means here

Every cycle of impl-phase work is:

1. RED. Write ONE failing test at the highest layer the feature lives at. The test exercises a real user-observable behavior, not an internal helper. Commit the failing test alone (or stage it before any code change), verify it fails for the right reason (the missing behavior), not for a compile error or a typo.
2. GREEN. Write the smallest code change that makes the test pass. No anticipating future tests, no scaffolding for tests that don’t yet exist. The codebase should compile and pass tests at this point.
3. REFACTOR. With the green test as the safety net, tighten the implementation: extract helpers, rename for clarity, replace primitives with newtypes, document tricky parts. Tests stay green throughout.
4. DRILL DOWN if needed. If the L3 (or L2) test passes but pinned the behavior less precisely than the contract requires (e.g., the L3 test asserts “exit 2 with some error” but the contract says “the specific MergeError variant must match”), add an L2 (or L1) test next that drills into the precise path. The drilled test FAILS at first against the green-but-imprecise impl, motivating the tighter impl.

Cycles must be atomic: one RED → one GREEN → optional REFACTOR → optional drill-down. Do not stack multiple tests on top of a single impl change; do not write impl ahead of tests. The discipline matters because the bug bar of this pipeline is high (CHAT-data byte-stable preservation, audit-trail reproducibility) and TDD is the cheapest way to catch regressions before they ship.

Three test layers + the adjudication layer

The merge pipeline’s behavior spans four substrates with different testing mechanisms.

L1 ⊂ L2 ⊂ L3 in terms of failure-mode coverage: a failing L1 test implies a failing L2 test which implies a failing L3 test. So when the same invariant could be tested at multiple layers, the starter test is the highest layer and lower-layer tests are supplements that pin the precise internal path. L4 sits beside L2/L3, same crate/file conventions but a dedicated layer because the prompter-injection pattern is specific to adjudication.

L1, Spec / fragment tests

Lives in spec/constructs/speaker-id/. Three subdirectories:

• token-cleaner/: what the Jaccard tokenizer strips and keeps
• jaccard-scoring/: fixed-input → fixed-score golden tests
• mapping-application/: header rewrite rules on real fragments

L1.1, Token cleaner

Each spec is a CHAT main-tier fragment + the expected token list after cleaning. Behavior pinned: bracket markup stripped, angle-bracket retracing unwrapped, terminator variants discarded, &-... / &+... discarded, xxx/yyy/www discarded, 0 discarded, @l / @n / @c suffix dropped, _-compound split to spaces, punctuation stripped, lowercased, ≥2-char alpha filter, NAK bullets stripped.

Each spec file in spec/constructs/speaker-id/token-cleaner/ has the standard # name, ## Input, ## Expected tokens, and ## Metadata sections per the spec authoring template at spec/CLAUDE.md in the workspace root (outside the book).

L1.2, Jaccard scoring

Fixed bag-of-tokens pairs with known multiset Jaccard. These guard against off-by-one errors in the sum_w min / sum_w max implementation and against any future “optimizations” that silently change scoring.

L1.3, Mapping application on fragments

Header-rewrite micro-tests. Each spec gives an input @Participants: or @ID: row and a small mapping; the expected output row is the rewritten form.

L2, Transform / AST tests

Lives in crates/talkbank-transform/tests/. Three test files:

• speaker_id_tests.rs
• transcript_merge_tests.rs
• override_file_tests.rs

Each tests behavior over parsed talkbank-model::ChatFile values, using inline synthetic CHAT strings parsed via talkbank_parser::parse_chat_file (no subprocess overhead).

L2.1, identify_mapping (reference mode)

L2.2, apply_mapping

L2.3, merge (core invariants)

These mirror the user-guide’s “What the merged output guarantees” section directly. Each invariant from that section maps to one or more L2 tests; the L3 tests then re-exercise the same invariant through the CLI.

L2.4, Override file I/O

L2.5, Domain-type unit tests

Smaller per-type tests. Each in its module’s #[cfg(test)] mod tests section.

L3, CLI / subprocess tests

Lives in crates/chatter/tests/merge_tests.rs (new file). Uses the same assert_cmd + predicates + tempfile pattern as the existing integration_tests.rs. Each test invokes chatter speaker-id or chatter merge as a subprocess against files written to a tempdir().

L3.1, chatter merge, success paths

L3.2, chatter merge, error paths

L3.3, chatter speaker-id, reference mode

L3.4, chatter speaker-id, explicit-mapping mode

L3.5, chatter speaker-id, override-file mode

L3.6, Pipeline composition

These exercise chatter speaker-id → chatter merge composed end-to-end through the file system, simulating the orchestrator workflow.

L4, Scripted adjudication tests

Lives in crates/talkbank-transform/tests/adjudication_tests.rs. Uses the Prompter trait and ScriptedPrompter documented in Adjudication Workflow §The prompter abstraction. Each test constructs a pending-adjudications input, scripts the operator’s decisions, runs run_adjudication, and asserts on the resulting override file plus the residual pending file.

L4.1, Speaker-id adjudication paths

L4.2, Parent-role-lookup adjudication paths

L4.3, Diarization-mix and sanity-scan paths

L4.4, Workflow plumbing

L4.5, Prompter contract conformance

These tests pin the contract that any Prompter impl must satisfy, so future UI backends (VS Code, web) can be developed against the same invariants.

Fixture catalog

These are the synthetic CHAT pairs that the tests above consume. Each is small (≤20 utterances), exercises a precise invariant, and is fully fictional (no real corpus content).

The fixtures live as inline const FIX_*: &str blocks in the respective test modules, following the precedent in chatter/tests/integration_tests.rs (which has const VALID_CHAT: &str = r#"..."# etc.).

FIX_REF_TWO_UTT_NO_MARKUP

The smallest possible valid CHAT pair input. Two *CHI: utterances, no markup beyond a simple terminator, time bullets on both. Used by cycle 1’s smoke test where the impl must work without yet handling any markup edge cases.

FIX_ASR_LABELED_TWO_UTT

The matching donor for FIX_REF_TWO_UTT_NO_MARKUP: two *INV: utterances at different time positions. Used by cycle 1.

FIX_REF_CHILD_ONLY_SIMPLE

A 6-utterance child-only hand transcript with rich CHAT markup (error code, retracing, filled pause, special-form letter, zero realization with paralinguistic). Used by every L2/L3 merge test from cycle 2 onward as the canonical “File 1”, the reference / authoritative file. Has time bullets on every utterance.

FIX_ASR_ANON_2SPEAKER_SIMPLE

The matching ASR-output file with anonymous PAR0 (clinician, asks questions) and PAR1 (child, says what FIX_REF_* shows plus some extra). Has %wor on every utterance. Used by every speaker-id test where auto-mode is expected to succeed cleanly (margin >> 2.0).

FIX_ASR_LABELED_INV_SIMPLE

FIX_ASR_ANON_2SPEAKER_SIMPLE after speaker-id has run with PAR1→drop, PAR0→INV:Investigator. Used by merge tests where we want to skip the speaker-id step and test merge alone.

FIX_ASR_BORDERLINE_VOCABULARY

ASR file where both speakers describe the same picture-book content (margin 1.6-1.9 against reference). Used by low-confidence tests.

FIX_REF_NO_BULLETS

A reference file with no time bullets at all. Used to test NoTimelineInFile1 precondition.

FIX_REF_LANG_ENG / FIX_ASR_LANG_YUE

Two files with conflicting @Languages. Used to test LanguageMismatch.

FIX_AMBIGUOUS_INV

Two files both containing *INV: utterances, with --retain CHI (INV not in retain set). Used to test AmbiguousSpeaker.

FIX_REF_MULTI_RETAIN

Reference file containing *CHI: and *SI2: utterances (sibling target). Used to test --retain CHI,SI2.

FIX_ASR_NO_MAIN_BULLET

Donor file where some utterances have no main-tier bullet, only %wor. Used to test bullet-lift behavior in normalization.

FIX_OVERRIDE_VALID / FIX_OVERRIDE_WRONG_SCHEMA / FIX_OVERRIDE_MALFORMED

Override files in valid, schema-rejected, and parse-rejected shapes. Used by override-file I/O tests.

FIX_PENDING_SPEAKER_ID / FIX_PENDING_PARENT_ROLE / FIX_PENDING_MIXED_KINDS

Pending-adjudications files exercising one kind, another kind, and a mix. Used by L4 adjudication tests.

FIX_SCRIPTED_ACCEPT_ALL / FIX_SCRIPTED_OVERRIDE_FIRST_DEFER_SECOND

Scripted-decisions TOML files for ScriptedTomlPrompter. Cover the canonical accept-suggested case and a mixed override+defer case.

The exact bytes of each fixture are pinned in their respective test modules when the implementation lands; this plan doesn’t freeze them yet, only their purpose. Drafting the actual bytes is the first step of impl-phase work.

Coverage matrix

Cross-checking that every behavioral invariant from the four design docs has at least one test:

Every invariant has at least one named test; many have multiple across layers. When the impl phase begins, the first commit should produce the fixtures, the second commit the highest-layer failing test for the simplest invariant, then drill down per the standard TDD progression.

What this plan does NOT cover

• Performance / scaling tests. Until the pipeline shows up on a measured workload, no targeted perf assertions. The reference corpus’s existing round-trip benchmarks remain the baseline.
• Fuzz testing. This repository now has a local fuzz/ workspace for parser/validation fuzzing. If the merge crate stabilizes enough to justify dedicated fuzzing, adding a merge-specific target for random parseable CHAT-pair inputs is a follow-up, not a v1 blocker.
• Cross-platform CI checks. Windows / Linux / macOS each build the workspace; the merge module rides the existing CI. No platform-specific tests needed (the merge operates on parsed AST and writes UTF-8; no path-or-line-ending quirks).
• Real-corpus regression sweeps. Once impl lands, running chatter merge over a curated subset of the reference corpus and snapshotting outputs is a smart follow-up. Lives in a separate tests/golden/ style mechanism if added; not designed here.

TDD authoring sequence

Each numbered item is one full RED → GREEN → REFACTOR cycle. Cycles must run in order; do not start cycle N+1 until cycle N is green and committed. Numbers are designed so the first working pipeline (cycle 8) emerges from the absolute minimum set of types + algorithms, then each later cycle extends.

The starter test for cycle 1 is intentionally tiny: a 2-utterance fixture pair with no markup, one retain speaker. The smoke test exercises every layer (parser, transform, CLI) but with the simplest possible CHAT bytes, so the first impl is small enough to land in one cycle.

Phase A, minimal end-to-end pipeline (cycles 1-8)

These cycles produce the simplest possible chatter merge working end-to-end with synthetic fixtures.

Phase A, actual cycle log

The four-precondition cycle 8 was deliberately split into four single-variant cycles (9a / 9b / 9c / 9d) so each MergeError variant lands with its own RED→GREEN cycle and L2 + L3 sibling tests. The numbering here is therefore finer-grained than the plan table above; the table records the shape of Phase A, the log records what was actually committed.

End of Phase A: chatter merge works on simple fixtures with all four preconditions (retain / timeline / language / ambiguous speaker) enforced. The pipeline is publishable as v0.

Phase B, actual cycle log

Phase B picks up at cycle 10 in the cycle log (Phase A used 9a-9d for the precondition split).

Phase B, speaker-id pipeline (cycles 9-16)

These cycles add chatter speaker-id and its three modes.

End of Phase B: full chatter speaker-id + chatter merge pipeline works auto + explicit + override modes.

Phase C, adjudication (cycles 17-22)

These cycles add the chatter adjudicate tool and its prompter-injection testability.

End of Phase C: scripted adjudication tested end-to-end with synthetic operator inputs. Interactive terminal UX still unimplemented (next phase).

Phase D, interactive UX (cycles 23-25)

End of Phase D: full v1 pipeline complete.

Phase E, non-speaker-id adjudication kinds (cycles 26-29)

Each adjudication kind gets its own RED→GREEN cycle.

Phase F, breadth pass (cycles 30+)

Fill in every remaining test from L1-L4 that hasn’t been written yet. These are coverage-deepening tests, not behavior adders. The impl from Phases A-E should pass them with at most minor refactoring; if a test fails meaningfully, that’s a gap in the impl that this cycle closes.

The breadth pass is the only phase where multiple cycles can proceed in parallel (different contributors take different test groups). Phases A-E are strictly serial.

Hard rules during impl phase

• No test stubs. Every test in this plan, when written, must FAIL before its impl exists and PASS after. Skipped or #[ignore]-marked tests are not allowed in the regression net (use #[ignore] only for genuinely slow or environment-dependent tests, not for “not implemented yet”).
• No test deletion to make CI green. If a test that was passing starts failing after a refactor, the refactor is wrong. Investigate; do not delete the test.
• Three cycle archetypes, distinguish them. A cycle is one of:
  • bug-fix: RED motivates new impl code (cycle N-1’s impl truly cannot satisfy the new test).
  • regression-guard: RED pins an invariant the impl inherits from upstream infrastructure (e.g. parse→serialize byte-stability inherited from talkbank-parser). The test passes against cycle N-1’s impl, but the cycle is valuable because it locks in the invariant against future “optimizations” that might break it. Verbose-output the actual behavior on first run to confirm the invariant holds for the right reasons, not by accident.
  • true no-op: RED tests something already pinned elsewhere. These ARE unnecessary; drop the cycle or sharpen the test. The difference between regression-guard and true no-op is whether the invariant is named explicitly anywhere else. If yes (e.g., the parser crate already has a roundtrip test that covers it), the cycle is true-no-op. If no, the cycle is a regression-guard and worth keeping.