Introduction

Status: Current Book last changed: 2026-09-15 (commit bb4bef82) This page last changed: 2026-09-01 (commit b1db0410)

TalkBank is the world’s largest open repository of spoken language data. This repository (TalkBank/chatter) is the standalone home of the CHAT format authority and the chatter tool family: the chatter CLI, the Rust crates for parsing/validation/transformation, the tree-sitter-talkbank grammar, the talkbank-lsp language server, and the desktop validation app.

chatter is publicly released. To get it right away:

• Command-line tool (macOS / Linux): curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.sh | sh (Windows and other options: Install).
• Desktop app: download for your platform from the latest release.
• Full installation guide (all platforms, package details): Install.

The Rust crates are source-available from this repository (not yet published to crates.io). As a 0.x release, APIs and flags may change before 1.0.

Choose the right surface

What’s In This Repo

• chatter CLI: validate, convert, normalize, and analyze CHAT files from the command line, with an interactive TUI for corpus-scale workflows
• Language Server (LSP): works with any LSP-compatible editor (Neovim, Emacs, Helix, Zed, etc.) to provide live validation and cross-tier alignment
• JSON data model: every CHAT structure as typed JSON with lossless roundtrip fidelity, backed by a published JSON Schema
• Rust API: parse, validate, inspect, and transform CHAT files programmatically via library crates

Who This Book Is For

Repository Layout

grammar/        Tree-sitter grammar for CHAT
spec/           Source of truth: CHAT specification + error specs
crates/         Rust crates for model, parser, transform, cache, CLI, LSP, tests, and FFI support
apps/           Tauri v2 desktop app (`chatter-desktop`)
corpus/         Reference corpus (must stay 100% valid under the regression gate)
schema/         JSON Schema for the CHAT AST
tests/          Integration tests and fixtures
docs/           Strategy docs, proposals, and investigations for this repo
book/           This documentation (mdBook)

Data flows: spec (source of truth) → grammar (tree-sitter) → Rust crates (parsers, model, validation, CLI, LSP) → applications (chatter, desktop app).

This page last changed: 2026-09-01 (commit b1db0410). The whole book last changed: 2026-09-15 (commit bb4bef82).

Install

Status: Current Last modified: 2026-07-31 06:13 EDT

Everything here comes from the latest release.

Just want to check CHAT files, and you are not a programmer? Get the desktop app below. You never need a terminal.

Chatter desktop app (recommended for most people)

The Chatter app checks CHAT transcripts in an ordinary window: open a file, see the problems highlighted, fix them, and re-check. No terminal and no setup, and it updates itself when a new version comes out.

macOS

The Mac app is signed and notarized by Apple, so it opens normally (no security warnings).

1. Download Chatter for your Mac:

   • Apple Silicon (M1/M2/M3/M4, essentially every Mac sold since late 2020): Download Chatter for Apple Silicon.
   • Intel (older models): Download Chatter for Intel Mac.

   Not sure which you have? Apple menu () then About This Mac: if it says “Apple M…”, it is Apple Silicon.

2. Open the downloaded .dmg file.

3. Drag Chatter onto the Applications folder in the window that appears.

4. Open Chatter from your Applications folder (or Launchpad).

Windows (Intel/AMD 64-bit, “x64”)

Download Chatter for Windows and run the installer. Windows binaries are not code-signed yet, so SmartScreen may warn on first run: choose More info, then Run anyway.

Linux (Intel/AMD 64-bit, “x86_64”)

Download Chatter (AppImage) (make it executable, then run it) or the .deb package (install with your package manager).

(The desktop app is x86_64-only on Windows and Linux today; macOS has both Apple Silicon and Intel builds. The chatter command-line tool below also ships a Linux ARM build.)

chatter, the command-line tool (for programmers and automation)

If you are comfortable in a terminal, the chatter CLI validates, normalizes, converts (JSON), watches, and batch-processes CHAT files, and is the right tool for scripting and CI.

macOS / Linux:

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.sh | sh

Windows (PowerShell):

irm https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.ps1 | iex

Then run chatter --help. Full reference: CLI installation and CLI Reference. chatter self-updates with chatter update.

talkbank-lsp language server (editor integration)

For live CHAT validation, hover, go-to-definition, and cross-tier alignment inside an LSP-aware editor (Neovim, Emacs, Helix, Zed, VS Code, and others), install the standalone, code-signed language server:

macOS / Linux:

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/talkbank-lsp-installer.sh | sh

Windows (PowerShell):

irm https://github.com/TalkBank/chatter/releases/latest/download/talkbank-lsp-installer.ps1 | iex

Or download the per-platform archive (talkbank-lsp-<target>.tar.xz, or .zip on Windows) from the release and point your editor’s LSP client at the talkbank-lsp binary (it speaks LSP over stdio on .cha files, language id chat).

Rust crates and the grammar (embed in your own program)

To embed CHAT parsing / validation / transformation in your own program, depend on the talkbank-* crates and the tree-sitter-talkbank grammar. They are source-available from this repository (not yet published to crates.io). See Library usage and the CHAT format overview.

As a 0.x release, APIs and flags may change before 1.0; see the Release Notes. For audio + ML pipelines (transcribe, force-align, morphotag), see the upstream batchalign3 project, which has its own installation flow.

This page last changed: 2026-07-31 (commit 50d0ca07). The whole book last changed: 2026-09-15 (commit bb4bef82).

Quickstart

Status: Current Last modified: 2026-06-21 21:33 EDT

Task-driven entry points. Pick the row that matches what you want to do today; each path starts at the narrowest useful documentation surface instead of dropping you into the whole book.

To download and install chatter, see Install.

For audio + ML workflows (transcribe / align media → CHAT), see the upstream batchalign3 project, outside the chatter repo.

This page last changed: 2026-06-21 (commit a05df6e4). The whole book last changed: 2026-09-15 (commit bb4bef82).

Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and the project follows Semantic Versioning. Before 1.0, breaking changes to the CLI or library APIs bump the minor version and are listed under “Changed” / “Removed”.

Unreleased

0.25.0 - 2026-09-15

Added

• Source-bound donor selection for structural merging preserves original header boundaries after utterance removal or splitting. Selected donor origins map back to original parents; recorded header brackets constrain order without synthesizing utterance timestamps.
• Selected-donor merges accept source-bound relative order constraints (RelativeOrderConstraint, with_relative_order), an opt-in timed gem exterior policy (with_timed_gem_exterior; the placements it decided are reported as GemExteriorPlacement), an opt-in flagged draft order that serializes unresolved frontiers reference first with a review comment (with_flagged_draft_order, DraftOrderReview), and header-only references.
• merge_chat_files_with_donor_selection_draft returns a MergeDraft before validation. Its only edit, set_terminal_bullet, replaces an end-of-line bullet and is recorded as a BulletEdit; Merged::bullet_edits reports the edits after validate.
• talkbank_model::validation exposes SPEAKER_OVERLAP_TOLERANCE_MS and has_transcribed_content.

Changed

• MergeError::InvalidDonorSelection reports inconsistent selection coordinates. Downstream exhaustive matches must handle this new variant.
• A repeated @Languages header is reported as E501.
• Source-order merges serialize utterances with exactly equal starts reference first; section markers at the same instant are refused as ambiguous.
• Merges refuse a missing, repeated or empty @Languages declaration in either input (MergeError::InvalidLanguageDeclaration), and selected-donor merges refuse a selected child whose bullet lies outside its parent’s.
• MergeError adds InvalidLanguageDeclaration, InvalidRelativeOrder, RelativeOrderTimingConflict and InvalidGemExterior. Downstream exhaustive matches must handle them; the CLI reports each as a precondition (exit 2).
• Releases publish only after the desktop installers are built and verified; the draft carries the CHANGELOG notes, and the app banner is added after publication.
• Release artifacts are built with cargo-dist 0.33.0. Its shell installer keeps the env PATH helper beside the install receipt (by default ~/.config/chatter) for flat installs, and moves an existing helper there.

Security

• rustls is updated to 0.23.45 (RUSTSEC-2026-0285), which rejects TLS 1.3 handshake messages accepted across encryption level boundaries. It reaches the CLI’s self-update and LLM client and the desktop app’s updater.

0.24.2 - 2026-09-12

Added

• An explicit source-order merge policy derives a unique interleaving from source ordering and available timing evidence without synthesizing timestamps. Ambiguous ordering is refused; the existing timed merge policy remains the default.

Changed

• MergeError includes AmbiguousUtteranceOrder; downstream exhaustive matches must handle the new variant.
• CLI release artifacts are published through an explicit release workflow dispatch.

0.24.1 - 2026-09-11

Fixed

• A @Media declaration with the missing medium no longer triggers E544 for absent timing. An explicitly absent recording does not promise linked media; expected recordings still require timing or an appropriate status.
• Added a specification example and generated regression fixture for the missing-medium declaration.

0.24.0 - 2026-09-10

Changed

• Structural merge now consumes ordered AST streams rather than collecting headers and sorting utterances. Interleaved body headers and dependent-tier order survive; selected utterances without timing or with reversed source starts are refused. Section markers use neighboring source timing bounds, and ambiguous cross-source placement is refused instead of guessed.
• MergeError has new variants for timing, section-placement, metadata-order, participant-join and output-validation failures. Exhaustive library matches must handle them. Merged and Reported retain a validated document; Reported::into_file relinquishes that proof for subsequent edits.

Fixed

• Documentation-date checks now include pending commit/squash changes before publication. The commit hook checks the actual index, preventing an unstaged repair or an older gate receipt from masking stale staged date headers.
• Merge builds the derived participant map through the canonical header join and validates the assembled AST, including tier alignment, before returning success. Callers no longer need serialization and reparsing to obtain a consistent participant map.
• Donor IDs extend the contiguous opening ID block; donor body comments remain at their source position. An opening ID after a comment is refused rather than silently reordered.

Added

• WorSlotMembershipPolicy::admits(&Word) -> bool, the public per-word %wor admission predicate. WorMainTierProjection::from_main admits its slots through it, so it is the projection’s own rule rather than a second statement of it. A downstream consumer that counts %wor-eligible words per content item (both Batchalign trees carried a hand copy of counts_for_tier(word, TierDomain::Wor) beside a walk_words for this) asks the policy and deletes the copy.

0.23.0 - 2026-09-09

Removed

• walk_overlap_points and OverlapPointVisit from talkbank_model::alignment::helpers: a visitor over overlap markers with no caller in any tree, carrying a third private walk of its own (and a position convention for intra-word closing markers that differed from the collector’s). extract_overlap_info is the one API.
• talkbank_transform’s corpus discovery and manifest API (discover_corpora, build_manifest, corpus_summary, format_manifest, CorpusManifest, CorpusEntry, FileEntry, CorpusFileStatus, FailureReason, ErrorDetail, ErrorLocation, ManifestError). No command in this repository and no known dependant used it; its only caller was its own test. TierContent’s to_content_string_no_bullets and write_tier_content_no_bullets, ValidationContext::with_quotation_validation and with_bullets_mode with the bullets_mode field they set (the bullets @Options was removed from CHAT and the flag was always false, so E362’s check on bullet monotonicity now simply runs), and the parser API’s always-false bullets_mode(); none had a caller anywhere. Breaking for a library user who called any of them.
• The Utterance builder helpers with no caller: with_preceding_headers, with_user_defined and every per-tier with_* except with_mor, with_gra, with_sin and with_com (add_dependent_tier is the one route they were sugar over and remains); the semantic-diff renderers short_summary, short_summary_with_source, render_with_source, render_comparison, render_comparison_short and render_tree_diff on SemanticDiffReport with the RenderMode they took and the tree renderer behind them, and the Utterance accessors mor, gra (the cloning aliases; mor_tier and gra_tier stay, as do pho and sin), mor_tier_mut, computed_language_metadata, wor_alignable_word_count, pho_alignable_word_count and sin_alignable_word_count. None had a caller in this repository’s root workspace, in talkbank-tools or in the downstream Batchalign, and a whole-workspace coverage run showed every one unreached; the two cloning aliases had one caller in the spec runtime tools (a separate workspace), moved to the borrowing accessors. Breaking for a library user who called any of them; SemanticDiffReport::render (also its Display) remains, and the alignable count that is used, mor_alignable_word_count, remains.
• TreeSitterParser::parse_utterance_cst. It forwarded to the free parse_utterance_node and had no caller in this repository or in any repository known to depend on it; a whole-workspace coverage run showed it unreached. Breaking for a library user who called it; the public routes are TreeSitterParser::parse_utterance (one utterance from its text) and TreeSitterParser::parse_utterance_fragment.
• talkbank_parser::parse_dependent_tier (the free function) and talkbank_parser::tiers::parse_mod_tier_from_unparsed. Neither had a caller in this repository or in any repository known to depend on it. The first returned an untyped UserDefinedTier for every tier, where the talkbank_model::ChatParser::parse_dependent_tier route returns the typed DependentTier; a whole-workspace coverage run showed both entirely unreached, %mod parsing having gone through the typed %pho tier parser for as long as the typed traversal has existed. Breaking for a library user who called either; use the talkbank_model::ChatParser trait’s methods.

Added

• talkbank_model::content::word::Word::new(NonEmptyString, WordText): the checked constructor, taking the two proofs a word’s texts must carry. The tree-sitter parser builds through it; new_unchecked remains for test support and the front ends not yet migrated.

Changed

• talkbank_parser::generated_traversal::NodeSlot is NodeSlot<'tree, T, M, U, A>: the payloads of Missing, Unexpected and Absent are the node (or NoChild) where the position can produce the state and the uninhabited Never where it cannot, and every generated accessor names its position’s kind through one of four aliases (ChildSlot, SeqSlot, ChoiceSlot, ClassifiedSlot). Absent carries NoChild; Recovery and SlotValue carry the same parameters; NodeSlot::view gives a borrowed slot’s states back by value as a SlotView, so a match through a reference can omit the arms the position kind rules out. Breaking for a library user matching the slot directly: an arm for a state the position cannot produce no longer compiles, which is the point. The parser’s own hand-written arms for those states, each carrying a diagnostic for a case that cannot happen, are gone with it.

• A content-bearing recovery node inside a %mor tier is E702 at any depth. It was E702 for a direct child of the tier and E316 for a node below one, because the utterance parser walked every dependent tier’s children before attaching it and reported them without the tier’s name, and the typed dispatch then reported the direct children again: a %wor line with an unparsable word carried the same E316 twice at the same span. One reporter now walks the whole tier in the tier’s words. Three E316 examples and E711’s first are E702’s (subsumed_by), E702’s first example is a violates claim, and the E342 text for a MISSING node inside a tier names the tier (“in gra tier”).

• Counting and extraction take PositionalDomain (Mor, Pho, Sin), a new type with no Wor: count_tier_positions, count_tier_positions_until, collect_tier_items, TierCountable, AlignableTier::DOMAIN, and talkbank_transform::extract::{extract_words, collect_utterance_content}. TierDomain keeps Wor and stays the vocabulary of the walkers and of counts_for_tier; PositionalDomain converts into it, and TryFrom<TierDomain> refuses Wor with NotPositional. The %wor count and pairing are WorMainTierProjection’s, and the two Wor arms in the counter and the one in the extractor were a second implementation of that count, agreeing with the projection only by test; a probe over every reference-corpus file and every spec example found them equal before they were deleted. The overlap-marker position walk in alignment::helpers::overlap still counts on the %wor scale with its own traversal and is not changed here. WorMainTierProjection::slots is crate-private (pair through bind_timing). A downstream caller passing TierDomain::Mor to any of these writes PositionalDomain::Mor; one asking a %wor count calls MainTier::wor_projection().slot_count(). This is a breaking Rust API change.

• Two validity rulings (maintainer, 2026-09-08), both grounded on real CLAN CHECK. A bullet INSIDE a main-tier utterance is timing evidence for the media-consistency family: hello \u{15}100_200\u{15} world . is E752 without an @Media header (CLAN CHECK 112 fires on it) and satisfies a declared @Media (it was E544 before, and valid without the header). And whitespace-only content on a bullet-payload tier (%com, %add, %exp, %gpx, %int, %sit, %spa, %act, %cod) declares nothing: E756 beside E758, as %eng already was (CLAN CHECK 31 rejects the same lines). Transcripts that relied on either gap now validate differently.

• Diagnostics inside an angle group, a quotation, a pho group or a sin group are now the ones the tier body gives for the same material. The four constructs parsed their contents through a second, hand-written walker with its own generic ERROR analysis; the one typed contents walker serves both now. Visible changes: a curly single quote inside a quotation is E256 (it was E331); a stray [ inside a construct is E316 at that byte (it was three E330 “expected X” messages); an ERROR fragment that opens a bracket or parenthesis and never closes it is E312 or E313 on the tier body as well as inside a construct (hel(lo . was E316; a parenthesis that opens the whole utterance still fails at file level), and “never closes” means no closer anywhere in the fragment, not merely not at its end. E331 (UnexpectedNodeInContext) has no known route from CHAT input any more and is recorded as unreachable.

• WordLengthening::count, its with_count argument, and re2c’s AST lengthening count now use NonZeroUsize instead of u8. JSON keeps the integer field and its omitted-one default, accepts longer runs, and rejects zero. This is a breaking Rust API and JSON-admission change.

Fixed

• The phonological tiers accept superscript one, two and three (U+00B9, U+00B2, U+00B3), the only superscript forms those digits have, wherever the other superscript digits were already accepted, and the last three modifier tone letters of their block (U+A71D to U+A71F, the raised and low exclamation-mark letters, ꜞ among them) as the rest of the block already was. A %pho or %mod word carrying one was E316 unparsable content (TalkBank/chatter#6, #7): 52 of 52 sessions of one tone-language corpus, 18 of 96 in a second, 3 of 58 in a third, and 2 in a fourth.

• E715 and E734 no longer report a %pho or %mod tier one token long when the main tier carries a pause inside a <...> group: a pause was counted as a phonological token at the top level of the utterance but not inside a group, while the phonological tiers carry it in both places, as the Phon team’s French corpora show at scale (72 records in 49 sessions of one corpus, every one a pause inside an overlap or retrace group; TalkBank/chatter#5). The counting walker’s own 2026-08-08 note had held that arm open for exactly this evidence.

• E740 and E741 no longer report a %mod or %pho word that carries the linking tie ‿ (U+203F) as a mismatch against its %xphoaln reconstruction: the tie joins two symbols into one segment or marks the absence of a break, is never a phone, and has no alignment column, so the source word is compared modulo the tie as it already was modulo the stress and syllable-boundary marks (a pair side carrying a tie is not a bare segment and is compared as written). TalkBank/chatter#3, with the inventory in #4: until now every such word in two PhonBank corpora was a spurious mismatch.

• A group with no code after it (<w> .) is E342 alone, and the model keeps the bare group that was written. The grammar requires a code there, tree-sitter inserts a MISSING placeholder, and the annotation decoder used to read the placeholder’s kind and build a full retrace nobody wrote: the validator then reported E757 and E370 against constructs the file does not contain, and the E342 spec example’s roundtrip diverged. The decoder skips MISSING nodes now. CHECK-parity for CHECK 51 expects E342; the E342 example leaves the backend-parity baseline, both parsers agreeing.

• E710 is reported only by the %gra relation parser. The dependent-tier recovery analyzer had a branch that fired on the substring %gra: anywhere in an ERROR node’s text and called it E710, “non-numeric index”: an %eng or %x body mentioning %gra:, or junk after a well-formed %gra relation, was reported as an invalid relation. The branch is gone; such a node is the generic E316 (E258 for a double comma, E760 for a %mor item with an empty part of speech, as before). The E760 branch’s own gate accepted %mor: anywhere in the text for the same reason and now needs the line, the tier context, or a text that starts with the prefix.

• A MISSING node is reported once. The whole-tree recovery backstop suppresses a candidate already covered by a region diagnostic by span overlap, and widened only its own zero-width MISSING span to a byte, so a region’s E342 for the same point (itself zero-width) never covered it: every MISSING node inside a dependent tier or a header list carried two E342 texts at one span. A zero-width region diagnostic at the same point now covers the candidate when it reports the same code; a different code there (E376 for an empty replacement beside its MISSING word segment) still leaves the E342 reported, as E208.md documents.

• Overlap-marker positions count a replaced word inside a group once. The collector behind extract_overlap_info (and so top_onset_fraction, estimate_onset_ms, and the cross-utterance E347 and E704 checks, which read the paired positions; E373 reads only the indices and was not affected) walked with two private traversals, and the bracketed one scanned a replaced word’s replacement words too, so <doggie [: dog]> under a marker counted two words where the %wor projection counts one and every later marker position, and the onset fraction, drifted. The collector now walks with the shared walk_content at the %wor domain, the projection’s own leaf set, so total_words is the projection’s slot count by construction; a snapshot of every marker-bearing utterance in the reference corpus and the spec examples was byte-identical across the change, and the group case is pinned.

• chatter debug sanitize redacts %act and %cod tiers. Both carry the same bullet payload as %com, and passed through the strict sanitizer with their text intact under a comment deferring their redaction; an action line is free text about the participant. A parse-backed table of every dependent-tier kind through the sanitizer is the pin.

• chatter debug sanitize redacts the words on %wor. The tier repeats every main-tier word beside its bullet and passed through untouched, so a sanitized file with a %wor tier still carried the whole utterance in clear. The tier is now judged the way timing recovery judges it, before the main tier is rewritten: WorMainTierProjection::bind_timing for the counts, then corroborate_wor_timing for the words. A tier that corroborated the main tier has each word rewritten as its paired main-tier word’s display text, now that word’s placeholder (w1w1 for a compound), so it corroborates the sanitized main tier exactly as before; a tier that drifted in count, or carried a word the main tier did not, takes fresh placeholders rather than a manufactured agreement, and disagrees after as it did before. Bullets are kept byte-exact, and sanitizing the output again reproduces it. The test that claimed to pin %wor offsets wrote them as bare 1000_1100 tokens, which the model parses as words; it passed only because nothing touched the tier. It now pins whole lines, drift and compounds included. CountMatchedWorTimings::pairs exposes the owner’s pairing.

• chatter validate --roundtrip counts a file’s roundtrip on every run. When the file’s validity and roundtrip verdicts were both served from the cache, the summary said Passed: 0 for a file whose roundtrip had passed, and Failed: 0 for one whose roundtrip had failed: the cached branch built the file’s status and never touched either roundtrip counter. The status now records whether the roundtrip ran (FileStatus::Valid { roundtrip: RoundtripVerdict }, a new public field and type) and both counters are derived from it.

• E370 (a retrace marker with nothing after it to retrace) is labelled at the marker’s own bytes whatever the spacing around it. The rule used to find the marker by rendering the main tier back to CHAT text and taking the offset there, which was right only when the source was already canonical: on <hello there> [/] . the label sat one byte early. The parser now records the marker token’s own span on the retrace (Retrace::marker_span, a new public field, None for a retrace built without a source) and the rule reports there.

• Overlap markers inside an angle group, a quotation, a pho group or a sin group are kept in the model and written back. The constructs’ old contents walker handed each item to a second walk over the item’s children, and an overlap_point is a single token with none, so <hello \u{2308} there \u{2309}> [/] hello there . parsed clean, validated clean and wrote back with both markers gone.

• %gra structural diagnostics (E721, E722, E723, E724) no longer fire on a tier the parser had to shorten. A relation the model cannot hold is rejected and dropped, and the rules for sequential indices, root count and cycles describe the graph the author wrote, not what survived; E722 in particular reported “no ROOT relation” against a tier whose only surviving relation was a ROOT. A new JudgeableGra witness is the sole route to those rules and asks both questions that decide it, parse recovery and prior alignment findings.

• The re2c backend records parse recovery on a dependent tier it could not build as written: a %gra that lost a relation, a %mor whose conversion failed, a %wor body it could not re-lex. Cross-tier alignment previously compared such a tier against its neighbours and reported the difference its own recovery had created, so %mor and %gra counts disagreed (E720) on a transcript where they agree.

• Both parsers preserve lengthening runs beyond 255 colons without integer overflow or truncation. The default model marker now consistently contains one colon, with no zero-count repair during serialization.

• Re2c retains malformed form suffixes for specific E202/E203 diagnostics; repeated dangling markers no longer produce both errors for one defect.

• Release lint checks application-version synchronization before compilation.

0.22.0 - 2026-09-06

Changed

• talkbank_lsp::backend::utils::LineIndex borrows its source. Its offset_to_position method accepts only the offset, preventing callers from pairing indexed line starts with another text. This is a breaking Rust API change.

Fixed

• LSP edits, formatting, semantic tokens, selection ranges, symbols and quick fixes consistently use UTF-16 coordinates. Multiline semantic captures split into individual lines, and whole-document formatting includes the final newline.
• Gem outlines use parsed header spans and matching labels, preserving CRLF positions and counting only actual utterances in the parent outline.
• Language-service initialization retains its result in OnceCell; nested highlighter access returns an error instead of panicking. Execute-command services accept only their own request enums, removing routing panic branches.

0.21.0 - 2026-09-06

Changed

• LSP backend cache fields are replaced by a private source-bound analysis. The unused public incremental-splice and validation-cache modules are removed. Syntax reuse remains incremental; models and validation results are rebuilt together using the shared model validator.

• DocumentRoot is a private-field classification with method accessors rather than a publicly constructible enum. It owns both document lowering and whole-source diagnostic scope.

• re2c parsed header lines carry HeaderProvenance in place of a standalone separator field, and box their header payload. The owned lexer extent now reaches model header spans; boxing keeps the file-line enum compact.

• re2c pause tokens and parsed pause variants retain a PauseLexeme instead of discarding the full lexical extent. This changes their Rust payload types.

Fixed

• A truncated document without a final newline retains its complete simple final main tier. Shared terminal recovery reuses the normal fragment parser and preserves caller coordinates while validation reports missing @End.

• LSP diagnostics after edits now agree with fresh-open text, including deleted headers, recovery suffixes, Unicode edits and skipped debounce revisions. Tree-sitter edits use the cached tree’s own source and UTF-8 byte coordinates. Feature and pull-diagnostic requests cannot reuse another revision’s spans. Published diagnostics carry editor versions; obsolete analyses are discarded.

• LSP validation includes shared file-level rules such as E752 instead of a separate incomplete validation sequence. Protocol regression tests share the existing executable harness, preserving interleaved responses/notifications.

• Recovery before or after a complete document receives localized diagnostics without discarding the document. A complete final main tier stranded outside its line wrapper when @End is missing is retained through normal utterance construction.

• Documents missing @UTF8 retain their headers and utterances and report E503, without cascades claiming present headers are absent. Both parsers locate the diagnostic at the end of the file, including rebased fragments.

• Both parser backends admit complete fragment coordinate ranges before parsing. Origins above 2 GiB retain correct model and diagnostic spans; overflowing 32-bit ranges are rejected instead of truncated. Synthetic wrapper text no longer consumes the caller’s document range.

• re2c header fragments reject extra headers, utterances and unsupported trailing lines instead of returning a partial result. Lowering consumes an admitted logical header; folded content and recovery diagnostics survive.

• re2c preserves pause spans, including timed-pause parentheses, through nested content and fragment rebasing. Shared validation now owns pause spacing; duplicate token scans are removed.

• Separator spacing validation visits nested groups, reporting E765 at the missing space just as it does for top-level content. Spaced group controls remain valid.

0.20.2 - 2026-09-06

Fixed

• Single-header parsing rejects extra headers instead of silently returning only the first. Lowering consumes a complete HeaderFragment that owns the selected node and its source; folded content and LF/CRLF remain accepted.
• Standalone header and dependent-tier parsing derive diagnostic coordinates from their owned synthetic source rather than separately supplied prefix lengths. Header lookup failures carry the caller’s text and document origin instead of empty context. Existing malformed-header diagnostics are retained.

0.20.1 - 2026-09-06

Fixed

• The standalone LSP exits after the editor’s exit notification even when the editor keeps stdin open. A completed shutdown permits exit code 0; exit before a successful shutdown returns code 1. Protocol completion now stops the transport, and runtime teardown does not wait on Tokio’s uncancellable stdin reader. Process-level regression tests cover the actual binary, rejected shutdown, early exit, and EOF.

0.20.0 - 2026-09-06

Changed

• Breaking: removed CachePool::open_or_else; use CachePool::new and handle its Result directly. Cache opening no longer splits failures between an optional handle and a callback; the CLI retains the concrete opening error.

• Validation caches include both parser implementation source fingerprints, closing stale verdict reuse after parser-only edits without a version bump. Shared build-only source hashing reads each crate’s own packaged files.

• Breaking: re2c Token::TierPrefix carries DependentPrefixToken, including the lexer-selected DependentBodyKind; dependent parsing matches that enum.

• Breaking: CacheStats::cache_dir is optional: in-memory storage has no filesystem directory. File-backed statistics retain the opening directory.

• Breaking: validation-cache constructors require CacheIdentity (rules and parser), and roundtrip cache methods use that bound identity instead of a parser string. ParserKind is shared from talkbank-model and re-exported. MaintenanceCache exposes administrative operations without verdict methods.

• Breaking: re2c prefix tokens carry PrefixToken payload/separator state. AST header lines, main tiers and DependentTierEntryParsed retain separator provenance. Access a prefix payload with text(); file AST snapshots reflect the new header and dependent-entry shapes.

• Breaking: re2c dependent-tier AST adds RejectedMor, retaining raw input after failed morphology admission without fabricating a model tier.

• Breaking: re2c Token::TierPrefix denotes a complete colon-tab prefix; the new IncompleteTierPrefix variant identifies recovery from a bare label.

• Breaking: re2c postcode tokens and main-tier AST postcodes carry checked payload state and lexer locations. main_tier_to_model and utterance_to_model now require an error sink; callers can no longer lower these structures without deciding where recovery diagnostics go.

• Breaking: re2c AST ParsedAnnotation separates Scoped annotations from retrace, replacement, language-code and postcode structures. Match ParsedAnnotation::Scoped(ScopedAnnotationParsed::...) for scoped kinds; their conversion to model annotations is now total. AST inspection snapshots reflect this category; serialized CHAT model output retains its shape.

Fixed

• Release bumping updates and checks both desktop npm lockfile version fields. Dependency versions remain unchanged; CLI regression checks run in the fast app-version gate.

• Owned fragment wrappers project secondary labels with primary locations and identify synthetic context by exact source text instead of a length heuristic. Independent diagnostic context is preserved even when longer than the input.

• Main-tier fragments reject trailing material instead of silently accepting their first tier. Aggregate spans exclude a synthetic final newline, while retaining caller-supplied LF and CRLF line endings. Root-admission diagnostics now describe the required source shape without raw CST-kind wording.

• Fragment APIs no longer subtract obsolete word/main-tier wrapper lengths or subtract caller offsets from diagnostics. Synthetic utterance, participant and dependent-tier wrappers own their input boundary for model and error projection. Complete CHAT documents are recognized by the utterance adapter.

• re2c reports unsupported lines as E326 with their original source spans and preserves following utterances. Diagnostic rebasing keeps context highlights relative to their own source text.

• Speaker-qualified @Birth of, @Birthplace of, and @L1 of headers retain their separator spans, including non-CA whitespace violations and CA exemptions.

• Tree-sitter’s whole-file fragment API now rebases model spans along with streamed diagnostics when parsing embedded CHAT at a nonzero offset.

• Tree-sitter recovery no longer reports E758 for spaces after rejected dependent-tier or header content. Separator provenance requires adjacency to the actual tab, preserving the original content diagnostics.

• Gate receipts verify the actual committed trees in every pushed ref, including annotated tags. Uncommitted fixes cannot authorize an older commit, and a gate whose source changes during verification cannot issue a receipt.

• re2c E760 highlights the original empty-POS morphology item, including across continuation lines and non-ASCII text. Recovery inspects source-owned items without rebuilding rich-token payloads or using a dummy diagnostic span.

• re2c no longer dispatches longer Phon labels through %mod or %pho body parsers. Bare and x-prefixed syllabification, alignment and interval tiers retain their own grammar without false E316 diagnostics.

• Cache statistics report the directory actually opened instead of resolving the current default again, including for explicitly located maintenance pools.

• Validation cache rows are isolated by parser in both CLI and desktop. Switching parser/rule combinations no longer risks serving another parser’s verdict or consumes extra retained generations. Maintenance opens do not prune rule generations or expire rows merely to display statistics.

• re2c records trailing separator spaces across headers and tiers. The shared validator reports E758 outside CA, and serialization canonicalizes separators in either mode. This removes the separate main-tier scan and CA probe.

• re2c enforces morphological lemma starts and nonempty features at lexing. Rejected %mor reports E316/E600 and preserves morphology taint instead of converting to an unsupported tier with E605.

• re2c rejects a replacement glued to its word with E375/E316, using the original bracket locations. A spaced replacement remains valid. The canonical parser’s malformed closing-bracket highlight excludes absorbed trailing whitespace and uses the original source for its diagnostic context.

• re2c reports E602 for malformed dependent-tier separators even when content follows the label, including a space in place of the required tab. Recovery uses the lexer-classified prefix and locates the complete malformed line; empty content no longer participates in deciding whether its prefix is valid.

• re2c rejects whitespace-only postcodes with E363 while preserving the rest of the tier. Valid postcodes preserve leading payload whitespace and trim only trailing whitespace, matching the canonical parser. Postcode diagnostics retain the complete token span through file, utterance and main-tier APIs.

• re2c utterance fragments forward parse diagnostics, and file diagnostics honor the caller’s offset. A shared streaming adapter replaces temporary diagnostic collectors in header and participant fragments.

• re2c reports E757 when rich bracketed annotations are glued to the following word, including [!]there and [= toy]there. The check uses the parser’s annotation categories and reports the following word’s original lexer span.

0.19.0 - 2026-09-05

Changed

• Breaking: the LLM response cache has one owning handle per path across processes. Share the handle across threads and drop it before reopening. CacheError distinguishes a busy cache and a visible replacement whose final directory sync failed.

• Breaking: removed SinToken::new_unchecked; use checked SinToken::new. SinTier::from_tokens now returns Result<SinTier, EmptyText>. re2c’s SinTierParsed and SinItemParsed own checked SinToken values and no longer take a source lifetime parameter.

• Breaking: re2c AST word raw_text is now Cow<str>, distinguishing borrowed source from owned reconstructed text. Parser combinators separate source and token-storage lifetimes.

• Breaking: re2c’s sin_tier_from_text returns ParseOutcome<SinTier>; malformed fragments can no longer appear as successfully parsed empty tiers.

• Breaking: diagnostic enrichment takes a source-bound index. Replace enhance_errors_with_line_map(errors, source, map) with enhance_errors_with_index(errors, &SourceIndex::new(source)), or retain one SourceIndex for repeated batches. Its immutable borrow prevents source edits while the index is used; callers cannot pair another file’s line boundaries with the source and trigger a UTF-8 slicing panic. The existing enhance_errors_with_source convenience API retains its signature.

Fixed

• LLM response-cache writes now prepare, flush, and atomically publish snapshots before updating memory. Failed writes preserve old entries; concurrent puts cannot publish stale snapshots. Unix builds also confirm directory durability.

• Both parsers now share source-bound control-character checking before parsing. re2c no longer silently accepts forbidden controls in free-text tiers; the lexical diagnostic retains its original source and exact byte range.

• E212 spec coverage now demonstrates its reachable CA-mode word-category boundary, alongside legal controls. Its existing implementation is marked implemented, replacing the misleading legal-only deferred fixture.

• Utterance validation now reaches bare and grouped %sin tokens admitted through JSON, reporting empty text at the tier’s span. Both parsers construct tokens through the checked constructor; the duplicate unchecked path is gone.

• re2c parsing no longer leaks copied source, token arrays, recovery buffers, or reconstructed words. The lexer safely handles unpadded input at EOF, and word-fragment conversion retains spans from the caller’s original source.

• re2c %sin fragment parsing now uses the whole-file grammar, preserving single-token gesture groups and rejecting unclosed groups with a diagnostic. The duplicate whitespace parser and its independent group state are removed.

• Diagnostic line/column lookup no longer caches by source address and length, which returned stale positions after same-length edits or allocation reuse. One-off lookup scans without allocation; indexed batches retain logarithmic lookups without retaining a hidden source copy or thread-local cache.

• Foundation publication checks now cover every workspace crate outside the approved first wave. talkbank-llm is explicitly held back; a metadata-only mode checks manifests and dependencies without packaging or registry access.

• Generated fixture and documentation directories now retain unchanged files and prune only obsolete output through an ownership capability. Conflicting ownership, nested human content and symlinks are refused before pruning.

• Spec regeneration preserves unchanged outputs in shared directories, including generated model code and Rust test bodies, while still removing explicitly retired files. Progress counts now report actual writes.

• Tree-sitter generation now stages all grammar artifacts and preserves unchanged files. The grammar currency check no longer rewrites source files, and also checks generated C headers.

• Node-type, traversal and conformance-inventory regeneration now preserves unchanged files and publishes changed output only after the generator succeeds. A failing generator no longer truncates those committed Rust files.

• Removed an unnecessary schema rewrite that treated valid Draft 2020-12 $ref siblings as invalid and could modify literal schema data. Generated schemas now retain schemars’ structure; enum tags and referenced payloads remain jointly validated.

• Schema generation now preserves unchanged files and runs only when explicitly requested by just schema-gen or just regen. Ordinary tests no longer rewrite a compile-time dependency and trigger avoidable recompilation. Schema currency failures report the repair command without dumping the complete schema.

• to-json --skip-schema-validation retains the transcript name and requested CHAT checks, including E531 for mismatched media filenames. Single-file and directory conversion now share one named parsing path before schema policy selects serialization.

• Directory JSON conversion prints individual parse/validation diagnostics and exits with failure when any file fails; successfully converted sibling files remain available.

Added

• JsonSchemaPolicy and chat_to_json_with_schema_policy let library callers select JSON Schema validation independently of CHAT validation and transcript identity. Existing conversion functions retain their signatures.

0.18.1 - 2026-09-05

Added

• Timing-producing transforms now have a typed media-link transition. reconcile_media_timing consumes a ChatFile and returns either an UntimedChatFile or a LinkedMediaChatFile. A timed document must have one usable @Media declaration; the transition removes unlinked, accepts an already-linked declaration, and returns typed errors for missing, ambiguous, or contradictory media. Both states expose only an immutable document and post-transition serialization. This prevents a forced-alignment pipeline from writing fresh timing bullets while retaining the contradictory @Media: ..., unlinked status rejected by E552. Validation verdicts are unchanged.

0.18.0 - 2026-09-04

Changed

• Warm development tests no longer scan unpacked macOS codegen objects. The root and specification workspaces embed line-table debug information in linked artifacts instead of retaining every .rcgu.o. This bounds the file count in both target directories and removes filesystem enumeration from the warm-test path while preserving source locations in diagnostics. Nine spec generator commands are also excluded as empty libtest harnesses; their library and integration tests remain in the suite.

• Breaking: validation returns owned evidence rather than a mutable phase marker. ChatFile is no longer generic. validate_into returns Result<ValidChatFile, ValidationFailure>; accepted payloads are read-only, errors retain the rejected model, and unknown/recovered tier provenance cannot pass. validate_with_policy records rules, alignment coverage and transcript name. parse_validated_with_parser additionally requires error-free source parsing. Remove the old NotValidated/Validated/ValidationState imports and consume into_unchecked() before editing an accepted document. Serialized transcript fields remain unchanged.

  Required-validation compatibility APIs now use the same proof-producing transition before returning an explicitly editable model. Their streaming variants return Err after parse or validation failure even when the caller’s diagnostic sink discards messages. Merge preflight retains ValidChatFile while reading the accepted reference and borrows its document for the merge.

• Breaking: utterance builders can retain an utterance comment. UtteranceDesc adds comment: Option<ComTier>; Rust struct literals must supply this field. A supplied comment is emitted as %com. A comment on an empty utterance is rejected instead of being silently discarded.

• %pho, %mod and %sin count mismatches are reported by one algorithm. The utterance metadata path used its own copy of the positional alignment with the diagnostic codes passed in as parameters; it now uses the AlignableTier route, which reads the tier’s own type (%pho or %mod) to choose E714/E715 or E733/E734. The codes are unchanged; the messages are the positional form the %sin route already used (a per-position table instead of a bare count).

• A control character is a lexical error anywhere in the file. E315 is now decided over the whole input before any parse, so a forbidden control character in a word, a %com line or a header value is reported at its own offset; previously a word’s surfaced as generic E316 and free-text tiers accepted it silently. Permitted are TAB, LF, CR, the bullet delimiter U+0015 and the CA underline pairs U+0002 U+0001 / U+0002 U+0002; CLAN’s italics pairs are reported (CHECK 102).

• E303 covers every header whose colon is not followed by a TAB. It used to fire only for @Comment:, every other header fell to E316, and the message said “space” whatever followed the colon; it now names what was found (a space, nothing, or the character).

• Word lexical content is read-only outside its owning type. Direct access to the former public content field is replaced by content(); callers that intentionally replace typed content use the named mutation APIs, which invalidate derived cleaned_text. This prevents a content edit from leaving stale lexical text in JSON. Direct crate-internal access to raw_text is closed as well, so recovery spelling changes use the explicit setter rather than bypassing the field boundary.

• Speaker-code structure now has one typed assessment across every model surface. Direct SpeakerCode::validate previously mislabeled an overlong code as undeclared (E308), mislabeled a reserved character as a missing CST node (E302), and enforced a different character policy from headers and main tiers. All three routes now consume the same producer-issued valid/invalid state and report E307. The seven-character limit counts Unicode scalar values rather than UTF-8 bytes, and diagnostic context records the offending code rather than an internal field label.

• Malformed regions now distinguish unpaired CHAT quotation delimiters from unrelated parser recovery. Structurally unpaired “ or ” delimiters report E242 even when tree-sitter encloses them in a larger error node. Balanced quotation delimiters inside some other malformed region no longer produce a false E242, and ASCII straight quotes are not mislabeled as CHAT quotation delimiters.

0.17.0 - 2026-08-30

Changed

• Reference-mode speaker identification now preserves typed lexical support. DonorMatchReport retains the reference, donor, shared, and union token counts that derive each Jaccard score; its winner, evidence, and confidence margin are no longer independently constructible public fields. Thresholds are checked ConfidenceThreshold values, while confidence is explicitly NoInformation, Finite, or Unbounded instead of overloading 0.0 and infinity. Use the read-only report accessors in place of direct field access. chatter speaker-id --write-match-report NEW.json writes the accepted, low-confidence, structural-refusal, or input-refusal evidence without replacing an existing report.

• %wor timing is now admitted through explicit typed evidence states. MainTier::wor_projection() defines the shared positional membership policy; count binding, canonical-token corroboration, and complete positive interval assessment are separate states, so equal word counts alone cannot be treated as trustworthy timing. The impossible tier-level WorTier::bullet field is removed: timing evidence exists only when an actual %wor word carries a bullet. Callers of the former alignment and tier-bullet APIs must migrate to the projection, binding, corroboration, sequence-assessment, and WorTier::timing_evidence() APIs.

• rediarize and rediarize_content require a DiarizationTimeline. The windowed overlap algorithm needs turns ordered by start time, but the former &[DiarizationTurn] API let every library caller bypass that precondition and silently obtain a wrong winner. DiarizationTimeline::new owns the sorting transition, retains the longest-turn window bound, and keeps its ordered storage private. TurnsFile now exposes source() and timeline() accessors instead of independently public fields.

• Tree-sitter 0.27.0 now drives the Rust parser, highlighting runtime, grammar crate, spec tooling, and grammar-generation CLI. The Node binding is independently current at 0.25.1. Generated parser artifacts and the complete parser/backend parity gates are regenerated and checked with this toolchain.

• The vendored Rust lexer is regenerated with re2c 4.6. re2c-version.toml is now the exact generator source of truth, and just verify-vendored-lexer refuses a different re2rust before comparing generated bytes. The 4.6 output differs from 4.5.1 only in its generator provenance header; upstream’s 4.6 implementation change is Zig-only.

• Merged::report is the only route from a merge to a file. into_file and file are gone from Merged; report(sink) yields a Reported, which owns them. Serializing a merge without asking what it dropped is no longer writable, which is what two commands did until each was fixed by hand.

• chatter merge and chatter pipeline now warn when a File 1 speaker is dropped. A speaker outside --retain loses every utterance while keeping its @Participants row, so the output declares someone who says nothing. AmbiguousSpeaker does not catch it: that fires only when a code appears in both files. Both commands print the same warning, naming the speakers and how many utterances each lost, from one shared reporter. chatter batch drives the pipeline path, so the silent one was the path that runs whole corpora.

• merge_chat_files returns the provenance of every merged utterance, not only the merged file. It always knew this and threw it away: it walks each input in order building two lists, then stable-sorts the combination by start_ms. A consumer joining the output back to its inputs had to reconstruct the mapping by matching (speaker, raw bullet), which is correct only while two facts hold that no caller can check, that the sort is stable and that inserted utterances are cloned unedited.

  The return type is now Merged, carrying the file, one MergeOrigin per output utterance in output order, one ReferenceFate per File 1 utterance and one DonorFate per File 2 utterance, each in its own input’s order. Ordinals are ReferenceIdx / DonorIdx, separate types because two same-signature accessors over one index type answered confidently about the wrong file. merge_chats, the string wrapper, is removed: it had no non-test caller, and a String return cannot carry the provenance, so every consumer that wants the report has to work on parsed files anyway. MergeError::Parse goes with it, since merge_chat_files takes files that are already parsed.

  Prefer Merged::utterances_with_origin() to pairing the accessors by hand: zipping origins() against file().lines type-checks and is wrong by the number of header lines.

  DonorFate is a partition rather than a list of exclusions, so “this donor utterance is unaccounted for” is not expressible. An earlier form returned only the excluded ordinals and proved completeness with arithmetic, which balances just as well when every ordinal is shifted by the donor’s header count.

  Both inputs are accounted for. DonorFate covers File 2; ReferenceFate covers File 1, where an utterance whose speaker is not retained is dropped and AmbiguousSpeaker does not catch it, because that fires only when a code appears in both files. A reference-only MOT with retain = [CHI] therefore passed every precondition, kept its @Participants row, and lost every utterance silently.

  DonorFate::Inserted carries tiers_stripped, because strip_tiers applies to the donor and only the donor: a bare Inserted claimed “carried over” for an utterance that was carried over AND edited.

  Still not a complete account of everything a merge omits, which is why the accessors are named excluded_by_retain and dropped_not_retained rather than excluded and dropped: donor headers other than @ID and @Comment are not carried.

Fixed

• chatter pipeline --override-file now refuses invalid override files. Malformed TOML, unsupported schema versions, and read failures previously disappeared into “no override configured”, silently triggering a fresh automatic speaker match. Explicit operator input now travels through the existing typed OverrideFileError exit path and no merged output is written.

• chatter rediarize no longer counts overlapping turns from one track twice. A track appearing in several turns is now measured by the UNION of its coverage: gaps remain gaps, while same-track overlaps count their shared interval once instead of manufacturing speaker time and distorting --contested-at. Cross-track overlap remains evidence for both simultaneous speakers, so ownership.total_ms is the sum of per-track union-held time and can exceed the utterance bullet’s duration. The JSON shape is unchanged; its corrected measurement semantics are documented in the user guide.

• chatter fix --apply can now repair E750 inside its recovered utterance. Ordinary splice edits remain barred from parser-tainted regions. The E750 catalog entry alone carries the typed state that it removes the delimiter whitespace responsible for that recovery, and the command still reparses and independently validates the resulting CHAT before writing it.

0.16.0 - 2026-08-27

Removed

• E214 is retired, and the reason is worth more than the code was. It began as “a bare [*] carries no error code” and was deliberately DISABLED as leniency Decision 1, because reference files use bare [*] as valid CHAT. Its number was then reused in the same file for a DIFFERENT rule, “the scoped-annotation list is empty”, while its spec file went on documenting the original. So one code carried a retired rule in its documentation and an unreachable one in its implementation, and its own spec example produced no diagnostic at all. Nothing detected the drift because neither rule could fire. ErrorCode::EmptyAnnotatedContentAnnotations is gone; code matching on it will not compile.

• rules::should_skip_group is absorbed into the descent module that was its only remaining caller.

Changed

• Scoped annotations are NON-EMPTY by construction. AnnotatedContentAnnotations::new returns Option<Self>, TryFrom<Vec<_>> replaces an infallible From that skipped the check, Deserialize rejects an empty list rather than accepting one off the wire, and there is no Default. Annotated::new(inner, annotations) takes the annotations instead of starting empty; Annotated::with_one(inner, annotation) is the single-annotation path, and with_scoped_annotations takes the newtype.

  The Option IS the bare-versus-annotated decision, so seven if scoped.is_empty() branches in the two parsers collapsed into it.

• UtteranceContent gains Action, and BracketedItem gains Group. Both enums had a gap where their sibling had a bare variant, and the parser filled it by wrapping the construct in an Annotated carrying nothing. That was 20,184,072 values across a 106,000-file corpus, 99.3% of all annotated_action nodes, almost all of them a bare 0 marking silence in daylong audio. The two content enums are symmetric now: every annotatable construct has a bare and an annotated form on both sides. Exhaustive matches over either enum will not compile until they handle the new variant, which is the intended outcome.

• ErrorCode is GENERATED from spec/codes/error-codes.toml, a new per-code registry that is the single owner of a code’s variant name, its rustdoc, its kind and its status, plus the retired numbers. kind and status are removed from all 236 files under spec/errors/; a spec’s code is a foreign key resolved at load, so a loaded spec proves its code exists. Anything reading kind or status out of a spec file must read the registry instead. Exhaustiveness moved from a generator check to the compiler: a wrong match arm fails the build rather than a lint.

• GoverningMarker is now pub(crate); the public face is GoverningMark. Its variants were public, so a caller could construct one directly and resolve a word’s language without ever saying what enclosed the word, which is the question the type exists to force. GoverningMark is opaque, with two constructors: of(word, enclosing) and without_own_marker(span, enclosing). This supersedes the 0.15.0 migration note below, which tells callers to use GoverningMarker::of(word, enclosing_span). That path is no longer public; use GoverningMark::of with the same arguments.

• talkbank_lsp::content_span is removed, and with it the free function content_span(&UtteranceContent). Deciding WHERE an item is now belongs to the model (WordRef::span, GroupRef::span), and deciding WHETHER the editor targets it belongs to talkbank_lsp::editor_target, which dispatches on ContentStructure rather than on 28 UtteranceContent variants. A new variant is therefore classified once, in the model, instead of once there and once in the LSP where the two could disagree.

• Parser entry points take the generated typed node wrappers, not a node plus a kind string. Every route from a loose node into a typed one is FromNodeKind::from_node, and the 411 sites that ASSERTED a node’s kind rather than testing it are gone. Callers passing (KIND_CONSTANT, raw_node) pairs pass the wrapper instead, so the proof is required where the value is born.

• JSON output changes, with no compatibility shim. An annotated_action or annotated_group carrying no annotations is now action or group. Previously-emitted JSON containing the empty annotated form will not deserialize. Regenerate rather than reading cached to-json output. CHAT text is byte-identical either way, so no file changes validity.

Known limitations

• --parser re2c is NOT READY to judge CHAT validity, and this release says so in the tool. A clean --parser re2c run is not evidence that a file is valid: the backend ACCEPTS constructs the default backend refuses. Measured 2026-08-27: an unrecognised scoped annotation on a quotation (“hello” [qq] .), on a pause (hello (.) [qq] .), and in utterance-initial position ([x 2] hey .) are all accepted, where the default backend reports E316 or E375.

  The cause is information lost before validation runs, not a missing rule. Both parsers build the same talkbank_model types and share one validator, but each has its own intermediate parse tree, and re2c’s does not carry annotations for every construct: ast::Group has an annotations field and ast::Quotation, six lines below it, does not. Three of the five hosts that regressed here ARE fixed in this release, because their annotations do reach the model; these three do not reach it.

  --parser help and book/src/architecture/parser-backends.md now say this at the point of use. That page also carried three claims this contradicts, including a parity table row reading “Re2c silent (misses error): 0”, and a recommendation to prefer re2c for batch and CI validation. All corrected; the parity figures are marked as not re-measured.

  Use re2c to COMPARE two implementations, which is what a specification oracle is for. Use the default backend to decide validity. The default backend is unaffected by any of this, and is what chatter validate, normalize, to-json and the LSP use unless you ask otherwise.

Fixed

• word@@ reported one defect twice, the specific diagnostic buried under the generic one. The parser names a repeated @ run as E203 with the run in hand (“a word may carry only one ‘@’ suffix, found ‘@@’”); check_inline_at_ markers then added its own E202 (“dangling ‘@’ marker”) for the same word, because the suppression that stops a double already existed for the E203-against-E203 case and was never applied to the E202 branch four lines above it. word@c@ was the same. Both now report E203 alone.

  A bare trailing @ (hello@) still reports E202: it carries no form type, so nothing else has named it, which is the case that branch exists for.

  Found by the release review; fixed by writing the spec example first, where the backend-parity gate stated it exactly: tree-sitter [E202, E203] ... spec expects [E203].

• chatter rediarize assigned an utterance to the track of its single LONGEST TURN, not the track holding the most of it. best_track took the greatest overlap_ms over the turn list with no per-track accumulator, so three short turns of one track lost to one longer turn of another even when the first held twice as much of the utterance. Its own docstring and the CLI help both said “the track with the greatest overlap”, which is what it now computes.

  This is the shape a diarizer actually produces: pyannote emits short turns with gaps inside a single speaker’s run. A track appearing in several turns is accumulated now, and ties break on the track code rather than on turn order, so the winner is a function of the input rather than of how the diarizer sorted its file.

  best_track is replaced by TrackOwnership, which keeps the whole distribution (winner(), shares(), total_ms(), runner_up_share()) rather than computing it and returning one name. Returning only the winner is why the defect was invisible: nothing downstream could tell a track that held 95% of an utterance from one that held 34% of a three-way split.

  Breaking: rediarize and rediarize_content take a further argument (below), and RediarizeOutcome gains a field.

Added

• chatter rediarize --contested-at SHARE reports utterances whose time is meaningfully split between tracks, in the stderr summary and in --summary-json under a new contested list. Each entry carries the utterance index, the track it was assigned to, and the full ownership distribution: every overlapping track with its summed milliseconds, descending, plus the total. The WHOLE distribution rather than a winner and a runner-up, because that narrower shape cannot tell a 55/45 split from 55/23/22.

  Contested utterances are still reattributed to their winner, so they are reported separately from flagged, which keeps its narrower meaning of “declined to reattribute”. Placement is byte-identical with and without the flag; this is a reporting change.

  There is deliberately no default. Omit the flag and nothing is reported. What share makes an utterance genuinely mixed has not been measured against human listening, and a default would hand every user a constant wearing this tool’s authority. A value outside 0.0 to 1.0, or NaN, fails the command before any file is read, rather than silently meaning “nothing is ever contested”.

  Known limitation, stated in the book page: summed milliseconds per track cannot distinguish a speaker change INSIDE an utterance from crosstalk across the whole of it, and those want opposite remedies.

• TimeSpanMs::start_ms() and end_ms(). The fields are private so that new() is the only route in and an inverted span cannot be built, which is right, but it left the type WRITE-ONLY through the public API: DiarizationTurn::span is a public field of a type a caller could hold and could not read, so a downstream consumer of parse_turns_json had to re-declare the same concept to get the numbers back out. Reading cannot invert anything.

• THREE COMMANDS COULD DELETE A TRANSCRIPT AND REPORT SUCCESS. The worst class in this release, found by review rather than by any gate, and every case exited 0 with a green line.

  chatter normalize notes.cha -o notes.cha, the documented in-place idiom, on a file of ordinary prose left a ZERO-BYTE FILE and printed ✓ Normalized. v0.15.0 refused it, so this was a regression. On a transcript missing its @End it deleted the LAST UTTERANCE, which is exactly the shape a file truncated mid-transfer has. On a malformed %gra tier it emptied the tier, producing a file it then refused to read again. Those two are unchanged from v0.15.0 and were shipping in both.

  chatter debug retag-language and debug fix-s wrote back a model that had DISCARDED an unparsable region: hello [[[[ test ]]]] world . became world ., in place, recursing over whole directories, with no --dry-run and no backup. debug join-retrace had the same shape and was found while fixing the other two.

  All four rewriters were the same three steps: parse, to_chat_string(), write. The return type was String, which cannot carry the one fact the caller needed, so no caller had it. chatter to-json refused all three normalize inputs, because it happened to route through a stricter path, and the two commands disagreeing about the same model is what made this findable.

  Two different proofs, because the commands promise different things. normalize reshapes and must lose nothing, so talkbank_transform::Rewrite refuses when a source line has no counterpart in the output, compared with whitespace removed so the canonicalisation it exists for still passes. The three EDITING commands change content on purpose, so that test would refuse every legitimate edit they make; they require instead that the model reproduce the source BYTE FOR BYTE before the edit, which is the only point where faithfulness is a clean question for them. A refused file is left untouched and the message names the line, or tells the operator to run chatter normalize first.

  Six CLI subprocess tests pin all of it, including the case that must NOT refuse: the six reference-corpus files normalize legitimately rewrites.

• chatter debug retag-language is new, and was missing from this section entirely. It retags a language code across all three notations it can reach (@Languages, the [- code] utterance precode, and word@s:code) and REFUSES a file naming the code in a <a b> [@s:code] span, which it cannot rewrite. --to deduplicates in @Languages. It is a tool and not a find-and-replace because a language code is also ordinary transcript content: its first use retagged sun to fin across a corpus where sun is also colloquial Finnish for “your” and appears 27 times as real speech.

• A nested quotation stopped being detected as soon as either quotation carried an annotation, on the default backend only, so “a “b” c” [//] hello . validated CLEAN while “a “b” c” . reported E372, and the two backends disagreed about a validity rule on identical bytes. [/], [*] and [% note] leaked the same way, and so did an annotation on the INNER quotation.

  A quotation has TWO spellings in the model, with and without its own scoped annotations, and each half of the rule named only the first. descent.rs named both; main_tier.rs named one. The annotated spelling was introduced by the same release that gave quotations scoped annotations, and the nesting rule was never taught about it.

  Fixed as a type rather than as two more match arms: GroupRef::Quotation and GroupRef::AnnotatedQuotation are folded into one Quotation(QuotationRef) variant, mirroring the RetraceRef beside it, so “is this a quotation” is a single arm that cannot be half-written. This is a breaking change to GroupRef; a caller matching AnnotatedQuotation will not compile, and the two spellings remain distinguishable one level down through QuotationRef. QuotationRef::span preserves the distinction that GroupRef::span drew between the two, which folding them could have lost silently. The outer scan that looked for a quotation to test now descends through ContentStructure as well, so a wrapper cannot hide either side of the relation again. Spec examples 4 and 5 of E372.md are the two directions.

• @Location and 14 other headers were REJECTED by the public fragment parser. parse_header_fragment, and the ChatParser::parse_header trait method behind it, dispatched through 19 hand-written arms plus a catch-all, while the grammar’s header supertype has 34 subtypes. @Activities, @Bck, @G, @Location, @Number, @Options, @Page, @Recording Quality, @Room Layout, @Time Duration, @Time Start, @Transcriber, @Transcription, @Thumbnail and @Unsupported all reached it and came back as errors rather than as Header::Unknown. The same headers parsed correctly inside a whole document, because that path matches the generated HeaderChoice exhaustively: two dispatchers for one job, one of them a drifted subset, and the tests covered only the arms that existed.

• Five call sites silently discarded a group and every word inside it. convert_to_group_content returned Result<BracketedItem, Group> where neither outcome is a failure, and the shape invited if let Ok(item), which five call sites duly wrote. It is a TOTAL function returning BracketedItem now, so there is no second case to ignore and those sites preserve the content. An intermediate two-variant enum was tried first and reverted: it moved the decision without closing it.

• The container descent rule had two implementations that had already drifted. The eight walkers and count.rs’s four traversals each carried their own container arms, about thirty per side; walk/bracketed.rs shipped four ungated AnnotatedQuotation arms while count.rs gated the same variant, so one node was walked by one and skipped by the other. One helpers::descent module owns it for every traversal now.

• A quotation could not carry a scoped annotation, which CLAN CHECK accepts. The grammar takes quotation_with_optional_annotations.

• An @ID age with a component too large for its field parsed as ZERO. AgeValue::from_text parsed each component with .parse::<u8>() behind an all-ASCII-digits guard, which leaves exactly one way to fail (a value above 255) and answered it with 0. 2;300. became Valid { years: 2, months: Some(0) }: two years and no months, presented as a successful parse, in the field that is the primary variable of most CHILDES research.

  Bounded honestly: no validation verdict changes. A component of three or more digits also fails the two-digit depfile pattern, so such a file was always reported invalid. What was wrong is what the typed model then SAID about it, which reaches library callers and anything reading the parsed age, not chatter validate’s answer. age_component returns Result<Option<u8>, Unrepresentable> now, so an ABSENT component (1; has no months) stays distinct from an unrepresentable one, and an unrepresentable one sinks the whole age to Unsupported, which preserves the original text byte for byte.

• ErrorCode’s string constructor with a silent fallback is REMOVED. It mapped any unrecognised string to UnknownError through a catch-all, so ErrorCode::new("E7O5") with a letter O compiled and would have shipped E999 with nothing to catch it. Three of the %mor alignment checks built their codes that way (E705, E706 and E716; the count mismatch among them is CLAN CHECK 140), which also meant nothing reasoning over the enum could see those three checks at all. The codes they emitted were correct, so no diagnostic changes; what changes is the API. All three return typed variants, and ErrorCode::parse_exact returns Option<Self> and is now the only route from a string. Callers of the old constructor must handle the None.

• A word carrying two @ suffixes was reported wrongly, and one such word was DELETED on the way out. Two distinct symptoms, which an earlier draft of this entry ran together:

  hello@@c and hello@c@d never formed a word at all, so the utterance fell to error recovery and reported the generic E316, “content could not be parsed”, while the model’s own rule for the shape could never fire.

  word@k@s:spa DID form a word and DID report E203 in v0.15.0. What was wrong there was the message, and what was dangerous was chatter normalize, which exited 0 and wrote the word back split in two: word@k@st became word@k@s t, silently, at exit 0. That is the deletion, and it is the reason this entry exists.

  Both shapes now parse, are refused as E203 with a message naming the actual defect (A word may carry only one '@' suffix, found '@c@s:spa'), and serialize back verbatim, so normalize REFUSES the file instead of rewriting it. hello@c, dog@j and hola@s:spa are unchanged from v0.15.0.

  A word may carry at most ONE @ suffix. Ruled 2026-08-27, asked because CLAN CHECK accepts multiple suffixes and chatter does not: “Multiple suffixes might make logical sense, but it is computationally messy. So, let’s disallow that.” word@k@s:spa is therefore invalid even though the form marker @k and the language suffix @s:spa are each fine alone. A documented divergence from CHECK, which passes these files; main-tier words with two @ runs number zero across the ~106,000 kept files.

  A bare trailing @ (hello@) deliberately keeps its existing E202: @ is the header sigil, and admitting a single one in word position moved the diagnostic for a doubled @End.

  --parser=re2c REFUSES all of these too, and agrees on the code for some of them. Measured: gumma@c@s:spa and bebe@k@st report E203 on both backends, because that lexer takes the two suffixes as separate tokens and the model’s own rule counts them. Not full agreement: on the @s:-bearing case the default backend now reports E203 alone where re2c reports E203 plus E255, because the word carries an undeclared form type and its @s:spa no longer registers as a language marker. Both refuse the file. dog@b@c (E209 plus E253) and hello@@c (E321) it refuses for other reasons, since its lexer cannot form those words at all. The named message above is the default backend’s. The divergence is recorded in the parser-parity baseline as a Conflicting row for E203.md.

• E207’s message asserted that a KNOWN annotation marker was unknown. It read "x" is not a known scoped annotation type. An annotation reaches the unknown path whenever no specific rule matched it WHOLE, which happens both when the marker really is unknown ([qq], [@ xyz]) and when a known marker carries content the rule refuses. Under --parser=re2c, whose rule set is narrower, [x 0] and [:] both land there, and the message then told the reader that x and : are not known scoped annotation types, when the marker is not the thing at fault. [: replacement] is ordinary valid CHAT on both backends, so the old message was plainly false there. The message now names the annotation as written, could not read [x 0] as a scoped annotation, which is true in every case and shows more than the marker alone.

  Scoped honestly: this reaches the --parser=re2c path only. On the default backend the diagnostic is issued by the parser rather than the model, and its message is byte-identical to 0.15.0’s. An earlier draft of this entry said “affects both backends”; measured, it does not.

  And [x 3] is NOT an example of a valid construct: hello [x 3] . is refused by both backends. Only the group and utterance-initial spellings parse, and only under re2c, which is its own divergence.

• Under --parser=re2c, an annotation on a top-level word reported E207 at byte 0, on line 1, pointing at @UTF8. Annotated::new starts at the dummy span and the tree-sitter parser follows it with .with_span(..); this converter never did. That path now takes the annotated construct’s span widened to cover every annotation whose text can be placed, and REFUSES rather than answering with the sentinel when nothing can be, so a span is either real or absent.

  SCOPED HONESTLY: this is one of eleven Annotated::new sites. An annotation on a bracketed word, a group, an event, an action or a retrace still reports at byte 0 under this backend. Those need the same AST work as the retrace spans, which is the queued change that carries the lexer’s own token ranges through the parser instead of re-deriving them.

• validate --list-checks advertised two checks that cannot fire. E361 (“invalid timestamp value in media bullet”) and E382 (“failed to parse %mor tier content”) were marked implemented in the code registry, and nothing can produce either. Both now list as Planned: --list-checks goes from 224 checks (184 Active, 40 Planned) to 223 checks (182 Active, 41 Planned), the one retired check being E214 above. The checks themselves are unchanged; the advertisement was wrong.

• Under --parser=re2c, every rule keyed on a span was silently unreachable. Separators and words both reached the model at Span::DUMMY, which is {0, 0} and therefore also a real position, and validation FILTERS on that value: a dummy span makes the model answer “there is no comma here”, so E258 (consecutive commas) and every other span-keyed rule never fired on that backend. Words and separators on the main tier now report the same spans on both backends, which is what was fixed and what was measured: 61 of 64 word and separator span sets match tree-sitter exactly.

  The backends are NOT span-identical in general, and this entry does not claim they are. Across the repository’s own .cha files, 434 (file, code) pairs are reported by both backends and 338 of them still differ, 329 because re2c answers at byte 0. Untouched here: the pause-glue mirror in parser/file.rs, every terminator, and every dependent-tier diagnostic. E370 is worse than byte 0, reporting an offset that is not a file position at all; that is unchanged from 0.15.0 and is not fixed here.

  This affects only the opt-in oracle parser; the default backend was never wrong.

• Under --parser=re2c, three diagnostics were reported TWICE. Giving words and separators real spans made three model rules reachable while the hand-written mirrors that existed BECAUSE they were unreachable were still in place, so E749, E764 and E765 arrived doubled. The parity gate stores codes in a BTreeSet, so multiplicity is structurally invisible to it and STILL is: a new six-utterance test (re2c_reports_no_diagnostic_twice) covers the mirrors it knows about instead. One doubled diagnostic survives that test’s case list, E307 on a bad speaker ID, unchanged from 0.15.0.

• Under --parser=re2c, an interposed word lost its form marker. &*SPK: took a bare word body where the grammar defines the payload as a whole standalone word, so a form marker, an @s language suffix or a $ POS tag on an interposed word was dropped. Found on real Spanish transcripts, where the two parsers had disagreed for as long as the rule existed.

• Under --parser=re2c, an unrecognised annotation killed the utterance. [@ xyz] reported E321 (“unparsable utterance”) rather than E207 (“unknown annotation”): every specific bracket form had a lexer rule and anything else fell to a bare [ no parser rule could use. E321 is a statement about the parser where E207 is a statement about the file.

• Under --parser=re2c, un++do reported a misplaced linker. The word body consumed + only when an atom followed, so the word ended at un and ++ matched the linker rule: E766, “a linker placed after utterance content”, on a construct containing no linker. It reports E233, “empty part in compound word”, as the specification says it should.

0.15.0 - 2026-08-25

Changed

• ExtractedWord.lang becomes ExtractedWord.language: ExtractedLanguage, and the word gains a span. The old field carried only a word’s OWN @s marker, so a word inside a <...> [@s:hin] span came out of extraction indistinguishable from an unmarked word in a plain English utterance. The extractor WALKS the tree and therefore knew about the span; it discarded what it had computed, and every NLP consumer downstream was left unable to recover it. Batchalign’s morphotag read the old field, so span-governed words fell out of second-language dispatch and were tagged against the tier language.

  ExtractedLanguage is Utterance | Own(marker) | Span(span). It is not an Option, because “the utterance governs” is a real answer rather than a missing one, and treating no-mark as no-language is the mistake the type exists to prevent. ExtractedLanguage::resolve(span, tier, declared) gives the resolved language directly.

• GoverningMarker::resolve_at(span, ..) resolves without a Word. The resolver only ever used the word for word.span, to place diagnostics, so a consumer holding an already-extracted word had to fabricate a Word::new_unchecked purely to satisfy the signature. Batchalign was doing exactly that. resolve_word_language_with_marker is deleted; one span-based core serves both paths.

0.14.0 - 2026-08-25

Removed

• resolve_word_language is gone from the public API. It answered “what language is this word?” without saying what SCOPE it was asking under, and silently assumed a word’s own marker was the whole story. Once <...> [@s] spans existed that was false, and the function had nowhere to put the span. Use GoverningMarker::of(word, enclosing_span) followed by .resolve(..): the constructor takes the scope, so a caller with none writes None explicitly. resolve_word_language_with_marker is no longer public either; it is a primitive of that operation.

• FileStem::from_str is renamed from_stem. It can never implement std::str::FromStr, whose signature has no input lifetime, while this type borrows its stem; the old name invited callers to expect a trait they could use generically.

Added

• Multi-word code-switch spans: <word word> [@s] and <word word> [@s:code]. Every word in the scope takes the switched language, exactly as if each carried the @s / @s:code suffix, so a switched stretch no longer has to be annotated word by word. Bare [@s] resolves the way a bare word@s does. As with any scoped annotation, a single content item needs no angle brackets: hallo [@s] is well-formed.

  A word inside the span may carry its own marker, and the word wins. This is attested usage rather than a case to reject: transcripts mark a switched stretch with the span and individual borrowed words inside it with the donor language. Resolution is innermost-first (word, then span, then utterance) and each layer records its own provenance, so language_metadata[].source gains span_shortcut and span_explicit alongside the existing word_* values. A span and a suffix can resolve to the same CODE, and that field is the only way to tell which mark decided it.

  Consumers reading a word’s lang field alone will under-report switches: a span is an annotation on the group, and the words inside keep lang: null unless suffixed. language_metadata carries the resolved answer for every word regardless of which mark produced it.

Fixed

• E220 and E763 gated on the wrong language inside a code-switch span. Span resolution reached metadata but not word validation, so a word’s recorded language and the language it was checked against could disagree: <ha# kelev> [@s:heb] in an English-headed file was reported E763 as English while its own metadata said Hebrew. Both paths now share one precedence decision.

• A [@s:code] span could not be serialized to JSON at all. The enum was internally tagged, which serde cannot use for a variant carrying a string, so chatter to-json failed at runtime on any transcript containing one while the bare [@s] form worked. The committed JSON Schema described a shape the serializer could never emit; it is regenerated and smaller.

• Every generated error fixture parsed with a spurious MISSING newline recovery node, because the generator stripped the trailing newline the grammar requires. Invisible for as long as each fixture also emitted a real diagnostic to hide it behind. Restoring it changed the diagnostics of zero of the 335 existing examples.

• chatter debug fix-s no longer rewrites an utterance containing a code-switch span. It strips each word’s @s suffix after writing the [- LANG] precode, so for a word inside a span the span would then govern it and its language would silently change: <how@s:fra to@s:fra> [@s:eng] . became [- fra] <how to> [@s:eng] ., whose words resolve to eng. It now refuses such utterances, which is lossless where rewriting was not. No released version could do this, because [@s:eng] did not parse before this release; it was found and fixed within the same cycle.

• E220 no longer fires on a word whose language is unresolved. It treated an empty candidate set as “no language permits digits”, so an unresolvable @s produced “illegal digits in language X” with no X. E763 already skipped in that case and the two are documented as agreeing. The visible effect: [- zho] ni3hao3@s . now reports E248 alone, which names the actual defect, rather than E248 plus a consequence of it.

• Language-gated word rules run whenever the language is KNOWN, not only when the file declares one. The gate asked whether @Languages exists, which is a different question: <...> [@s:eng] names a word’s language with no header present, and nothing was checking those words.

• A word carrying its own @s inside a span, and a span on a replaced word or a retrace, are now validated under the span like any other span-governed word. Only annotated GROUPS were handled, so hallo [@s] was recorded as switched in metadata while being validated against the tier language.

Changed

• “CA” named three different things, and two names picked the wrong one. The symbol registry’s exported arrays are computed from parse_role and were called ca_element_symbols and ca_delimiter_symbols, naming PROVENANCE on a value holding a PARSE ROLE. They are now word_attached_symbols and paired_stretch_symbols, and their union, which builds the set forbidden inside a word_segment, is ALL_MARKER_SYMBOLS. Library consumers reading talkbank_model::generated::symbol_sets see the renamed constants.

  The registry already carried both facts: notation_family says where a symbol’s notation came from, and 2 of the 25 are disfluency rather than Conversation Analysis. Those two are ≠ (blocking) and ↫ (segment repetition), which every fluency corpus depends on, so the old name was false for exactly its load-bearing members. The ca_element and ca_delimiter NODE names are unchanged, and parser.c, grammar.json and node-types.json regenerate byte-identical.

• ChatOptionFlag::enables_ca_mode is replaced by has_effect(CaOptionEffect). The old name asserted that @Options: CA turns Conversation Analysis parsing on. It does not: CA-originated markup needs no option at all, and the flag’s scope is material judged specifically weird CA. A predicate that reads “is this file CA” is what invites gating SYMBOL ADMISSIBILITY on the option, which would be wrong for every symbol, including the genuinely CA-originated ones.

  The two effects are named separately because they are not the same kind of thing: TerminatorRequirementWaived waives a requirement, while ParentheticalIsCaOmission changes what a construct MEANS. Calling both leniency would be a quieter version of the same conflation. The match on the effect is exhaustive, so a third effect, or a second flag granting one, breaks compilation rather than silently inheriting CA’s answer.

  Validation verdicts: UNCHANGED. Renames and one predicate; every call site computes what it computed before.

• Tree-sitter 0.26.13 across the workspace, the grammar crate, the spec workspace and the tree-sitter-cli devDependency, plus desktop npm devDependency bumps and jsonschema 0.51.

  Validation verdicts: UNCHANGED over the sampled corpus, and that needed measuring rather than assuming. 0.26.13 avoids wide error nodes on unparseable input, which is a change to RECOVERY, so it can move what validate reports on malformed CHAT without changing one byte of the regenerated parser.c (which is in fact byte-identical here) and without any fixture in the suites noticing, because they all parse. The corpus differential is what can see it: over 2,147 files at stride 50, stratified per repo, against the v0.13.0 released build, there were no new error codes, no per-code count increases, no newly failing roundtrips and no new cross-backend disagreements. That is a statement about the sample, not about the whole corpus; at this stride a defect in a few dozen of ~106,000 files could still hide.

  The generated typed CST traversal is byte-identical apart from its generator provenance stamp, and just spec-gen moved no artifact.

0.13.0 - 2026-08-21

Validation verdicts: UNCHANGED. Nothing here moves what validate reports on a CHAT file. Every prior entry states this either way, and the published promise is that an entry without the note did not move its verdicts, so an entry that omits it cannot be told from one nobody filled in.

Removed

• talkbank_transform::capitalize is GONE. The English capitalization transform announced in 0.7.0 (capitalize_english, capitalized_pronoun_i, is_capitalizable_initial, capitalize_first) is deleted. It is the only change here that affects a library consumer.

  Why: chatter is the CHAT-format authority, and English orthography is a convention of one language rather than a fact about CHAT. Nothing inside chatter ever called it; its two users were downstream generators, which wanted different policies. One of them had already written its own version of is_capitalizable_initial and documented that chatter’s answered a different question. The module also had no stopping rule: pronoun “I” and sentence capitals today, then contractions and proper nouns on request.

  If you used it: copy it into your own generator, where the policy belongs. It is built entirely on public API (walk_words_mut, Word::category, Word::untranscribed), so nothing about the move needs chatter internals. Note that the version shipped here had three defects in is_capitalizable_initial, all from deciding a structural question from cleaned_text(), which strips the very prefixes the question needs: a non-letter-initial word did not consume the utterance-initial slot, so the capital landed on the following word; an apostrophe-initial word received no capital at all; and the &-fragment guard could never fire, so a filler took the sentence capital. Ask the typed model instead.

  num_words is unaffected and stays: it serves E220, a rule chatter enforces.

Changed

• chatter new-file builds its template through the typed model. The emitted skeleton is produced by parsing and serializing a typed ChatFile rather than formatting text, so it is roundtrip-proven by construction; the default output is unchanged.

• docs/errors/index.md is one table, sorted by code. It emitted one ## section per spec, 236 of them with 31 exact duplicates, each over a single-row table, and reprinted every description. It is now a flat table with Code, Name, Category, Kind, Level and Status columns: 234 lines where it was 2,247. Anything that scraped the old section structure will need updating; anything that followed the E###.md links is unaffected.

• The error-spec format is TOML frontmatter with a required CLAIM per example. Landed in stages within this unreleased window, superseding earlier entries’ details: metadata moved from ## Metadata bullets to +++ frontmatter (an unknown or missing field is a load error); the authored Layer field was then DELETED (which pipeline stage catches a rule is recorded per example in the generated spec/observations/ snapshot, and every example is a fixture in the validation corpus); and Expected Error Codes was replaced by claim = 'violates' | 'legal' | { subsumed_by = ... }, whose negative halves (a code that must NOT fire) are enforced. level moved from the spec file to the example, where it is required: a code can be violated at one level in one example and another in the next (E519 has header-level and utterance-level violations), so the fault site is a fact about the example; a code’s page renders the distinct set. A non-empty Description remains required. This matters only if you author specs against spec/errors/.

• Corpus tests require TALKBANK_DATA. The re2c integration tests defaulted to a hard-coded directory under $HOME, which could only ever be right on one machine and silently sent everyone else to a path that does not exist. The variable is now required and its absence fails loudly. just corpus-tests needs it set; the default test suite is unaffected, since those tests are #[ignore]d.

Internal

Not part of any published API, listed because the commits are marked breaking: spec/errors/*.md now has ONE parser in the spec workspace rather than two (ErrorCorpusSpec and its types are deleted), and the spec format’s vocabulary moved to a new dependency-light talkbank-spec-vocabulary crate that both cargo workspaces share. The generators and talkbank-parser-tests crates are publish = false.

0.12.0 - 2026-08-16

Validation verdicts: CHANGED. Four rules report where they were silent: E241 on illegal untranscribed spellings, E756 on any empty dependent tier, the participants check on files declaring an empty set, and the re2c backend on empty tiers it used to paper over. If you gate a pipeline on validate, diff your own corpus before upgrading; see What a Version Bump Promises.

Adjudicated against real corpus data before shipping, per the standing grammar-change gate. The full-stride differential against the shipped 0.11.0 build covers all 106,507 corpus files and reports EVERY error code unchanged except E241, whose 661 new instances are every one an illegal short or miscased spelling of an untranscribed marker: 624 ww, 18 Www, 10 XX, 6 Ww, 2 Xxx, 1 Xx. All adjudicated INTENDED, the rule correctly flagging invalid data, and the 194 affected files join the cleanup queue. No new cross-backend disagreements and no newly-failing roundtrips.

Added

• E241 rejects the illegal untranscribed spellings. The corpus authority ruled that ww is not legal CHAT and www is canonical, adding yy against yyy unprompted. Which spellings are wrong is now DERIVED from the canonical set rather than listed, so ww cannot be missed while xx and yy are caught, which is what happened before. Eight instances in the differential sample, every one adjudicated as the rule correctly flagging invalid data.

• E756 covers every dependent tier, not only %x*. A tier line whose payload is absent or whitespace-only declares nothing. The rule always said that; only its name was %x-specific, and it could not be applied to a standard tier until the model could represent an empty one. Before this, an empty %eng: was read as VALID by the re2c backend and rejected by tree-sitter through an undescribed code, so the two backends disagreed about a file neither could explain. Zero instances in the differential sample: the construct is invalid CHAT and correspondingly rare.

  The rule now reaches EVERY tier whose grammar body is free text, which is every dependent tier except the structured ones (%mor, %gra, %pho, %mod, %sin, %wor), whose bodies are not free text and whose empty case fails earlier and more specifically. That boundary is a grammar fact, not a list: a tier qualifies exactly when its rule marks its body optional(...). %tim gained an Empty state to make this expressible, since both of its content variants hold a non-empty string; the Phon tiers (%xmodsyl, %xphosyl, %xphoaln, %xphoint) answer from the word or group count they already reported.

Fixed

• An empty dependent tier is no longer papered over. The re2c backend met %eng: with no content and substituted a single space, which made the tier look well formed and the whole FILE read as valid where tree-sitter reported errors. The model can now say that a tier declares nothing, so the parser reports what the file contains and E756 judges it.
• An empty %x tier survives a roundtrip, and normalize no longer swallows the file. %xtst: with no content reported E756 from the PARSE path and returned without adding the tier to the model, so the line vanished on roundtrip while an empty %eng: was preserved. Worse, because the report came from parsing rather than validation, chatter normalize treated the whole file as unparseable and wrote NOTHING. The parser now says what the file contains and the validator judges it, as it does for every other tier kind.
• An empty %tim: is a %tim tier. The re2c backend lowered it to an unsupported DEPENDENT TIER and reported E605, “unsupported dependent tier ‘%tim’”, about a tier name that is perfectly supported; a whitespace-only body additionally drew E603 (“Invalid %tim tier format: ‘’”) alongside E756, two codes for one fact and the more specific of them false. Same for an empty %xphoaln: and %xphoint:, which conflated an absent body with a malformed one. All four now report E756 on both backends.
• The participants check reads the declaration. An empty participant set used to disable the check rather than fail it, so the files least likely to be well formed were the ones exempted from the rule.
• An annotation’s separator is not part of its text. [=! contacts], written with two spaces, parsed as " contacts" in one backend and "contacts" in the other. That was the last content-level disagreement between the two parser backends across all 107,403 corpus files.
• chatter validate --format json no longer writes cache housekeeping to stderr. Two facts leaked there: Cleared N cache entries on every --force run, and note: pruned N unreachable cache row(s)... whenever a prune fired. Both broke the documented promise that JSON mode’s stderr is empty, and the test suite contained two tests with contradictory expectations about it, one requiring stderr empty and one asserting it contained the cleared count. The first only failed when a prune happened to fire, which is why both shipped green through four releases.

Changed

• Breaking (library): TimTier gained a third variant. TimTier::Empty { span } represents a %tim: line that declares nothing, which neither Parsed nor Unsupported could hold: both carry a NonEmptyString. Code matching on TimTier exhaustively must add an arm. TimTier::empty() constructs one, declared_content() returns None for it (as_str() still flattens to "" for Display and serialization), and the serde form is unchanged apart from "" now deserializing to Empty instead of erroring.

• Breaking (library): the test-utils feature is REMOVED, and with it ChatCleanedText::test_unchecked and ChatRawText::test_unchecked. Not renamed: gone. This is the breaking change that bites FIRST, because cargo refuses to resolve a graph that asks for a feature which no longer exists, so it fails before anything compiles and is invisible to a “what will fail to build” scan. A consumer sees:

  package `X` depends on `talkbank-model` with feature `test-utils`
  but `talkbank-model` does not have that feature
  

  Build fixtures through the parser instead: TreeSitterParser::parse_word followed by ChatCleanedText::from_word. The hatch was removed because a type whose existence proves “this text came from a parsed AST” is only as strong as its weakest constructor, and one any dev-dependency could switch on was that constructor. Downstream adoption on the day of release found three fixtures that had been asserting on a shape production cannot emit (a terminator in a words list), passing only because the hatch let them fabricate it.

• Breaking (library): BulletContent::empty(). A named constructor for a payload that carries nothing, distinct from from_text(""), which fabricates an empty text segment that is not in the file. Additive; no existing call site changes.

• NDJSON surface: a new record type. Those facts now arrive on stdout as {"type":"cache","action":"clear"|"prune"|"warning",...}, emitted only when cache maintenance did something. Silencing them under --format json was considered and rejected: they are results a caller can act on. A consumer that ignores unknown type values needs no change; one that errors on an unrecognised type will see these. The type field’s documented value set is now "file", "summary", "cache", and the contract page says to treat unknown values as ignorable. See Diagnostic contract.

0.11.0 - 2026-08-13

Validation verdicts: CHANGED, in BOTH directions. Six error codes that had silently degraded to E316 “unparsable content” now report themselves again (E202, E307, E311, E314, E370, E375), and two false positives are gone. If you gate a pipeline on validate, diff your own corpus before upgrading; see What a Version Bump Promises.

Adjudicated against real corpus data before shipping: the operator’s corpus differential over a 2158-file stratified sample reports byte-identical per-code counts and no newly-failing roundtrips against v0.10.0. The changes below are all on malformed input, which a curated corpus contains almost none of.

Library APIs: BREAKING. This release changes the public API in several ways. The list below is from a mechanical diff of the public surface between the two tags, made after the notes first shipped saying “additive” and then being corrected twice as a downstream consumer hit one break after another. A release note written from memory of a 415-file change is a guess; this one is a measurement.

Removed items (6):

• FormType::A. The @a marker was retired by the corpus authority in 2024 and is absent from the form-marker registry that now generates every site of that closed set; the variant survived only because sixteen hand-written copies of the list disagreed. No replacement: the construct is not CHAT.
• ALL_MARKERS, all_markers_string. Superseded by the same registry.
• collect_bracketed_content, collect_bracketed_item. Superseded by the typed traversal.
• counts_for_tier_in_context. Use counts_for_tier, now re-exported at talkbank_model::alignment.
• iso.

Added, and breaking for an exhaustive match:

• FormType::Undeclared(String) carries the raw text of a marker naming no declared form, so word@zz roundtrips instead of being silently rewritten to word@z:zz.

Changed signatures:

• ChatFile::validate and validate_into take TranscriptName<'_> rather than Option<&str>. None becomes TranscriptName::Anonymous; a real name becomes TranscriptName::Named. The Option could not say which of “no name” and “a name we failed to read” it meant, and both reached the same branch.

Library APIs: additive. talkbank_model::alignment re-exports walk_words, walk_words_mut and counts_for_tier, which previously required naming the helpers module.

Fixed

• A recovery node could displace an entire tier_body. An utterance ending in “ .“ was told its terminator was missing (E305), and a retrace or bracket at utterance start took the rest of the line with it. The parser was reading its own recovery artefact as evidence about the user’s file. The generated typed CST traversal is regenerated from a generator that no longer absorbs an ERROR child at whatever position its cursor had reached.

• Six codes degraded to the E316 catch-all. Which classifier a recovery node reached was decided by WHERE tree-sitter had put it, so the same construct was named precisely at utterance start and generically after spoken material. MainTierRegion is now stated by the caller that knows it, and every main-tier Unexpected sink routes through one owner.

• E246 blamed a lengthening marker for a stray tab. The classifier saw a : before the recovery node, and that : was the SPEAKER’s. A tab inside the main tier now reports the tab.

• E758 pointed at whitespace nowhere near a tab. It claims “extra whitespace between the tab and the tier content”; filling that slot never established the adjacency the sentence asserts, so ordinary space between two words was reported as a leading-space violation. The span is now built only when it starts at the tab’s end byte.

• E754 retired. @l and @ls no longer require a single character.

• Windows: the content-catch-all gate reported every exempted file as new, because repo-relative paths were compared with the host separator against a forward-slashed list.

Changed

• Two wall-clock test assertions became hang detectors with order-of-magnitude ceilings; both were tuned to one machine and one of them turned the Windows matrix red doing correct work.

• chatter-desktop and talkbank-llm set doctest = false. Neither has doc examples, and each was paying a full rustdoc compile to run zero doctests.

0.10.0 - 2026-08-07

Validation verdicts: CHANGED, in the stricter direction. Two new error codes reject retrace constructions that previously passed silently, and two existing rules stopped being suppressed by the shape of the content they were asked about. Adjudicated over all 107,376 corpus files: E377 fires 53 times in 42 files and E378 15 times in 12 files, all real transcription defects and queued for data cleanup; restoring E372 and E704 costs zero new instances, so those two are pure correctness.

Library APIs: BREAKING. Retrace loses its annotations field, both content enums gain an AnnotatedRetrace variant, and per-word language records lose word_index. Pre-1.0, so this is a minor bump.

Every fix below except the merge one is the same defect: a traversal carrying its own private list of which content variants contain other content, plus a catch-all arm for everything the list forgot. Five such traversals existed.

Added

• E377 RetraceWithNoMaterial. A retracing marker whose material is another marker, so it retraces nothing of its own. One rule covers the unbracketed на [//] [/] на and the bracketed <<a> [/]> [//], because the lowering folds both into the same tree; naming it for the shape rather than for a spelling is what makes that possible. Deliberately narrow: 11,163 retraces in the corpora sit inside another retrace and only 4 wrap a lone marker, so a “no retrace inside a retrace” rule would have rejected ordinary stutter chains (<the [/] the piece> [//] the people) in exactly the aphasia and fluency corpora that study them.
• E378 RetraceWithoutWords. The retraced material must contain a word at some depth. Phrased against absent WORDS rather than a present event, so <the floor on the &=laughs water> [//] stays legal while <&=sigh> [/] does not. The boundary falls out of what the model already calls a word: 0det [/] 0det dog is legal (an omitted determiner is lexical content), <xxx> [/] xxx is legal (untranscribed speech is speech), and 0 [=! snuffles] [/] ok is not.

Fixed

• A retracing marker’s position among its annotations was discarded. dog [* p:w] [/] dog and dog [/] [* p:w] dog are different claims: the first codes the error on the abandoned attempt, the second on the retrace. chatter built the identical model for both and wrote the first back as the second. 12,226 places in the corpora put an annotation immediately before a retrace marker. A second adjacent marker had nowhere to go at all, so на [//] [/] на round-tripped as на [/] на, losing a marker outright in 105 places across 46 files, 31 of them bilingual or language-impairment corpora where disfluency is the research variable.
• E704 (overlapping bullets) was silently disabled on any utterance whose content held a retrace or a group. The predicate for “does this utterance say anything timeable” recursed into neither, so two speakers’ bullets could overlap by a full second and report nothing whenever either line contained a retrace.
• E372 (nested quotation) was invisible below every container except an annotated group. “a <“b”> [/] c” is a quotation inside a retrace inside a quotation, and reported nothing.
• Per-word language metadata skipped every word inside a quotation, phonological group, sign group or retrace, in a tool whose per-word language resolution is the point. hao3 “ni3” <ma> [/] ma produced records for two words out of four.
• The re2c backend diverged from tree-sitter on marker runs. It split a run where tree-sitter folds it, dropped retrace markers on events, and never raised E377 at all, despite three doc comments saying it did. The parser-equivalence gate now covers all three.
• chatter merge dropped donor @Comment rows when the reference file had none of its own.
• Five missing CHANGELOG link references. Every release from v0.6.0 to v0.9.1 shipped a ## [X.Y.Z] heading with no matching [X.Y.Z]: definition. It renders as literal bracketed text rather than as a broken link, so the book’s link check reports zero errors and cannot see it. The version gate now requires both halves of the entry.

Changed

• Retrace::annotations is gone. Annotated retraces are AnnotatedRetrace(Box<Annotated<Retrace>>) in both content enums, parallel to the existing Group/AnnotatedGroup pair. Parser lowering is now a left fold over the marker run, one wrapper per marker, which absorbed three hand-rolled copies of the same tail.
• Adjacency is validated, not refused at parse time. Folding the offending input faithfully means it still round-trips, so a file that trips E377 stays recoverable rather than being partly discarded during recovery.
• Per-word language records no longer carry word_index, and get_word_language is removed. The records are a #[serde(transparent)] list, so a consumer reads position with enumerate(). The stored index was a second representation of that position whose documentation claimed it matched the tier-alignment domains; it cannot, because %mor excludes retraces and %pho counts them, so no single integer indexes both.
• --parser re2c is documented as reporting unreliable diagnostic positions. The lexer emits spans and the parser discards them; until that is plumbed through, the flag is for cross-checking verdicts, not for locating them.

Internal

• Design rule 3 (no _ => catch-all over the content enums) is now enforced by the compiler, through #![deny(clippy::wildcard_enum_match_arm)] added per file as each is cleaned, seven so far. A reintroduced catch-all is a compile error at the exact line, which no scalar count could be. cargo run -p talkbank-parser-tests --bin audit_content_catch_alls inventories the 24 modules still to clean.
• ContentStructure is the single owner of which content contains what, carrying WordRef and GroupRef payloads so a caller can ask not merely whether something is a container but which one. That set had been encoded independently in 18 files, and two copies disagreeing about phonological and sign groups is what let E377 escape from inside ‹...›.

0.9.1 - 2026-08-05

Validation verdicts: UNCHANGED. No rule was added, removed or altered, and no file changes its valid/invalid verdict. This release completes the library API that v0.9.0 closed the fields on, and every entry below was found by compiling a real downstream consumer against v0.9.0, which is a gate this project did not previously have.

Added

• into_vec(), take() and retain() on every collection newtype. v0.9.0 made these types’ inner fields private but shipped only the READING half of the resulting API. With no consuming accessor there was no way to move the items out, so a consumer rebuilding a content list or resegmenting a file could only clone through as_slice().to_vec(), on paths that run per utterance; and with no retain, every caller wrote take-edit-rebuild by hand, which hands a closure a &mut Vec<_> and is DerefMut under another name. Downstream, these three methods delete three helper functions and sixteen copies of one incantation.
• One owner for that API. collection_newtype_ops! now emits the accessor set for all seventeen Vec-backed newtypes. They had drifted while hand-written: into_vec was on 6 of 17, as_slice on 11, as_mut_slice on 6, so what a consumer could do depended on which type it happened to hold.
• TierContentItems and BracketedItems are re-exported from model. Both were pub but reachable only through a glob, so a consumer could not name the type to reconstruct one after its field closed.

Fixed

• A doc-comment claim that was not true. Several comments said reconstruction “goes through new, where a future invariant would be enforced”. Every one of these seventeen types also has impl From<Vec<T>> and impl Deref<Target = Vec<T>>, so new is not the only door and no invariant is enforceable on them today. Closing the fields prevents literal construction and destructuring, and nothing more. The docs now say that, and name the open question (whether From should become TryFrom) rather than implying it is already answered.
• A test whose “unique” temporary directory repeated 98% of the time. The name was the pid plus SystemTime::now(), but the pid is constant across a test binary and 19,584 of 20,000 consecutive SystemTime samples measured identical, so parallel tests shared a directory and one’s cleanup deleted the other’s file. Now a process-wide counter.

0.9.0 - 2026-08-05

Validation verdicts: CHANGED, in the permissive direction. Files that earlier versions wrongly REJECTED now parse: an unquoted @Media filename may contain dots, parentheses, interior spaces and non-ASCII characters. Nothing that used to pass now fails. A comparison over a 2,136-file stratified sample of the reference corpora reports no new error code and no count increase on any code, and no newly-failing roundtrip file.

Three rules changed with no effect on any known transcript. E767 (new) reports whitespace before the @Media comma; those files were already invalid, and what changes is the diagnostic. E768 (new) cannot be reached from a .cha file at all. E602 became E756 on empty user-defined tiers, a construct that occurs zero times in the wild corpus.

This release closes the library’s newtype surface ahead of 1.0, so it carries a lot of breaking API change and very little behaviour change.

Fixed

• @Media rejected legal media filenames. media_filename was an ASCII allowlist ([a-zA-Z0-9_-]+), so a dot, a space, a parenthesis or any non-ASCII character made the header fail to match. The failure surfaced as E330 “Missing media_type node” on a line that visibly ended in , audio, and as E525 about a header chatter had recognised perfectly well. A filename is now defined the way the format defines it, as everything up to the comma that introduces the media type, with the quoted form still available for URLs (which may contain commas). This was costing real transcription runs: a media file named in Chinese, or containing a space, could not be referenced at all.
• A %mor tier could be silently dropped on an empty user-defined tier. UserDefinedDependentTier::content was a NonEmptyString, so the model could not represent a %x tier with empty content and the two parsers disagreed about what to do with one. The state is now representable and rejected by a validation rule (E756) rather than being unrepresentable and handled twice.
• The validation cache could panic on drop inside an async runtime. This was the second half of the nesting bug fixed in 0.8.0: the first half covered the call, this one covers teardown.
• A %mor clone was a no-op, cloning a reference rather than the owning vector it was meant to copy.
• A declared speaker with no @ID was reported as undeclared. The “Speaker *X not declared in @Participants” check read the @Participants-to-@ID join rather than the @Participants header, so for a speaker declared without an @ID it asserted the opposite of the file. The missing @ID is a real fault and E522 already reported it correctly. The neighbouring “@Participants header missing or has no participants” check had the same confusion: an empty join and an absent header are different facts.
• E767 never fired in the editor. It was implemented as a file-level sweep, and the LSP calls validate_headers_only, which does not run those. Both @Media payload rules now live on the per-header dispatcher that every entry point calls, so the CLI and the editor report the same thing. The LSP’s per-speaker code lens had a quieter version of the roster bug: a speaker without an @ID got no lens while speaking.
• A spec file had been failing to load silently. E502_wor_cascade_regression.md carried a malformed title, and the loader downgraded every load failure to a warning on stderr, so it simply left the corpus unnoticed. The loader now fails closed on a spec it cannot parse, and distinguishes a spec from the prose that shares its directory.

Added

• ChatFile::declared_speakers() returns every speaker declared in @Participants, in declaration order, each enriched with its @ID metadata when present. participants is populated from the @Participants-to-@ID join, so a speaker declared without an @ID raised E522 and was then absent from the map: consumers saw fewer speakers than the file declares. Prefer this for “who is in this transcript”; all_participants() remains the @ID join.
• ChatFile::participant_entries(), the named @Participants extraction, alongside the existing id_headers().
• MorWord::analysis() borrows the analysis half of a %mor item (lemma[-Feature]*) so a consumer whose token model keeps the tag and the analysis in separate fields need not serialize the whole item and strip the POS| prefix back off. MorWord::write_chat now delegates to it, so the two renderings cannot drift.
• DependentTierEntry::kind(), span() and content_span(), the last giving the byte range of a tier’s content without its label or terminator.
• MediaFilename::parse(), unquoted(), and MediaFilenameProblem.
• E767: whitespace between the @Media filename and its comma. Reported from the validation layer so both parser front ends raise it from one implementation.
• E768: an @Media filename that cannot be written to a header and read back unchanged. Unreachable from CHAT by construction; it guards the JSON ingress, where a document can carry a value no transcript could express.
• string_newtype_read_impls!, the read and render surface shared by every string newtype, so a newtype WITH an invariant can share it instead of copying it.
• Status: unreachable_from_chat for error specs: a rule that IS implemented but that no CHAT input can trigger, so it carries no corpus fixture and owes a named out-of-corpus test instead. This closes a hole in the gate meant to stop an implemented rule shipping untested: a spec with no example used to fail to parse, and the loader turned that into a warning, so the gate never saw the one case it names. Both directions are now checked, a spec marked unreachable that carries an example is also an error.

Changed

• BREAKING: no model newtype exposes its inner field. Every newtype in the model, including every one generated by string_newtype!, now has a private field. Code reading .0 uses as_str() / as_slice() / raw(); code mutating through it uses the named accessors.
• BREAKING: DerefMut is gone from the collection newtypes. While it existed, a private field bought nothing: any caller could still push, clear or replace the contents. as_mut_slice() allows element mutation without allowing the collection to be resized.
• BREAKING: MediaHeader::new takes a MediaFilename, not impl Into<MediaFilename>, and MediaFilename has no new, no From<&str> and no From<String>. parse is the only way in. An @Media filename containing the delimiter was constructible, and build_chat built one.
• BREAKING: build_header_lines and build_media_header are fallible, and BuildChatError gains a MediaFilename variant, because a caller-supplied media name is external input that @Media cannot always represent.
• BREAKING: UserDefinedDependentTier::content is no longer a NonEmptyString.
• BREAKING: the crates are edition 2024.
• Deserialization of MediaFilename is lenient, like every other checked newtype in the model: the serde boundary reconstructs what the document held and validation reports the violation with a code and a span.

0.8.0 - 2026-08-03

Validation verdicts: UNCHANGED. No rule was added, removed or altered, and no file changes its valid/invalid verdict in this release. What changes is that the desktop app can run at all, that runs differing only in --suppress share a cache again, and the library API named under “Changed” below.

Fixed

• Chatter Desktop can validate again. Since v0.6.0 the desktop app could not start a run at all: it stopped on “Starting…” forever, on every machine and every folder. Tauri drives a command on its async runtime, and the validation cache bridges its synchronous API to an async database by owning a runtime and blocking on it; nesting runtimes panics, the panic unwound out of the command, and the IPC call then never resolved OR rejected, so the window had nothing to report and no error to show. The cache now runs such a call on a thread with no ambient runtime, so nesting cannot arise, and the desktop validate command always produces an outcome, reporting a panic as a failed run rather than as silence. The CLI was never affected. Introduced 2026-07-07; shipped in v0.6.0 and v0.7.0.

Changed

• Desktop commands return typed errors instead of String. Each command now names the failures it actually has (TargetError, ValidationStartError, ClanError, InstallCliError, RevealError, ExportError, OpenExternalError), so a failure can be matched on and carries its source error. Errors still cross the IPC boundary as the same display text, so nothing the user sees changes.

• --suppress no longer throws the validation cache away. Suppression is a presentation preference: it changes which diagnostics are printed, never which ones the validator computes. v0.6.0 folded the suppression set into the cache key, so every distinct --suppress list got its own private cache and chatter validate ~/corpus followed by chatter validate --suppress xphon ~/corpus re-validated all ~106,000 files from cold instead of hitting the cache. Runs that differ only in --suppress now share one cache; --strict-linkers, which genuinely turns extra checks on, still validates afresh. Suppression behaviour itself is unchanged: a suppressed code is not reported, and a file with other diagnostics still counts invalid.

• The cache no longer grows without bound across releases. Every read binds the current rules version, so rows written under a superseded one can never be matched again, yet nothing deleted them: only a 30-day age cutoff existed, which answers a different question. Each release therefore stranded a complete copy of the corpus in the database, which had reached 464,773 rows across 88 versions (about 190 MB of a 243 MB file) for a corpus of ~106,000 files. Opening the cache now deletes rows outside a two-generation window (the current version plus the most recently written previous one, so a rollback or a bisect is not cold), rewrites the file so the space actually returns to the filesystem, and reports what it reclaimed.

• Rule selection and presentation policy are now separate types. ValidationConfig held both “which rules run” and “how diagnostics are shown”, and the validation cache key was derived from the whole thing, which is what let a display preference partition the cache. It is replaced by talkbank_model::RuleSelection (what is computed; the only input to the cache key) and talkbank_transform::PresentationPolicy (what is shown). The cache crate cannot name the second, since the crate that owns it depends on the cache, so folding a display preference into the key is now a compile error rather than a judgement call.

  Library callers: ChatFile::validate_with_config and validate_with_alignment_and_config are now validate_with_rules and validate_with_alignment_and_rules, taking a RuleSelection, and they report the complete diagnostic set with nothing filtered. ConfigurableErrorSink moved to talkbank_transform and takes a PresentationPolicy. The validation runner’s config field model_config is now the pair rules and presentation.

0.7.0 - 2026-08-03

Validation verdicts: UNCHANGED. No rule was added, removed or altered, and no file changes its valid/invalid verdict in this release. What changes is what a run REPORTS about itself when it does not complete normally.

Changed

• A validation run now always terminates its event stream, and says how. Previously a run whose thread died emitted no terminal event at all, and the three surfaces each guessed differently: the CLI exited non-zero, the desktop app waited forever showing “Discovering files”, and the TUI marked the run COMPLETE, so a dead run was presented as a finished one. Terminality now belongs to the runner, which guarantees a terminal event on every exit path, so all three surfaces inherit the same guarantee instead of reconstructing it.

• A run that lost files can no longer report success. A panicking worker was caught, logged where no graphical user could see it, and then ignored: the run reported Finished with partial statistics, so a 500 file corpus could validate 480 and be presented as “all valid”. Finished now means every discovered file was accounted for and is the only basis for a claim about the whole input; a run that covered less reports FinishedIncomplete with the number of files lost. chatter validate exits non-zero in that case, where it previously exited 0.

• Cancelling a run now cancels it. The cancel request was a single token on a channel that the dispatch loop, every worker, and the end of run check each consumed destructively, so exactly one of them observed it: cancelling stopped one worker while the rest drained the queue, and the run’s own statistics usually recorded that it had not been cancelled.

• ValidationEvent gains Aborted and FinishedIncomplete (BREAKING for library consumers). The enum is deliberately NOT #[non_exhaustive]: a consumer that upgrades gets a compile error and has to decide what a dead or partial run means for its own interface, rather than silently inheriting “pretend it finished”, which is the defect these variants exist to fix.

Fixed

• Desktop: a run that never started looked identical to one in progress. The app showed “Discovering files” from the moment it sent the request, and the backend’s own discovery event set the same state, so a backend that never answered was indistinguishable from one still working. The two are now separate states: the app shows “Starting” until the validator actually responds, and says so if that takes more than a few seconds. A run stuck there is a start-up fault rather than anything about your files, which is worth quoting in a bug report.

• Desktop: an aborted run is no longer a dead end. It reports why it stopped and offers Re-validate, instead of leaving the window with no way forward.

0.6.0 - 2026-07-31

Validation verdicts: CHANGED. Files that earlier versions accepted may now be rejected, and files they rejected may now be accepted. Both directions occur in this release: the Phon %x fixes below remove false rejects, while the removed error codes and the suppression fix change what validate reports and what exit code it returns. Pin with ~ if you depend on a fixed rule set.

Added

• chatter fix, built on the span-splicing engine. It supersedes the deleted chatter lint: the old lint --fix is now fix --apply. fix covers the full fix catalog rather than three codes, applies fixes at exact byte spans validated against the source text, and repairs a clean utterance in a file whose other regions did not parse (the utterance containing an edit must have parsed clean, or the edit is refused and reported, never silently dropped). Every catalog entry carries a batch-safety tier and a bare --apply writes only the mechanical ones; a semantic fix is written only when its code is named with --code; an ambiguous fix is only ever reported, never written by this command.

Removed

• chatter lint (the --fix auto-fixer) is deleted. It was a live span-driven byte writer built before the splice engine’s safety guarantees existed: it read error.location.span with no dummy-span guard (Span::DUMMY is {0,0}, a real file offset), called String::replace_range with no is_char_boundary check (a panic on a non-character-boundary span), inserted its E301 terminator fix at zero width with no dummy-span guard either (corrupting the @UTF8 header had one ever fired at offset 0), and detected no overlap between fixes. An audit found zero production callers (no talkbank-tools reference, no workspace script, no IISRP pipeline usage; only its own tests and the book mentioned it). chatter fix is its successor; see Added above.

• Five ErrorCode variants, each unreachable or redundant: LongFeatureLabelMismatch, NonvocalLabelMismatch, UnexpectedTierNode, UnexpectedMorphologyNode, and LegacyWarning (with its generated spec entry). Consumers matching on ErrorCode will see these gone.

• Twelve of the language server’s twenty-one quick fixes. Each was attached to a code it did not repair: the action offered for a duplicate header inserted a missing one, and eleven others were similarly mismatched. The nine that remain repair the diagnostic they are attached to. Quick-fix matching now goes through parsed error codes rather than string literals, so a renamed code is a compile error instead of a silently dead action.

Fixed

• Phon %x dependent tiers, reconciled against the upstream spec. Three fixes, two of which were false rejects on valid Phon output:

  • %xphoaln no longer requires its word count to equal %mod/%pho exactly. The spec allows a pause present on only one of the two tiers to consume a word slot only on the tier that contains it, so the counts legitimately differ by one.
  • Numeric inter-word pauses ((1.5), (1:05.2)) are accepted on the syllabification tiers, alongside the three untimed forms. They were rejected on the grounds of being unattested in available corpora, which is not a basis for refusing a construct the spec declares legal.
  • Intra-word pauses (^, U+005E) are tokenized rather than absorbed into the neighbouring phone. A word-final ^ previously produced a spurious error, and a mid-word ^ silently became part of the following phone. Reconstruction preserves the pause in place, per the spec’s rule that stripping each unit’s :CODE and concatenating must reproduce the source word exactly.

• validate --suppress no longer zeroes the invalid count and the exit code. Suppressing a code removed it from the report AND from the tallies, so a file with genuine OTHER errors could be counted valid and the command could exit 0. A file that still has unsuppressed diagnostics now counts invalid and the command exits non-zero, as it should. A file whose every diagnostic was suppressed does count valid: that is what asking for those codes to be suppressed means.

• The validation cache key covers every dimension of the verdict, including parse behaviour and the active rule set, as a required parameter rather than a hand-picked subset. A cached verdict from one configuration could previously be served for another.

• Strict parsing no longer discards the model it built on failure, so a caller can inspect what parsed alongside the diagnostics.

Changed

• Diagnostic classification happens once, from the active rule set, rather than being recomputed at three call sites that could disagree.

• The diagnostic kind is generated from the spec instead of mirrored in a hand-maintained match, and a divergence between the spec and the ErrorCode enum now fails the build in both directions rather than falling through to a default.

0.5.1 - 2026-07-30

Fixed

• validate --force was unusable at corpus scale (v0.5.0 DOA): the cache refresh called clear_prefix once per resolved FILE, and each call scanned every file_path in the cache, so a corpus-sized invocation did quadratic work (on a 136k-file cache, effectively forever) at 100% CPU behind a blank screen before the progress display started. The refresh is now one batched DELETE ... IN (...) pass over the resolved file list, and clear_prefix itself became a single range-predicate statement instead of a scan-and-loop. Pinned by a real-CLI regression test that warms a 6,000-file cache and bounds the forced pass (old: 34s at that size; new: seconds, dominated by validation itself).

0.5.0 - 2026-07-30

Removed

• Two ungrounded CA-mode validation exemptions. @Options: CA no longer disables the E241 illegal-untranscribed checks, nor the E701/E704 temporal checks (which it had skipped wholesale via an early return, while E362 bullet monotonicity kept running on the same files). Neither skip had a CLAN CHECK counterpart: CHECK’s whole CA behavior is three suppressed errors (21 terminator, 155 parenthesized word, 123 leading space), and chatter keeps exactly those three. Measured before removal on ALL 994 kept CA-declared files: both gates protected zero occurrences. The temporal skip’s recorded rationale (leniency-policy Decision 6, false positives on CA reference files) no longer reproduces, since the temporal rules gained the 500 ms tolerance and per-speaker semantics; its Revisit line anticipated this removal. CA files with genuine timing defects or illegal untranscribed markers are now diagnosed like any other file.

Fixed

• E326 now says when the skipped line looks like a CHAT line pushed off column 1. An indented dependent tier ( %mor: ...) was reported as “Unsupported line skipped”, accurate but useless: the reader hunts for junk when the fix is deleting one space. The message now names the shape (“looks like a dependent tier line pushed off column 1; it must begin at column 1”) for tier-, main-tier-, and header-shaped lines, with a suggestion to remove the leading whitespace.

• An annotated word’s wrapper span was never set, left Span::DUMMY at construction while the annotated event, action, and group paths all set a real one. Two consequences: any diagnostic located on an annotated word pointed at byte zero, and E757 could not see a bracketed code glued to the following word (hello [!]there) at all, because its detection is span adjacency. The wrapper now spans the word through the enclosing node’s end, covering its trailing [...] codes, exactly as the retrace paths do.

Changed

• E757 now covers every bracketed code, not only retraces. hello [/]x was rejected; hello [!]x and bobo [= toy]x were silently accepted, though they are the same defect and the code’s own description already said “bracketed code”. Juxtaposition-matrix cell 8, ruled REJECT 2026-07-18. Mirrored in the re2c front end, where a bare closing bracket joins the retrace tokens. The 2026-07-18 matrix scan found ][letter unattested corpus-wide and the differential confirms no new instances, so no kept file is affected.

Added

• talkbank-transform gained a default-on validation-runner feature. The corpus-scale validation runner is the crate’s only SQL consumer (sqlx, via talkbank-cache), so it, that dependency, and the runner-only crossbeam-channel/num_cpus now sit behind the feature. Default builds are unchanged; a consumer that wants the transform surface without a SQL stack opts out with default-features = false. The path predicate is_chat_transcript_path moved to the feature-independent talkbank_transform::paths (still re-exported from validation_runner), since the corpus walk and CLI walks need it on every build.

• E766, a linker placed after utterance content (yeah that go +" okay .). Linkers connect an utterance to the previous one, so they are utterance-initial by definition; a misplaced one used to surface as generic unparsable content (E316), which gave the transcriber nothing to act on. The grammar now parses the misplaced linker into the CST (the same strict+catch-all pattern as the curly-quote rule) so the diagnostic names the construct at the exact token, in both parser front ends. One deliberate carve-out: a ++ glued to words on both sides (un++do) is a word run with an empty compound part and keeps its E233 diagnosis. A side effect of the grammar change is finer error recovery on several unparsable-content shapes: diagnostics that used to blame a whole line now land on the exact offending region (e.g. an unmatched < now yields E316 on <word with the rest of the utterance parsed normally).

• E765, a free-standing : or ; separator, or a pause, glued to the item after it (:and, ;;, (.)dog). Same family and same span-adjacency mechanism as E764; the preceding side stays valid, since word↘ and dog, are documented convention and dog: fuses into the word.

  Juxtaposition-matrix cell 7 was ruled REJECT for the whole separator class, against an estimate of roughly six affected files. A real-corpus comparison measured that reading at 270 new instances on a 2%, 2,134-file sample (about 13,500 corpus-wide), every inspected one legitimate CA notation rather than a missing space: ≡ is latching and is written glued on both sides, and the intonation arrows attach to the material they mark, including directly before an overlap close. Adjudicated UNINTENDED, so the rule ships narrowed to the plain punctuation separators and pauses, where the differential is clean. Whether any CA mark should forbid trailing glue is left open, with receipts in the spec.

• E764, a &-prefixed form glued to the preceding word (dog&-um, dog&~gaga, dog&+fr). The shape parses as two words, because & cannot continue a word, so a missing space silently manufactures a word boundary that the transcriber did not write and nothing reported. Style rule in the E749/E751/E757 family, detected by span adjacency, mirrored in the re2c front end as a token scan. Glued omission (dog0is) is not this code: it yields one malformed word and E220 already rejects it.

  Juxtaposition-matrix cell 6, ruled REJECT 2026-07-18; zero main-tier attestations in the kept corpus at adoption, so no existing file is affected. Validator-only: grammar, model shape, and roundtrip behavior are untouched.

0.4.1 - 2026-07-27

Fixed

• talkbank_transform::dependent_tiers::replace_or_add_tier could not be called on an utterance. It still took SmallVec<[DependentTier; 3]> after DependentTierEntry was introduced and Utterance::dependent_tiers became SmallVec<[DependentTierEntry; 3]>, so the one thing the helper exists to do no longer type-checked. It shipped in this state in 0.3.6 and 0.4.0.

  It compiled because it was internally consistent, and no test caught it because this workspace has no callers of it: the helper is public API for downstream consumers, and the only one was pinned to an older release. The regression guard added with the fix is a compile-time function taking &mut Utterance, so any future drift between the signature and the field fails the build rather than passing silently.

  On replace, the existing entry’s TierSeparator is preserved (only the payload is regenerated, and the separator is the provenance E758 is detected from); on append the new entry is CLEAN. Serialization canonicalizes to a single tab either way, so this affects diagnostics, not output.

0.4.0 - 2026-07-27

Added

• Three validation rules that catch real, previously-invisible defects in transcript data. All three live entirely in the validator: the grammar, the model’s serialized shape, and roundtrip behavior are untouched.

  • E761, %gra relation head is not a Universal Dependencies relation. A %gra label is HEAD or HEAD-SUBTYPE; UD fixes the head set at 37 universal relations and leaves subtypes open and language-specific, so the head is checked against that closed set and the subtype is never checked. Nothing validated relation labels before, in chatter or in CLAN CHECK, so a corrupted label rode silently into every downstream analysis that reads the dependency graph. Grounded in a survey of the entire corpus (138,565,864 relation instances across 106,158 files): all 37 universal heads are attested, 150 distinct labels occur, and exactly three heads fall outside the set, all of them defects (IOB for IOBJ, PAD, PUNCTT for PUNCT).

  • E762, the prefix marker # stands alone as a word or opens one. The marker attaches to the END of the prefix it marks, and the prefix is a word of its own (Hebrew ha# kelev), so neither shape can be that construct in any language. Language-independent, and zero-attested corpus-wide.

  • E763, prefix marker in a language that does not use it. Gated on the WORD’s resolved language rather than the file’s @Languages header, exactly as the digits rule (E220) is, so a code-switched word brings its own rules with it. Languages that write the marker: heb, ara. Word-internal markers stay legal wherever the language allows the marker at all.

• TreeSitterParser now implements the shared ChatParser trait (talkbank_model::ChatParser), making the two parser backends interchangeable behind one generic bound at every granularity (file, header, utterance, tier, word, relation). Previously only Re2cParser implemented the trait, so consumers selecting a backend per target (tree-sitter natively, pure-Rust re2c on wasm) had to hand-roll a cfg-gated facade. Every trait method delegates to the matching inherent parse_*_fragment method, so trait-path and inherent-path behavior are identical; conformance is pinned by talkbank-parser/tests/chat_parser_trait.rs.

• Dedicated error codes for two malformations that previously fell through to the generic E316 unparsable-content catch-all, from the CHECK-parity adjudication of CLAN CHECK errors 52 and 11: E759 (an utterance beginning with a postfix annotation such as [/], [<], or [: text], which has no preceding material to scope over) and E760 (a %mor item with an empty part-of-speech field, |we). Both are recognized by the tree-sitter front end’s error analysis and mirrored in the re2c oracle’s front end; both files were already rejected, so no validity verdict changes, only the diagnosis.

Removed

• TalkBank XML support, in full. The to-xml command, the talkbank_transform::xml emitter, the corpus/reference-xml/ golden corpus, the xml_golden and xml_schema_validate suites, the bundled talkbank.xsd/xml.xsd schemas, the XML Emitter book chapter, and the quick-xml dependency.

  TalkBank stopped generating TalkBank XML on 2025-10-29, when its last consumer said he no longer used it, and the published data-xml/ distribution has been offline since. Phon moved off the format some time ago. Nothing produced by this emitter had a consumer.

  Breaking: chatter to-xml no longer exists and there is no replacement. Use chatter to-json, which is the format the toolchain actually maintains. talkbank_transform::xml::XmlWriteError is gone from the public API surface.

Changed

• The %gra documentation, examples, reference corpus, and error-spec fixtures no longer use retired TalkBank relation labels (SUBJ, JCT, POBJ, COM, VOC, MOD, NEG, PRED, COMP, ADV, INCROOT, QUANT, LINK), which E761 now rejects. None of them occurs anywhere in the real corpora; they were fixture inventions that would have taught readers of the API docs a vocabulary the validator rejects. Replaced throughout by the UD relations the corpora actually use (NSUBJ, OBL, CASE, DISCOURSE, VOCATIVE, AMOD, ADVMOD-NEG, CCOMP, EXPL, DET, DEP). The gra_incroot grammar construct deliberately keeps INCROOT: it pins the property that relation labels are open text at the grammar layer, which is why the vocabulary is a validation policy and not a syntax.

Fixed

• Word validation now reaches words nested inside groups. Main-tier validation iterated content items flatly and matched only Word, AnnotatedWord and ReplacedWord, with a catch-all that silently discarded every container, so a word inside a retrace, a reformulation, an angle group or a quotation was never word-validated at all.

  The symptom: the identical token was rejected outside a group and accepted inside one. In English hello3 dog . was invalid (E220) while hello3 [/] hello dog . was valid, on every release up to this one. Every word-level rule inherited the hole, so E220 has carried it for as long as the rule has existed; the newer prefix-marker rules inherited it on arrival.

  Corpus impact, measured over all 106,158 files: 341 to 348 invalid files, 8 new error instances across 7 files (E241 x2, E252 x4, E248 x1, E763 x1), each a pre-existing data defect that had been hiding inside a group rather than any change in what counts as valid CHAT.

• ErrorCollector::is_empty() violated the standard Rust contract len() == 0 <=> is_empty(): it answered “is the internal buffer unallocated?”, so a collector created with with_capacity (which pre-allocates) reported non-empty while holding zero errors. Found by the 1.0 contract-set API audit; now implemented as len() == 0 with a regression test.

• TreeSitterParser::parse_gra_relation_fragment (and the trait’s parse_gra_relation) rejected EVERY bare %gra relation and leaked a spurious E709 diagnostic into the caller’s sink, because the wrapper appended a scaffold terminator with the never-valid index 0 (0|0|PUNCT) and the tier wrapper rejects on any internal diagnostic. The scaffold is now valid CHAT (2|1|PUNCT), and a scaffold-region filter guarantees diagnostics against wrapper scaffolding can never reach the caller. The re2c backend was unaffected (it parses the relation directly); the fix restores backend agreement. Caught by the new ChatParser trait conformance test.

• Validation cache: initialization is now concurrency-safe across processes, not just threads. Every opener takes an exclusive advisory file lock (talkbank-cache.init.lock, beside the database) around first-time create + migrate, so parallel chatter runs (or parallel test processes) sharing one cache directory can no longer race sqlx’s SQLite migration (UNIQUE constraint failed: _sqlx_migrations.version, the 2026-07-13 flake) or collide on first-connection WAL setup. Lock acquisition is bounded: on timeout, opening fails with the new typed CacheError::InitLockTimeout and the CLI degrades to running uncached instead of blocking. The 2026-07-13 bounded retry is retained as a backstop for older builds that share the cache directory without honoring the lock protocol. Regression coverage: a cross-process stress test races 8 processes over a fresh cache directory for 4 rounds under a hard deadline, so both failure modes (constraint error and hang) fail the suite instead of flaking or wedging it.

• Desktop release: the macOS updater bundle is now uploaded under a per-arch asset name (Chatter-<target>.app.tar.gz). Previously both the aarch64 and x86_64 macOS jobs uploaded the arch-independent Chatter.app.tar.gz, which raced on the shared release asset (the v0.3.6 release-desktop upload failure) and pointed both darwin entries in latest.json at a single URL holding one arch’s binary. Fresh .dmg downloads were unaffected; the desktop auto-updater is the surface this corrects. (Ships with the next release.)

• Public API: a downstream crate that depends only on talkbank-parser can now name the error type of its parse methods. The six TreeSitterParser::parse_* methods return ParseResult<T> = Result<T, ParseErrors>, but ParseErrors / ParseResult were not reachable from the talkbank-parser crate root (only via a pub(crate) module), forcing consumers to add a separate talkbank-model dependency or stringify at the boundary; both are now re-exported. Also re-exported talkbank_model::SylWordError (the error of classify_syl_word / tokenize_syl_word), which was omitted from the model root while its sibling phon parse-error types were present. Completes the BUG-3 audit: a compile-test now names every public fallible constructor’s error type so this class cannot regress.

0.3.6 - 2026-07-17

Fixed

• The Phon %x-tier content checks (introduced with the %x fold-in) no longer mass-flag valid Phon exports. Two wild-corpus conventions the original specification never confronted are now accepted: (1) pause fillers ((.), (..), (...)) mirrored at the same word position on %mod/%pho/%xmodsyl/%xphosyl (and as pause pairs on %xphoaln) to keep word-aligned tiers in index lockstep, which E735 previously rejected as malformed phone:CODE units (roughly 13,000 spurious errors across the PhonBank corpora); and (2) ^ and IPA . syllable-boundary notation in %mod/%pho words, which the segment-level %xphoaln reconstruction comparison now ignores exactly as it already ignored stress markers (roughly 770 spurious E740/E741). Genuine misalignments (index-shift chains, pause fillers standing in for real words) are still reported. Users who adopted --suppress xphon to silence the storm can remove it and regain the genuine %x-tier checks.
• Generated error-documentation pages (docs/errors/) no longer fuse words across wrapped spec lines or drop backticked text: the spec text extractor now renders soft line breaks as spaces and includes inline code spans.

Added

• New validation rule E752: timing bullets without an @Media header. A transcript carrying timing evidence (utterance bullets or %wor word timing) must declare the media those timestamps index; completes the media-consistency family (E544: declared linkage without timing; E552: declared unlinked contradicted by timing). Mirrors CLAN CHECK error 112.

• New validation rule E753: a word consisting only of a repetition segment (fully ↫...↫-wrapped, no stem outside the delimiters) is rejected; word-category prefixes (&- filler, &~ nonword, 0 omission) count as a stem. Adopted from GUI CLAN CHECK error 151 as a chatter-authority rule (the unix CHECK build never enforced it).

• New validation rule E754: the @l letter form must carry exactly one letter of stem (b@l); multi-letter content belongs under @k / @ls. Repeated-segment material (↫b^↫b@l) does not count toward the stem, matching real CLAN CHECK behavior. Mirrors CLAN CHECK error 76.

• New validation rule E755: a [- CODE] utterance-level language must be declared in @Languages (utterance-level presence is substantial). Mirrors CLAN CHECK error 152.

• Word-level explicit language codes (word@s:CODE) are now validated against the ISO 639-3 registry (E519), the same rule that guards @Languages and @ID; declaration in @Languages remains not required.

• @L1 of values are now typed ISO 639-3 language codes and validated against the registry (E519), completing registry validation at every position language codes appear. Wild usage was already uniformly codes; generation via build_chat now takes a LanguageCode for the participant first language.

• E756 (empty user-defined %x tier) replaces W601: the rejection is unchanged; the old code fired as a hard error despite its warning prefix, so the number was the bug. The diagnostic message also no longer double-prefixes the tier name (%xfoo, not %xxfoo).

Removed

• The E254 warning (word-level @s:CODE not listed in @Languages) is retired: an explicit word-level language code is self-contained and deliberately carries no declaration requirement. @Languages declares the transcript’s substantial languages; a one-word insertion is not substantial presence. (This matches CLAN CHECK, which dropped its own @s declaration requirement in 2019.)

0.3.5 - 2026-07-15

Emergency release restoring corpus-correct word parsing. Versions 0.3.3 and 0.3.4 have been YANKED (releases and tags removed).

Fixed

• Reverted the whitespace-boundary overlap-custody grammar introduced in 0.3.3. Its GLR-arbitrated word readings fragmented words carrying four or more glued markers (for example multi-syllable-pause chains like or^ga^ni^zi^ra), causing spurious E252/E331/E600/E705 validation errors across real corpora and, worse, a serialization mutation (a space inserted into such words on rewrite). Word parsing is restored to the 0.3.2 grammar, verified by an error-code differential and a roundtrip comparison against the 0.3.2 binary over a corpus sample: identical profiles.
• A regression test pins that multi-marker words parse as one word and validate cleanly.

Retained from the yanked releases

• Typed @u phonetic word forms (UNIBET).
• The build_chat header emitters and @ID demographics fix.
• The shared English capitalization transform.
• The long-tier stack-overflow fix and its regression test.
• The SQLite cache concurrency-safety fix; CI runs under nextest.

0.3.4 - 2026-07-15 [YANKED]

Added

• @u phonetic forms are now typed phonetic content. A @u word (a UNIBET/IPA phonetic transcription standing in a word slot, e.g. the spoken side of an aphasia [: target] replacement) now models its content as a dedicated WordContent::Phonetic(WordPhonetic) node instead of orthographic text, in both parsers. Orthographic word-hygiene rules structurally cannot apply to phonetic content; the phonetic string itself stays deliberately lenient (IPA, ASCII UNIBET, X-SAMPA), matching the %pho tier’s stance. to-json emits {"type": "phonetic", ...} for these nodes (schema updated); cleaned_text remains the phonetic string verbatim; the sanitizer redacts phonetic forms like spoken text. Scope is @u only; sibling special forms remain orthographic words.

• build_chat now emits the full standard header set. The general CHAT-generation schema (TranscriptDescription / ParticipantDesc) gained typed optional fields for @Date, @Situation, @Options, @Transcriber, @Comment, per-speaker @L1 of, and @PID (preserved from a source, never minted), each emitted in canonical header order. @ID demographics (age, sex, group, SES, education, custom) are now carried through ParticipantDesc instead of being silently dropped, fixing empty demographic slots in generated @ID headers.

• Shared English capitalization transform (talkbank_transform::capitalize): capitalizes the pronoun “I” family and the first real word of each utterance on the typed model, for generators whose sources are all-lowercase (improves downstream %mor accuracy). Token-level helpers are public for generators that capitalize their own word representation.

Fixed

• chatter validate no longer headlines a warnings-only file as an error. A file whose findings are all warnings (which is valid CHAT, and was already counted valid in the summary) now prints ⚠ Warnings in <file> instead of the contradictory ✗ Errors found in <file>, and the “fix structural errors first” hint fires only on hard errors. Presentation only; validation logic unchanged.
• The validation cache no longer fails to initialize when opened concurrently. Two chatter runs sharing a cache directory (or a multi-threaded consumer) could race the one-time SQLite setup and hit UNIQUE constraint failed: _sqlx_migrations.version or a WAL init collision, silently disabling caching for that run. Concurrent opens on a fresh cache directory now retry the transient init race and all succeed.

0.3.3 - 2026-07-13 [YANKED]

Added

• Desktop app: a “Check for Updates…” menu item and a periodic background update check. The app previously checked for a new release only at launch, so an app that was rarely relaunched could sit far behind. It now also checks every six hours in the background, and the app menu has a manual “Check for Updates…” item that reports when you are already up to date.
• Desktop app: a real “About Chatter” panel with the version, a short description, and clickable links to the TalkBank site and the source repository, replacing the bare version-only default.
• talkbank_transform::build_chat: assemble a validated CHAT file from a typed transcript description. Given participants, optional media, and utterances as pre-formatted CHAT main-tier text (TranscriptDescription), it synthesizes the header block, parses each utterance through the tree-sitter parser, and returns a ChatFile. The description carries a media_status, so a transcript that names its media but has no timing bullets yet (pre-forced-alignment) can emit @Media: <id>, audio, unlinked and stay valid instead of falsely claiming linkage (E544).
• talkbank_transform::num_words::expand_number: spell digit tokens as language-appropriate number words (13 lookup-table languages, CJK, and English ordinals/decades), so generated CHAT satisfies E220 (numeric digits are not allowed in words for languages that do not permit them).

Changed

• Overlap custody now follows whitespace boundaries, with canonical overlap serialization. Overlap markers bind to the token on the correct side of a whitespace boundary, and serialization emits a single canonical form.
• tree-sitter updated to 0.26.11 across the workspace (CLI, grammar bindings, and the generated parser).

Fixed

• Long dependent-tier reconstruction is now linear-time. A quadratic blowup on very long utterance tiers is eliminated; pathological inputs that previously stalled the parser now reconstruct in linear time.
• Desktop app: the validation settings popover no longer opens hidden behind the results panel. It was rendered below the panels in the stacking order; it now sits above them.
• Desktop app: the “up to date” dialog now dismisses on the first OK. A listener leak (an async menu subscription whose cleanup could run before it resolved) let duplicate listeners accumulate, so one menu click stacked several identical dialogs.

0.3.2 - 2026-07-10

Added

• chatter rediarize: repair speaker attribution from external diarization turns. Takes a transcript whose utterance timing is trusted but whose speaker labels are not, plus a speaker-turns JSON file ({"source": ..., "turns": [{"track", "start_ms", "end_ms"}]}) from an external diarizer, and re-attributes each timed utterance to the dominant overlapping turn. Utterances with no turn coverage are flagged, never guessed. Reconciled @ID rows are inserted in the header block. --summary-json emits a machine-readable outcome summary (per-utterance reattributions and flag reasons) for downstream tooling.
• Four validation rules for constructs that do not make sense, each adjudicated against real CLAN CHECK behavior and the wild corpus: E748 leading-zero media-bullet times; E749 comma glued to the following word; E750 whitespace inside angle-group delimiters; E751 pause marker glued to a word.

Fixed

• The re2c oracle lexer now tokenizes short-form parenthesized material the same way the canonical parser does (its catch-all previously swallowed a trailing delimiter), keeping the two independent parsers in cross-check agreement on the new spacing rules.

Changed

• Rust toolchain pin bumped to 1.97.0 (CI workflow pins synced); workspace and spec lockfiles refreshed; desktop dependency bumps (jsonschema 0.47, TypeScript 7).
• Documentation: an architecture page on overlap-marker binding (why edge-adjacent overlap markers bind into words, the ideal top-level model, and the conversion-layer path); the grammar’s empty-extras (all-whitespace-explicit) design rationale is now recorded at the declaration site.

0.3.1 - 2026-07-08

Fixed

• Every public fallible constructor’s error type is now publicly nameable. LanguageCodeError (from LanguageCode::new), XphointParseError, and PhoalnParseError were not re-exported, so downstream crates could not store them in typed #[source] fields and had to stringify at the boundary; found by the first real downstream consumption of the 0.3.0 API. A new API-surface guard test pins the contract so a constructor error type can never silently become unnameable again.

0.3.0 - 2026-07-07

Added

• --llm-cache <file> (env CHATTER_LLM_CACHE) for holistic speaker-id judgment. A persistent, write-through JSON response cache for speaker-id / pipeline / batch --judgment holistic: an identical request (same endpoint, model, and rendered prompt) is served from the cache instead of making another LLM call, so re-running a batch after a crash or an unrelated code change does not re-pay completed sessions. Absent flag and env variable means uncached, unchanged from before.

Fixed

• chatter batch no longer reports holistic suggestions as merges. In holistic-judgment mode the per-session pipeline exits 0 after writing a suggestion to the pending file without merging (the operator adjudicates first); the batch summary counted those as “merged” and reported zero pending work. Outcomes are now classified by whether the merged output actually exists, and the summary separately counts merges, suggestions awaiting adjudication, and low-confidence refusals awaiting adjudication.
• E552 (@Media says unlinked but timing exists) now says where the timing was found and how to fix it. When the only timing evidence is word-level bullets inside a %wor tier (invisible in normal display), the message names the %wor tier and offers both remedies (the media is in fact aligned: remove unlinked; or the %wor tier is stale: remove it) instead of asserting the media is linked and pointing at bullets the user cannot see. The main-tier-bullet case keeps its direct advice.
• Chatter Desktop’s single-file validation now shares the CLI’s validation engine. Previously, validating a single .cha file in the desktop app (as opposed to its parent folder) bypassed the on-disk cache entirely, skipped the @Media-filename check (E531), and could not honor --roundtrip / --parser / --strict-linkers. All of these now work identically to chatter validate and to the desktop’s own folder validation, and a new Settings panel exposes the equivalent options.
• Chatter Desktop no longer shows “N files, all valid” before a run has actually finished. The file tree previously derived this message from the partial, still-streaming result set, so it could flash “all valid” mid-run whenever no error had streamed in yet.

0.2.1 - 2026-06-24

Added

• The talkbank-lsp language server now ships as a standalone release artifact. Prebuilt, code-signed talkbank-lsp binaries for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64, static musl), and Windows are attached to the GitHub Release, each with its own talkbank-lsp-installer.sh / talkbank-lsp-installer.ps1. Any LSP-aware editor can now install the server without building it from source; it is a first-class artifact in its own right, not only the binary the VS Code extension bundles per platform.

0.2.0 - 2026-06-23

Added

• More of CLAN CHECK’s invalidity is now enforced. A batch of CHECK-parity rules was implemented so chatter validate rejects more invalid CHAT:
  • E514: an @ID line’s corpus field is required (CHECK 63).
  • E547: a constant participant header must follow the @ID block.
  • E548: closes the case CHECK 126 covers.
  • E549: a speaker may not be declared twice (CHECK 13).
  • Duplicate @ID lines and out-of-order @Options fields (CHECK 13, 125).
  • A dependent tier used without being declared (CHECK 17).
  • An out-of-range @Time Duration (CHECK 35).
  • An @Media header marked unlinked while the transcript still carries timing bullets (CHECK 124), and an @Media filename that does not match the data file (CHECK 157).
  • A replacement [: ...] now requires a preceding space (CHECK 161).
  • Tree-sitter recovery nodes are surfaced as invalidity rather than silently repaired: a surviving ERROR node maps to E316 and a MISSING node to E342 (with the re2c oracle mirroring it), covering a group with no annotation and swallowed recovery nodes inside comma-list headers (CHECK 5/6/106/108).
• Phon: U (unknown) is accepted as a legal syllable-constituent code on the %xmodsyl and %xphosyl tiers.
• A formal behavioral CHECK-validity parity test suite that runs real CLAN CHECK and chatter on the same fixtures and fails if either side drifts.

Changed

• chatter update now self-updates in process. It embeds the axoupdater self-updater as a library, reads the cargo-dist install receipt (keyed by the package name), and replaces the running binary from GitHub Releases. This removes the package-name coupling that previously made chatter update report “not installed” on a correctly installed binary.

• The CLI package is renamed talkbank-cli to chatter (the crate now lives at crates/chatter/). The generated install scripts are therefore chatter-installer.sh and chatter-installer.ps1 (previously talkbank-cli-installer.*); update any pinned install URL accordingly. The binary is still chatter, and the library/API crates keep their talkbank-* names.

• Validation is stricter. Because of the new CHECK-parity rules above, some files that passed chatter validate under 0.1.1 may now report errors. This is intended: chatter is the CHAT-validity authority and is at least as strict as CLAN CHECK.

• Word-level explicit language codes (word@s:CODE) are now validated against the ISO 639-3 registry (E519), the same rule that guards @Languages and @ID; declaration in @Languages remains not required.

Removed

• The standalone self-updater binary (cargo-dist install-updater = false). The chatter update subcommand is unchanged for users; it now updates in process instead of shelling out to a separate program.

Fixed

• The recovery-node invalidity backstop is scoped to localized errors so it does not over-flag, and several malformed @ID test fixtures were corrected.
• Hardened the CHECK-parity audit and corrected a CHECK 126 verdict it had falsely certified; the curated CHECK error-code map is restored in place of a brittle keyword heuristic.

0.1.1 - 2026-06-22

Fixed

• Validation cache could serve a stale verdict across rule-set changes. chatter validate keyed its result cache on the cache crate’s package version, which does not change when validation rules change, so a “Valid” result cached before a new rule (such as a retrace-marker check) existed kept being served, while a fresh conversion of the same bytes correctly rejected them. The cache key now folds in a fingerprint over every error-code rule, so adding, removing, or renaming any rule invalidates stale entries; the cache is kept and still functions, only keyed correctly.
• CLI usage lines pin the binary name to chatter regardless of the invoked path (clap bin_name).
• The book renders Mermaid diagrams again (restored mdbook-mermaid assets).
• Desktop app version is now locked to the release version. The desktop bundle (.dmg / .exe / .deb) and the Tauri auto-updater manifest now report the same version as the CLI. A version-sync gate (scripts/sync-app-version.py, enforced in CI and at release time) keeps tauri.conf.json, package.json, the workspace version, and this changelog from drifting, so the updater can never again advertise a version the installed bundle does not match.

Changed

• CI book toolchain bumped to mdBook 0.5.3 and mdbook-mermaid 0.17.0.
• Build: force serialize-javascript >= 7.0.5 to clear advisories, and bump rand in the spec crate.
• Docs: the book intro is de-staged for the public release (download-first).

0.1.0 - 2026-06-15

First public release.

Added

• CHAT-format core. A strict, incremental tree-sitter parser (talkbank-parser) with an independent re2c oracle parser (talkbank-parser-re2c) that cross-checks it on every file; a typed CHAT data model with structured validation, error codes, and tier alignment (talkbank-model); and CHAT-to-JSON / JSON-to-CHAT / XML conversion, normalization, transcript-merge, and redaction pipelines (talkbank-transform).
• Phon extension tiers. The four Phon %x dependent tiers (%xmodsyl, %xphosyl, %xphoaln, %xphoint) are parsed and validated as first-class CHAT tiers, on by default (pass --suppress xphon to opt out): syllabification constituent codes and phone-vs-source reconstruction, model-to-actual phone alignment, and per-phone time intervals, with dedicated error codes.
• chatter CLI. validate, normalize, to-json / from-json / to-xml, merge, speaker-id, batch, pipeline, adjudicate, sanity-scan, lint, clean, watch, new-file, show-alignment, validate-utseg, schema, update, and a content cache.
• Language server (talkbank-lsp): real-time validation, hover, go-to-definition, and cross-tier alignment for any LSP-aware editor.
• Desktop app (Chatter): a Tauri-based CHAT validation app, shipping in the coordinated release alongside the CLI.
• Auto-update. The chatter CLI self-updates with chatter update (the bundled cargo-dist / axoupdater self-updater), and the desktop app checks for and installs new releases on launch (Tauri updater). Both pull from GitHub Releases. The CLI self-updater is experimental.
• Prebuilt binaries for macOS (Apple Silicon and Intel), Linux, and Windows, plus desktop installers, attached to the GitHub Release. The macOS desktop .dmg is signed and notarized.

Known limitations

• The merge and adjudication surface is experimental. merge, adjudicate, speaker-id, and sanity-scan work, but their interfaces and heuristics may change before 1.0.
• Windows binaries are not code-signed yet, so Windows SmartScreen warns on first run (choose “More info” then “Run anyway”). macOS CLI binaries are codesigned but not notarized; install via the release installer script to avoid the Gatekeeper quarantine prompt.
• Not on crates.io yet. crates.io publication is deferred.

This page last changed: 2026-06-24 (commit 34abe802). The whole book last changed: 2026-09-15 (commit bb4bef82).

Installation

Status: Current Last modified: 2026-08-30 14:11 EDT

chatter targets Windows, macOS, and Linux. There are two ways to install it: the prebuilt binaries (recommended for most people, including clinicians and researchers) and a from-source build (for contributors or unsupported platforms).

Prebuilt binaries (recommended)

Every GitHub Release attaches prebuilt binaries for macOS (Apple Silicon and Intel), Linux (x86_64 and ARM64), and Windows (x64), plus desktop-app installers.

chatter CLI

One-line installers (they download the binary for your platform and place it on your PATH):

• macOS and Linux:

  curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.sh | sh
  

• Windows (PowerShell):

  powershell -ExecutionPolicy Bypass -c "irm https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.ps1 | iex"
  

On Windows the binary is not yet code-signed, so SmartScreen may warn on first run: choose More info, then Run anyway. The macOS binaries are codesigned, and the installer above does not set the quarantine attribute, so Gatekeeper does not prompt.

Prefer a manual download? Grab the archive for your platform from the latest release and extract chatter onto your PATH. (On macOS, a browser-downloaded archive is quarantined; right-click the binary and choose Open once, or run xattr -d com.apple.quarantine ./chatter.)

Verify:

chatter --version
chatter --help

chatter desktop app

The desktop app (“Chatter”) is for people who prefer a window to a terminal. Download the installer for your platform from the latest release:

• macOS: the .dmg is signed and notarized; open it and drag the app to Applications. No Gatekeeper override is required.
• Windows: the installer is not yet signed (same SmartScreen note as above: More info then Run anyway).
• Linux: an AppImage and a .deb are provided.

Updating chatter

chatter keeps itself current so you do not have to track releases by hand.

• CLI: run

  chatter update
  

  This self-update runs in-process: chatter update embeds axoupdater as a library and downloads the newest release from GitHub Releases directly, without a separate bundled chatter-update program. (The self-update facility is experimental and works the same way regardless of how you installed the CLI.)

• Desktop app: the app checks for updates on launch and offers to install a new version when one is available.

From source

Building from source needs only a stable Rust toolchain (install via rustup, which supports Windows, macOS, and Linux). Node.js and the Tree-sitter CLI are needed only when working on the grammar or generated artifacts. Use the Node version in grammar/.nvmrc, then run npm ci in grammar/; its lockfile pins the CLI to 0.27.0 so regeneration uses the same toolchain as CI.

Clone and install the CLI:

git clone https://github.com/TalkBank/chatter.git
cd chatter
cargo install --path crates/chatter --locked

This installs the chatter binary to ~/.cargo/bin/ (macOS/Linux) or %USERPROFILE%\.cargo\bin\ (Windows). To update a source install, pull and re-run the cargo install command above (chatter update is only for installer-based installs).

Building the libraries

If you are developing with the Rust crates directly, from your chatter checkout root:

cargo build --workspace --all-targets --locked
cargo test --workspace --locked
cargo clippy --all-targets -- -D warnings

See the contributor setup for additional commands.

Directory layout

Everything lives in a single repository:

<your-chatter-checkout>/
├── grammar/            # Tree-sitter grammar
├── crates/             # All Rust crates (talkbank-* + the chatter binary)
├── spec/               # CHAT specification
├── apps/               # Tauri desktop app (chatter-desktop)
└── book/               # Chatter mdBook (this book)

The CLI, grammar, crates, and the LSP/desktop integrations all live in this single repository.

This page last changed: 2026-08-30 (commit 733da964). The whole book last changed: 2026-09-15 (commit bb4bef82).

Quick Start

Status: Current Last updated: 2026-07-13 17:59 EDT

This page gets you from zero to productive with chatter in five minutes. Install chatter first if you haven’t already.

Validate a CHAT file

Check a single transcript for errors:

chatter validate transcript.cha

If the file is valid you get a summary (a cache-statistics block follows it; use --quiet to suppress all output and rely on the exit code):

=== Summary ===
Total files: 1
Valid: 1
Invalid: 0

If there are problems, you’ll see rich diagnostics with the exact location and a stable error code. For example, a *CHI: line missing its terminator:

✗ Errors found in transcript.cha

E305 (https://talkbank.org/errors/E305)

  × error[E305]: Expected terminator not found (line 6, column 1)
   ╭─[input:6:1]
 6 │ *CHI:   hello world
   · ─────────┬─────────
   ·          ╰── here
   ╰────
  help: Add a terminator at the end: Standard (. ? !), Interruption
        (+... +/. ...), or CA intonation (⇗ ↗ → ↘ ⇘ ...)

Every error code (E305, E705, etc.) is documented with fix guidance in the validation error reference.

Not every diagnostic is an error. Some codes are warnings: the file is valid CHAT, but something is worth flagging (for example E254, a word-level @s: language override that is not listed in @Languages). A file whose only diagnostics are warnings is reported as valid, and its heading reflects that:

⚠ Warnings in transcript.cha

E254 (https://talkbank.org/errors/E254)

  ⚠ warning[E254]: Explicit word language 'spa' is not listed in @Languages
   ╭─[input:6:15]
 6 │ *CHI:   hello hola@s:spa .
   ·               ─────┬────
   ·                    ╰── here
   ╰────
  help: Add 'spa' to @Languages or confirm the word-level override is intentional

The summary still counts this file under Valid, and the exit code stays 0.

Validate an entire corpus

Point chatter at a directory, it walks recursively, validates in parallel, and caches results:

chatter validate corpus/

The interactive TUI shows progress and lets you browse errors per file. Use --format json for machine-readable output, or --quiet for CI (exit code 1 on errors).

Convert to JSON

Get a structured representation of any CHAT file:

chatter to-json transcript.cha

The output conforms to the TalkBank CHAT JSON Schema. Convert back with chatter from-json.

Watch for changes

Edit a file and get live validation feedback:

chatter watch transcript.cha

Every time you save, chatter re-validates and shows updated diagnostics.

What next?

• CLI Reference: all commands, flags, and output formats
• Validation Errors: every error code, with examples and fix guidance
• Batch Workflows: corpus-scale validation and analysis

This page last changed: 2026-07-13 (commit e0f8cebb). The whole book last changed: 2026-09-15 (commit bb4bef82).

CLI Reference

Status: Current Last modified: 2026-08-12 22:20 EDT

The chatter CLI is the primary command-line surface for the TalkBank CHAT toolchain.

The following diagram shows the command dispatch structure. Each top-level command dispatches to a handler in the corresponding crate.

flowchart TD
    chatter(["chatter"])

    chatter --> validate["validate\n(chatter)"]
    chatter --> normalize["normalize\n(chatter)"]
    chatter --> tojson["to-json\n(talkbank-transform)"]
    chatter --> fromjson["from-json\n(talkbank-transform)"]
    chatter --> showalign["show-alignment\n(chatter)"]
    chatter --> watch["watch\n(chatter)"]
    chatter --> fix["fix\n(talkbank-transform splice)"]
    chatter --> clean["clean\n(chatter)"]
    chatter --> newfile["new-file\n(chatter)"]
    chatter --> cache["cache\n(stats, clear)"]
    chatter --> schema["schema\n(JSON Schema output)"]
    chatter --> debug["debug\n(overlap-audit, linker-audit,\nfind, sanitize, fix-s)"]
    chatter --> update["update\n(self-update, experimental)"]

    chatter --> merge["merge\n(experimental)"]
    chatter --> speakerid["speaker-id\n(experimental)"]
    chatter --> rediarize["rediarize\n(experimental)"]
    chatter --> adjudicate["adjudicate\n(experimental)"]
    chatter --> pipeline["pipeline\n(experimental)"]
    chatter --> batch["batch\n(experimental)"]
    chatter --> sanityscan["sanity-scan\n(experimental)"]

Top-Level Commands

chatter validate PATH...
chatter normalize INPUT
chatter to-json INPUT
chatter from-json INPUT
chatter show-alignment INPUT
chatter watch PATH
chatter fix PATH... --apply
chatter clean PATH
chatter new-file
chatter cache stats
chatter cache clear --prefix PATH
chatter schema
chatter debug ...
chatter update                     # experimental: self-update to the latest release
chatter merge FILE1 FILE2          # experimental: combine two transcripts
chatter speaker-id INPUT           # experimental
chatter rediarize INPUT --turns T  # experimental
chatter adjudicate ...             # experimental
chatter pipeline ...               # experimental
chatter batch ...                  # experimental
chatter sanity-scan ...            # experimental

Use chatter --help or chatter <command> --help for the exact live surface.

validate

Validate CHAT file(s) or directory tree(s). Accepts multiple paths.

Usage: chatter validate [OPTIONS] <PATH>...

chatter validate file.cha                         # single file
chatter validate file1.cha file2.cha file3.cha    # multiple files
chatter validate corpus/                          # directory (recursive, parallel)
chatter validate file.cha corpus/ other.cha       # mix of files and directories
chatter validate corpus/ -f json                  # structured JSON output
chatter validate corpus/ --force                  # ignore cache, revalidate everything
chatter validate corpus/ --force --audit out.jsonl # bulk audit to JSONL file
chatter validate corpus/ --suppress xphon         # suppress named error group
chatter validate corpus/ --suppress E726,E727     # suppress specific error codes
chatter validate corpus/ -j 8                     # use 8 parallel workers
chatter validate corpus/ --max-errors 50          # stop after 50 errors

Options:

--parser re2c reports unreliable diagnostic positions.

The re2c lexer DOES produce a source span for every token; the parser discards it (parser/mod.rs, lexer.map(|(tok, _span)| tok)), so the converter assigns every model node a dummy span and diagnostics that compute a position from those spans point somewhere arbitrary. The same file validated both ways:

tree-sitter   error[E370] ... (line 7, column 13)     <- the offending tier
re2c          error[E370] ... (line 2, column 7)      <- points at @Begin

The VERDICT is trustworthy on both backends and the two are held to structural equivalence by the parity oracle; only the reported location is not. A wrong position that looks plausible is worse than none, so treat --parser re2c as suitable for batch pass/fail and use the default backend when you need to find the error in the file.

Restoring the positions means carrying the lexer’s spans through the token slice rather than re-deriving them, which is bounded work rather than a redesign.

Suppress groups: xphon expands to the whole Phon %x dependent-tier validation surface (%xmodsyl/%xphosyl/%xphoaln/%xphoint, codes E725-E728 and E735-E746). These checks run by default; pass --suppress xphon to silence the group. (The old --check-xphon flag is a deprecated no-op kept only so existing scripts do not break.) The --suppress flag can mix groups and codes: --suppress xphon,E316.

Suppression does not cost you the cache. It changes what is printed, not what is validated, so runs that differ only in --suppress share cached results: chatter validate corpus/ followed by chatter validate corpus/ --suppress xphon reuses the first run’s work. --strict-linkers is the other kind of flag, since it turns extra checks on, so it validates afresh.

normalize

Serialize a CHAT file into canonical formatting.

chatter normalize input.cha
chatter normalize input.cha -o normalized.cha
chatter normalize input.cha --validate
chatter normalize input.cha --validate --skip-alignment

Flags:

• -o, --output <PATH>: write to a file instead of stdout.
• --validate: validate (including alignment by default) before writing the normalized output.
• --skip-alignment: when paired with --validate, skip the dependent-tier alignment checks (still validates the rest).

normalize writes to stdout unless you pass -o/--output. There is no --in-place flag.

JSON Conversion

# Single file
chatter to-json input.cha                          # pretty-printed JSON to stdout
chatter to-json input.cha --compact                # minified JSON to stdout
chatter to-json input.cha -o output.json           # JSON to file

# Directory (recursive, preserves structure)
chatter to-json corpus/ --output-dir json/          # incremental by default (mtime check)
chatter to-json corpus/ --output-dir json/ --compact # minified output (saves disk)
chatter to-json corpus/ --output-dir json/ --force   # full rebuild
chatter to-json corpus/ --output-dir json/ --prune   # remove orphaned .json files
chatter to-json corpus/ --output-dir json/ --jobs 4  # parallel workers

# Reverse and schema
chatter from-json input.json -o output.cha
chatter schema
chatter schema --url

Single-file mode: to-json validates by default. Use --skip-validation, --skip-alignment, or --skip-schema-validation to bypass checks.

Directory mode: Walks recursively, converting each .cha to .json under --output-dir with the same relative path. Incremental by default: skips files whose JSON is already newer than the source. Use --force to rebuild all. Use --prune to remove .json files with no matching .cha (handles renames/deletions). Use --jobs N for parallel conversion (defaults to number of CPUs).

Editing and Inspection Commands

show-alignment

Print the dependent-tier alignment for a CHAT file (debugging aid).

chatter show-alignment file.cha
chatter show-alignment file.cha -t mor          # one tier type
chatter show-alignment file.cha -t gra -c       # compact one-line-per-alignment output

Flags: -t/--tier <mor|gra|pho|sin> (omit to show all available tiers); -c/--compact (one line per alignment).

watch

Watch a CHAT file or directory and re-validate on every save.

chatter watch file.cha
chatter watch corpus/
chatter watch corpus/ --skip-alignment --clear

Flags: --skip-alignment (faster reruns); -c/--clear (clear the terminal between runs).

fix

Apply catalog fixes to CHAT file(s) at exact byte spans. Every file is parsed and validated, each diagnostic is resolved against a per-code fix catalog, and the resulting edits are admitted only into utterances that parsed clean (a broken region elsewhere in the file never blocks a fix, and is never itself rewritten) before being spliced in.

chatter fix file.cha                      # report only, writes nothing
chatter fix corpus/ --apply               # write the mechanical fixes
chatter fix corpus/ --apply --dry-run     # preview without writing
chatter fix file.cha --apply --code E259  # opt a semantic fix into writing

Every catalog entry carries a batch-safety tier, and this command enforces it rather than trusting the caller:

• Mechanical (one right answer, no semantic judgment): written by a bare --apply.
• Semantic (deterministic, but changes meaning enough to need a human naming it): written only when its code is named with --code.
• Ambiguous (several valid answers, no evidence in the file picks one): never written by this command, regardless of --code; only reported.

Flags: --apply (write; without it, fix only reports what it would do); --dry-run (preview, requires --apply); --code <CODE> (repeatable; narrows the diagnostics considered to exactly the named codes, and is how a semantic-tier code opts into being written); --skip-alignment.

Header-scoped fixes are currently reported, not applied. Edits are admitted only into utterances that parsed clean, so a catalog fix whose edit lands in the header region (E501, E502, E503, E504, E506, E507) never has an enclosing utterance to be admitted into; fix reports it as skipped instead of writing it. This is today’s limit of the admission gate, not a missing catalog entry; a header-scoped admission path is separate future work.

clean

Show the cleaned text for each word (a debugging aid for the text-normalization pipeline).

chatter clean file.cha
chatter clean file.cha --diff-only       # only words where raw differs from cleaned
chatter clean file.cha --format json

Flags: --diff-only; --format text|json.

new-file

Create a new minimal valid CHAT file from defaults.

chatter new-file
chatter new-file -o starter.cha --speaker CHI --language eng
chatter new-file -o adult.cha -s MOT -l eng -r Mother
chatter new-file -c brown -u "hello world ."

Flags:

• -o, --output <PATH>: stdout if omitted
• -s, --speaker <CODE>: default CHI
• -l, --language <ISO 639-3>: default eng
• -r, --role <ROLE>: default Target_Child
• -c, --corpus <CORPUS>: corpus identifier in the @ID header (default corpus)
• -u, --utterance <TEXT>: optional initial main-tier utterance content

Cache Commands

chatter cache stats
chatter cache stats --json
chatter cache clear --prefix /path/to/corpus
chatter cache clear --all --dry-run

The validation cache lives under the platform cache directory and stores per-file validation results. validate --force refreshes cache state for the specified path.

What the cache does and does not speed up

Files that passed are remembered; files with errors are re-checked every time. This is deliberate, and it is worth knowing because it decides how fast a re-run feels.

A file that validated cleanly is skipped entirely on the next run, as long as its contents have not changed. A file that had errors is validated again from scratch, because the cache remembers only THAT a file had errors, never what they were: the codes, the line numbers, the quoted source and the suggestions have to be produced by actually reading the file. Storing them instead would mean showing you an older release’s wording for an error that has since been improved, which is worse than waiting.

In practice this costs nothing on a corpus in good shape. A full run over the ~106,000 kept TalkBank transcripts takes about 6 seconds when cached, because only ~141 files have errors to re-check. It is noticeable in the opposite situation, part way through cleaning up a corpus where most files still fail, or just after a new release tightens a rule. Two things help there: narrow the target to the directory you are working in rather than the whole corpus, and fix files as you go, since each one that passes joins the fast path permanently.

Two other things reset the cache, both expected:

• Editing a file. The cache follows file contents, so a changed file is always re-validated, and reverting a change restores the earlier result.
• Upgrading Chatter. A new release can change what counts as valid, so every cached result from an older version is retired and the first run after an upgrade is a full one. Later runs are fast again. The previous version’s results are kept, so downgrading does not force another full run.

--suppress does not reset anything: it changes what is printed, not what is checked, so runs differing only in --suppress share the same cached results.

debug

Developer / debugging subcommands for CHAT analysis. Not intended for routine end-user workflows; surface and behavior may change between releases. Run chatter debug --help for the live list. Current subcommands include:

• overlap-audit: analyze CA overlap markers (⌈⌉⌊⌋): pairing, temporal consistency, orphans.

• linker-audit: audit linker / special-terminator usage across a corpus (cross-utterance pairing for +<, ++, +^, +", +,, +≋, +≈, plus +..., +/., +//., +"/. etc.).

• find: filter CHAT files by @Languages and body content (token / substring counts) across a corpus tree; emits paths, JSONL, or CSV.

• sanitize: strip contributor lexical content while preserving structure, for protected-corpus debugging. See the Sanitize user-guide page for the full workflow.

• fix-s: normalize whole-utterance same-language @s runs into a [- lang] precode, clear the per-word @s markers (including those on fillers and nonwords), and append any missing explicit @s:LANG codes to @Languages. Trigger conditions and safety rules:

  • Every word-bearing item in the utterance, including fillers (&~, &-, &+), nonwords, and retraced material, must carry an explicit language marker AND every marker must resolve to the same target language. If a single filler such as &~dang3 lacks a marker, the utterance is left untouched (the predicate cannot prove it is monolingual).
  • Bare @s shortcuts on fillers must be cleared when the rewrite fires. A bare @s resolves relative to the surrounding tier language, so adding a [- LANG] precode without clearing the shortcut would flip the filler’s language to the precode target. fix-s clears the shortcut to keep the original meaning intact.
  • The pre-validation rule that catches the unrewritten pattern is E255 (whole-utterance same-language @s run); fix-s is the canonical repair. The companion warn-only E254 reports @s:LANG codes missing from @Languages; fix-s appends them.
  • True no-op on already-correct files: a file is rewritten only when a [- lang] conversion or @Languages repair can be proved necessary.

• join-retrace: auto-repair dangling-retrace (E370) utterances. An utterance whose last main-tier content is a retrace marker with nothing after it is joined with the next same-speaker utterance. The --scope flag (value-enum, default repetition) selects which retrace kinds qualify:

  • --scope repetition (default, Wave 1): only [/] partial-repetition retraces qualify, and only when the successor’s leading words repeat the retraced material. This is the conservative, OBVIOUS-only repair suitable for most automated use.
  • --scope corrections (Wave 3a, opt-in): also joins correction retraces: [//] (Full), [///] (Multiple), and [/-] (Reformulation). Corrections replace rather than repeat the retraced material, so the leading-words prefix check is skipped; same-speaker presence alone is the gate. Use --dry-run first to review every proposed correction-join before writing.
  • --scope all (Wave 3b, broadest, opt-in): joins ANY dangling retrace kind, including [/] Partial where the successor does NOT repeat the retraced material. This covers genuine child-language disfluencies: false starts, partial words, disfluent repetitions, expansions, and fillers where the transcriber correctly coded a [/] but the successor cannot repeat the abandoned material. Same-speaker presence alone is the gate. Always use --dry-run first when running this scope on new data.

  Shared behavior for all joined pairs:

  • The join produces one utterance: the first utterance’s content (keeping the trailing retrace marker) followed by the successor’s content, terminated by the successor’s terminator. Main-tier time bullets are unioned (start from the first, end from the successor).
  • Dependent tiers are dropped. If either side carried %mor, %gra, or any other dependent tier, the joined utterance drops all of them (a naive %gra merge would yield two ROOT relations, which chatter validate rejects as E723). Such joins are reported as “needs re-morphotag” so the file can be re-run through morphotagging afterwards; the main tier alone remains valid CHAT.
  • --dry-run reports what would be joined without modifying files.

Merge and Reconciliation Commands (experimental)

These commands combine, reconcile, and relabel CHAT transcripts of the same recording, in the tradition of CLAN’s reliability and comparison tools (rely, trnfix). They are experimental and in active development: flags and behavior may change, and several modes are not yet complete. Work on copies and validate the output.

Full guides: Merge, Speaker ID, Rediarize, and the Merge Workflow walkthrough. The holistic-judgment mode of speaker-id / pipeline / batch can call an LLM provider (talkbank-llm) when configured via --llm-endpoint / --llm-model (plus --llm-timeout-secs, --llm-max-retries, and a persistent response cache via --llm-cache or CHATTER_LLM_CACHE); the deterministic modes need no network access. Flag-level detail: Merge, LLM holistic judgment.

Exit Codes

chatter validate exits with code 1 if any file has validation errors or parse errors. This makes it safe to use in scripts and CI pipelines:

chatter validate corpus/ --quiet --tui-mode disable || echo "Validation failed"

Use --quiet to suppress per-file success output while still relying on exit codes. Use --format json for machine-readable structured output (JSON objects go to stdout; exit code still reflects pass/fail).

Output Contracts

• Text output is intended for humans.
• JSON output is intended for automation and downstream tools.
• Error codes and the JSON Schema are documented public contracts; see the Integrating section of this book.

This page last changed: 2026-08-12 (commit 0752f182). The whole book last changed: 2026-09-15 (commit bb4bef82).

Validation Errors

Status: Current Last modified: 2026-09-09 07:46 EDT

The CHAT validator produces diagnostics at two severity levels: errors (must fix) and warnings (should fix). Each diagnostic has an error code that maps back to a documented spec and validator rule.

chatter validate is the binding judgment on whether a byte sequence is valid CHAT. When it reports an error, the file is invalid CHAT: clean the data rather than working around the check.

But that is a conclusion, not a reflex, and it cuts both ways. chatter is under active development and its current behaviour is not sacred. If a diagnostic does not make sense against your file, the fault may be ours, and we would rather hear about it than have you edit a transcript to silence it. Tells that a diagnostic is a chatter bug: the message names a construct your line does not contain, the code is a generic “unparsable content” rather than a specific rule, or no documented rule justifies it. Report it with the smallest file that reproduces it. Editing data to quiet a wrong check destroys the evidence and leaves the bug in place for the next person. A warning flags a questionable but parseable construct you should review. Where chatter and an older tool such as CLAN’s check disagree on whether a file is valid, chatter validate is authoritative (see CHECK Parity Audit for how the two are reconciled).

Reading Error Output

The validator emits rich diagnostics that include the error code, a source-pointed snippet, and a suggested fix:

  × error[E304]: Missing speaker in main tier (line 15, column 3)

15 │ *	hello world .
   ·  ╰── here
   ╰────
  help: Add a speaker code between * and : (e.g., *CHI:)

Each diagnostic contains:

• File path and location (line:column)
• Severity: error or warning
• Error code: E prefix for errors, W prefix for warnings, with a URL pointing at the per-code documentation page
• Message: human-readable description
• Suggestion: actionable fix guidance where available

Error Code Ranges

Common Errors and Fixes

E256: Curly single quote used as a word character

A curly single quotation mark (U+2018 or U+2019), commonly introduced by autocorrect or speech-to-text, is not a legal CHAT word character. CHAT words use the ASCII apostrophe (U+0027, the plain '). For example, a contraction typed as don + U+2019 + t is rejected; write don't with the ASCII apostrophe instead. chatter flags the curly form wherever it appears in word content and points the diagnostic at the exact character. This mirrors CLAN CHECK errors 138 and 139.

E243: Private-use or non-standard Unicode in a word

A word may contain only standard Unicode. Characters from the Unicode Private Use Area and the other non-standard code points in the U+E000-U+FFFF block are rejected, including the replacement character U+FFFD that marks a botched text encoding. The most common cause is a file saved in the wrong encoding: re-save it as UTF-8 and replace any private-use or compatibility-area character with its standard Unicode equivalent. chatter points the diagnostic at the exact character. This mirrors CLAN CHECK error 86.

E304: Missing speaker code

A main tier line must have a speaker code after the *:

*CHI:	hello world .

An empty speaker code (*: hello .) triggers E304.

E308: Undeclared speaker

Every *SPEAKER: code must be listed in @Participants. Add the missing speaker to the header:

@Participants:	CHI Target_Child, MOT Mother

E370: Retrace marker with nothing to retrace

A retrace or repetition marker ([/], [//], [///]) must be followed by the repeated or corrected material; per the CHAT manual the marker always refers to the text that follows it. A marker followed only by a terminator has nothing to retrace:

*CHI:	<the> [/] .          ← invalid: [/] is not followed by repeated material
*CHI:	<the> [/] the cat .  ← valid: the repeated material follows the marker

This mirrors CLAN CHECK error 119 (and the related retrace checks 151 and 159).

E505: Invalid @ID format

Check that pipe-separated fields are correct and the speaker code matches @Participants:

@ID:	eng|corpus|CHI|2;6.||||Target_Child|||

E705: Main/%mor alignment mismatch

The number of %mor items must match the number of alignable words on the main tier. Retraces, pauses, and events are not counted. The validator shows a columnar diff:

  Main tier       %mor tier
  ──────────────  ──────────────
  I               pro|I
  want            v|want
  to              inf|to
  go              v|go
  home, ⊖

E714 / E715 and E733 / E734: phonology count mismatch

E714 and E715 report too few or too many %pho tokens. E733 and E734 report the corresponding %mod mismatch. The tier names use distinct codes because actual and model phonology are different evidence layers.

%wor does not use any of these codes. It is a timing sidecar, and a count mismatch does not make legacy CHAT invalid. Timing consumers handle the state explicitly:

• Missing: no %wor tier;
• Drifted: current main-tier and %wor slot counts differ;
• CountMatched: counts match, but no timing is exposed yet;
• Uncorroborated: canonical display tokens differ;
• Corroborated: positional slots are available, with lexical identity from the main tier and timing from %wor word bullets.

The current FilteredLexicalV1 policy includes regular words, fillers, retraced regular words, and the original spoken word of a replacement. It excludes fragments, nonwords, xxx/yyy/www, omissions, pauses, and actions. Changing that policy requires a new named policy and evaluation; it must not silently reinterpret existing %wor data.

And this is valid too:

*EXP:	what's is dis [: this] ?
%wor:	what's •37050_37471• is •37491_37631• dis •37631_38131• ?

E721: %gra sequential index error

%gra entries must have sequential 1-based indices: 1|...|... 2|...|... 3|...|...

E748: Leading zero in bullet timestamp

A media bullet time component is written with a leading zero before another digit, for example \u{15}012_200\u{15}. Bullet times are plain millisecond integers; write 12, not 012. A bare 0 (as in 0_200) is legal. This mirrors CLAN CHECK error 90 (“Illegal time representation inside a bullet.”). The bullet’s numeric value still parses, so downstream tooling sees the intended times; the diagnostic alone makes the file invalid.

E749: Comma glued to the following word

A comma on a speaker tier must be followed by a space or end-of-line: write hey , you, not hey ,you. Mirrors CLAN CHECK error 92. The check looks at the word immediately after the comma in document order (including inside <...> groups); constructs that place their own character after the comma (group and overlap marks, CA symbols) are not flagged.

E750: Space inside angle-bracket group delimiters

Group delimiters hug their content: write <dog> [/], never < dog> or <dog >. Mirrors CLAN CHECK error 160. Each offending space gets its own diagnostic; the group still parses, so downstream tooling sees the intended structure. chatter fix --apply --code E750 removes only the offending delimiter-adjacent space. This mechanical catalog repair carries a distinct recovery-safe edit state, so it may repair the parser recovery that tainted its own utterance while ordinary edits remain barred from recovered content; the standard post-splice reparse still has to prove the result.

E751: Pause glued to the preceding word

A pause marker must be space-delimited from the word before it: write hello (.) there, not hello(.) there. Mirrors CLAN CHECK error 57.

E752: Timing bullets without an @Media header

The transcript carries timing evidence (an utterance-final bullet, a bullet inside an utterance, or %wor word timing) but no @Media header declares the recording those timestamps index. Add an @Media header naming the media file (or remove the timing bullets if the transcript is genuinely unlinked). Completes the media-consistency family: E544 covers declared linkage without timing, E552 covers a declared unlinked contradicted by timing. Mirrors CLAN CHECK error 112.

E753: Word consisting only of a repetition segment

A word whose entire material sits inside segment-repetition delimiters (↫hi↫ with nothing outside the arrows) marks the repetition of a word that is not there; attach the repeated segment to its host word (↫p↫parents) or transcribe a stand-alone fragment as a filler or nonword form. Filler and other word-category prefixes (&-, &~, 0) count as material outside the arrows. Adopted from GUI CLAN CHECK error 151 as a chatter rule.

E519 at word level: language codes must be real everywhere

The ISO 639-3 registry check that guards @Languages and @ID also applies to explicit word-level switch codes (word@s:CODE, including +/& multi-code forms) and to @L1 of values: the code needs no declaration, but it must name a real language. Utterance-level [- CODE] precodes are covered by E755 plus the header check.

E755: Utterance language not declared in @Languages

A [- CODE] precode marks a whole utterance as being in another language, which is substantial presence: declare that language in @Languages. Deliberate contrast: a word-level @s:CODE insertion needs NO declaration (ok@s:eng in a Cantonese transcript is valid as-is), because @Languages lists the transcript’s substantial languages, not every language that appears. Mirrors CLAN CHECK error 152.

E756: Empty dependent tier

A dependent tier with empty or whitespace-only content declares an annotation that is not there; add the content or remove the line. Whitespace-only counts as nothing on every free-text tier, %com and %add included (they were exempt by accident until 2026-09-08); CLAN CHECK 31 rejects the same lines.

This covers every tier whose body is free text, which is every dependent tier except the structured ones (%mor, %gra, %pho, %mod, %sin, %wor). An empty structured tier fails earlier and more specifically, because its body is not free text and there is no “you declared nothing” to report.

The rule read only user-defined %x tiers until 2026-08-15. That was never the rule, only its name: the model could not represent an empty standard tier, so an empty %eng: had nowhere to be recorded and the two parser backends disagreed about it, one calling the file valid and the other rejecting it through an undescribed code. (Formerly W601; renumbered because it always was a hard error.)

E757: Bracketed code glued to the following word

A bracketed code’s closing ] must be space-delimited from what follows: write hello [/] x, not hello [/]x. The parse is unambiguous either way, which is exactly why this is a style rule: the corpus stays canonically spaced. Mirrors CLAN CHECK error 19.

E758: Leading space before tier content in a non-CA file

A space between the tier’s tab delimiter and the first content item (*CHI:<tab><space>dog .) is invalid unless the file declares @Options: CA; CA transcripts legitimately column-align content with spaces after the tab. Mirrors CLAN CHECK error 123.

E759: Annotation at utterance start

Postfix annotations (retraces [/] [//], overlap markers [<] [>], replacements [: text], the quotation code ["]) scope over the material BEFORE them; an utterance whose content begins with one has nothing for the code to attach to. Mirrors CLAN CHECK error 52.

E760: %mor item with an empty part-of-speech field

A %mor item beginning with the | separator (|we) declares no part of speech; every item is pos|stem with a non-empty POS. The modern reading of CLAN CHECK error 11 (the depfile mechanism is legacy; the non-empty-symbol invariant is real).

E761: %gra relation head is not a Universal Dependencies relation

A %gra label is HEAD or HEAD-SUBTYPE. UD fixes the head set at 37 universal relations and defines subtypes as language-specific and open-ended, so only the head is checked; a subtype such as NMOD-POSS or ACL-RELCL passes untouched. Nothing validated relation labels before, in chatter or in CLAN CHECK, so a typo like PUNCTT for PUNCT rode silently into every analysis that reads the dependency graph. Common causes: truncation (IOB for IOBJ), typos, and the retired TalkBank labels (SUBJ, JCT, POBJ, INCROOT), none of which occurs in the corpora any more.

E762: prefix marker # stands alone or opens a word

The prefix marker separates a bound prefix from its stem and attaches to the END of the prefix, which is a word of its own (Hebrew ha# kelev, “the dog”). So a word that is nothing but #, or one that opens with it (#dog), cannot be that construct in any language. This covers CLAN CHECK 71 and the #-undeclared facet of CHECK 11.

E763: prefix marker # in a language that does not use it

Languages that write the marker are heb and ara; anywhere else it is a stray character, usually a typo or a conversion artifact. The gate reads the WORD’s resolved language, not the file’s @Languages header, exactly as the digits rule (E220) does, so a code-switched word marked @s:heb inside an English file is accepted. Word-internal markers (mi#ha#shuk) stay legal wherever the language allows the marker at all.

E765: separator glued to the following content

A free-standing : or ;, or a pause, must have a space after it: :and, ;;, (.)dog are invalid. The preceding side is untouched: word↘ and dog, are documented CHAT convention, and dog: is not two items at all (the colon fuses as lengthening).

Every CA mark is out of scope, on corpus evidence. Implementing the whole separator class flagged 270 instances in a 2% corpus sample (about 13,500 corpus-wide), all legitimate notation: ≡ is latching and is written glued on both sides (y≡I≡) because that is what it encodes, and the intonation arrows attach to the material they mark, including directly before an overlap close (⌊I don't know⇗⌋). Whether any CA mark should forbid trailing glue is unresolved.

E766: linker not utterance-initial

Linkers (+", ++, +<, +^, +,, +≈, +≋) tie an utterance to the PREVIOUS one, so they may only open the utterance. One placed after content (yeah that go +" okay .) is meaningless and is named here, at the exact token, instead of surfacing as generic unparsable content (E316).

One deliberate carve-out: a ++ glued to words on both sides (un++do) is not a linker but a word run with an empty compound part, and keeps its E233 diagnosis.

E767: whitespace before the @Media comma

In @Media the comma separates the filename from the media type, so the filename ends where the comma begins and a space between them belongs to neither. Real CLAN rejects it too (CHECK 148), and an unambiguous style violation that CLAN rejects is an error here.

The construct is unambiguous, so the grammar deliberately PARSES it rather than failing, which is the only way to name the rule and point at the exact space. That is a change of diagnostic, not of verdict: before, the @Media line failed to match and the whole header fell back to Unknown, reporting E525 about a header chatter had recognised perfectly well alongside E330 “Missing media_type node” on a line visibly ending in , audio. Deleting the one space now validates clean.

E768: @Media filename cannot be written and read back

The filename is delimited by the comma that introduces the media type, so a few strings cannot survive a round trip through the header: an unquoted comma, surrounding whitespace, a line break, a stray double quote, or an empty name. A quoted remote URL may contain a comma.

You will not see this one from a transcript. Both parsers end the filename at the comma, so no .cha file can express a violating value; the rule guards a ChatFile that arrived as JSON, where deserialization is deliberately lenient and validation is what reports the violation.

E757 widening: every bracketed code, not only retraces

hello [/]there was caught; hello [!]there and bobo [= toy]there were not, because the rule only examined retraces. It now applies to any item that ends in a bracketed code, which is what the code always claimed to mean.

The cause was a parser omission rather than a missing rule: an annotated word’s wrapper span was left DUMMY at construction (its annotated event, action, and group siblings all set a real one), so the glue was invisible to a span-adjacency check, and any diagnostic reported on an annotated word pointed at byte zero. Both are fixed by giving that wrapper the span it should always have had.

E764: prefixed form glued to the preceding word

dog&-um parses as TWO words, because & cannot continue a word. So a single missing space silently manufactures a word boundary the transcriber never wrote, and until this rule nothing reported it. Applies to the three & prefixes (&- filler, &~ nonword, &+ fragment).

Glued omission (dog0is) is a different shape: 0 is ordinary word text, so it yields one malformed word and is already rejected by E220.

E243 addition: the pipe character

| is the %mor tier’s delimiter and has no meaning in main-tier word text; a bare or embedded pipe in a word now reports E243 (IllegalCharactersInWord). Covers the grounded shape of CLAN CHECK error 48.

Generated Error Documentation

The source of truth for error-code details is spec/errors/. Maintainers can The browsable error catalog under docs/errors/ is generated from those specs and committed, so it is regenerated with every other artifact:

just spec-gen

That generated reference includes the error description, example inputs, suggested fixes, and the layer that catches the diagnostic.

This page last changed: 2026-09-09 (commit b6bfc5d2). The whole book last changed: 2026-09-15 (commit bb4bef82).

Chatter Desktop

Status: Current Last modified: 2026-08-03 09:06 EDT

Chatter Desktop is a native graphical validation app for CHAT files, released alongside the chatter CLI. Prefer the chatter CLI for scripted or batch validation; use the desktop app when you want a standalone graphical validation experience without a terminal.

When to use Chatter Desktop

Chatter Desktop (apps/chatter-desktop/) is the right tool when you want to:

• Validate CHAT files through a graphical interface, no terminal required
• Drag and drop a file or folder and read errors with source snippets
• Work on the desktop without setting up a terminal workflow

Related surfaces:

• Validate CHAT from the command line: use chatter validate

This page documents the desktop surface:

• Chatter Desktop (apps/chatter-desktop/), the CHAT validation GUI

Current status

• Release contract: released alongside the CLI in the public chatter release
• Distribution: ships in the coordinated chatter release alongside the CLI; also buildable from source (below)
• Platforms: macOS, Windows, and Linux

Staying up to date

Chatter Desktop keeps itself current. When you launch it, it quietly checks for a newer release; if one is available it asks whether to update, and on your confirmation it downloads, installs, and restarts into the new version. If the check cannot reach the network it simply does nothing and the app keeps working on the version you have. You never have to track releases or re-download by hand.

Getting Started

Build from source

cd apps/chatter-desktop
npm ci
cargo tauri dev       # launches the app with hot reload
cargo tauri build     # produces a distributable app bundle

Requires: Rust (stable, edition 2024), Node.js, and npm.

Using the App

Opening files

Chatter validates one target at a time: a single .cha file or one folder.

Three ways to start validating:

1. Choose File: opens a file picker filtered to .cha files
2. Choose Folder: opens a folder picker; validates all .cha files recursively
3. Drag and drop: drag one .cha file or one folder onto the app window

When idle, if you’ve previously validated a target, the drop zone shows “Last: corpus/reference/, Re-validate?” as a clickable shortcut.

Reading results

The main window has three areas:

┌──────────────────────────────────────────────────────────────┐
│  [Choose File] [Choose Folder] or drag here  [System|Light|Dark] │
├──────────────────┬───────────────────────────────────────────┤
│ 3 FILES WITH     │  Filter by code… [All|Errors|Warnings]    │
│ ERRORS / 120     │                                           │
│                  │  ▾ [E302] Missing @End header              │
│  📁 corpus/      │  ┌───────────────────────┐                │
│    ✗ file1 (3)   │  │ 41 │ *CHI: hello .    │                │
│    ✗ file3 (1)   │  │ 42 │                   │                │
│                  │  │    │ ^                 │                │
│                  │  └───────────────────────┘                │
│                  │  💡 Add @End on the last line             │
│                  │  [Copy] [Open in CLAN]                    │
├──────────────────┴───────────────────────────────────────────┤
│  Progress: 45/120 │ 4 errors │ ~2m 30s remaining │ [Cancel]  │
└──────────────────────────────────────────────────────────────┘

• File tree (left), collapsible directory tree showing only files with errors (valid files are hidden to reduce clutter). A header shows “N files with errors / M total”. Files are sorted alphabetically.

• Error panel (right), for the selected file, shows each error with its code in [E001] format, severity color, message, source snippet with caret underlines, and multi-span labels for complex errors (e.g., alignment mismatches across tiers). CHAT-specific formatting is handled: tabs expanded to 8-column boundaries, \x15 bullets rendered as •, underline markers shown as styled underlined text. Suggestions prefixed with 💡.

• Status bar (bottom), streaming progress during validation, ETA after 5+ files, total error count, and action buttons.

Filtering errors

A compact filter bar appears above the error cards when a file has diagnostics:

• Code filter: type “E7” to show only alignment errors, “W” for warnings, etc.
• Severity toggle: switch between All / Errors / Warnings

The file header updates to show filtered vs. total count (e.g., “3 errors (7 total)”).

Collapsible error cards

Each error card has a clickable header that toggles between expanded and collapsed view. Collapsed cards show only the error code and first line of the message. When a file has 5 or more errors, an Expand All / Collapse All button appears.

Validation settings

A ⚙ Settings popover next to the file picker exposes the same knobs the CLI’s flags do, since both surfaces build the same underlying validation config:

Settings are disabled while a validation run is in progress and apply to the next run (including Re-validate).

Dark mode

Chatter follows your system appearance by default. A System / Light / Dark toggle in the drop zone area lets you override. Your preference is remembered across sessions.

The dark palette uses muted Apple-style colors, readable miette error highlighting on dark backgrounds.

Clickable file paths

Click the file name in the error panel heading to reveal the file in Finder (macOS), Explorer (Windows), or the default file manager (Linux).

Copy errors

Each error card has a Copy button that copies the full miette-rendered error text (plain text, not HTML) to your clipboard for pasting into issue reports or messages.

Actions

“Open in CLAN” only appears when the CLAN application is detected on your system (macOS and Windows only). It adjusts line numbers to account for headers that CLAN hides (@UTF8, @PID, @Font, @ColorWords, @Window).

Keyboard shortcuts

All other navigation is mouse-driven (click files, scroll errors).

Window title

The window title updates to reflect the current state:

• Idle: “Chatter”
• Starting: “Chatter, Starting…”
• Discovering: “Chatter, Discovering files…”
• Running: “Chatter, Validating (45/120)”
• Finished: “Chatter, 14 errors in 3 files” or “Chatter, All 74 files valid”
• Incomplete: “Chatter, Incomplete (2 files not checked)”
• Stopped: “Chatter, Run stopped unexpectedly”

The last two are failures, and they never claim anything about your whole folder. Incomplete means the validator finished but some files were never opened, so the counts it shows describe only the rest; you will see how many were missed, and re-validating is the right response. Stopped means the run died without producing results at all. Neither one can show “All N files valid”, because that sentence is a claim about every file and neither run examined every file.

“Starting” and “Discovering” are different states, and the difference is worth knowing if you ever need to report a problem. Starting means the app has asked the validator to begin and has not heard back; nothing has been scanned yet, so a run stuck there is a fault in start-up rather than anything about your files. Discovering means the validator is walking the folder, which legitimately takes time on a large one. If the app sits on “Starting” for more than a few seconds it says so in the status bar, and that message is worth quoting in a bug report.

ETA

After 5 or more files have been processed, the status bar shows an estimated time remaining (e.g., “~2m 30s remaining”). The estimate updates every second.

Notifications

When validation finishes while the app is not focused, a system notification shows the summary (“Validation complete, 14 errors in 3 files”).

First launch

On first launch, an onboarding overlay explains the four main interactions: drag files, error panel, keyboard shortcuts, and export. Dismiss with “Got it”, it won’t appear again.

CLI Bundling

The desktop app can bundle the chatter CLI binary so power users who download the GUI can also run the CLI from their terminal (like VS Code ships the code command).

An Install CLI Command menu item (when available) symlinks the bundled binary to /usr/local/bin/chatter (macOS/Linux) or copies it to a PATH directory (Windows).

To build with the bundled CLI:

cargo build --release -p chatter
mkdir -p apps/chatter-desktop/src-tauri/resources
cp target/release/chatter apps/chatter-desktop/src-tauri/resources/
cargo tauri build

Architecture

The desktop app lives in apps/chatter-desktop/:

apps/chatter-desktop/
  src-tauri/          Rust backend (Tauri v2)
    src/
      main.rs         Bin entry, calls chatter_desktop_lib::run()
      lib.rs          Tauri app setup (Builder + module wiring)
      protocol.rs     Shared command/event names + request types
      commands.rs     validate, cancel, open_in_clan, export, reveal, install_cli
      events.rs       ValidationEvent → frontend event bridge
      validation.rs   Desktop validation orchestration for one target
  src/                React + TypeScript frontend
    components/       DropZone, FileTree, ErrorPanel, ProgressBar, OnboardingOverlay
    hooks/            useValidation, validationState, useTheme
    protocol/         Command/event names + TypeScript transport mirrors
    runtime/          Tauri transport + capability-focused runtime seam

The Rust backend calls validate_directory_streaming() and validate_files_streaming() from talkbank-transform directly (folder vs. single-file targets respectively), the same streaming validation pipeline and on-disk cache used by the CLI and TUI. Events flow over crossbeam channels to the Rust side, then are serialized to JSON and emitted to the frontend via Tauri’s event bridge.

Cancellation uses ArcSwapOption for lock-free atomic swap of the cancel sender, no mutex.

The frontend keeps Tauri-specific code confined to src/runtime/tauriTransport.ts. React components and hooks consume narrower capabilities (validationRunner, validationTarget, clan, exports) instead of reaching for one broad desktop service object.

Comparison with TUI

Both use the identical validation engine and produce the same error codes.

When to Use Which Tool

The TalkBank toolchain offers validation through three interfaces. Each serves a different workflow:

Chatter Desktop focuses on validation only.

This page last changed: 2026-08-03 (commit 8ee91c3c). The whole book last changed: 2026-09-15 (commit bb4bef82).

CLAN Line Numbering

Status: Current Last modified: 2026-05-29 17:31 EDT

When you click “Open in CLAN” in the desktop app or press Enter in the TUI, chatter sends the error location to the CLAN editor. CLAN opens the file and places the cursor at the error. This usually works seamlessly, but there is one caveat: CLAN and chatter count lines differently.

Hidden Headers

CLAN hides five header types from its editor display:

These headers are present in the .cha file but invisible in CLAN’s editor. CLAN’s line numbers skip them entirely. A file that starts with @UTF8 on line 1 will show @Begin as “line 1” in CLAN’s display, even though it’s actually line 2 in the file.

What Chatter Does

Chatter automatically adjusts line numbers before sending to CLAN:

1. Compute the error’s line number in the source file
2. Count how many hidden headers appear before that line
3. Subtract the hidden count to get CLAN’s line number
4. Send the adjusted line number to CLAN

This happens transparently, you don’t need to do anything.

Edge Case: Errors on Hidden Lines

If an error is on a hidden header itself (e.g., a malformed @UTF8 line), CLAN cannot navigate to it because CLAN doesn’t display that line. In this case, “Open in CLAN” will show an error message explaining why.

For Developers

The shared resolution logic lives in talkbank_model::resolve_clan_location(). Both the TUI and the desktop app call this function, it resolves line/column from byte offsets when needed and adjusts for hidden headers.

See clan_location.rs for the implementation and tests.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Batch Workflows

Status: Current Last modified: 2026-06-12 21:05 EDT

The chatter CLI is designed for processing large CHAT corpora efficiently. This page covers common batch workflows.

Validating a Corpus

Validate all .cha files in a directory tree:

chatter validate /path/to/corpus/

The validator recursively discovers .cha files and processes them in parallel. Results are cached, subsequent runs skip unchanged files.

Forcing Revalidation

To bypass the cache and revalidate everything:

chatter validate /path/to/corpus/ --force

Filtering Output

Show only errors (hide warnings):

chatter validate /path/to/corpus/ --quiet

Stop after the first reported error:

chatter validate /path/to/corpus/ --max-errors 1

Write a JSONL audit file while validating:

chatter validate /path/to/corpus/ --audit validation.jsonl

CHAT-JSON Roundtrip

Convert an entire corpus to JSON and back:

# CHAT → JSON
for f in corpus/**/*.cha; do
  chatter to-json "$f" > "${f%.cha}.json"
done

# JSON → CHAT
for f in corpus/**/*.json; do
  chatter from-json "$f" > "${f%.json}.roundtrip.cha"
done

The roundtrip is designed to preserve the ChatFile model. In regression tests, compare normalized output rather than assuming byte-for-byte identity after parser or serializer changes.

Cache Management

The validation cache stores results for previously validated files (keyed by content hash). The cache database file is named talkbank-cache.db and lives in the OS cache directory:

• macOS: ~/Library/Caches/talkbank-chat/talkbank-cache.db
• Linux: ~/.cache/talkbank-chat/talkbank-cache.db
• Windows: %LocalAppData%\talkbank-chat\talkbank-cache.db

It can hold results for large file collections.

To relocate the cache (a different disk, a per-project cache, or an isolated cache for scripted runs), set the TALKBANK_CHAT_CACHE_DIR environment variable to a directory; the database is created directly inside it. This is the supported override on every platform, and the only effective one on Windows, where the default location comes from the system Known Folder API rather than environment variables.

chatter cache stats    # Show hit rates and entry count
chatter cache clear --all

Do not delete the cache file manually while chatter is running.

Reference Corpus Validation

This repository includes a reference corpus at corpus/reference/ (currently ~100 .cha files; verify by find corpus/reference -name '*.cha' | wc -l). The parser must handle every file in this corpus at 100%:

cargo test -p talkbank-parser-tests reference_corpus_parses

This runs the parser equivalence test; each .cha file is its own test, so reports individual failures.

Integration with batchalign

The Batchalign pipeline uses the same Rust core (via PyO3) for CHAT parsing and serialization. Since the 2026-04-28 monorepo merge, Batchalign source lives inside this repository under crates/batchalign-* (the standalone batchalign3 GitHub repo was archived). Files processed by Batchalign produce valid CHAT that passes chatter validate.

This page last changed: 2026-08-27 (commit 8b445304). The whole book last changed: 2026-09-15 (commit bb4bef82).

CI Integration

Status: Current Last updated: 2026-04-13 19:23 EDT

How to use chatter in continuous integration pipelines.

Exit Codes

All examples below rely on exit code 1 to signal validation failure.

Basic Usage

chatter validate corpus/ --quiet --tui-mode disable

• --quiet suppresses per-file success output
• --tui-mode disable prevents interactive TUI (required in non-TTY environments)
• Exit code 0 means all files valid; 1 means errors found

GitHub Actions Example

- name: Validate CHAT corpus
  run: |
    chatter validate corpus/ --quiet --tui-mode disable --format json --audit results.jsonl

- name: Upload validation report
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: validation-report
    path: results.jsonl

The --audit results.jsonl flag streams per-error JSON lines to a file, which is useful for archiving or downstream analysis even when the step fails.

JSON Output for Automation

chatter validate corpus/ --format json --tui-mode disable 2>/dev/null

Each file produces a JSON object on stdout with status, error_count, and errors array. The exit code still reflects overall pass/fail.

Pre-commit Hook

#!/bin/sh
# .git/hooks/pre-commit
chatter validate . --quiet --tui-mode disable

This blocks commits that introduce invalid CHAT files. The hook runs quickly on cached files; only modified files are re-validated.

Suppressing Specific Errors

Some corpora have known issues that should not block CI. Use --suppress to ignore specific error codes or named groups:

chatter validate corpus/ --suppress E726,E727,E728 --tui-mode disable

Or use the named group shorthand:

chatter validate corpus/ --suppress xphon --tui-mode disable

Suppressed errors do not appear in output and do not affect the exit code.

Audit Mode for Large Corpora

For bulk corpus validation where you want a full error database without caching overhead:

chatter validate corpus/ --audit errors.jsonl --tui-mode disable

The --audit flag streams one JSON object per error to the specified file. A summary is printed to stderr at the end.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

CHAT Processing Playbook for Editors and Analysts

Status: Current Last updated: 2026-03-24 00:01 EDT

Objective

Provide practical guidance for non-compiler users who create, edit, and validate CHAT files, with emphasis on error interpretation and correction workflow.

Who This Is For

• Transcript editors,
• corpus curators,
• QA reviewers,
• linguists using tooling outputs but not parser internals.

Core Editing Workflow

1. Open file in editor with CHAT diagnostics enabled.
2. Run validation (single file first, then batch).
3. Fix highest-severity structural issues first (headers, tier markers, unmatched delimiters).
4. Re-run validation and inspect warnings.
5. Only then address style and normalization suggestions.

Error Triage Heuristic

• Errors at file start: likely header formatting or encoding issues.
• Errors at tier prefix: likely malformed */% tier syntax.
• Errors inside words: likely symbol, marker, or annotation boundary issues.
• Repeated same error class: likely one systemic rule violation pattern.

Fast Interpretation Guide

• Error: parser/validator could not accept structure; must fix.
• Warning: valid but suspicious or non-canonical; review strongly recommended.
• Info: advisory normalization or convention hints.

Common Fix Recipes

• Header spacing problems:
  • Ensure expected separators and avoid accidental tabs/spaces drift.
• Unclear language/form markers:
  • Confirm @s usage and suffix ordering with house style guide.
• Duration/annotation confusion:
  • Verify bracketed annotation form and avoid malformed punctuation.
• Dependent tier attachment issues:
  • Ensure % tiers follow intended main tier and keep indentation consistent.

Batch Validation Workflow

1. Validate a small sample first.
2. Group failures by error code.
3. Fix by pattern, not file-by-file random order.
4. Re-run and confirm error count decreases monotonically.
5. Save run report for audit trail.

Collaboration Workflow with Developers

When reporting parsing issues, include:

• exact file path,
• minimal excerpt around failing span,
• observed diagnostic code/message,
• expected behavior (if known).

This reduces back-and-forth and speeds defect triage.

Quality Checklist Before Publishing Corpus Updates

• No unresolved error-level diagnostics.
• Warning classes reviewed and accepted or fixed.
• Participant headers and IDs internally consistent.
• Roundtrip serialization check passes for representative samples.
• Changelog note recorded for major normalization edits.

Training Recommendations

• Maintain short examples for each common error class.
• Provide editor cheat sheet for tier prefixes and marker syntax.
• Run periodic QA calibration sessions across editors.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Sanitize (chatter debug sanitize)

Status: Current Last updated: 2026-09-09 08:49 EDT

chatter debug sanitize strips contributor lexical content from a CHAT file while preserving structure (timing bullets, %wor per-word bullets, speaker codes, dependent-tier scaffolding, structural counts, POS tags, language markers). Output is structurally identical to the input but contains no participant words, names, or free-text annotations.

The command exists so engineering tooling, including LLM-assisted debugging, can operate on protected-corpus files (aphasia/, dementia/, rhd/, fluency/Password/, clinical-children corpora, etc.) without exposing contributor speech to commercial LLM services.

When to use it

Run chatter debug sanitize on the source file before loading it into any tool (LLM-backed debugger, scratch directory, screen-shareable session) where you don’t want participant content visible.

When you need to ask a contributor for help debugging a specific file, frame the request as “run the sanitizer locally and send me the output” rather than asking for the raw file.

Usage

# Write sanitized output to stdout
chatter debug sanitize input.cha

# Write sanitized output to a file
chatter debug sanitize input.cha --output sanitized.cha

Working location for sanitized files: prefer a stable, non-/tmp scratch directory (e.g. set TB_SCRATCH_DIR to a per-project dir under your workstation’s persistent storage) for any state that should outlive a single command. macOS clears /tmp on reboot.

What is preserved (byte-exact)

• Timing bullets •start_end• on the main tier.
• %wor per-word bullets (•start_end• after each word); the words beside them become the main tier’s placeholders, below.
• Speaker codes (*PAR, *INV, *CHI, …).
• Utterance count, word count per utterance, dependent-tier count.
• Structural markers: compound +, clitic ~, CA elements, overlap points, lengthening, stress markers, syllable pause, underline begin/end, proper-noun @n markers.
• Language markers (@s:LANG), form types (@a, @b), POS tags ($adj, $n).
• Headers: @Languages, @Birth, @Date, @Media, @PID, @L1Of, @Begin/@End/@UTF8.
• %mor POS categories and morphological features (e.g., n|, -Past).
• %gra (numeric grammatical relations) and %tim (timing).
• Untranscribed tokens xxx / yyy / www, preserving them changes semantic meaning, so they pass through unchanged.

What is replaced or redacted

Determinism + Idempotence

Placeholder generation uses one monotonic counter in deterministic document traversal order. Two consequences:

• Deterministic: sanitizing the same input twice produces byte-identical output.
• Idempotent: sanitizing a sanitized file produces the same file again, no double-replacement, no shifting placeholder numbers.

Pipeline

flowchart LR
    Input["Source .cha\n(protected corpus)"] --> Parser["TreeSitterParser\n(talkbank-parser)"]
    Parser --> Model["ChatFile model\n(talkbank-model)"]
    Model --> Sanitize["sanitize()\n(talkbank-transform::redact)"]
    Sanitize --> Walker["walk_words_mut\n+ header walker\n+ dep-tier walker\n+ scoped-annot walker"]
    Walker --> WordMutation["Typed Word replacement\n+ derived-text refresh"]
    WordMutation --> Mutated["Mutated ChatFile\n(placeholders + redactions)"]
    Mutated --> Writer["WriteChat\n(byte-exact bullets)"]
    Writer --> Output["Sanitized .cha\n(scratch path)"]

The walker step replaces lexical segments in the typed word-content sequence, mutates MorWord.lemma fields, redacts free-text header / dep-tier / scoped-annotation strings, and drops phonological tiers. WriteChat then re-serializes, and because it serializes from typed content (not from Word.raw_text), every CA element, compound marker, clitic boundary, and timing bullet round-trips byte-exact.

Word content is externally read-only. Replacement must use Word’s mutation methods, which invalidate the derived cleaned_text cache. The sanitizer also rebuilds raw_text from the complete sanitized typed word on every path. That includes parser-recovery words containing only structural elements and the xxx/yyy/www pass-through path. This prevents either JSON-facing string field from retaining source lexical material or losing nonlexical word markers.

Out of v1 scope

Documented for transparency; v2 work:

• Speaker-code anonymization (graph rewrite across @Participants, @ID, *SPK:, @Birth, @L1Of).
• @Birth / @Date fuzzing (exact birth dates can be identifying).
• @Media filename redaction.
• Audio-side sanitization. (Audio bytes are never touched by the sanitizer; the audio stays at its original path.)
• “Unsanitize” or round-trip mapping. Explicitly not built, the sanitizer is one-way, the mapping table that would reverse it is the exact artifact we don’t want to exist.

Implementation

Library module: talkbank_transform::redact. CLI surface: chatter debug sanitize. The strict policy is the only public preset in v1; future variants can grow on SanitizationPolicy.

This page last changed: 2026-09-09 (commit a30c20c4). The whole book last changed: 2026-09-15 (commit bb4bef82).

Speaker-ID (chatter speaker-id)

Status: Draft Last modified: 2026-08-30 16:02 EDT

chatter speaker-id assigns CHAT-conformant speaker codes and role tags to a CHAT file whose speakers carry anonymous or placeholder labels (typically the output of an ASR system that labels speakers as PAR0, PAR1, …). It is the bridge between an ASR pipeline that does not understand speaker roles and a CHAT pipeline that does.

The command is structural: it does not modify utterance content, does not run audio analysis, does not infer speaker identity from voice features. Its inputs are the CHAT file to relabel plus an identification signal (reference transcript, explicit mapping, or saved override record); its output is the same CHAT file with speaker codes rewritten and @Participants / @ID headers reconciled.

When to use it

Whenever you have a CHAT file with placeholder speaker codes that need to become CHAT-conformant codes before downstream tooling can process the file meaningfully. The canonical case is an ASR system that emits CHAT but does not know which speaker is the child, parent, clinician, etc.

A complete pipeline that consumes ASR output and produces a publishable CHAT file goes:

flowchart LR
    Media --> Transcribe
    Transcribe["batchalign3 transcribe<br/>ASR"] --> AsrAnon["asr.cha<br/>PAR0, PAR1, ..."]
    Ref["reference.cha<br/>target speakers only"] -.->|reference signal| SpkId
    AsrAnon --> SpkId
    SpkId["chatter speaker-id<br/>(this page)"] --> AsrLabeled["asr-labeled.cha<br/>CHI, INV, MOT, ..."]
    AsrLabeled --> Merge["chatter merge"]
    Ref --> Merge
    Merge --> Aligned["batchalign3 align"]

The speaker-id stage is the single point in the pipeline where “which anonymous speaker corresponds to which CHAT role” is decided. Downstream stages (chatter merge, batchalign3 align, batchalign3 morphotag) all trust that the labels they receive are correct.

Identification modes

Three mutually-exclusive modes, exactly one of which must be selected:

1. Reference mode

The most common case: a separate CHAT file already exists that covers the same media and contains an authoritative speaker (typically the hand-transcribed target speaker). The reference file’s anchor speaker tells us what that speaker’s content looks like; speaker-id finds the matching speaker in the input by text similarity.

The matching algorithm is multiset Jaccard over bags of content tokens, see “Algorithm” below for the full specification. The ASR speaker whose bag-of-words best matches the reference anchor’s bag-of-words is taken as the same speaker, and is marked for drop in the output (because the reference file authoritatively covers them, the downstream chatter merge stage will pull their utterances from the reference, not from this file). The remaining speakers are renamed to the role specified by --inserted-role.

If the Jaccard margin between the winning speaker and the runner-up is below --confidence-threshold, the command refuses to auto-decide. The operator must either lower the threshold (not recommended without spot-checking), supply an explicit mapping (--mapping), or load a previously-adjudicated override (--override-file).

2. Explicit-mapping mode

The operator already knows the mapping (typically because they listened to the audio, or because the contributor’s data sheet documents it). They supply it directly.

chatter speaker-id input.cha \
  --mapping "PAR0=INV:Investigator,PAR1=drop" \
  -o relabeled.cha

The grammar for --mapping:

• One or more comma-separated assignments.
• OLD=CODE:ROLE renames OLD to CODE with role tag ROLE.
• OLD=drop removes OLD’s utterances entirely.
• Every speaker present in the input must be named in the mapping (no defaulting). This is intentional, we want operator decisions to be explicit.

3. Override-file mode

The operator has previously adjudicated this session (perhaps through an interactive review tool) and saved the decision to a shared override file. speaker-id reads the file, finds the entry for this session, and applies it. See “Override file format” below.

chatter speaker-id input.cha \
  --override-file batch-2026-05-27.overrides.toml \
  --session-id NF203-2 \
  -o relabeled.cha

This mode is the production substrate for batch workflows: the orchestrator first runs chatter speaker-id in reference mode for every session; for any session that exits with low-confidence, the operator works through an adjudication tool that writes to the override file; the orchestrator then re-runs chatter speaker-id in override-file mode for those sessions.

CLI contract

chatter speaker-id <INPUT> [OPTIONS]

ARGUMENTS:
  <INPUT>  Path to the CHAT file to relabel.

OPERATION MODES (exactly one required):

  REFERENCE MODE:
    --reference <FILE>
    --anchor <SPEAKER>
    --inserted-role <CODE>:<TAG>[,<CODE>:<TAG>...]

  EXPLICIT-MAPPING MODE:
    --mapping <SPEC>

  OVERRIDE-FILE MODE:
    --override-file <FILE>
    --session-id <ID>

REFERENCE-MODE OPTIONS:
  --confidence-threshold <FLOAT>
      Minimum Jaccard margin (winner_score / loser_score) for the
      command to auto-decide. Below threshold: exit code 4. The
      command prints per-speaker scores to stderr so the operator
      can inspect. Default: 2.0.

  --write-match-report <NEW-FILE.json>
      Write a typed report for the complete reference-mode attempt.
      Matched outcomes record reference, donor, shared, and union token
      counts plus the derived score and margin. Structural and input
      refusals record their own outcome-specific evidence. The report
      never overwrites an existing file.

  --write-override <FILE>
      When auto-decide succeeds, append the decision to FILE in
      override-file format (creates if missing). Captures the
      audit trail of a batch run.

COMMON OPTIONS:
  -o, --output <PATH>
      Write relabeled CHAT to PATH. Default: stdout.

The operator identity and any free-text note for a session are set
when an operator confirms it through `chatter adjudicate` (see the
merge workflow), not on this command.

Exit codes:

What the output guarantees

These are testable invariants. Every release verifies them against the reference corpus.

Speaker codes match the supplied mapping

For every speaker in the input file:

• If the mapping marks the speaker for drop, none of their utterances appear in the output, AND their @ID row (if any) is removed from the headers, AND their entry is removed from the @Participants header.
• If the mapping marks the speaker for rename, every main-tier line *OLD:\t... becomes *NEW:\t... byte-stable except for the speaker code prefix. The @ID row’s third pipe-separated field (speaker code) and eighth field (role tag) are rewritten; other @ID fields are preserved. The @Participants entry’s code and role-tag tokens are rewritten; any intervening tokens (corpus ID, participant name) are preserved.
• Speakers not in the mapping are passed through unchanged. (In modes 1 and 3, all speakers are assigned automatically; in mode 2, “all speakers must be in the mapping” is a precondition.)

Utterance content is byte-stable except for the speaker prefix

For every retained utterance, every byte EXCEPT the leading *CODE:\t prefix is preserved verbatim. Dependent tiers attached to the utterance are preserved exactly. NAK-delimited time bullets, CHAT markup, special-form annotations, paralinguistic codes, retracing scopes, all untouched.

Headers reconcile per a fixed table

Provenance is captured if --write-override is set

When --write-override <FILE> is supplied AND the command succeeds in reference mode, an entry is appended to FILE recording the session ID (derived from the input filename stem unless overridden), the per-speaker Jaccard scores, the chosen mapping, the operator, and an ISO 8601 timestamp. The format is specified in “Override file format” below. The operator identity and any free-text note are set later, when a session is confirmed via chatter adjudicate.

This is the audit-trail mechanism: a year from now, a researcher who asks “why was PAR0 labeled INV in this session?” can read the override entry and see the scores, the operator, and any notes the operator added.

Algorithm (reference mode)

Token cleaning

Both the reference anchor’s bag of words and each input speaker’s bag of words are built by walking the typed CHAT AST and emitting content tokens. The cleaner strips:

• NAK-delimited time bullets
• bracket-annotated markup [*], [//], [/], [=! ...], etc.
• angle-bracket retracing scope (<...>, unwrap, keep inner text)
• terminator variants +//., +..., +/., +!?, etc.
• filled-pause and phonological-fragment markers &-..., &+...
• unintelligible placeholders xxx, yyy, www
• zero-realization markers 0
• special-form suffixes (word@l → word)
• CHAT compound underscores (Valentine's_Day → Valentine s Day)
• punctuation, then lowercase, then filter to alpha-only tokens of length ≥ 2

Both sides are cleaned identically so the comparison is apples-to-apples. This is the same cleaner specified in the reference corpus under spec/constructs/speaker-id/token-cleaner/.

Multiset Jaccard

For two bags-of-words A and B (counted multisets):

J(A, B) = sum_w min(A[w], B[w])  /  sum_w max(A[w], B[w])

Range [0, 1]. The multiset (rather than set) form rewards speakers who say similar things to the anchor in similar volume, not just speakers whose vocabulary happens to intersect.

Decision

scores  = { speaker: J(anchor_bag, speaker_bag) for speaker in input }
winner  = argmax(scores)
loser   = argmax(scores - {winner})
margin  = scores[winner] / scores[loser]    # ∞ when loser score = 0

• winner is the input speaker whose content matches the reference anchor’s content best → marked for drop (the reference authoritatively covers them).
• loser (and any other lower-scoring speakers, in the multi-speaker case) → renamed to the role given by --inserted-role.

Match-evidence report

--write-match-report preserves the observations behind the scalar score and the typed reason when matching could not begin. This matters because a ratio alone cannot distinguish a winner supported by one shared token from one supported by hundreds. The JSON schema records, for each donor speaker, reference_tokens, donor_tokens, shared_tokens, union_tokens, and the derived score. Its margin is one of no_information, finite, or unbounded; no-information and a zero-scoring runner-up are not represented by floating-point sentinels.

The outer outcome is accepted, low_confidence, reference_missing_anchor, donor_too_few_speakers, or input_rejected. Matched outcomes contain the lexical report; other outcomes cannot pretend to contain one. Input rejection records donor versus reference, the typed pipeline-failure category, and TalkBank diagnostic codes when available.

The option is intended for audit and calibration tooling. It does not change which speaker wins or the confidence threshold. Chatter stages the complete JSON beside its destination and persists it without clobbering, so a later run cannot silently replace the evidence used for an earlier decision and a failed write cannot leave a partial final report. The no-clobber persist is not guaranteed to be atomic on every platform; an interrupted persist can leave the staging link behind, but it never replaces an existing report.

If margin < --confidence-threshold (default 2.0), the command exits with code 4 and prints per-speaker scores to stderr. The operator must inspect, adjudicate, and re-run with --mapping or --override-file.

Why this algorithm

The choice was empirical, not theoretical, and was made against a calibration set of CHAT files paired with their corresponding ASR output. Two earlier candidates were tested first and rejected:

• Raw temporal-overlap (sum of ms of an input speaker’s activity inside the anchor’s bullet windows): too weak on real data. Hand transcripts often place per-utterance time bullets as end-to-end segmentation boundaries covering 95-99% of the session timeline, rather than as tight “speaker active here” windows. Both input speakers fall almost entirely “inside” the anchor’s bullet windows and the signal disappears.
• Speaker purity (fraction of each input speaker’s activity falling inside anchor windows): same root cause, same failure.

Multiset Jaccard over content tokens succeeded on every session of the calibration set. The borderline cases (margin below 2.0x) clustered around tasks where the non-anchor speaker shares vocabulary with the anchor by the structure of the task, e.g. a clinician describing the same scene the child is also describing in a picture-narrative task. These borderline cases are the reason for the conservative threshold and the --mapping/--override-file escape hatches; the algorithm correctly refuses to auto-decide them rather than silently picking wrong.

Override file format

The override file is a UTF-8 TOML document with one [<session_id>] table per decision. A minimal entry:

schema_version = 2

[session-101-t1]
mode = "auto"
adult_roles = { PAR0 = { code = "INV", tag = "Investigator" } }
mapping = { PAR0 = "rename", PAR1 = "drop" }
scores = { PAR0 = 0.1931, PAR1 = 0.7347 }
margin = 3.81
operator = "alice"
decided_at = 2026-05-27T08:41:00-04:00

The complete schema specification, every field, every type, every mode-semantics rule, the strict refuse-with-clear-error versioning policy, and worked examples for auto/explicit/replay/diarization-mixed cases, is on the dedicated reference page: Merge Override File Format.

Highlights from the reference:

• mode = "auto" | "explicit" | "override" records how the decision was made (informational for audit trail; behavior at apply time is the same).
• adult_roles maps each renamed speaker’s donor code to its own role assignment: adult_roles[<donor_code>].code is the CHAT speaker code (INV, MOT, FAT, PAR, …); .tag is the CHAT role-tag (Investigator, Mother, …). Renamed speakers in one entry may share a role or each carry a distinct one.
• mapping must cover every speaker in the input, no defaulting.
• scores and margin are optional but the writer always records them when an auto attempt produced them (even when the final decision was operator-supplied).
• flags carries operator-supplied markers like "diarization-mixed" for unusual cases. Unknown strings are preserved verbatim.

Preconditions

chatter speaker-id refuses (exit code 2) if any hold:

Reference mode

• The reference file has no utterances for --anchor
• The reference file fails to parse
• The input file has fewer than 2 distinct speakers (no discrimination problem)

Explicit-mapping mode

• A speaker in the mapping is not present in the input
• A speaker in the input is not covered by the mapping (no defaulting)

Override-file mode

• The override file does not contain a <session-id> entry
• The entry’s mapping references a speaker not in the input
• The entry’s mapping does not cover every speaker in the input

What chatter speaker-id is NOT

• Not voice diarization. Use Batchalign’s ASR pipeline upstream; the labels this command consumes are the labels Batchalign emits.
• Not content correction. If the speaker the command identifies has been mis-transcribed by ASR, this command does not fix that , re-run ASR with a better engine.
• Not a merge. This command operates on a single CHAT file. To combine the relabeled file with the reference, use chatter merge.
• Not interactive. chatter speaker-id is batch-only: it succeeds, refuses, or fails. The interactive review that resolves a low-confidence refusal into an override-file entry is a separate command, chatter adjudicate, run as part of the merge workflow.

Worked example

A typical fully-automated reference-mode call from an orchestrator script:

chatter speaker-id asr-anonymous.cha \
  --reference hand-transcript.cha \
  --anchor CHI \
  --inserted-role INV:Investigator \
  --confidence-threshold 2.0 \
  --write-override batch.overrides.toml \
  -o asr-labeled.cha

For a session this refused (e.g., shared-vocabulary narrative task with margin 1.82x), the orchestrator captures the failure and the operator later resolves it:

# Inspect the scores the command emitted to stderr:
#   PAR0=0.6286  PAR1=0.3457  margin=1.82x  threshold=2.0
# Operator listens to a few seconds of audio and confirms PAR0 is
# the child:

chatter speaker-id asr-anonymous.cha \
  --mapping "PAR0=drop,PAR1=INV:Investigator" \
  --write-override batch.overrides.toml \
  -o asr-labeled.cha

Later, if anyone re-runs the batch, they use override-file mode:

chatter speaker-id asr-anonymous.cha \
  --override-file batch.overrides.toml \
  --session-id NF204-2 \
  -o asr-labeled.cha

The same asr-labeled.cha content is produced; the audit trail remains intact.

Implementation notes (for contributors)

• Source: crates/talkbank-transform/src/speaker_id/.
• CLI surface: crates/chatter/src/commands/speaker_id/.
• CHAT-domain types such as SpeakerCode and ParticipantRole live in talkbank-model. Speaker-identification types such as MappingSpec, MergeOverride, LexicalMatchEvidence, JaccardScore, ConfidenceThreshold, and ConfidenceMargin live beside the algorithms in talkbank-transform::speaker_id.
• The Jaccard cleaner walks talkbank-model::ChatFile directly via the existing content walker (talkbank-model::walk_words); it does NOT re-implement CHAT parsing or use regex on raw bytes for tokenization.
• Spec entries for the cleaner and the algorithm live in spec/constructs/speaker-id/. Every invariant on this page has a spec; regenerate them with the current spec/tools commands from Spec Workflow.
• The override-file reader/writer is a typed serde round-trip on a TOML representation owned by talkbank-transform::speaker_id, so consumers use one shared parser rather than duplicating the format.

This page last changed: 2026-08-30 (commit 733da964). The whole book last changed: 2026-09-15 (commit bb4bef82).

Rediarize (chatter rediarize)

Status: Draft Last updated: 2026-08-30 16:26 EDT

chatter rediarize re-attributes utterance speakers in a CHAT file from an external diarization. Given a transcript whose utterances carry media time bullets and a JSON file of timestamped speaker turns produced by a dedicated diarizer (for example pyannote), it reassigns each utterance’s main-tier speaker to the diarization track that covers the utterance’s time span the most, keeping the utterance content (the words) byte-stable.

The command exists for a specific, common failure shape: ASR systems with bundled diarization (Rev.AI and others) auto-detect the speaker count and can under-count on hard material such as child-adult overlap, collapsing three or four real voices into two tracks. The ASR words are usually fine; the attribution is what is wrong. A dedicated diarizer recounts the voices correctly, and rediarize reconciles its turns with the existing transcript so you keep the good words and replace only the bad attribution.

The command is structural and audio-free: it never touches the recording. The diarizer runs elsewhere (any tool, any model) and hands its result across a documented JSON boundary.

Pipeline position

flowchart LR
    Media["recording\n(audio)"] --> Diarizer["external diarizer\n(e.g. pyannote)"]
    Diarizer --> Turns["turns.json\n(documented format below)"]
    Media --> Asr["ASR with bundled\ndiarization"]
    Asr --> AsrCha["asr.cha\ngood words,\nsuspect speaker tracks"]
    AsrCha --> Rediarize["chatter rediarize\n(this page)"]
    Turns --> Rediarize
    Rediarize --> Fixed["rediarized.cha\nPAR0..PARn correctly\nseparated tracks"]
    Fixed --> SpkId["chatter speaker-id\n(assign real roles)"]

rediarize fixes WHICH anonymous track owns each utterance; it does not decide who each track is. Role assignment (child, mother, investigator, …) is chatter speaker-id’s job, downstream.

Usage

chatter rediarize INPUT.cha --turns TURNS.json -o OUTPUT.cha

Omitting -o prints the rewritten CHAT to stdout.

A summary is reported on stderr after the rewrite (stderr so that a stdout CHAT stream stays clean when -o is omitted):

rediarize: 214 reassigned, 671 unchanged, 7 flagged

Flagged utterances (see below) are listed individually with their utterance index, kept speaker, and reason.

--contested-at SHARE additionally reports utterances whose time is split between tracks; see Contested utterances.

Machine-readable summary (--summary-json)

Batch drivers looping rediarize over a corpus should not scrape the stderr text. --summary-json PATH additionally writes the outcome as JSON:

chatter rediarize INPUT.cha --turns TURNS.json \
    -o OUTPUT.cha --summary-json SUMMARY.json

{
  "source": "pyannote/speaker-diarization-community-1",
  "reassigned": 747,
  "unchanged": 145,
  "flagged": [
    {"utterance_index": 12, "kept_speaker": "PAR1",
     "reason": "no_overlapping_turn"}
  ],
  "contested": [
    {"utterance_index": 41, "assigned": "PAR2",
     "ownership": {"shares": [["PAR2", 600], ["PAR1", 400]],
                   "total_ms": 1000}}
  ]
}

• source: the turns file’s provenance, passed through (null if the turns file carried none).

• reassigned / unchanged: utterance counts. unchanged includes flagged utterances (they kept their speaker), so the file’s total bulleted-tier utterance count is reassigned + unchanged.

• flagged: every declined reattribution, never truncated (the stderr listing caps at 20 detail lines; this list is complete). utterance_index is the 0-based position among main-tier lines; reason is "no_bullet" or "no_overlapping_turn".

• contested: utterances whose time was meaningfully split between tracks, empty unless --contested-at was given. These were still reattributed, to assigned, so they are NOT in flagged, which means “declined”. ownership.shares is every overlapping track with its union-held milliseconds, descending; overlapping turns for the SAME track count their shared interval once. ownership.total_ms is the sum of those per-track values and is the denominator. Simultaneous DIFFERENT tracks each retain the shared interval, so total_ms can exceed the utterance bullet’s duration. The whole distribution is emitted rather than a winner and a runner-up, because that narrower shape cannot tell a 55/45 split from 55/23/22 and the difference is the point.

Field names and the reason strings are a stable output contract. The summary is written only on exit 0, after the CHAT output.

Contested utterances (--contested-at)

An utterance’s bullet can overlap turns from more than one track. The tool assigns it to the track holding the most of it, which is the best available answer, but “most” can mean 95% or 34%, and those are different situations that the output otherwise reports identically.

chatter rediarize INPUT.cha --turns TURNS.json --contested-at 0.25

Reports an utterance as contested when the RUNNER-UP track holds at least that share of the total track-held time. Two rivals at 20% each is a different situation from one at 40%, and this is the latter question. Same-track duplicate coverage never inflates the denominator; cross-track overlap remains evidence for both simultaneous speakers.

There is no default, deliberately. Omit the flag and nothing is reported as contested. What share makes an utterance genuinely mixed has not been measured against human listening, so shipping a number here would hand every user a constant wearing this tool’s authority. Supply one you can defend, or none.

The flag changes reporting only: placement is byte-identical with and without it. A value outside 0.0 to 1.0, or NaN, fails the command before any file is read, rather than silently meaning “nothing is ever contested”.

Known limitation: per-track union-held totals cannot distinguish a speaker change INSIDE an utterance (one track holds the first half, the other the second) from crosstalk (both across the whole), and those want opposite remedies. Contested says ownership is divided, not how it is arranged on the timeline.

The turns JSON format

The --turns file is the corpus-agnostic seam between the diarizer and chatter. Producing it from any given diarizer’s native output is the caller’s concern; the format is:

{
  "source": "pyannote/speaker-diarization-community-1",
  "turns": [
    {"track": "PAR0", "start_ms": 12063, "end_ms": 17024},
    {"track": "PAR1", "start_ms": 13379, "end_ms": 14375}
  ]
}

• source (optional): free-form provenance, typically the diarizer model name. Not interpreted, but useful in audit trails.
• turns (required): the timestamped segments. Each has:
  • track: the anonymous CHAT speaker code this segment belongs to (PAR0, PAR1, …). The producer chooses the codes; a deterministic mapping from diarizer-native labels (for example pyannote’s SPEAKER_00) is recommended.
  • start_ms / end_ms: the segment’s media time span in integer milliseconds, half-open [start_ms, end_ms), with end_ms >= start_ms.

Turns MAY overlap each other (diarizers that permit overlapping speech produce such turns). Overlap between turns for the same track is unioned; overlap between different tracks is retained for each track as crosstalk evidence. Input order is immaterial: chatter admits the turns to a typed, start-ordered timeline before attribution. Unknown fields anywhere in the file are rejected, so a misspelled field fails loudly instead of being silently ignored.

Behavior contract

• Every utterance with a time bullet is assigned to the track with the greatest union of millisecond coverage against the bullet’s span. An utterance already on its max-overlap track counts as unchanged.
• An utterance with no bullet, or whose bullet overlaps no turn at all, keeps its existing speaker and is flagged in the summary. Ambiguity is surfaced, never silently guessed.
• @Participants and @ID headers are reconciled to declare exactly the set of tracks the output actually uses: new tracks get entries cloned from an existing participant (same role), declarations for tracks no longer used are dropped.
• Utterance content, dependent tiers, and all other headers are preserved as-is.

Exit codes

On any non-zero exit, no output file is written.

Worked example

A recording of one child and two parents, transcribed by an ASR whose bundled diarization auto-detected two speakers (the two adults were merged into one track). A dedicated diarizer found three voices and produced turns.json with PAR0/PAR1/PAR2. Then:

chatter rediarize session.cha --turns turns.json -o session-3spk.cha
chatter validate session-3spk.cha

splits the merged adult track by time, declares PAR2 in the headers, and leaves every word as the ASR wrote it. The output then flows into chatter speaker-id (or the merge workflow) to name the three tracks.

This page last changed: 2026-08-30 (commit 733da964). The whole book last changed: 2026-09-15 (commit bb4bef82).

Merge (chatter merge)

Status: Draft Last modified: 2026-09-15 12:06 EDT

chatter merge combines two CHAT transcripts that cover the same media recording into one. The caller designates which speakers’ utterances are authoritative in which file; the merged output interleaves them by time while byte-preserving every utterance from its designated source.

The command is structural: it does not invent or rewrite utterance content, does not run ASR, does not run forced alignment, does not infer speaker identity. It is the moment in a multi-input CHAT workflow where two parsed transcripts become one.

The library also exposes merge_chat_files_by_source_order for inputs with untimed utterances. It preserves source-relative order and derives cross-source placement only from genuine time anchors in the selected AST utterances. It never adds a missing time bullet. If the source chains and strict anchor comparisons do not uniquely order competing utterances or section markers, it refuses the merge. Two utterances from different sources with exactly equal start times are serialized reference first: a stable convention, not a claim about which was spoken first. Section markers at the same instant are not tie-broken; that ambiguity is refused. Intervals may overlap when their distinct starts establish order. This API does not establish common-media identity or speaker authority for the caller. The command and existing merge_chat_files timing contract remain unchanged.

Every library merge first assembles a MergeDraft, then validates it; only a validated draft becomes a Merged result. For a selected-donor merge, merge_chat_files_with_donor_selection_draft returns the draft before validation. A draft admits exactly one edit, replacing an output utterance’s end-of-line bullet, so it cannot gain, lose or reorder utterances. A caller can use it to repair timing that would fail validation and then call validate. The draft records every edit itself: MergeDraft::bullet_edits and, after validation, Merged::bullet_edits give each edited utterance’s assembled and replacement timing, one record per utterance in output order. Evidence the merge computed at assembly (gem exterior placements, draft order reviews, donor fates) describes the assembled timing; for an edited utterance the record says what replaced it.

When to use it

Whenever you have two valid CHAT files of the same recording and you want a single combined CHAT file out, with explicit per-speaker provenance.

Two recurring shapes from real TalkBank workflows:

• Hand-coded target speaker + ASR everyone else. A contributor has hand-transcribed only the target speaker (often the child in child-language research) with rich disfluency and error coding, and separately someone runs ASR on the same media to produce a rough-but-complete transcript with all speakers. chatter merge combines them with the hand-coded target speaker’s utterances byte-preserved and the other speakers spliced in from the ASR file.
• Older hand transcript + later supplementary transcription. A legacy CHAT file covers most of the recording; a newer pass transcribes additional content (an investigator’s turns, a parent’s turns, a second target child). Merge with --retain listing the speakers whose content lives in the legacy file.

In both shapes the speakers are the unit of authority, not the files. chatter merge’s job is to express that mapping cleanly.

Conceptual model

A CHAT file describes utterances on a shared media timeline. Two CHAT files of the same media share the same timeline; their utterance sets may overlap (same speech transcribed twice) or be disjoint (each file covers different speakers). The merged output is a single CHAT file on the same timeline whose utterance set is the disjoint union of:

• the utterances of every speaker listed in --retain from the first input file, and
• the utterances of every speaker NOT listed in --retain from the second input file.

Retained-speaker utterances from the first file are kept byte-for-byte identical, including every dependent tier they own (%wor, %mor, %gra, %com, %pho, …). Inserted-speaker utterances from the second file have their downstream-generated dependent tiers (%wor/%mor/%gra/%pho, anything a later pipeline stage will regenerate) stripped before insertion, so the merged file is in a clean state for batchalign3 align and batchalign3 morphotag to own those tiers authoritatively post-merge.

flowchart LR
    File1["File 1<br/>any CHAT file"] --> Merge
    File2["File 2<br/>any CHAT file<br/>(same media)"] --> Merge
    Retain["--retain CHI[,SPK,…]"] -.-> Merge
    Merge["chatter merge<br/>(structural)"] --> Out["Merged CHAT file<br/>retained speakers: byte-stable from File 1<br/>inserted speakers: from File 2,<br/>derived tiers stripped"]

CLI contract

chatter merge <FILE1> <FILE2> --retain <SPEAKER_LIST> [OPTIONS]

ARGUMENTS:
  <FILE1>  Path to the first CHAT file. Speakers listed in --retain are
           taken from here, byte-preserved.
  <FILE2>  Path to the second CHAT file. All other speakers are taken
           from here.

REQUIRED OPTIONS:
  --retain <SPEAKER>[,<SPEAKER>...]
           Comma-separated list of speaker codes (e.g. CHI, or
           CHI,SI2). These speakers' utterances come from <FILE1>;
           everything else comes from <FILE2>.

OPTIONS:
  -o, --output <PATH>
           Write merged output to PATH. Default: stdout.

The CLI uses the default stripping set. The Rust merge_chat_files API accepts an explicit stripping list, including an empty list to preserve all dependent tiers. The CLI does not expose --strip-tiers or --allow-bullet-drift.

Exit codes:

What the merged output guarantees

These are testable invariants. Every release verifies them against the reference corpus.

Retained speakers are byte-stable

For every speaker code in --retain, every main-tier line and every dependent-tier line attached to that speaker in <FILE1> appears byte-for-byte identical in the merged output, in the same relative order they appeared in <FILE1>. CHAT markup, NAK-delimited time bullets, paralinguistic annotations, retracing scope, terminator variants, special-form @l/@n/@c suffixes, all preserved.

This is the core semantic guarantee of merge: if you hand-coded disfluency on the target speaker, the disfluency coding survives the merge without any structural change.

Inserted speakers’ downstream-generated tiers are stripped

For every speaker code in <FILE2> that is NOT in --retain, the utterance is included in the merged output with its main tier preserved verbatim BUT with %wor, %mor, %gra, and %pho removed (configurable through the Rust API). Other dependent tiers (%com, %spa, %act, %sit, %add, contributor-specific tiers) are preserved.

The rationale: batchalign3 align and batchalign3 morphotag are the authoritative source stages for these tiers in the post-merge pipeline. Carrying inserted-speaker %wor across the merge would leave the merged file in a half-state, some utterances would have %wor, others would not, and downstream behavior on mixed inputs is undefined. The contract is: enter the post-merge stages in a clean state, exit with the tier present and consistent across every utterance.

Source order is preserved

Utterances in the merged output appear in ascending order by their start time bullet (\\x15START_END\\x15, milliseconds). Where two utterances have identical start times, the first-file utterance comes first. This is a forward merge of admitted source sequences, never a global sort. Each selected utterance must have a time bullet, and selected utterance starts must be nondecreasing within each source. Missing positions or a source time reversal are precondition failures; resolve placement before merging rather than changing source order inside merge. Overlapping spans are allowed when their start times remain ordered.

Headers remain events in the source sequence. Dependent tiers remain ordered inside their owning utterance. Ordinary headers are emitted when their source reaches them; reference ordinary headers win simultaneous frontiers. Section markers (@Bg, @Eg, @G) instead carry a timing bracket from the preceding selected utterance’s end and following selected utterance’s start. A competing turn starting before the bracket precedes the marker; one starting at or after its upper bound follows it. A start inside an uncertain gap, overlapping neighbor bounds, or indeterminate cross-source section order causes refusal. Missing one-sided bounds are not invented. Whole utterances are placed by onset; the transform does not split speech spanning a boundary.

The reference @End follows all donor events. These rules preserve source-relative order without inventing section timestamps. An assembled transcript must pass full model validation, including tier alignment, before it can become a Merged result. Conflicting section markers are refused with diagnostics; independently valid sources are not proof of a valid merge.

Merged and Reported retain an immutable ValidChatFile proof internally. Reported::into_file deliberately consumes that proof to allow further edits; the caller must validate again after editing.

Time bullets are pass-through

chatter merge does NOT recompute, smooth, or refresh time bullets. The bullets in the merged output are exactly those that appeared in the source files. If <FILE2> had %wor rows whose first/last word times implied a slightly different utterance span than the main-tier bullet, the main-tier bullet wins (it was the contract before merge).

Missing main-tier bullets are refused. This transform does not infer a replacement from %wor; any timing repair belongs before placement admission.

Header reconciliation

The merged file’s headers are constructed deterministically from the two inputs:

Opening metadata ends at the first utterance or gem marker. Donor body headers are streamed in their original order, including comments after utterances; they are not collected into opening metadata. Reference headers retain their position relative to reference speech. The participant and ID reconciliation above is an explicit metadata operation, not permission to reorder body content. Donor IDs extend the reference’s contiguous ID block; donor opening comments follow the reference opening metadata. A donor ID following an opening comment is refused rather than silently reordered to satisfy the ID-block rule.

The @Media modality field (audio vs video) is a known divergence point: when ASR runs against an mp4, it may write video on its input but emit audio on its output. File 1’s modality wins, as with all @Media content; no warning is emitted for modality mismatch.

Overlap markup is NOT injected

When an inserted-speaker utterance temporally overlaps a retained-speaker utterance, chatter merge does NOT inject CHAT [>] / [<] / angle-bracket-scoped overlap markers. The time bullets carry overlap information; markers are a CLAN-era surface convention that the output of chatter merge deliberately omits.

The retained speakers’ existing overlap markers (if File 1 already contains some) are preserved byte-stably under the byte-preservation rule above.

Preconditions

chatter merge refuses (exit code 2) if any of these hold:

• File 1 declares no utterances for any speaker in --retain.
• File 1 has no time-bulleted utterances at all (no shared timeline to merge against).
• The two files’ @Languages headers disagree.
• A speaker code appears in both files but not in --retain (use --retain to disambiguate).
• File 2 is missing or unparseable.

chatter merge WARNS and proceeds on:

• A File 1 speaker not in --retain. Every one of its utterances is dropped, but its @Participants row survives, so the output declares a speaker who says nothing. The warning names each such speaker and how many utterances it lost. AmbiguousSpeaker does not cover this: that refuses only when a code appears in BOTH files. chatter pipeline prints the same warning from the same reporter.

chatter merge accepts these silently BY DESIGN, because a diagnostic on every run is one nobody reads:

• Small backward-time bullets in either input (one utterance ends slightly after the next starts), common in hand transcripts, not corrupting; downstream batchalign3 align cleans these.
• File 1 has fewer utterances than File 2, or vice versa. Every merge has this; it is the normal case.
• Donor headers other than @ID and @Comment are not carried into the output. That is the documented policy, fixed rather than data-dependent, so this page is the right place to learn it.

chatter merge accepts these silently and ARGUABLY SHOULD NOT. Listed separately because “settled” and “not done yet” are different states, and a single list hides which is which:

• --strip-tiers removes dependent tiers from inserted donor utterances. The merge now records how many it removed per utterance, and no command reports it.
• File 2’s @Media modality disagrees with File 1’s (audio vs video). The merge does not inspect @Media at all, which is worth stating plainly: a donor over-claiming a LANGUAGE is a hard refusal, reasoned as evidence of a wrong-file pairing, and a modality disagreement is the same evidence treated as nothing.

Speaker identity in File 2 must already be coherent

chatter merge does NOT identify or rename speakers. If File 2 came from ASR and carries anonymous codes like PAR0, PAR1, run chatter speaker-id first to assign CHAT-conformant codes. The merge step trusts whatever speaker codes appear in its inputs.

What chatter merge is NOT

• Not ASR. Use batchalign3 transcribe.
• Not forced alignment. Use batchalign3 align.
• Not morphological tagging. Use batchalign3 morphotag.
• Not speaker identification. Use chatter speaker-id.
• Not content reconciliation. If two files disagree about what a speaker said at the same time, chatter merge does not adjudicate; it trusts --retain to designate one file as authoritative per speaker.
• Not three-way or n-way merge in this release. The 2-input case composes into the n-input case by chaining (chatter merge a b --retain X -o tmp.cha && chatter merge tmp.cha c --retain Y -o out.cha). A future release may add native n-ary merging if a workflow appears for which chained 2-way merges are awkward.

Worked example

A speech-pathology lab hand-transcribed a child’s spontaneous-speech session, marking disfluency carefully, but did not transcribe the clinician’s turns. They send the media and the child-only transcript; the project runs ASR on the media to produce a full-coverage transcript with anonymous speaker codes; then chatter speaker-id labels the ASR file’s adult speaker as INV; then chatter merge combines.

# After ASR labeling: asr.cha has speakers CHI and INV.
chatter merge child-only.cha asr.cha \
  --retain CHI \
  -o merged.cha

# Then alignment regenerates %wor cleanly across all speakers:
batchalign3 align merged.cha

# Then morphotag regenerates %mor and %gra:
batchalign3 morphotag merged.cha

The merged file contains:

• Every *CHI utterance byte-stable from child-only.cha, including every disfluency marker, every retracing scope, every paralinguistic annotation, every %com session-structural comment.
• Every *INV utterance from asr.cha, in their original time order, interleaved with the *CHI utterances by start time.
• One @Participants row listing CHI and INV; @ID rows for both; the union of @Comment rows including any ASR provenance comments from asr.cha.

Relationship to other commands

flowchart TB
    Media[Media file mp4 / wav] --> Transcribe
    Transcribe["batchalign3 transcribe<br/>ASR"] --> AsrAnon["asr-anonymous.cha<br/>PAR0, PAR1, ..."]
    HandTranscript["hand-transcript.cha<br/>target speakers only"] --> SpeakerId
    AsrAnon --> SpeakerId
    SpeakerId["chatter speaker-id<br/>label anon speakers"] --> AsrLabeled["asr-labeled.cha<br/>CHI, INV, MOT, ..."]
    HandTranscript --> Merge
    AsrLabeled --> Merge
    Merge["chatter merge<br/>(this page)"] --> Merged[merged.cha]
    Merged --> Align
    Align["batchalign3 align"] --> Aligned[aligned.cha]
    Aligned --> Morph
    Morph["batchalign3 morphotag"] --> Final[final.cha]

chatter merge sits between speaker-identity resolution and forced alignment. It assumes its inputs have coherent CHAT-conformant speaker codes (no anonymous PAR0/PAR1) and emits a file ready for batchalign3 align to refresh timing and produce %wor.

Inputs must be valid CHAT (pipeline / batch)

The per-session chatter pipeline shortcut and the directory-level chatter batch driver validate every input as CHAT before doing any speaker-id or merge work. Each donor and the reference it is merged against must pass the same validation chatter validate runs; an input that fails is never merged. Clean invalid transcripts to valid CHAT first (run chatter validate <file> to see the errors), then re-run.

• chatter pipeline refuses (exit 2, no output written) if its donor or reference is invalid CHAT.
• chatter batch is fail-closed and whole-batch: if any input under the donor/reference directories is invalid CHAT, it reports every offending file and aborts the entire run without merging a single session. “All inputs are chatter-valid” is a hard precondition of the batch, not something discovered session-by-session mid-run.

This gate catches validation-only invalidity (files that parse but fail chatter validate, e.g. a malformed @ID), which the lower-level chatter merge parse is otherwise lenient about.

LLM holistic judgment (pending-only)

--judgment holistic is now reachable from pipeline and batch (not just speaker-id). In holistic mode the command is pending-only: it writes an engine = "llm" review-gated entry via --write-pending and produces no merged file. The operator supplies the LLM connection with --llm-endpoint / --llm-model (or the environment variables CHATTER_LLM_ENDPOINT / CHATTER_LLM_MODEL); an optional --session-context <file.json> provides per-session context that the LLM prompt includes to sharpen its judgment.

Response caching (--llm-cache)

--llm-cache <file> (env fallback CHATTER_LLM_CACHE) points holistic judgment at a persistent, write-through JSON response cache. When set, a request identical to one already answered (same endpoint, model, and rendered prompt) is served from the cache file instead of making another LLM call, so re-running a batch after a crash, or after fixing an unrelated bug, does not re-pay every already-completed session. The cache key folds in the exact wire request, so any prompt or PromptVersion change invalidates stale entries automatically, no separate version bump is needed. A cache file that exists but is not valid JSON is a hard error (the run refuses rather than silently ignoring or overwriting it); a missing file is treated as an empty cache and created on first write. Absent flag and env variable means uncached, today’s default behavior. chatter batch threads --llm-cache to every per-session chatter pipeline subprocess it spawns, so one cache file accumulates entries across the whole batch.

Only one process may open a cache file at a time. Finish the current run before starting another with the same cache, or use separate cache paths. Library callers should share one ResponseCache handle across threads. The normal batch driver runs its subprocesses sequentially and releases ownership between sessions.

Writes replace a flushed temporary file rather than truncating live responses. A failed write before replacement preserves both existing disk entries and lookup results. Unix builds also sync the parent directory; if that final sync fails, the error explicitly says the replacement is visible but durability is unconfirmed. Windows builds flush the file before replacement without claiming portable directory-sync durability.

Session-context JSON (--session-context)

The session-context file is a corpus-agnostic JSON object mapping session IDs (the donor file’s basename stem) to context records. Every record field is optional, and the label fields are free vocabulary: chatter imposes no closed set, the labels are surfaced verbatim into the LLM prompt.

{
  "SESSION-ID": {
    "sample_type": "clinician interview",
    "declared_roles": ["Investigator"],
    "consent_tier": "video+audio",
    "age_months": 52
  }
}

• sample_type: what kind of speech sample the session is (e.g. "narrative retell").
• declared_roles: adult roles declared present in the session.
• consent_tier: media-consent tier governing what may be shared.
• age_months: child age in months at the session.

When --session-context is absent, the CHATTER_SESSION_CONTEXT environment variable supplies the path (empty counts as unset). Per session, each context field resolves in order: the explicit record from the file; for the age only, the donor’s CHAT @ID age header (pure CHAT, no external metadata needed); otherwise unknown. Absent sessions or fields are passed to the judgment as unknown, never guessed. A configured-but-malformed file is a hard error, and labels must contain at least one non-whitespace character. Configuring session context on a non-holistic run prints a warning (the deterministic judgment never consults it).

Conversion from a contributor’s own records format (a spreadsheet, a database export) to this JSON happens outside chatter.

The two-pass operator flow is:

1. batch --judgment holistic --session-context context.json --write-pending P accumulates one engine = "llm" pending entry per session in P.
2. Operator reviews P, accepts or corrects each entry.
3. chatter adjudicate promotes reviewed entries to the override file.
4. batch (deterministic, reading the override file) replays every confirmed mapping and writes the merged files.

Note: the MLU sanity-scan is unreliable for the FluencyBank clinical-interview corpus (children out-narrate the adult, so MLU ratios invert relative to typical child-language recordings). Holistic-pending review via --judgment holistic is the trustworthy alternative there.

Selected-donor merges (library)

A donor transcript is often edited before merging: some of its utterances are removed, and some are split into shorter children. SourceBoundDonorSelection::bind takes the original donor, the selected donor and each selected utterance’s original parent, and checks that the selection still describes the original: parents appear in order, each child keeps its parent’s speaker, and a timed child’s bullet lies inside its timed parent’s bullet. Header brackets are taken from the original timeline. merge_chat_files_with_donor_selection and its draft form merge a reference with such a selection. Four opt-in refinements build on it.

• Relative order (with_relative_order, RelativeOrderConstraint). The caller states that a reference utterance comes before or after a donor utterance, for example from matched words. When the two utterances’ intervals are disjoint and contradict the stated order, the merge refuses with MergeError::RelativeOrderTimingConflict, naming both. When their intervals overlap, the stated order is followed; if their starts then run backwards, validation reports it (E362), and a caller using the draft can repair the bullets before validating.
• Timed gem exterior (with_timed_gem_exterior). A donor utterance timed strictly before or after a paired, fully timed reference gem is placed outside it instead of being refused as an ambiguous section placement. Utterances that cross or touch the gem’s span are still refused. Merged::gem_exterior_placements reports only the placements this policy decided.
• Flagged draft order (with_flagged_draft_order). A frontier the sources cannot order is serialized reference first, with a generated @Comment asking for review, and recorded in Merged::draft_order_reviews. Known contradictions between stated order and timing are still refused.
• Header-only references. With an empty retain set, a reference that has headers but no utterances can be merged with a selection: the output keeps the reference headers, every selected donor utterance, and the donor’s opening metadata, placed before @End.

Implementation notes (for contributors)

• Source: crates/talkbank-transform/src/transcript_merge.rs.
• CLI surface: crates/chatter/src/commands/transcript_merge.rs.
• Domain types: SpeakerCode lives in talkbank-model (a core model type); the merge/override types (MergeOverride, MappingSpec) live in talkbank-transform (speaker_id/override_file.rs, speaker_id/mapping.rs) so the override-file format is sharable across the speaker-id stage, the orchestrator, and any future adjudication UI. There is no separate RetainSet type: retained speakers are passed as &[SpeakerCode].
• The merge operates on talkbank-model::ChatFile; both inputs are parsed via talkbank-parser. The byte-preservation guarantee on retained-speaker utterances relies on the parser’s existing round-trip serialization.
• Spec entries exercising the merge live in spec/constructs/, every behavioral invariant on this page has a spec; tests are regenerated via the current spec/tools workflow documented in Spec Workflow.
• This page is the user contract; book/src/chatter/reference/ carries the override-file reference for the speaker-id stage that this merge consumes.

This page last changed: 2026-09-15 (commit bb4bef82). The whole book last changed: 2026-09-15 (commit bb4bef82).

The Merge Workflow (pipeline, batch, adjudicate, sanity-scan)

Status: Draft (experimental) Last modified: 2026-07-07 21:20 EDT

The merge workflow combines, at scale, the two structural primitives documented elsewhere, chatter speaker-id (assign CHAT-conformant speaker codes to an anonymous donor) and chatter merge (combine two transcripts of the same recording), and adds the operator loop needed when the automatic speaker decision is not confident enough to trust.

Four commands make up the workflow. They are experimental and in active development; flags and behavior may change.

If you only have one pair of files and one clean answer, reach for pipeline. Everything else here is about doing that safely across a directory of sessions where some answers are not clean.

The big picture: a two-pass loop

The hard part of merging at scale is not the merge; it is deciding, per session, which anonymous ASR speaker is the child the reference already covers. speaker-id’s multiset-Jaccard match (see its page) answers that automatically when the winner clearly beats the runner-up, and refuses (exit code 4) when it does not. The workflow turns that refusal into a reviewable queue.

flowchart TD
    subgraph Pass1["Pass 1: automatic"]
        B1["chatter batch DONOR_DIR REF_DIR\n--write-override audit.toml\n--write-pending pending.toml"]
        B1 --> Clean["confident sessions:\nmerged file written,\ndecision logged to audit.toml"]
        B1 --> Refused["low-confidence sessions:\nNO merge, appended to pending.toml\n(exit code 4)"]
    end
    Refused --> Adj["chatter adjudicate pending.toml\n--override-file audit.toml\n(operator decides)"]
    Adj --> Pass2["Pass 2: chatter batch ... --override-file audit.toml\n(replays the operator's decisions,\nmerges the previously-refused sessions)"]
    Clean --> Done["all sessions merged"]
    Pass2 --> Done

Pass 1 merges everything it is confident about and parks the rest. The operator works the parked queue once. Pass 2 replays their decisions. The same chatter batch (or chatter pipeline) command runs both passes; what changes is whether an override file with entries exists yet.

chatter pipeline (one session)

The per-session shortcut: run speaker-id in reference mode to relabel an anonymous donor, then merge the relabeled donor with the reference, in one command instead of two.

chatter pipeline <DONOR> <REFERENCE> \
  --anchor <SPEAKER> --inserted-role <CODE>:<ROLE> --output <PATH> [OPTIONS]

ARGUMENTS:
  <DONOR>      Donor CHAT file with anonymous speaker codes (the ASR output).
  <REFERENCE>  Reference CHAT file carrying the authoritative anchor speaker
               (typically the hand-coded child transcript).

REQUIRED:
  --anchor <SPEAKER>            Anchor code in the reference (typically CHI).
  --inserted-role <CODE>:<ROLE> Role for the donor's non-anchor speakers
                                (e.g. INV:Investigator).
  -o, --output <PATH>           Output path for the merged CHAT file.

KEY OPTIONS:
  --retain <SPEAKER>            Speaker(s) taken from the reference in the
                               final merge (typically the same as --anchor).
  --confidence-threshold <F>    Minimum winner/runner-up Jaccard margin to
                               auto-decide (default 2.0x).
  --write-override <FILE>       On a confident auto-decision, append a
                               mode = "auto" audit entry for this session.
  --write-pending <FILE>        On a low-confidence refusal, append a pending
                               entry (exit code 4 still fires).
  --override-file <FILE>        If the file has an entry for this session
                               (the donor's basename stem), replay that
                               decision instead of running reference mode.

The same command serves pass 1 (no override entry yet, run reference mode) and pass 2 (entry present, replay it). Validation is a hard precondition: a donor or reference that fails chatter validate is never merged (exit 2, nothing written).

chatter batch (a directory pair)

Loops pipeline over matched files: the reference for DONOR_DIR/X.cha is REFERENCE_DIR/X.cha. Donors without a matching reference are warned and skipped. It is fail-closed and whole-batch on validity: if any input under either directory is invalid CHAT, the batch reports every offending file and aborts without merging a single session.

chatter batch <DONOR_DIR> <REFERENCE_DIR> \
  --anchor <SPEAKER> --inserted-role <CODE>:<ROLE> --output <DIR> [OPTIONS]

PASS-1 AUDIT + QUEUE:
  --write-override <FILE>  Append every confident auto-decision (mode =
                          "auto"). Required if you want --sanity-scan.
  --write-pending <FILE>   Aggregate every low-confidence refusal into one
                          pending file. One `chatter adjudicate` run resolves
                          them all. Refusals do NOT abort the batch.

PASS-2 REPLAY:
  --override-file <FILE>   Threaded to every per-session pipeline call.
                          Sessions with an entry replay it; the rest fall
                          through to reference mode.

POST-MERGE QA:
  --sanity-scan            Run `sanity-scan` after the loop. Requires
                          --write-override (it reads the auto-decisions) and
                          --write-pending (flagged sessions are appended).
                          Exit code 4 fires if it flags any session.
  --sanity-scan-threshold <F>  Heuristic ratio (default 1.5).

OPERATIONAL:
  --skip-existing          Skip donors whose merged output already exists, to
                          resume an interrupted batch.

batch also accepts the same --judgment deterministic|holistic and LLM / --session-context options as pipeline; see Merge, LLM holistic judgment for that mode and the session-context JSON format.

Reading the batch summary

Every run ends with one summary line on stderr that accounts for every matched donor exactly once:

batch summary: 345 matched donor(s); 0 merged, 345 suggestions awaiting
adjudication, 0 low-confidence refusals awaiting adjudication, 0 errored,
0 unmatched (no reference), 0 skipped (output existed)

• merged: the pipeline produced a merged output file (deterministic mode, or a session already covered by an override decision).
• suggestions awaiting adjudication: holistic mode judged the session confidently and wrote a suggestion to the pending file; no merge happens until an operator accepts it via chatter adjudicate.
• low-confidence refusals awaiting adjudication: the engine declined to suggest (below the confidence threshold); the entry is in the pending file for a human call.
• errored / unmatched / skipped: per-session failures, donors with no same-named reference file, and outputs that already existed under --skip-existing, respectively.

The distinction between the first three matters operationally: a holistic run that ends 0 merged, N suggestions awaiting adjudication has done its job; the merge itself happens after adjudication.

chatter adjudicate (the operator step)

Reads the pending file a pass produced, walks the operator through the unresolved sessions, and appends the resolved decisions to the override file. On success the pending file is rewritten to drop the entries that were resolved, so re-running adjudicate only ever shows what is left.

chatter adjudicate <PENDING> --override-file <FILE> [--interactive | --scripted <TOML>]

ARGUMENTS:
  <PENDING>  The pending-adjudications TOML a pass wrote.

REQUIRED:
  --override-file <FILE>  Override file to append resolved decisions to
                         (created if absent). This is the same file pass 2
                         reads back.

DECISION SOURCE (one of):
  --interactive           Prompt per pending entry on stdin. See "The
                         interactive decision language" below for the
                         three decision verbs and their syntax.
  --scripted <TOML>       Pre-canned operator decisions, for replayable /
                         tested runs. Mutually exclusive with --interactive.

  --operator <NAME>       Recorded in each override entry (defaults to $USER).

The interactive decision language

Each pending entry is printed with its full context (the sessions, the suggested mapping, the engine’s confidence scores and reasoning), then one line is read from stdin. Three decision verbs are accepted:

SPK:CODE:TAG groups are repeatable, so multi-adult sessions are expressed naturally, one group per speaker:

choose A:CHI:Target_Child B:INV:Investigator C:MOT:Mother reviewed against the recording

Anything after the structured arguments is recorded verbatim as the operator’s note. Every decision (verb, mapping, note, operator, and the engine’s original scores) is appended to the override file, so the audit trail survives the session.

This is the interactive review tool the speaker-id and merge pages refer to: the audit trail (who decided, the scores, any note) lands in the override file so a later reader can see why a session was labeled the way it was. The decision schema is the same override-file format used everywhere in the workflow; see Merge Override File Format, and the Adjudication Workflow architecture page for the design.

chatter sanity-scan (post-merge QA)

A confident auto-decision can still be wrong, the runner-up was simply even further off. sanity-scan re-reads the merged output and the pass-1 audit file and flags sessions that pass an out-of-band check: the mean utterance word count of the anchor speaker versus the inserted speaker. In a typical child-language recording the adult out-talks the child, so an anchor (child) mean that is much higher than the inserted (adult) mean is suspicious, possibly the two were swapped.

chatter sanity-scan <MERGED_DIR> \
  --override-file <FILE> --anchor <SPEAKER> --write-pending <FILE> [OPTIONS]

REQUIRED:
  --override-file <FILE>  The pass-1 audit file. Only auto-decided sessions
                         are scanned; explicit-mode entries are skipped (the
                         operator already signed off).
  --anchor <SPEAKER>      Anchor code in the merged files (typically CHI).
  --write-pending <FILE>  Flagged sessions are appended here as
                         sanity-scan-misclassification pending entries for
                         `chatter adjudicate`. Required.

  --threshold <F>         Flag when anchor_mean >= inserted_mean * threshold
                         (default 1.5).

A flag is a question, not a verdict: the session goes back into the adjudication queue for an operator to confirm or correct. Whether to run the scan at all is a judgment about the corpus. It assumes the typical “adult out-talks child” shape, and is unreliable where that inverts (e.g. a clinical-interview corpus where children out-narrate the adult); there, prefer the LLM holistic-pending review described on the merge page.

End-to-end worked example

A directory of ASR donors (asr/) and the matching hand-coded child references (ref/), child anchor CHI, adults labeled INV:

# Pass 1: merge what we are sure of; queue the rest; keep an audit trail.
chatter batch asr/ ref/ \
  --anchor CHI --inserted-role INV:Investigator \
  --output merged/ \
  --write-override audit.toml \
  --write-pending pending.toml \
  --sanity-scan

# Exit 0: every session merged confidently and the scan was clean.
# Exit 4: some sessions are pending (low-confidence and/or scan-flagged).

# Operator resolves the queue once (audit trail recorded):
chatter adjudicate pending.toml --override-file audit.toml --interactive --operator alice

# Pass 2: replay the operator's decisions; the previously-pending
# sessions now merge.
chatter batch asr/ ref/ \
  --anchor CHI --inserted-role INV:Investigator \
  --output merged/ \
  --override-file audit.toml \
  --skip-existing

Exit codes

The workflow commands share the convention used across the merge surface:

Exit code 4 is the normal “there is operator work to do” signal, not an error: a batch that parks ten sessions still merged the rest.

See also

• Speaker-ID, the speaker-relabeling primitive and the Jaccard matching algorithm pass 1 uses.
• Merge, the structural merge primitive and the LLM holistic-judgment mode.
• Merge Override File Format, the shared decision schema.
• Adjudication Workflow and Merge Pipeline, Crate Architecture, the developer-facing design.

This page last changed: 2026-07-07 (commit bf80c629). The whole book last changed: 2026-09-15 (commit bb4bef82).

CHAT Format Overview

Status: Reference Last updated: 2026-05-11 21:51 EDT

CHAT (Codes for the Human Analysis of Transcripts) is a standardized transcription format for spoken language data, developed by MacWhinney as part of the CHILDES and TalkBank projects. It is the most widely used format in child language research and conversational analysis.

File Anatomy

Every CHAT file follows this structure:

@UTF8
@Begin
@Languages:	eng
@Participants:	CHI Target_Child, MOT Mother
@ID:	eng|corpus|CHI|2;6.||||Target_Child|||
@ID:	eng|corpus|MOT|||||Mother|||
*MOT:	what do you want ?
%mor:	ADV|what AUX|do PRON|you VERB|want ?
%gra:	1|4|OBJ 2|4|AUX 3|4|NSUBJ 4|0|ROOT 5|4|PUNCT
*CHI:	I want cookie .
%mor:	PRON|I VERB|want NOUN|cookie .
%gra:	1|2|NSUBJ 2|0|ROOT 3|2|OBJ 4|2|PUNCT
@End

A CHAT file consists of:

1. @UTF8: required first line, declares UTF-8 encoding
2. @Begin: marks the start of the transcript
3. Headers: lines starting with @ that provide metadata (participants, languages, IDs, etc.)
4. Utterances: blocks consisting of:
   • A main tier (line starting with *SPEAKER:) containing the transcribed speech
   • Zero or more dependent tiers (lines starting with %tier:) containing annotations
5. @End: marks the end of the transcript

Key Conventions

• Tab separation: a tab character separates the tier prefix from its content (e.g., *CHI:⟶content)
• Terminators: every utterance ends with a terminator (., ?, !, or special forms like +...)
• Line continuation: long lines wrap with a tab at the start of continuation lines
• Speaker codes: short identifiers; the validator accepts up to seven characters from A-Z, 0-9, _, -, '; three uppercase letters is the convention (e.g., CHI, MOT, FAT, INV)
• Media linking: timestamps link transcripts to audio/video via bullet markers

CHAT vs Other Formats

References

• CHAT Manual: the canonical reference
• TalkBank: the data repository

This page last changed: 2026-07-27 (commit 905227c9). The whole book last changed: 2026-09-15 (commit bb4bef82).

Headers

Status: Reference Last updated: 2026-05-11 20:30 EDT

Headers are lines beginning with @ that provide metadata about the transcript. They appear between @Begin and the first utterance (though some headers like @Comment can appear anywhere).

Required Headers

@UTF8

Must be the very first line of every CHAT file. Declares UTF-8 encoding.

@UTF8

@Begin / @End

Mark the start and end of the transcript body. Every CHAT file must have exactly one @Begin and one @End.

@Participants

Declares all speakers in the transcript. Format: CODE [Name] Role, comma-separated. The role is required; the name is optional, so each entry is either CODE Role or CODE Name Role.

@Participants:	CHI Target_Child, MOT Mother, FAT Father
@Participants:	CHI Alex Target_Child, MOT Mary Mother

In the first line, Target_Child, Mother, and Father are roles, not names. In the second line, Alex and Mary are optional names sitting between the speaker code and the role.

Speaker codes are short identifiers; the validator accepts up to seven characters from A-Z, 0-9, _, -, and '. The convention is three uppercase letters; the most common codes are:

• CHI: target child
• MOT: mother
• FAT: father
• INV: investigator
• OBS: observer

@ID

Provides detailed metadata for each participant. One @ID line per participant.

@ID:	eng|corpus|CHI|2;6.||||Target_Child|||

Fields (pipe-separated): language, corpus, speaker code, age, sex, group, SES, participant role, education, custom field.

Age format: years;months.days (e.g., 2;6. = 2 years, 6 months).

SES field: ethnicity (White, Black, Asian, Latino, Pacific, Native, Multiple, Unknown), socioeconomic code (UC, MC, WC, LI), or combined with comma separator (e.g., White,MC).

Optional Headers

@Languages

Declares the language(s) used in the transcript.

@Languages:	eng, fra

@Date

Recording date in DD-MON-YYYY format.

@Date:	15-JAN-2024

@Location

Where the recording took place.

@Location:	Boston, MA, USA

@Situation

Description of the recording context.

@Situation:	free play with toys in lab

@Activities

Activities during the recording.

@Activities:	toyplay, reading

@Comment

Free-form comments. Can appear anywhere in the file (before, between, or after utterances).

@Comment:	child was tired during this session

@Media

Links the transcript to an audio or video file.

@Media:	session01, audio

@Transcriber / @Coder

Identifies who created or coded the transcript.

@Transcriber:	JDS
@Coder:	ABC

Header Ordering

Headers should follow this conventional order:

1. @UTF8 (required, first line)
2. @Begin (required)
3. @Languages
4. @Participants (required)
5. @ID lines (one per participant)
6. Other metadata headers (@Date, @Location, etc.)
7. @Comment lines (can also appear later)

Validation

The parser validates header structure including:

• @UTF8 must be the first non-empty line
• @Begin and @End are required and must appear exactly once
• @Participants is required and must declare all speakers used in utterances
• @ID participant codes must match @Participants declarations
• Age format validation in @ID lines

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Utterances

Status: Reference Last updated: 2026-05-11 23:22 EDT

An utterance is the fundamental unit of a CHAT transcript. It consists of a main tier (the transcribed speech) followed by zero or more dependent tiers (annotations).

Main Tier

The main tier begins with *SPEAKER: followed by a tab and the utterance content, ending with a terminator.

*CHI:	I want a cookie .

Speaker Codes

Speaker codes are short identifiers (up to seven characters from A-Z, 0-9, _, -, '; three uppercase letters is the convention) matching a code declared in @Participants:

@Participants:	CHI Target_Child, MOT Mother
*MOT:	what do you want ?
*CHI:	cookie .

Terminators

Every utterance must end with a terminator:

Line Continuation

Long utterances wrap to the next line with a leading tab:

*MOT:	well I think that we should probably go to
	the store and get some more cookies .

Content Items

The content between *SPEAKER: and the terminator consists of content items separated by whitespace:

• Words: regular words, potentially with annotations
• Groups: bracketed content like <word word> for overlap, retrace, etc.
• Special forms: pauses (.), events &=laughs, fillers &-uh
• Separators: commas , and other punctuation

Words

Words are the primary content unit. See Word Syntax for full details.

Groups

Angle brackets < > group words for annotations:

*CHI:	<I want> [/] I want cookie .

Common group annotations:

• [/]: partial retrace (speaker repeats the same words)
• [//]: full retrace (speaker restarts with different words)
• [///]: multiple retracing (multiple false starts)
• [/-]: reformulation (speaker rephrases with different structure)
• [?]: uncertain transcription

Special Forms

*CHI:	um (.) I want &-uh cookie .

• (.): short pause
• (..): medium pause
• (...): long pause
• (1.5): timed pause in seconds
• &=laughs: paralinguistic event
• &-uh: filler

Media Linking

Utterances can include media timestamps (bullets) that link to audio/video:

*CHI:	I want cookies . •1234_5678•

The numbers represent start and end times in milliseconds. The bullets delimiting the pair render as • in most editors; on disk they are the NAK control character (U+0015). See grammar/grammar.js rule bullet.

Dependent Tiers

See Dependent Tiers for documentation on %mor, %gra, %pho, %wor, and other annotation tiers that follow the main tier.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Retraces and Repetitions

Status: Current Last updated: 2026-09-09 08:49 EDT

Retraces mark content that the speaker said but then corrected, repeated, or abandoned. They are one of the most consequential constructs in CHAT because they affect how every dependent tier aligns to the main tier.

CHAT Syntax

A retrace has two parts: the retraced content (what the speaker said first) and the correction (what follows). The retraced content is marked with a trailing bracket code:

Single-Word Retraces

When only one word is retraced, no angle brackets are needed:

*CHI: I [/] I want that .
*CHI: ana [//] an .
*MOT: the book [/-] the magazine is here .

Group Retraces

When multiple words are retraced, angle brackets delimit the scope:

*MOT: <the dog> [//] the cat ran .
*CHI: <I want> [/] I need cookie .
*CHI: <I want the> [///] give me that .

Retraces with Replacements

A retraced word often has a replacement [: target] and/or error code [* code]. This is common in aphasia and child language corpora where the speaker produces an incorrect form:

*PAR: tika@u [: kitty] [* p:n] [//] kitty is nice .
%mor: noun|kitty aux|be-Fin-Ind-Pres-S3 adj|nice-S1 .

*PAR: lɛɾɪ@u [: later] [* p:n] [//] later in the day .
%mor: adv|late adp|in det|the-Def-Art noun|day .

*CHI: male [: female] [* s:r] [/] male [: female] [* s:r] .
%mor: adj|female-S1 .

In each case, the retraced word (before the [//] or [/]) is excluded from %mor alignment. Only the correction (after the marker) is counted.

Data Model

Retraces are a first-class variant of UtteranceContent:

flowchart TD
    UC["UtteranceContent"]
    UC --> Word
    UC --> RW["ReplacedWord"]
    UC --> Retrace
    UC --> AG["AnnotatedGroup"]
    UC --> Other["...20 other variants"]

    Retrace --> BC["BracketedContent"]
    Retrace --> RK["RetraceKind"]
    BC --> BIW["BracketedItem::Word"]
    BC --> BIRW["BracketedItem::ReplacedWord"]

    style Retrace fill:#f96,stroke:#333

The Retrace struct wraps the retraced content in a BracketedContent container, which can hold any combination of words, replaced words, and other content items:

// crates/talkbank-model/src/model/content/retrace.rs
pub struct Retrace {
    pub content: BracketedContent,  // the retraced words
    pub kind: RetraceKind,          // Partial, Full, Multiple, Reformulation
    pub is_group: bool,             // <word> [/] vs word [/]
    pub span: Span,
}

Where the annotations live

There is deliberately no annotations field. A content item followed by a run of scoped markers is a LEFT-ASSOCIATIVE CHAIN: each marker scopes over everything to its left, so these two lines are different claims about the same two words.

dog [* p:w] [/] dog     the error is on the abandoned attempt
dog [/] [* p:w] dog     the error is on the retrace

Annotations written BEFORE the marker therefore annotate the retraced material and live inside content; annotations written AFTER it annotate the retrace and live on an AnnotatedRetrace(Box<Annotated<Retrace>>) wrapper, exactly parallel to Group / AnnotatedGroup.

A single flat annotations field used to hold both, which made the two indistinguishable. dog [* p:w] [/] was silently written back as dog [/] [* p:w], and a second adjacent marker overwrote the first, so на [//] [/] на became на [/] на. Neither was visible to validate, to --roundtrip (which tests idempotence of serialize(parse(x)), not fidelity to the input) or to SemanticEq (the two orderings WERE the same model).

Why First-Class?

Before the retrace refactor, retraces were represented as annotations on words or groups. This meant every match on content had to inspect annotation lists to determine whether a word was retraced. This led to a class of bugs where retraced content was accidentally included in alignment counting, word extraction, or retokenization.

Making Retrace a top-level UtteranceContent variant means:

1. The compiler enforces handling, WHERE the match is exhaustive. Every match on UtteranceContent must have a Retrace arm. The caveat is real: when AnnotatedRetrace was added, five sites matching a retrace behind a _ => arm compiled unchanged and silently answered wrong, one of them the gate in front of all retrace validation. A first-class variant guarantees exhaustiveness only where the matches are already exhaustive, which is why this codebase bans catch-alls over content enums.
2. Domain-aware gating is centralized. The content walker checks the Retrace variant once, not at every annotation-inspection site.
3. Alignment counting is simple. The count function returns 0 for Retrace in Mor domain, no annotation inspection needed.

Parser Conversion

The tree-sitter grammar parses retrace markers ([/], [//], etc.) as annotations on word_with_optional_annotations. The Rust parser converts them to structural Retrace nodes in parse_word_content():

flowchart LR
    subgraph "Tree-sitter CST"
        WOA["word_with_optional_annotations"]
        SW["standalone_word"]
        BA["base_annotations"]
        RP["retrace_partial / retrace_complete / ..."]
        WOA --> SW
        WOA --> BA
        BA --> RP
    end

    subgraph "Rust Model"
        RET["UtteranceContent::Retrace"]
        BC2["BracketedContent"]
        W2["Word or ReplacedWord"]
        RET --> BC2
        BC2 --> W2
    end

    WOA -->|"parse_word_content()\n(word.rs)"| RET

Three cases in parse_word_content():

1. Word + retrace (I [/]), wrap Word in BracketedItem::Word inside Retrace
2. Word + replacement + retrace (tika@u [: kitty] [* p:n] [//]), build ReplacedWord, then wrap in BracketedItem::ReplacedWord inside Retrace
3. Word + replacement, no retrace (tika@u [: kitty]), emit bare ReplacedWord

Group retraces (<content> [/]) are handled in group/parser.rs via the same structural wrapping.

Alignment Behavior

Retraces interact differently with each dependent tier domain:

flowchart TD
    RT["Retrace node\n(e.g. 'tika@u [: kitty] [* p:n] [//]')"]

    RT -->|"Mor domain"| SKIP["SKIP\n(return 0)\nNot morphologically analyzed"]
    RT -->|"Pho domain"| COUNT["COUNT\nPhonologically produced"]
    RT -->|"Sin domain"| COUNT2["COUNT\nGesturally produced"]
    RT -->|"Wor domain"| COUNT3["RECURSE\napply retrace-aware %wor leaf rule"]

    style SKIP fill:#faa,stroke:#333
    style COUNT fill:#afa,stroke:#333
    style COUNT2 fill:#afa,stroke:#333
    style COUNT3 fill:#afa,stroke:#333

Why %mor skips retraces: The %mor tier represents the morphological analysis of what the speaker meant to say. Retraced content is a false start or error; it was produced phonologically but is not part of the intended linguistic structure. The correction after the retrace marker carries the morphological analysis.

Why %pho/%sin/%wor include retraces: These tiers document what was actually produced, the sounds, gestures, and timing of the speech as it happened, including false starts. The retrace was physically spoken, so it appears in these tiers.

For %wor, retrace ancestry does not change leaf-level membership:

• spoken word tokens count both inside and outside retrace
• that includes fillers, fragments, nonwords, and untranscribed placeholders
• overlap annotations do not affect %wor membership

Exact corpus-shaped contrast:

*CHI:	<one &+ss> [/] one play ground .
%wor:	one •321008_321148• ss •321148_321368• one •321809_321969• play •322049_322310• ground •322390_322890• .

*CHI:	&+ih <the what> [/] what's letter &+th is this ?
%wor:	ih •49063_49103• the •49103_49163• what •49183_50205• what's •50205_50405• letter •50405_50685• th •50886_50946• is •50946_51046• this •51086_51586• ?

Implementation

Both the counter and the walker ask one owner, alignment/helpers/descent.rs, what a domain does with a retrace: %mor excludes it, every other domain enters it. The counter takes a PositionalDomain and converts it for the descent rule; the walker takes the TierDomain directly.

Counting and extraction: walk_alignable_item() in alignment/helpers/count.rs, one walk whose sink either counts or collects:

UtteranceContent::Retrace(_) | UtteranceContent::AnnotatedRetrace(_) | /* other containers */ => {
    match descend(item.structure(), Some(domain.into())) {
        Descent::Into(entered) => walk_alignable_bracketed(entered.content(), domain, sink),
        Descent::Atomic(unit) => sink(AlignablePosition::Atomic(unit)),
        Descent::Excluded => {} // %mor: a retrace is not aligned
    }
}

Walking: walk_words() in alignment/helpers/walk/mod.rs:

UtteranceContent::Retrace(_) | UtteranceContent::AnnotatedRetrace(_) | /* other containers */ => {
    if let Some(into) = descend(item.structure(), domain).entered() {
        walk_bracketed_content(&into.content().content, domain, f);
    }
}

%wor generation and overlap counting still use dedicated recursive helpers, but now for %wor-specific sequencing details like replacement handling rather than for retrace-sensitive membership.

Validation

The three retrace rules

validation/retrace/ runs three checks over ONE traversal of the tier (visit.rs), which reaches every retrace including those nested inside another retrace’s content.

E377 and E378 are disjoint despite the neighbouring names. In a [//] [/] a the inner retrace still holds a word, so E378 stays silent; in <&=sigh> [/] &=sigh there is no second marker, so E377 stays silent. The repairs differ too: drop a marker for E377, retrace the words rather than the vocalization for E378.

Both were adjudicated with the CHAT maintainer on 2026-08-07. On adjacent markers: “clearly a mistake … It’s an error.” On a marker over an event: “No, not legal. You can’t retrace a laugh.” He gave the legal alternative half an hour earlier in the same thread, and it is why E378 tests for absent WORDS rather than for a present event:

*PAR:	<the floor on the &=laughs water> [//] the floor on the xxx .

E378 recurses because 205 corpus retraces hold their words one level down, in an annotated group or a quotation (<<the dog> [?]> [/] the dog), and a rule testing the immediate children would reject every one of them. Untranscribed material counts as words on purpose: xxx, yyy and www lower as words, so retracing speech nobody could make out stays valid.

Which variants are containers is owned in one place for these rules, model::content::structure::ContentStructure. It is written that way because two hand-written copies of that knowledge disagreed about PhoGroup and SinGroup, which silently stopped E377 firing inside ‹...› with no test able to see it.

Be precise about the scope, because an earlier draft of this paragraph was not: the retrace validators classify through it, and the alignment walkers do not. They need to know WHICH container they are in, so a tier domain can skip a phonological group but not a quotation, and Container deliberately does not carry that. Settling those payloads is the prerequisite for migrating them.

Alignment Validation (E705)

E705 fires when the main tier has more alignable items than %mor. If retraces are correctly parsed as Retrace nodes; they are excluded from the count and E705 does not fire. If a retrace is accidentally parsed as a bare ReplacedWord (the bug fixed in c90b9bf), it is counted and triggers a false E705.

Regression Tests

tests/retrace_replaced_word_regression.rs contains 6 targeted tests:

Reference corpus entries: corpus/reference/annotation/retrace.cha

See Also

• Alignment Architecture: full alignment system docs
• The %mor Tier: morphological tier format and alignment rules
• CHAT Manual: Retracing

This page last changed: 2026-09-09 (commit a30c20c4). The whole book last changed: 2026-09-15 (commit bb4bef82).

Replacements

Status: Current Last modified: 2026-05-29 17:47 EDT

A replacement is a CHAT annotation [: ...] that pairs a single spoken word on the main tier with one or more “intended” words. It records both what the speaker actually said and what the analysis should treat the utterance as containing.

*CHI:	wanna [: want to] go .
*CHI:	dis [: this] is fun .
*CHI:	rocking+house [: rocking+horse] [*] ?

This page is the canonical reference for what replacements mean in TalkBank, both as a CHAT-manual construct and as a typed AST in this repo. The most important load-bearing fact, which the rest of the page expands on:

Replacements are word-level, not group-level. Each tier domain chooses one side of the pair: %mor analyzes the replacement (right side); %wor, %pho, %sin align to the original (left side). %gra follows %mor.

CHAT Syntax

Word-Level Scope

A replacement attaches to a single standalone_word on the main tier and contains one or more replacement words inside the brackets:

*CHI:	gonna [: going to] eat lunch .
*CHI:	dis [: this] toy .
*CHI:	rocking+house [: rocking+horse] [*] ?

The grammar rules are word_with_optional_annotations and replacement in grammar/grammar.js grep for the rule names rather than line numbers so this stays accurate as the grammar evolves. Replacement words can be separated by whitespace, so [: going to] is a single replacement of gonna with two words.

There Is No Group-Level Replacement

<dat is> [: that is] is not valid CHAT. A replacement does not attach to a group; it attaches to a single word. The grammar enforces this by typing: ReplacedWord.word: Word, never Group. To replace words inside a group, attach the replacement to the inner word:

*CHI:	<dat [: that] is> [/] is broken .

This shape, replacement inside a group inside a retrace, is legal because each annotation operates at its own scope.

There Is No [::] Form

Some literature on CHILDES tooling references a [::] annotation; it does not exist in this repo’s grammar, parser, or model, and is not defined by the current CHAT manual. Only [:] exists. If you encounter [::] in legacy data, treat it as a parse error to investigate, not a construct to support.

The Per-Domain Alignment Rule

This is the rule contributors most often get wrong. Different tier domains align to different sides of a replacement pair:

The mnemonic: the replacement encodes the intended form (what the speaker meant or what a corrected transcript would read). Tiers analyzing intent (%mor/%gra) use the replacement; tiers documenting realization (%wor/%pho/%sin) use the original.

flowchart LR
    spoken["Original word\n(left of [:)\n'dis'"]
    target["Replacement words\n(inside [: ])\n'this'"]

    spoken -->|"%wor (timing)"| wor["%wor: dis"]
    spoken -->|"%pho (phonology)"| pho["%pho: dɪs"]
    spoken -->|"%sin (spelling)"| sin["%sin: dis"]
    target -->|"%mor (UD parse)"| mor["%mor: pron|this"]
    target -->|"%gra (paired with %mor)"| gra["%gra: 1|0|ROOT"]

For multi-word replacements like gonna [: going to], the rule generalizes consistently:

• %wor / %pho / %sin produce one entry, for gonna.
• %mor produces two entries, for going and to.
• %gra produces two entries, paired to the two %mor items.

The alignment-counting code that enforces this is in alignment/units.rs look for the UtteranceContent::ReplacedWord arm. The full table of per-domain rules is in spec/docs/ALIGNMENT_RULES.md.

Rust AST

A replacement is modeled as a first-class UtteranceContent variant, not as a flag on Word:

// crates/talkbank-model/src/model/annotation/replacement.rs
pub struct ReplacedWord {
    pub word: Word,                       // left side: original spoken word
    pub replacement: Replacement,         // right side: 1+ intended words
    pub scoped_annotations: ReplacedWordAnnotations,
}

Two consequences of this shape:

1. A replacement is a wrapper around a Word, not a kind of Word. ReplacedWord lives as its own variant of UtteranceContent (and BracketedItem), holding an inner word: Word plus the replacement payload. Contrast with retraces: Retrace is also a variant of UtteranceContent/BracketedItem, but it wraps a group of content (a single word or a <...> group), not a single Word. Different mechanism, different scope, same top-level slot in the AST.
2. The walk_words() content walker yields WordItem::ReplacedWord as a distinct leaf (defined in crates/talkbank-model/src/alignment/helpers/walk/mod.rs). Domain-aware extraction code branches on this leaf type and chooses original or replacement per the table above.

Validation

Each Replacement Word Is Validated Like a Main-Tier Word

The replacement is a Vec<Word>. Each Word inside it goes through the same validator that runs on main-tier words:

*CHI:	dog [: C-3PO] .

This produces [E220] "C-3PO" is not a legal word in language(s) "eng": numeric digits not allowed, exactly as if C-3PO had appeared on the main tier directly. The replacement does not provide an escape from word-level validation. The implementation is in replacement.rs.

This is critical for any code generating replacements programmatically: do not assume [: ...] lets you smuggle arbitrary text past the word validator. If your producer emits a replacement, both sides must be CHAT-legal under the utterance’s declared language.

Replacement-Specific Error Codes

Three error codes are specific to replacements and do not apply to main-tier words:

The principle: a replacement must be a concrete intended form. Empty, omitted, or unintelligible content defeats that purpose.

Interactions with Other Annotations

Replacements and Retraces Are Orthogonal

A retrace ([/], [//], [///], [/-]) and a replacement ([:]) are distinct annotations operating at different structural levels:

• Retraces wrap content (a single word or a group). They are first- class UtteranceContent variants and represent post-hoc speaker correction.
• Replacements attach inside a Word slot via ReplacedWord. They are editorial metadata about an individual spoken word.

Both can coexist:

*CHI:	<dat [: that] is> [/] is broken .   (replacement inside retrace)

A retrace cannot live inside a replacement (the grammar wraps replacements around standalone_word, not arbitrary content).

Replacements and Error Coding

Error codes follow the replacement and operate on the replaced word as a unit:

*CHI:	rocking+house [: rocking+horse] [*] ?

Here [*] marks rocking+house as containing a phonological/lexical error; the [: rocking+horse] records the intended form. The two annotations cooperate: the replacement encodes what was meant, the error code classifies how it deviates. Implementation: scoped_annotations field on ReplacedWord.

Common Misconceptions

These are bugs we have repeatedly written down then forgotten, recording them here so future contributors don’t reinvent them.

1. “[: ...] lets me put any text I want.” No. Each replacement word is validated. [: C-3PO] fails E220 in English just as C-3PO would.
2. “[:] is the right mechanism for ASR sanitization.” Usually no. ASR-introduced normalization typically wants [% ...] (free- form comment) or [= ...] (free-form explanation), neither of which validates word grammar. Use [:] only when you have a concrete CHAT-legal intended form.
3. “%mor analyzes the original.” No. %mor analyzes the replacement. This is the correction’s morphology, not the error’s.
4. “%wor count must equal %mor count.” No. For gonna [: going to], %wor has 1 entry and %mor has 2. They align to different sides. The validator’s per-domain rule respects this.
5. “<a b> [: c d] is a group-level replacement.” No. Group-level replacements don’t exist. Either replace inside (<a [: c] b [: d]>) or rephrase the transcription.

Source Citations

See Also

• Retraces and Repetitions: the orthogonal post-hoc correction mechanism.
• The %mor Tier: UD-syntax morphosyntactic analysis that aligns to the replacement form.
• Word Syntax: the word grammar that replacement words must satisfy.
• Dependent Tiers: overview of %mor, %wor, etc., with their alignment relationships.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Untranscribed Markers: xxx, yyy, www

Status: Reference Last updated: 2026-06-14 19:57 EDT

CHAT reserves three short word-level markers for material the human transcriber cannot or chose not to render as words on the main tier. Each one has a specific meaning. Tools that emit CHAT, including ASR pipelines, format converters, and editor heuristics, must respect those meanings, because every downstream consumer (researchers, validators, and aggregate-statistics tools like CLAN’s freq, kideval, mlu) reads them at face value.

The shared property: each marker is the human transcriber telling later readers something specific about their experience listening to the audio. None of them mean “tooling could not process this token”.

Why this matters

When a researcher loads a CHAT corpus and counts xxx occurrences, the result is a measure of human listening difficulty: it tells them how much of the audio resisted human transcription. That number feeds into methodology decisions (“can we get reliable MLU from this corpus?”, “what’s the noise floor on this child’s speech?”, “should we re-record in a quieter environment next time?”). It is a load-bearing signal in language-development research.

If an ASR pipeline emits xxx whenever it can’t sanitize a token, for example, substituting xxx for any word that fails CHAT validation under a strict language profile, every xxx count in the corpus becomes a meaningless mixture of “human couldn’t tell” and “pipeline gave up”. Researchers then reading those counts are silently misled. The signal is destroyed for the entire history of that corpus, because the corruption is indistinguishable from real unintelligibility once committed.

The same reasoning applies to yyy and www. A converter or post-processor that emits any of these three markers because the tooling couldn’t handle a token is committing semantic vandalism against the whole field.

Rules for tooling

1. Never emit xxx, yyy, or www from a tool to mean “could not process”. These markers are reserved for human transcriber judgment.
2. When a token cannot be validated as legal CHAT under the declared language, prefer one of:
   • Pass the token through verbatim and let the CHAT validator (or CLAN’s check) flag it for human review. The transcriber listens, decides, and corrects.
   • Fail loud, abort the file rather than emit corrupted output.
   • Apply only purely orthographic, semantically null repairs (e.g., stripping a stray boundary quote mark from "My). These are safe because no information is lost.
3. Never sanitize a token by replacing it with one of the three markers. That is exactly the corrupting behavior this document prohibits.
4. Never delete a token to “fix” a validation failure. Deletion loses data without any flag.

What tools synthesizing CHAT should do instead

Any tool that builds CHAT from an external source (ASR output, an importer, a format converter) should follow the same division of labor:

1. Silently fix only orthographically inarguable problems (for example, stripping a stray boundary quote mark from "My).
2. For tokens that fail language-level validation but are structurally legal CHAT (e.g., C-3PO under English: tree-sitter accepts the digit-hyphen compound but Word::validate fires E220 “numeric digits not allowed”), ship the token verbatim. The full-file validator and check fire E220 on the same word, the file ends up in the human review queue, and the transcriber listens to the audio and decides what was actually said.
3. For tokens that fail structural parsing (tree-sitter rejects), fail loud: emitting malformed CHAT would corrupt the file beyond the validator’s ability to flag it.

The division of labor is: the tool fixes only what is mechanically unambiguous; CHECK and the human transcriber handle everything that requires judgment about what the speaker said.

Related rules

• xxx / yyy / www survive the transcript through all NLP passes (morphotag, utseg, translate, coref) without re-interpretation. Tools that walk the AST treat them as opaque tokens; they have no POS tag, no lemma, no dependency parent, no translation.
• %wor excludes all three (no phoneme sequence to align). %pho may reference yyy directly because the phonetic content is the whole point of the marker.
• See word-syntax.md for grammar; this document is the policy reference for who is allowed to emit them and why.

This page last changed: 2026-06-21 (commit 1952fb27). The whole book last changed: 2026-09-15 (commit bb4bef82).

Postcodes ([+ ...])

Status: Reference Last updated: 2026-06-25 07:30 EDT

A postcode is a tagged annotation token that attaches to an utterance as a whole and appears after the terminator. The canonical CHAT syntax is [+ <text>]. Postcodes carry researcher / analysis tags about the utterance, whether it should be excluded from analysis, how it should be coded, what kind of speech act it represents, without modifying the utterance’s word content.

Syntax and Scope

*CHI:   I want cookie .  [+ exc]
*MOT:   what did you say ?  [+ imp]
*CHI:   no I don't want it !  [+ neg] [+ trn]

Three structural facts to internalize:

1. Postcodes attach to the utterance, not to a word. They sit after the terminator, on the main tier, alongside (but distinct from) any utterance-level bullet. Unlike word-scoped annotations ([: ...] replacement, [% ...] comment, [= ...] explanation, [* ...] error code), a postcode does not modify the interpretation of any single word, it tags the whole utterance.
2. Multiple postcodes may follow a single terminator. They are ordered, but the order is not semantically privileged.
3. The body is free-form text. The CHAT word grammar is not applied to postcode contents. Researchers can write arbitrary tags, codes, descriptions, comments, or analytic notes. The model stores the raw text and leaves interpretation to downstream tooling and conventions.

Common Postcodes, Empirical Survey

The postcode vocabulary is open-ended: the CHAT format imposes no closed set, and an audit of every [+ ...] token across a JSON-mirrored snapshot of the TalkBank corpora (~99k files, 23+ data-repo families) found 488 distinct values in active use.

The findings split into three tiers ranked by repo spread (in how many distinct corpus families the code appears), the more useful ranking than raw count, because high-count codes can be concentrated in a single corpus.

Tier 1, Cross-corpus codes (in 7+ repos)

These are the conventions every CHAT consumer should expect to encounter across collections:

Tier 2, Multi-corpus protocol codes (in 4-6 repos)

Codes deployed across several CHILDES sub-collections, typically encoding picture-narration / story-reading / imitation experimental conditions. Substantial raw counts (often tens of thousands), but their meaning is set by the originating protocol, consult per-corpus documentation rather than assuming a global definition:

Tier 3, Single-corpus and long-tail codes

About 80% of the 488 distinct values appear in one repo only. The single-corpus codes include high-volume protocol vocabularies (e.g. [+ uncued] ~19,500 in one repo, [+ NAC] ~3,500 in one repo, [+ diary] ~2,800 in a Romance/Germanic diary-study collection, [+ noatt] ~2,300 in one repo, [+ inter-utter-switch] ~720 flagging code-switching turns).

The long tail also includes researcher-private notes, typos that survived check, and per-study coding schemes. Tooling MUST treat any unknown postcode value as opaque text, the corpus author may know what it means, the format does not.

Caveats

• Numbers are from a snapshot audit and will drift as corpora are added or revised. Treat the broad shape (open vocabulary, ~4 truly cross-corpus codes, ~10 multi-corpus protocol codes, ~hundreds of single-corpus or long-tail codes) as the load-bearing finding, not the exact counts.
• “Repo spread” counts data-repo families, not individual files. Two corpora curated by the same group inside one data-repo count as one for spread; researchers using the same code in two different family-of-corpora packages count as two.
• The CHAT manual remains the source of truth for standard conventions. The empirical survey above shows what is actually deployed; when ingesting a new corpus, consult its own documentation for the postcodes in use.

What Postcodes Are NOT

Postcodes are easy to confuse with several other CHAT annotation forms because they all use square brackets. The differences are substantive and load-bearing.

Two consequences worth pinning down explicitly:

• A postcode cannot carry per-word semantics. If you want to attach a comment, replacement, or error code to a single word, use the appropriate word-scoped form. Stretching a postcode to mean “this word is X” loses the per-word position downstream tools depend on.
• A word-scoped annotation cannot tag an utterance. If you want to mark an entire utterance for exclusion or translation, use a postcode. A [% exclude this] after a word does not mean “exclude the utterance” to any consumer.

Not Postcodes: Quotation Markers

Quotation marking in CHAT is not a postcode form. The constructs +"/. (quotation end), +"/, and +" (quotation linkers / continuations) are tier-level terminators and linkers, not [+ ...] postcodes, the grammar rule postcode in grammar/grammar.js is strictly [+ <text>], and the quotation forms live under separate grammar rules (quoted_new_line, linker_quotation_follows).

See Utterances → Terminators for the syntactic forms, and the talkbank-model::validation::cross_utterance validator family (gated by ValidationContext::enable_quotation_validation) for the cross-utterance balance checks.

A walker in talkbank-model::validation::utterance::quotation (check_quotation_balance) does scan the postcode list for text "/ and "/., but a sweep over the data-json corpus mirror (101,414 files, 2026-05-11) returned zero such postcodes, that code path is effectively dead, retained presumably as defence against hand-edited oddities. The real quotation-balance work happens in the cross-utterance family above.

Position in the AST

An utterance’s main tier is MainTier, whose content: TierContent field carries the actual tier payload, including postcodes, as a typed list:

pub struct MainTier {
    pub speaker: SpeakerCode,
    pub content: TierContent,
    // spans omitted for brevity
}

pub struct TierContent {
    pub linkers: TierLinkers,                  // utterance-leading +<, ++, etc.
    pub language_code: Option<LanguageCode>,   // [- code]
    pub content: TierContentItems,             // word-level items (newtype over Vec<UtteranceContent>)
    pub terminator: Option<Terminator>,        // ., ?, !, +..., etc.
    pub postcodes: TierPostcodes,              // [+ ...] tokens after the terminator
    pub bullet: Option<Bullet>,                // optional terminal media bullet
    // content_span omitted for brevity
}

(See talkbank-model/src/model/content/main_tier.rs and tier_content.rs for the exact shape. The postcodes slot lives on TierContent, which is the main-tier payload. Dependent tiers do not use TierContent: each has its own type (for example %com is a text tier, %wor is a list of timed items), and none carries a postcode slot. So a [+ ...]-shaped token on a dependent tier is parsed as ordinary tier content, never a Postcode. This is why chatter does not, and structurally cannot without banned raw-text scanning, reproduce CLAN CHECK 109 (“postcodes are not allowed on dependent tiers”); that deliberate divergence is recorded in the CHECK Parity Audit.)

Because postcodes live at the utterance level, the per-word traversal helpers (walk_words, walk_words_mut) do not visit them. Code that needs to read or rewrite postcodes accesses the list directly.

The model stores postcode text as SmolStr and preserves it verbatim through CHAT roundtrips. Downstream tooling, including CLAN command implementations such as freq, mlu, kideval, is responsible for interpreting individual postcode values per its own conventions.

Tooling Rules

Tools that emit or consume CHAT must respect the scope distinction.

• Emitters: when adding a researcher tag to an utterance, attach a Postcode to the utterance’s MainTierContent, not a ContentAnnotation to a word. Both serialize, but only the former reaches downstream consumers as utterance-level metadata.
• Consumers: when reading utterance-level tags (e.g., implementing an “exclude” filter), iterate main.content.postcodes on each utterance, not the word-level annotations in UtteranceContent. The two lists are populated by different parser branches and have different semantics.
• Round-trip preservers (extract→modify→inject pipelines such as the NLP injection passes in crates/batchalign-*): preserve the postcode list unchanged. None of the standard NLP passes have a reason to add, remove, or reorder postcodes.

References

• CHAT manual: Postcodes
• CHAT manual: Excluded Utterance Postcode
• CHAT manual: Included Utterance Postcode
• Model: talkbank-model/src/model/content/postcode.rs
• Quotation validator: talkbank-model/src/validation/utterance/quotation.rs

This page last changed: 2026-07-07 (commit ca8c5b8e). The whole book last changed: 2026-09-15 (commit bb4bef82).

Dependent Tiers

Status: Reference Last updated: 2026-09-09 08:49 EDT

Dependent tiers appear on lines beginning with % immediately after an utterance. They provide annotations linked to the main tier content.

CHAT defines four structural categories of dependent tiers:

1. Structured linguistic tiers: parsed into typed AST nodes with word-level alignment
2. Phon phonological tiers: syllabification and segmental alignment from the Phon project
3. Bullet-content tiers: free-form text with optional inline timing markers
4. Text tiers: plain text with no structural alignment

Structured Linguistic Tiers

These tiers have rich, parsed representations in the data model. Each token aligns 1-to-1 with an alignable word on the main tier (excluding retraces, pauses, and events). Terminators (., ?, !) must match the main tier terminator.

%mor, Morphological Analysis

The %mor tier carries part-of-speech tags, lemmas, and morphological features for each word on the main tier. See The %mor Tier for full documentation covering the UD-style format, data model, divergences from Universal Dependencies, and migration from traditional CHAT MOR.

Format: POS|lemma[-Feature]*, with ~ separating post-clitics.

*CHI:	she's eating cookies .
%mor:	PRON|she~AUX|be-Pres-S3 VERB|eat-Prog NOUN|cookie-Plur .

%gra, Grammatical Relations

The %gra tier encodes dependency syntax using Universal Dependencies relation labels. Each entry has the format index|head|relation, where indices are 1-based and head 0 indicates ROOT.

*CHI:	I want cookies .
%mor:	PRON|I VERB|want NOUN|cookie-Plur .
%gra:	1|2|NSUBJ 2|0|ROOT 3|2|OBJ 4|2|PUNCT

The %gra tier aligns with %mor chunks (clitics expand into multiple chunks). Validation checks sequential indices (E721), ROOT structure (E722 missing root, E723 multiple roots), and circular dependencies (E724). Two of those describe the tier AS A WHOLE and are withheld when the tier in hand is not the one you wrote: E721 and E722, when the parser had to reject a relation it could not represent, or when %mor-to-%gra alignment has already reported a count or index fault. E723 and E724 are always reported, because dropping a relation cannot create a second root or close a cycle, so a violation among the relations that survive is one the transcript contains.

%pho / %mod, Phonological Transcription

The %pho tier records actual pronunciation; %mod records target/model pronunciation. Both use the same format: space-separated phonetic tokens aligned 1-to-1 with main tier words.

*CHI:	I want three cookies .
%pho:	aɪ wɑnt fwi kʊkiz .
%mod:	aɪ wɑnt θri kʊkiz .

Phonological tiers support IPA, UNIBET, X-SAMPA, or custom notation systems. They are used for child language, speech disorders, L2 learning, and dialectal variation studies.

Parsing strategy: We deliberately parse only the minimal word/group-level structure in %pho and %mod needed for coarse alignment with the main tier. The full IPA phoneme content is stored as opaque strings, deep phonological analysis is handled by Phon, and we avoid duplicating that work. The Phon extension tiers (%modsyl, %phosyl, %phoaln) follow the same strategy.

%sin, Gesture and Sign Annotation

The %sin tier codes gestures and signs aligned with speech. Each token is either 0 (no gesture) or g:referent:type (e.g., g:ball:dpoint for a deictic point at a ball).

*CHI:	that ball .
%sin:	g:ball:dpoint 0 .

Multiple simultaneous gestures use bracket grouping: 〔g:toy:hold g:toy:shake〕.

%wor, Word Timing

The %wor tier carries word-level timing annotations for media synchronization. Words may include inline bullets with millisecond timestamps. Timing data comes from the bullet fields. Word text is not lexical authority, but consumers may compare it with chatter’s canonical generated display sequence to refuse stale same-count timing reuse.

⚠ IMPORTANT: %wor word text is the cleaned form, by design. When chatter serializes a %wor word it writes the word’s cleaned text, the spoken form with surface markers removed, NOT the raw main-tier surface form. This is a deliberate convention (see WorTier::write_chat in crates/talkbank-model/src/model/dependent_tier/wor.rs), chosen for human readability and because %wor exists to anchor timing, not to re-state the main tier’s orthography. The generated %wor text and the TextGrid export both use this cleaned form.

Consequence you must know: surface markers carried on a word, prosodic lengthening (wabe:), and similar in-word notation, are not preserved in %wor output. A main-tier word wabe: becomes wabe on %wor. This means a %wor line containing such words does not byte-roundtrip (parse, serialize, reparse changes the surface text), and that is expected, not a bug. %wor is a cleaned, timing-only view; the main tier remains the faithful record of surface forms. Do not “fix” the %wor serializer to emit raw text without an explicit decision to change this convention.

%wor is not a flat “all tokens except punctuation” tier. It follows a word-level alignment rule:

• Regular words count.
• Fillers (&-um, &-uh, &-you_know) count; they are real spoken words with known phoneme sequences.
• Fragments (&+...) do NOT count: incomplete phoneme sequences; the FA engine cannot reliably anchor partial phonological material.
• Nonwords (&~...) do NOT count: interactional/gestural sounds without stable lexical phoneme content for alignment.
• Untranscribed placeholders (xxx, yyy, www) do NOT count: they have no known phoneme sequence; CTC forced alignment cannot produce timings for unknown material.
• Replacements keep the original spoken word slot for %wor; the replacement text matters for %mor, not %wor. If the original slot is untranscribed or a fragment/nonword, it is still excluded.
• Retrace scope does not change %wor membership.
• Overlap markers do not change %wor membership.

%wor is a timing-annotation tier. Its word count equals the number of Wor-domain words and may differ from a naive main-tier word count. CHAT validation does not require the %wor count to match the current main tier. Timing consumers first require equal counts and then require every %wor display token to match the canonical token derived from its main-tier position. A mismatch refuses timing reuse without making the legacy CHAT file invalid.

*CHI:	I want cookies .
%wor:	I want cookies .

Exact corpus-shaped contrast:

*CHI:	<one &+ss> [/] one play ground .
%wor:	one •321809_321969• play •322049_322310• ground •322390_322890• .
# &+ss is a fragment, excluded from %wor regardless of retrace context.

*EXP:	&+ih <the what> [/] what's letter &+th is this ?
%wor:	the •49103_49163• what •49183_50205• what's •50205_50405• letter •50405_50685• is •50946_51046• this •51086_51586• ?
# Fragments &+ih and &+th excluded; regular words remain.

*EXP:	what's is dis [: this] ?
%wor:	what's •37050_37471• is •37491_37631• dis •37631_38131• ?

*CHI:	xxx snack .
%wor:	snack •884668_885168• .
# xxx has no phoneme sequence, excluded from %wor; only snack appears.

*CHI:	&~um a boat .
%wor:	a •1073779_1073799• boat •1076861_1077361• .
# &~um is a nonword, excluded from %wor.

*CHI:	&-mm [<] bananas are good .
%wor:	mm •1949506_1949566• bananas •1949566_1949766• are •1949846_1949987• good •1950067_1950567• .
# &-mm is a filler, included in %wor (real spoken word with alignable phoneme sequence).

flowchart TD
    A["Main-tier word candidate"] --> B{"Timestamp token /\nomission / empty?"}
    B -->|Yes| OUT["Excluded from %wor"]
    B -->|No| C{"Untranscribed?\n(xxx/yyy/www)"}
    C -->|Yes| OUT
    C -->|No| D{"Fragment or nonword?\n(&+ or &~)"}
    D -->|Yes| OUT
    D -->|No| IN["Counts for %wor\n(word or filler &-)"]

    style IN fill:#afa,stroke:#333
    style OUT fill:#faa,stroke:#333

Phon Phonological Tiers

These tiers originate from the Phon project and provide syllable-annotated phonological transcription and segmental alignment. They were originally serialized as %x-prefixed user-defined tiers (%xmodsyl, %xphosyl, %xphoaln) and are being promoted to official CHAT tiers. Phon stores phonological data in its own XML format. As of Phon 4.0.0-beta.9 (2026-06-25), Phon reads and writes CHAT natively.

%modsyl / %phosyl, Syllabified Phonology

%modsyl is a syllabified version of %mod (target pronunciation); %phosyl is a syllabified version of %pho (actual pronunciation). Each phoneme is annotated with a syllable position code (N=nucleus, O=onset, C=coda, etc.). Words are space-separated and align 1-to-1 with the corresponding %mod or %pho tier.

*CHI:	the best .
%mod:	ðə bɛst .
%modsyl:	ð:Oə:N b:Oɛ:Ns:Ct:C .
%pho:	ðə bɛs .
%phosyl:	ð:Oə:N b:Oɛ:Ns:C .

Alignment: Content-based, stripping position codes (:N, :O, :C, etc.) and stress markers (ˈ, ˌ) from %modsyl should yield the same phonemes as %mod. Same for %phosyl → %pho.

%phoaln, Phone Alignment

%phoaln provides segmental alignment between target and actual IPA, showing phoneme-by-phoneme correspondence. Each pair uses source↔target notation; ∅ marks insertions or deletions.

*CHI:	the best .
%phoaln:	ð↔ð,ə↔ə b↔b,ɛ↔ɛ,s↔s,t↔∅

Alignment: Positional, word-by-word, word N in %phoaln aligns with word N in both %mod and %pho.

Parsing strategy: Same as %pho/%mod, we parse just enough structure for alignment (word boundaries for %modsyl/%phosyl, alignment pairs for %phoaln). IPA phoneme content is treated as opaque strings.

Validation (E725-E728)

Because these are derived views, word counts must match between each syllabification tier and its parent IPA tier: