Introduction

Status: Current Last modified: 2026-06-21 21:33 EDT

TalkBank is the world’s largest open repository of spoken language data. This repository (TalkBank/chatter) is the standalone home of the CHAT format authority and the chatter tool family: the chatter CLI, the Rust crates for parsing/validation/transformation, the tree-sitter-talkbank grammar, the talkbank-lsp language server, and the desktop validation app.

chatter is publicly released. To get it right away:

• Command-line tool (macOS / Linux): curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.sh | sh (Windows and other options: Install).
• Desktop app: download for your platform from the latest release.
• Full installation guide (all platforms, package details): Install.

The Rust crates are source-available from this repository (not yet published to crates.io). As a 0.x release, APIs and flags may change before 1.0.

Choose the right surface

What’s In This Repo

• chatter CLI: validate, convert, normalize, and analyze CHAT files from the command line, with an interactive TUI for corpus-scale workflows
• Language Server (LSP): works with any LSP-compatible editor (Neovim, Emacs, Helix, Zed, etc.) to provide live validation and cross-tier alignment
• JSON data model: every CHAT structure as typed JSON with lossless roundtrip fidelity, backed by a published JSON Schema
• Rust API: parse, validate, inspect, and transform CHAT files programmatically via library crates

Who This Book Is For

Repository Layout

grammar/        Tree-sitter grammar for CHAT
spec/           Source of truth: CHAT specification + error specs
crates/         Rust crates for model, parser, transform, cache, CLI, LSP, tests, and FFI support
apps/           Tauri v2 desktop app (`chatter-desktop`)
corpus/         Reference corpus (must stay 100% valid under the regression gate)
schema/         JSON Schema for the CHAT AST
tests/          Integration tests and fixtures
fuzz/           Fuzz testing targets (separate Cargo workspace)
docs/           Strategy docs, proposals, and investigations for this repo
book/           This documentation (mdBook)

Data flows: spec (source of truth) → grammar (tree-sitter) → Rust crates (parsers, model, validation, CLI, LSP) → applications (chatter, desktop app).

Install

Status: Current Last modified: 2026-06-21 21:33 EDT

Installation paths for each surface of chatter. Pick the row that matches what you want to do and the audience you belong to.

chatter is publicly released: the CLI and desktop app are available from the latest GitHub release. The Rust crates and grammar are source-available from this repository (not yet published to crates.io). As a 0.x release, APIs and flags may change before 1.0.

For audio + ML pipelines (transcribe, force-align, morphotag, benchmark), see the upstream batchalign3 project, that lives outside the chatter repo and has its own installation flow.

Quickstart

Status: Current Last modified: 2026-06-21 21:33 EDT

Task-driven entry points. Pick the row that matches what you want to do today; each path starts at the narrowest useful documentation surface instead of dropping you into the whole book.

To download and install chatter, see Install.

For audio + ML workflows (transcribe / align media → CHAT), see the upstream batchalign3 project, outside the chatter repo.

Installation

Status: Current Last modified: 2026-06-16 07:55 EDT

chatter targets Windows, macOS, and Linux. There are two ways to install it: the prebuilt binaries (recommended for most people, including clinicians and researchers) and a from-source build (for contributors or unsupported platforms).

Prebuilt binaries (recommended)

Every GitHub Release attaches prebuilt binaries for macOS (Apple Silicon and Intel), Linux (x86_64 and ARM64), and Windows (x64), plus desktop-app installers.

chatter CLI

One-line installers (they download the binary for your platform, place it on your PATH, and also install the chatter-update self-updater):

• macOS and Linux:

  curl --proto '=https' --tlsv1.2 -LsSf https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.sh | sh
  

• Windows (PowerShell):

  powershell -ExecutionPolicy Bypass -c "irm https://github.com/TalkBank/chatter/releases/latest/download/chatter-installer.ps1 | iex"
  

On Windows the binary is not yet code-signed, so SmartScreen may warn on first run: choose More info, then Run anyway. The macOS binaries are codesigned, and the installer above does not set the quarantine attribute, so Gatekeeper does not prompt.

Prefer a manual download? Grab the archive for your platform from the latest release and extract chatter onto your PATH. (On macOS, a browser-downloaded archive is quarantined; right-click the binary and choose Open once, or run xattr -d com.apple.quarantine ./chatter.)

Verify:

chatter --version
chatter --help

chatter desktop app

The desktop app (“Chatter”) is for people who prefer a window to a terminal. Download the installer for your platform from the latest release:

• macOS: the .dmg is signed and notarized; open it and drag the app to Applications. No Gatekeeper override is required.
• Windows: the installer is not yet signed (same SmartScreen note as above: More info then Run anyway).
• Linux: an AppImage and a .deb are provided.

Updating chatter

chatter keeps itself current so you do not have to track releases by hand.

• CLI: run

  chatter update
  

  This runs the bundled chatter-update program, which checks GitHub Releases and installs the newest release in place. (The self-update facility is experimental. It is installed only by the one-line installers above; if you installed another way, update the same way you installed.)

• Desktop app: the app checks for updates on launch and offers to install a new version when one is available.

From source

Building from source needs only a stable Rust toolchain (install via rustup, which supports Windows, macOS, and Linux). Node.js and the tree-sitter CLI (cargo install tree-sitter-cli) are needed only when working on the grammar or generated artifacts.

Clone and install the CLI:

git clone https://github.com/TalkBank/chatter.git
cd chatter
cargo install --path crates/chatter --locked

This installs the chatter binary to ~/.cargo/bin/ (macOS/Linux) or %USERPROFILE%\.cargo\bin\ (Windows). To update a source install, pull and re-run the cargo install command above (chatter update is only for installer-based installs).

Building the libraries

If you are developing with the Rust crates directly, from your chatter checkout root:

cargo build --workspace --all-targets --locked
cargo test --workspace --locked
cargo clippy --all-targets -- -D warnings

See the contributor setup for additional commands.

Directory layout

Everything lives in a single repository:

<your-chatter-checkout>/
├── grammar/            # Tree-sitter grammar
├── crates/             # All Rust crates (talkbank-* + the chatter binary)
├── spec/               # CHAT specification
├── apps/               # Tauri desktop app (chatter-desktop)
└── book/               # Chatter mdBook (this book)

The CLI, grammar, crates, and the LSP/desktop integrations all live in this single repository.

Quick Start

Status: Current Last updated: 2026-06-15 15:00 EDT

This page gets you from zero to productive with chatter in five minutes. Install chatter first if you haven’t already.

Validate a CHAT file

Check a single transcript for errors:

chatter validate transcript.cha

If the file is valid you get a summary (a cache-statistics block follows it; use --quiet to suppress all output and rely on the exit code):

=== Summary ===
Total files: 1
Valid: 1
Invalid: 0

If there are problems, you’ll see rich diagnostics with the exact location and a stable error code. For example, a *CHI: line missing its terminator:

✗ Errors found in transcript.cha

E305 (https://talkbank.org/errors/E305)

  × error[E305]: Expected terminator not found (line 6, column 1)
   ╭─[input:6:1]
 6 │ *CHI:   hello world
   · ─────────┬─────────
   ·          ╰── here
   ╰────
  help: Add a terminator at the end: Standard (. ? !), Interruption
        (+... +/. ...), or CA intonation (⇗ ↗ → ↘ ⇘ ...)

Every error code (E305, E705, etc.) is documented with fix guidance in the validation error reference.

Validate an entire corpus

Point chatter at a directory, it walks recursively, validates in parallel, and caches results:

chatter validate corpus/

The interactive TUI shows progress and lets you browse errors per file. Use --format json for machine-readable output, or --quiet for CI (exit code 1 on errors).

Convert to JSON

Get a structured representation of any CHAT file:

chatter to-json transcript.cha

The output conforms to the TalkBank CHAT JSON Schema. Convert back with chatter from-json.

Watch for changes

Edit a file and get live validation feedback:

chatter watch transcript.cha

Every time you save, chatter re-validates and shows updated diagnostics.

What next?

• CLI Reference: all commands, flags, and output formats
• Validation Errors: every error code, with examples and fix guidance
• Batch Workflows: corpus-scale validation and analysis

CLI Reference

Status: Current Last modified: 2026-06-15 15:00 EDT

The chatter CLI is the primary command-line surface for the TalkBank CHAT toolchain.

The following diagram shows the command dispatch structure. Each top-level command dispatches to a handler in the corresponding crate.

flowchart TD
    chatter(["chatter"])

    chatter --> validate["validate\n(chatter)"]
    chatter --> normalize["normalize\n(chatter)"]
    chatter --> tojson["to-json\n(talkbank-transform)"]
    chatter --> fromjson["from-json\n(talkbank-transform)"]
    chatter --> showalign["show-alignment\n(chatter)"]
    chatter --> watch["watch\n(chatter)"]
    chatter --> lint["lint\n(chatter)"]
    chatter --> clean["clean\n(chatter)"]
    chatter --> newfile["new-file\n(chatter)"]
    chatter --> cache["cache\n(stats, clear)"]
    chatter --> schema["schema\n(JSON Schema output)"]
    chatter --> debug["debug\n(overlap-audit, linker-audit,\nfind, sanitize, fix-s)"]

    chatter --> merge["merge\n(experimental)"]
    chatter --> speakerid["speaker-id\n(experimental)"]
    chatter --> adjudicate["adjudicate\n(experimental)"]
    chatter --> pipeline["pipeline\n(experimental)"]
    chatter --> batch["batch\n(experimental)"]
    chatter --> sanityscan["sanity-scan\n(experimental)"]

Top-Level Commands

chatter validate PATH...
chatter normalize INPUT
chatter to-json INPUT
chatter from-json INPUT
chatter to-xml INPUT
chatter show-alignment INPUT
chatter watch PATH
chatter lint PATH
chatter clean PATH
chatter new-file
chatter cache stats
chatter cache clear --prefix PATH
chatter schema
chatter debug ...
chatter merge FILE1 FILE2          # experimental: combine two transcripts
chatter speaker-id INPUT           # experimental
chatter adjudicate ...             # experimental
chatter pipeline ...               # experimental
chatter batch ...                  # experimental
chatter sanity-scan ...            # experimental

Use chatter --help or chatter <command> --help for the exact live surface.

validate

Validate CHAT file(s) or directory tree(s). Accepts multiple paths.

Usage: chatter validate [OPTIONS] <PATH>...

chatter validate file.cha                         # single file
chatter validate file1.cha file2.cha file3.cha    # multiple files
chatter validate corpus/                          # directory (recursive, parallel)
chatter validate file.cha corpus/ other.cha       # mix of files and directories
chatter validate corpus/ -f json                  # structured JSON output
chatter validate corpus/ --force                  # ignore cache, revalidate everything
chatter validate corpus/ --force --audit out.jsonl # bulk audit to JSONL file
chatter validate corpus/ --suppress xphon         # suppress named error group
chatter validate corpus/ --suppress E726,E727     # suppress specific error codes
chatter validate corpus/ -j 8                     # use 8 parallel workers
chatter validate corpus/ --max-errors 50          # stop after 50 errors

Options:

Suppress groups: xphon expands to E725/E726/E727/E728 (%xphosyl/%xphoaln/%xmodsyl cross-tier alignment). These are suppressed by default since 2026-04-21; pass --check-xphon to include them. The --suppress flag can mix groups and codes: --suppress xphon,E316.

normalize

Serialize a CHAT file into canonical formatting.

chatter normalize input.cha
chatter normalize input.cha -o normalized.cha
chatter normalize input.cha --validate
chatter normalize input.cha --validate --skip-alignment

Flags:

• -o, --output <PATH>: write to a file instead of stdout.
• --validate: validate (including alignment by default) before writing the normalized output.
• --skip-alignment: when paired with --validate, skip the dependent-tier alignment checks (still validates the rest).

normalize writes to stdout unless you pass -o/--output. There is no --in-place flag.

JSON Conversion

# Single file
chatter to-json input.cha                          # pretty-printed JSON to stdout
chatter to-json input.cha --compact                # minified JSON to stdout
chatter to-json input.cha -o output.json           # JSON to file

# Directory (recursive, preserves structure)
chatter to-json corpus/ --output-dir json/          # incremental by default (mtime check)
chatter to-json corpus/ --output-dir json/ --compact # minified output (saves disk)
chatter to-json corpus/ --output-dir json/ --force   # full rebuild
chatter to-json corpus/ --output-dir json/ --prune   # remove orphaned .json files
chatter to-json corpus/ --output-dir json/ --jobs 4  # parallel workers

# Reverse and schema
chatter from-json input.json -o output.cha
chatter schema
chatter schema --url

Single-file mode: to-json validates by default. Use --skip-validation, --skip-alignment, or --skip-schema-validation to bypass checks.

Directory mode: Walks recursively, converting each .cha to .json under --output-dir with the same relative path. Incremental by default: skips files whose JSON is already newer than the source. Use --force to rebuild all. Use --prune to remove .json files with no matching .cha (handles renames/deletions). Use --jobs N for parallel conversion (defaults to number of CPUs).

to-xml

Export one CHAT transcript to TalkBank XML. The transcript is validated before any XML is emitted, so an invalid input fails (exit 1) and writes nothing to stdout; a failed export never leaves a partial document. This command is export-only: XML ingest is not implemented, so there is no from-xml.

chatter to-xml input.cha                  # XML to stdout
chatter to-xml input.cha -o output.xml    # XML to a file
chatter to-xml input.cha --skip-alignment # skip dependent-tier alignment checks

The output is TalkBank XML in the http://www.talkbank.org/ns/talkbank namespace (referencing talkbank.xsd). Writing to --output prints a one-line ✓ Converted ... to ... confirmation on stderr; writing to stdout prints only the XML.

Flags: -o, --output <PATH> (stdout if omitted); --skip-alignment (disable dependent-tier alignment validation during export).

Editing and Inspection Commands

show-alignment

Print the dependent-tier alignment for a CHAT file (debugging aid).

chatter show-alignment file.cha
chatter show-alignment file.cha -t mor          # one tier type
chatter show-alignment file.cha -t gra -c       # compact one-line-per-alignment output

Flags: -t/--tier <mor|gra|pho|sin> (omit to show all available tiers); -c/--compact (one line per alignment).

watch

Watch a CHAT file or directory and re-validate on every save.

chatter watch file.cha
chatter watch corpus/
chatter watch corpus/ --skip-alignment --clear

Flags: --skip-alignment (faster reruns); -c/--clear (clear the terminal between runs).

lint

Run lint checks and optionally auto-fix.

chatter lint corpus/
chatter lint corpus/ --fix
chatter lint corpus/ --fix --dry-run         # preview without modifying files
chatter lint corpus/ --skip-alignment

Flags: --fix (apply fixes); --dry-run (show what would change without writing); --skip-alignment.

clean

Show the cleaned text for each word (a debugging aid for the text-normalization pipeline).

chatter clean file.cha
chatter clean file.cha --diff-only       # only words where raw differs from cleaned
chatter clean file.cha --format json

Flags: --diff-only; --format text|json.

new-file

Create a new minimal valid CHAT file from defaults.

chatter new-file
chatter new-file -o starter.cha --speaker CHI --language eng
chatter new-file -o adult.cha -s MOT -l eng -r Mother
chatter new-file -c brown -u "hello world ."

Flags:

• -o, --output <PATH>: stdout if omitted
• -s, --speaker <CODE>: default CHI
• -l, --language <ISO 639-3>: default eng
• -r, --role <ROLE>: default Target_Child
• -c, --corpus <CORPUS>: corpus identifier in the @ID header (default corpus)
• -u, --utterance <TEXT>: optional initial main-tier utterance content

Cache Commands

chatter cache stats
chatter cache stats --json
chatter cache clear --prefix /path/to/corpus
chatter cache clear --all --dry-run

The validation cache lives under the platform cache directory and stores per-file validation results. validate --force refreshes cache state for the specified path.

debug

Developer / debugging subcommands for CHAT analysis. Not intended for routine end-user workflows; surface and behavior may change between releases. Run chatter debug --help for the live list. Current subcommands include:

• overlap-audit: analyze CA overlap markers (⌈⌉⌊⌋): pairing, temporal consistency, orphans.

• linker-audit: audit linker / special-terminator usage across a corpus (cross-utterance pairing for +<, ++, +^, +", +,, +≋, +≈, plus +..., +/., +//., +"/. etc.).

• find: filter CHAT files by @Languages and body content (token / substring counts) across a corpus tree; emits paths, JSONL, or CSV.

• sanitize: strip contributor lexical content while preserving structure, for protected-corpus debugging. See the Sanitize user-guide page for the full workflow.

• fix-s: normalize whole-utterance same-language @s runs into a [- lang] precode, clear the per-word @s markers (including those on fillers and nonwords), and append any missing explicit @s:LANG codes to @Languages. Trigger conditions and safety rules:

  • Every word-bearing item in the utterance, including fillers (&~, &-, &+), nonwords, and retraced material, must carry an explicit language marker AND every marker must resolve to the same target language. If a single filler such as &~dang3 lacks a marker, the utterance is left untouched (the predicate cannot prove it is monolingual).
  • Bare @s shortcuts on fillers must be cleared when the rewrite fires. A bare @s resolves relative to the surrounding tier language, so adding a [- LANG] precode without clearing the shortcut would flip the filler’s language to the precode target. fix-s clears the shortcut to keep the original meaning intact.
  • The pre-validation rule that catches the unrewritten pattern is E255 (whole-utterance same-language @s run); fix-s is the canonical repair. The companion warn-only E254 reports @s:LANG codes missing from @Languages; fix-s appends them.
  • True no-op on already-correct files: a file is rewritten only when a [- lang] conversion or @Languages repair can be proved necessary.

Merge and Reconciliation Commands (experimental)

These commands combine, reconcile, and relabel CHAT transcripts of the same recording, in the tradition of CLAN’s reliability and comparison tools (rely, trnfix). They are experimental and in active development: flags and behavior may change, and several modes are not yet complete. Work on copies and validate the output.

Full guides: Merge and Speaker ID. The speaker-id holistic-judgment mode can call an LLM provider (talkbank-llm) when configured; the deterministic modes need no network access.

Exit Codes

chatter validate exits with code 1 if any file has validation errors or parse errors. This makes it safe to use in scripts and CI pipelines:

chatter validate corpus/ --quiet --tui-mode disable || echo "Validation failed"

Use --quiet to suppress per-file success output while still relying on exit codes. Use --format json for machine-readable structured output (JSON objects go to stdout; exit code still reflects pass/fail).

Output Contracts

• Text output is intended for humans.
• JSON output is intended for automation and downstream tools.
• Error codes and the JSON Schema are documented public contracts; see the Integrating section of this book.

Validation Errors

Status: Current Last modified: 2026-06-17 11:29 EDT

The CHAT validator produces diagnostics at two severity levels: errors (must fix) and warnings (should fix). Each diagnostic has an error code that maps back to a documented spec and validator rule.

chatter validate is the binding judgment on whether a byte sequence is valid CHAT. When it reports an error, the file is invalid CHAT: clean the data rather than working around the check. A warning flags a questionable but parseable construct you should review. Where chatter and an older tool such as CLAN’s check disagree on whether a file is valid, chatter validate is authoritative (see CHECK Parity Audit for how the two are reconciled).

Reading Error Output

The validator emits rich diagnostics that include the error code, a source-pointed snippet, and a suggested fix:

  × error[E304]: Missing speaker in main tier (line 15, column 3)

15 │ *	hello world .
   ·  ╰── here
   ╰────
  help: Add a speaker code between * and : (e.g., *CHI:)

Each diagnostic contains:

• File path and location (line:column)
• Severity: error or warning
• Error code: E prefix for errors, W prefix for warnings, with a URL pointing at the per-code documentation page
• Message: human-readable description
• Suggestion: actionable fix guidance where available

Error Code Ranges

Common Errors and Fixes

E256: Curly single quote used as a word character

A curly single quotation mark (U+2018 or U+2019), commonly introduced by autocorrect or speech-to-text, is not a legal CHAT word character. CHAT words use the ASCII apostrophe (U+0027, the plain '). For example, a contraction typed as don + U+2019 + t is rejected; write don't with the ASCII apostrophe instead. chatter flags the curly form wherever it appears in word content and points the diagnostic at the exact character. This mirrors CLAN CHECK errors 138 and 139.

E304: Missing speaker code

A main tier line must have a speaker code after the *:

*CHI:	hello world .

An empty speaker code (*: hello .) triggers E304.

E308: Undeclared speaker

Every *SPEAKER: code must be listed in @Participants. Add the missing speaker to the header:

@Participants:	CHI Target_Child, MOT Mother

E370: Retrace marker with nothing to retrace

A retrace or repetition marker ([/], [//], [///]) must be followed by the repeated or corrected material; per the CHAT manual the marker always refers to the text that follows it. A marker followed only by a terminator has nothing to retrace:

*CHI:	<the> [/] .          ← invalid: [/] is not followed by repeated material
*CHI:	<the> [/] the cat .  ← valid: the repeated material follows the marker

This mirrors CLAN CHECK error 119 (and the related retrace checks 151 and 159).

E505: Invalid @ID format

Check that pipe-separated fields are correct and the speaker code matches @Participants:

@ID:	eng|corpus|CHI|2;6.||||Target_Child|||

E705: Main/%mor alignment mismatch

The number of %mor items must match the number of alignable words on the main tier. Retraces, pauses, and events are not counted. The validator shows a columnar diff:

  Main tier       %mor tier
  ──────────────  ──────────────
  I               pro|I
  want            v|want
  to              inf|to
  go              v|go
  home, ⊖

E714 / E715: %pho, %mod, or %wor count mismatch

The same two codes are reused for “too few” / “too many” count mismatches on %pho, %mod, and %wor.

For %wor, the main-tier side is a spoken-token inventory:

• regular words and fillers count
• fragments, nonwords, and xxx/yyy/www count
• retrace does not change %wor membership
• replacements keep the original spoken surface word for %wor

That context-sensitivity decides membership, not leniency. Once an item is in the %wor set, alignment is still strict 1:1. So if a filler like &-mm counts on the main tier and %wor omits it, E714 is the correct result.

So this is valid:

*CHI:	<one &+ss> [/] one play ground .
%wor:	one •321008_321148• ss •321148_321368• one •321809_321969• play •322049_322310• ground •322390_322890• .

But this is also valid:

*EXP:	&+ih <the what> [/] what's letter &+th is this ?
%wor:	ih •49063_49103• the •49103_49163• what •49183_50205• what's •50205_50405• letter •50405_50685• th •50886_50946• is •50946_51046• this •51086_51586• ?

And this is valid too:

*EXP:	what's is dis [: this] ?
%wor:	what's •37050_37471• is •37491_37631• dis •37631_38131• ?

E721: %gra sequential index error

%gra entries must have sequential 1-based indices: 1|...|... 2|...|... 3|...|...

Generated Error Documentation

The source of truth for error-code details is spec/errors/. Maintainers can also regenerate a local error-reference set from those specs when working on diagnostics:

cargo run --manifest-path spec/tools/Cargo.toml --bin gen_error_docs

That generated reference includes the error description, example inputs, suggested fixes, and the layer that catches the diagnostic.

Chatter Desktop

Status: Current Last modified: 2026-06-21 21:33 EDT

Chatter Desktop is a native graphical validation app for CHAT files, released alongside the chatter CLI. Prefer the chatter CLI for scripted or batch validation; use the desktop app when you want a standalone graphical validation experience without a terminal.

When to use Chatter Desktop

Chatter Desktop (apps/chatter-desktop/) is the right tool when you want to:

• Validate CHAT files through a graphical interface, no terminal required
• Drag and drop a file or folder and read errors with source snippets
• Work on the desktop without setting up a terminal workflow

Related surfaces:

• Validate CHAT from the command line: use chatter validate

This page documents the desktop surface:

• Chatter Desktop (apps/chatter-desktop/), the CHAT validation GUI

Current status

• Release contract: released alongside the CLI in the public chatter release
• Distribution: ships in the coordinated chatter release alongside the CLI; also buildable from source (below)
• Platforms: macOS, Windows, and Linux

Staying up to date

Chatter Desktop keeps itself current. When you launch it, it quietly checks for a newer release; if one is available it asks whether to update, and on your confirmation it downloads, installs, and restarts into the new version. If the check cannot reach the network it simply does nothing and the app keeps working on the version you have. You never have to track releases or re-download by hand.

Getting Started

Build from source

cd apps/chatter-desktop
npm ci
cargo tauri dev       # launches the app with hot reload
cargo tauri build     # produces a distributable app bundle

Requires: Rust (stable, edition 2024), Node.js, and npm.

Using the App

Opening files

Chatter validates one target at a time: a single .cha file or one folder.

Three ways to start validating:

1. Choose File: opens a file picker filtered to .cha files
2. Choose Folder: opens a folder picker; validates all .cha files recursively
3. Drag and drop: drag one .cha file or one folder onto the app window

When idle, if you’ve previously validated a target, the drop zone shows “Last: corpus/reference/, Re-validate?” as a clickable shortcut.

Reading results

The main window has three areas:

┌──────────────────────────────────────────────────────────────┐
│  [Choose File] [Choose Folder] or drag here  [System|Light|Dark] │
├──────────────────┬───────────────────────────────────────────┤
│ 3 FILES WITH     │  Filter by code… [All|Errors|Warnings]    │
│ ERRORS / 120     │                                           │
│                  │  ▾ [E302] Missing @End header              │
│  📁 corpus/      │  ┌───────────────────────┐                │
│    ✗ file1 (3)   │  │ 41 │ *CHI: hello .    │                │
│    ✗ file3 (1)   │  │ 42 │                   │                │
│                  │  │    │ ^                 │                │
│                  │  └───────────────────────┘                │
│                  │  💡 Add @End on the last line             │
│                  │  [Copy] [Open in CLAN]                    │
├──────────────────┴───────────────────────────────────────────┤
│  Progress: 45/120 │ 4 errors │ ~2m 30s remaining │ [Cancel]  │
└──────────────────────────────────────────────────────────────┘

• File tree (left), collapsible directory tree showing only files with errors (valid files are hidden to reduce clutter). A header shows “N files with errors / M total”. Files are sorted alphabetically.

• Error panel (right), for the selected file, shows each error with its code in [E001] format, severity color, message, source snippet with caret underlines, and multi-span labels for complex errors (e.g., alignment mismatches across tiers). CHAT-specific formatting is handled: tabs expanded to 8-column boundaries, \x15 bullets rendered as •, underline markers shown as styled underlined text. Suggestions prefixed with 💡.

• Status bar (bottom), streaming progress during validation, ETA after 5+ files, total error count, and action buttons.

Filtering errors

A compact filter bar appears above the error cards when a file has diagnostics:

• Code filter: type “E7” to show only alignment errors, “W” for warnings, etc.
• Severity toggle: switch between All / Errors / Warnings

The file header updates to show filtered vs. total count (e.g., “3 errors (7 total)”).

Collapsible error cards

Each error card has a clickable header that toggles between expanded and collapsed view. Collapsed cards show only the error code and first line of the message. When a file has 5 or more errors, an Expand All / Collapse All button appears.

Dark mode

Chatter follows your system appearance by default. A System / Light / Dark toggle in the drop zone area lets you override. Your preference is remembered across sessions.

The dark palette uses muted Apple-style colors, readable miette error highlighting on dark backgrounds.

Clickable file paths

Click the file name in the error panel heading to reveal the file in Finder (macOS), Explorer (Windows), or the default file manager (Linux).

Copy errors

Each error card has a Copy button that copies the full miette-rendered error text (plain text, not HTML) to your clipboard for pasting into issue reports or messages.

Actions

“Open in CLAN” only appears when the CLAN application is detected on your system (macOS and Windows only). It adjusts line numbers to account for headers that CLAN hides (@UTF8, @PID, @Font, @ColorWords, @Window).

Keyboard shortcuts

All other navigation is mouse-driven (click files, scroll errors).

Window title

The window title updates to reflect the current state:

• Idle: “Chatter”
• Discovering: “Chatter, Discovering files…”
• Running: “Chatter, Validating (45/120)”
• Finished: “Chatter, 14 errors in 3 files” or “Chatter, All 74 files valid”

ETA

After 5 or more files have been processed, the status bar shows an estimated time remaining (e.g., “~2m 30s remaining”). The estimate updates every second.

Notifications

When validation finishes while the app is not focused, a system notification shows the summary (“Validation complete, 14 errors in 3 files”).

First launch

On first launch, an onboarding overlay explains the four main interactions: drag files, error panel, keyboard shortcuts, and export. Dismiss with “Got it”, it won’t appear again.

CLI Bundling

The desktop app can bundle the chatter CLI binary so power users who download the GUI can also run the CLI from their terminal (like VS Code ships the code command).

An Install CLI Command menu item (when available) symlinks the bundled binary to /usr/local/bin/chatter (macOS/Linux) or copies it to a PATH directory (Windows).

To build with the bundled CLI:

cargo build --release -p chatter
mkdir -p apps/chatter-desktop/src-tauri/resources
cp target/release/chatter apps/chatter-desktop/src-tauri/resources/
cargo tauri build

Architecture

The desktop app lives in apps/chatter-desktop/:

apps/chatter-desktop/
  src-tauri/          Rust backend (Tauri v2)
    src/
      main.rs         Bin entry, calls chatter_desktop_lib::run()
      lib.rs          Tauri app setup (Builder + module wiring)
      protocol.rs     Shared command/event names + request types
      commands.rs     validate, cancel, open_in_clan, export, reveal, install_cli
      events.rs       ValidationEvent → frontend event bridge
      validation.rs   Desktop validation orchestration for one target
  src/                React + TypeScript frontend
    components/       DropZone, FileTree, ErrorPanel, ProgressBar, OnboardingOverlay
    hooks/            useValidation, validationState, useTheme
    protocol/         Command/event names + TypeScript transport mirrors
    runtime/          Tauri transport + capability-focused runtime seam

The Rust backend calls validate_directory_streaming() from talkbank-transform directly, the same streaming validation pipeline used by the TUI. Events flow over crossbeam channels to the Rust side, then are serialized to JSON and emitted to the frontend via Tauri’s event bridge.

Cancellation uses ArcSwapOption for lock-free atomic swap of the cancel sender, no mutex.

The frontend keeps Tauri-specific code confined to src/runtime/tauriTransport.ts. React components and hooks consume narrower capabilities (validationRunner, validationTarget, clan, exports) instead of reaching for one broad desktop service object.

Comparison with TUI

Both use the identical validation engine and produce the same error codes.

When to Use Which Tool

The TalkBank toolchain offers validation through three interfaces. Each serves a different workflow:

Chatter Desktop focuses on validation only.

CLAN Line Numbering

Status: Current Last modified: 2026-05-29 17:31 EDT

When you click “Open in CLAN” in the desktop app or press Enter in the TUI, chatter sends the error location to the CLAN editor. CLAN opens the file and places the cursor at the error. This usually works seamlessly, but there is one caveat: CLAN and chatter count lines differently.

Hidden Headers

CLAN hides five header types from its editor display:

These headers are present in the .cha file but invisible in CLAN’s editor. CLAN’s line numbers skip them entirely. A file that starts with @UTF8 on line 1 will show @Begin as “line 1” in CLAN’s display, even though it’s actually line 2 in the file.

What Chatter Does

Chatter automatically adjusts line numbers before sending to CLAN:

1. Compute the error’s line number in the source file
2. Count how many hidden headers appear before that line
3. Subtract the hidden count to get CLAN’s line number
4. Send the adjusted line number to CLAN

This happens transparently, you don’t need to do anything.

Edge Case: Errors on Hidden Lines

If an error is on a hidden header itself (e.g., a malformed @UTF8 line), CLAN cannot navigate to it because CLAN doesn’t display that line. In this case, “Open in CLAN” will show an error message explaining why.

For Developers

The shared resolution logic lives in talkbank_model::resolve_clan_location(). Both the TUI and the desktop app call this function, it resolves line/column from byte offsets when needed and adjusts for hidden headers.

See clan_location.rs for the implementation and tests.

Batch Workflows

Status: Current Last modified: 2026-06-12 21:05 EDT

The chatter CLI is designed for processing large CHAT corpora efficiently. This page covers common batch workflows.

Validating a Corpus

Validate all .cha files in a directory tree:

chatter validate /path/to/corpus/

The validator recursively discovers .cha files and processes them in parallel. Results are cached, subsequent runs skip unchanged files.

Forcing Revalidation

To bypass the cache and revalidate everything:

chatter validate /path/to/corpus/ --force

Filtering Output

Show only errors (hide warnings):

chatter validate /path/to/corpus/ --quiet

Stop after the first reported error:

chatter validate /path/to/corpus/ --max-errors 1

Write a JSONL audit file while validating:

chatter validate /path/to/corpus/ --audit validation.jsonl

CHAT-JSON Roundtrip

Convert an entire corpus to JSON and back:

# CHAT → JSON
for f in corpus/**/*.cha; do
  chatter to-json "$f" > "${f%.cha}.json"
done

# JSON → CHAT
for f in corpus/**/*.json; do
  chatter from-json "$f" > "${f%.json}.roundtrip.cha"
done

The roundtrip is designed to preserve the ChatFile model. In regression tests, compare normalized output rather than assuming byte-for-byte identity after parser or serializer changes.

Cache Management

The validation cache stores results for previously validated files (keyed by content hash). The cache database file is named talkbank-cache.db and lives in the OS cache directory:

• macOS: ~/Library/Caches/talkbank-chat/talkbank-cache.db
• Linux: ~/.cache/talkbank-chat/talkbank-cache.db
• Windows: %LocalAppData%\talkbank-chat\talkbank-cache.db

It can hold results for large file collections.

To relocate the cache (a different disk, a per-project cache, or an isolated cache for scripted runs), set the TALKBANK_CHAT_CACHE_DIR environment variable to a directory; the database is created directly inside it. This is the supported override on every platform, and the only effective one on Windows, where the default location comes from the system Known Folder API rather than environment variables.

chatter cache stats    # Show hit rates and entry count
chatter cache clear --all

Do not delete the cache file manually while chatter is running.

Reference Corpus Validation

This repository includes a reference corpus at corpus/reference/ (currently ~100 .cha files; verify by find corpus/reference -name '*.cha' | wc -l). The parser must handle every file in this corpus at 100%:

cargo nextest run -p talkbank-parser-tests -E 'test(parser_equivalence)'

This runs the parser equivalence test, each .cha file is its own test, so nextest runs them in parallel and reports individual failures.

Integration with batchalign

The Batchalign pipeline uses the same Rust core (via PyO3) for CHAT parsing and serialization. Since the 2026-04-28 monorepo merge, Batchalign source lives inside this repository under crates/batchalign-* (the standalone batchalign3 GitHub repo was archived). Files processed by Batchalign produce valid CHAT that passes chatter validate.

CI Integration

Status: Current Last updated: 2026-04-13 19:23 EDT

How to use chatter in continuous integration pipelines.

Exit Codes

All examples below rely on exit code 1 to signal validation failure.

Basic Usage

chatter validate corpus/ --quiet --tui-mode disable

• --quiet suppresses per-file success output
• --tui-mode disable prevents interactive TUI (required in non-TTY environments)
• Exit code 0 means all files valid; 1 means errors found

GitHub Actions Example

- name: Validate CHAT corpus
  run: |
    chatter validate corpus/ --quiet --tui-mode disable --format json --audit results.jsonl

- name: Upload validation report
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: validation-report
    path: results.jsonl

The --audit results.jsonl flag streams per-error JSON lines to a file, which is useful for archiving or downstream analysis even when the step fails.

JSON Output for Automation

chatter validate corpus/ --format json --tui-mode disable 2>/dev/null

Each file produces a JSON object on stdout with status, error_count, and errors array. The exit code still reflects overall pass/fail.

Pre-commit Hook

#!/bin/sh
# .git/hooks/pre-commit
chatter validate . --quiet --tui-mode disable

This blocks commits that introduce invalid CHAT files. The hook runs quickly on cached files; only modified files are re-validated.

Suppressing Specific Errors

Some corpora have known issues that should not block CI. Use --suppress to ignore specific error codes or named groups:

chatter validate corpus/ --suppress E726,E727,E728 --tui-mode disable

Or use the named group shorthand:

chatter validate corpus/ --suppress xphon --tui-mode disable

Suppressed errors do not appear in output and do not affect the exit code.

Audit Mode for Large Corpora

For bulk corpus validation where you want a full error database without caching overhead:

chatter validate corpus/ --audit errors.jsonl --tui-mode disable

The --audit flag streams one JSON object per error to the specified file. A summary is printed to stderr at the end.

CHAT Processing Playbook for Editors and Analysts

Status: Current Last updated: 2026-03-24 00:01 EDT

Objective

Provide practical guidance for non-compiler users who create, edit, and validate CHAT files, with emphasis on error interpretation and correction workflow.

Who This Is For

• Transcript editors,
• corpus curators,
• QA reviewers,
• linguists using tooling outputs but not parser internals.

Core Editing Workflow

1. Open file in editor with CHAT diagnostics enabled.
2. Run validation (single file first, then batch).
3. Fix highest-severity structural issues first (headers, tier markers, unmatched delimiters).
4. Re-run validation and inspect warnings.
5. Only then address style and normalization suggestions.

Error Triage Heuristic

• Errors at file start: likely header formatting or encoding issues.
• Errors at tier prefix: likely malformed */% tier syntax.
• Errors inside words: likely symbol, marker, or annotation boundary issues.
• Repeated same error class: likely one systemic rule violation pattern.

Fast Interpretation Guide

• Error: parser/validator could not accept structure; must fix.
• Warning: valid but suspicious or non-canonical; review strongly recommended.
• Info: advisory normalization or convention hints.

Common Fix Recipes

• Header spacing problems:
  • Ensure expected separators and avoid accidental tabs/spaces drift.
• Unclear language/form markers:
  • Confirm @s usage and suffix ordering with house style guide.
• Duration/annotation confusion:
  • Verify bracketed annotation form and avoid malformed punctuation.
• Dependent tier attachment issues:
  • Ensure % tiers follow intended main tier and keep indentation consistent.

Batch Validation Workflow

1. Validate a small sample first.
2. Group failures by error code.
3. Fix by pattern, not file-by-file random order.
4. Re-run and confirm error count decreases monotonically.
5. Save run report for audit trail.

Collaboration Workflow with Developers

When reporting parsing issues, include:

• exact file path,
• minimal excerpt around failing span,
• observed diagnostic code/message,
• expected behavior (if known).

This reduces back-and-forth and speeds defect triage.

Quality Checklist Before Publishing Corpus Updates

• No unresolved error-level diagnostics.
• Warning classes reviewed and accepted or fixed.
• Participant headers and IDs internally consistent.
• Roundtrip serialization check passes for representative samples.
• Changelog note recorded for major normalization edits.

Training Recommendations

• Maintain short examples for each common error class.
• Provide editor cheat sheet for tier prefixes and marker syntax.
• Run periodic QA calibration sessions across editors.

Sanitize (chatter debug sanitize)

Status: Current Last updated: 2026-04-28 22:18 EDT

chatter debug sanitize strips contributor lexical content from a CHAT file while preserving structure (timing bullets, %wor per-word offsets, speaker codes, dependent-tier scaffolding, structural counts, POS tags, language markers). Output is structurally identical to the input but contains no participant words, names, or free-text annotations.

The command exists so engineering tooling, including LLM-assisted debugging, can operate on protected-corpus files (aphasia/, dementia/, rhd/, fluency/Password/, clinical-children corpora, etc.) without exposing contributor speech to commercial LLM services.

When to use it

Run chatter debug sanitize on the source file before loading it into any tool (LLM-backed debugger, scratch directory, screen-shareable session) where you don’t want participant content visible.

When you need to ask a contributor for help debugging a specific file, frame the request as “run the sanitizer locally and send me the output” rather than asking for the raw file.

Usage

# Write sanitized output to stdout
chatter debug sanitize input.cha

# Write sanitized output to a file
chatter debug sanitize input.cha --output sanitized.cha

Working location for sanitized files: prefer a stable, non-/tmp scratch directory (e.g. set TB_SCRATCH_DIR to a per-project dir under your workstation’s persistent storage) for any state that should outlive a single command. macOS clears /tmp on reboot.

What is preserved (byte-exact)

• Timing bullets •start_end• on the main tier.
• %wor per-word offsets (word START_END triples).
• Speaker codes (*PAR, *INV, *CHI, …).
• Utterance count, word count per utterance, dependent-tier count.
• Structural markers: compound +, clitic ~, CA elements, overlap points, lengthening, stress markers, syllable pause, underline begin/end, proper-noun @n markers.
• Language markers (@s:LANG), form types (@a, @b), POS tags ($adj, $n).
• Headers: @Languages, @Birth, @Date, @Media, @PID, @L1Of, @Begin/@End/@UTF8.
• %mor POS categories and morphological features (e.g., n|, -Past).
• %gra (numeric grammatical relations) and %tim (timing).
• Untranscribed tokens xxx / yyy / www, preserving them changes semantic meaning, so they pass through unchanged.

What is replaced or redacted

Determinism + Idempotence

Placeholder generation is keyed off (utterance_index, word_index) tree position, not a global counter. Two consequences:

• Deterministic: sanitizing the same input twice produces byte-identical output.
• Idempotent: sanitizing a sanitized file produces the same file again, no double-replacement, no shifting placeholder numbers.

Pipeline

flowchart LR
    Input["Source .cha\n(protected corpus)"] --> Parser["TreeSitterParser\n(talkbank-parser)"]
    Parser --> Model["ChatFile model\n(talkbank-model)"]
    Model --> Sanitize["sanitize()\n(talkbank-transform::redact)"]
    Sanitize --> Walker["walk_words_mut\n+ header walker\n+ dep-tier walker\n+ scoped-annot walker"]
    Walker --> Mutated["Mutated ChatFile\n(placeholders + redactions)"]
    Mutated --> Writer["WriteChat\n(byte-exact bullets)"]
    Writer --> Output["Sanitized .cha\n(scratch path)"]

The walker step replaces WordContent::Text segments inside Word.content, mutates MorWord.lemma fields, redacts free-text header / dep-tier / scoped-annotation strings, and drops phonological tiers. WriteChat then re-serializes, and because it serializes from Word.content (not from Word.raw_text), every CA element, compound marker, clitic boundary, and timing bullet round-trips byte-exact.

Out of v1 scope

Documented for transparency; v2 work:

• Speaker-code anonymization (graph rewrite across @Participants, @ID, *SPK:, @Birth, @L1Of).
• @Birth / @Date fuzzing (exact birth dates can be identifying).
• @Media filename redaction.
• Audio-side sanitization. (Audio bytes are never touched by the sanitizer; the audio stays at its original path.)
• “Unsanitize” or round-trip mapping. Explicitly not built, the sanitizer is one-way, the mapping table that would reverse it is the exact artifact we don’t want to exist.

Implementation

Library module: talkbank_transform::redact. CLI surface: chatter debug sanitize. The strict policy is the only public preset in v1; future variants can grow on SanitizationPolicy.

Speaker-ID (chatter speaker-id)

Status: Draft Last modified: 2026-06-15 12:18 EDT

chatter speaker-id assigns CHAT-conformant speaker codes and role tags to a CHAT file whose speakers carry anonymous or placeholder labels (typically the output of an ASR system that labels speakers as PAR0, PAR1, …). It is the bridge between an ASR pipeline that does not understand speaker roles and a CHAT pipeline that does.

The command is structural: it does not modify utterance content, does not run audio analysis, does not infer speaker identity from voice features. Its inputs are the CHAT file to relabel plus an identification signal (reference transcript, explicit mapping, or saved override record); its output is the same CHAT file with speaker codes rewritten and @Participants / @ID headers reconciled.

When to use it

Whenever you have a CHAT file with placeholder speaker codes that need to become CHAT-conformant codes before downstream tooling can process the file meaningfully. The canonical case is an ASR system that emits CHAT but does not know which speaker is the child, parent, clinician, etc.

A complete pipeline that consumes ASR output and produces a publishable CHAT file goes:

flowchart LR
    Media --> Transcribe
    Transcribe["batchalign3 transcribe<br/>ASR"] --> AsrAnon["asr.cha<br/>PAR0, PAR1, ..."]
    Ref["reference.cha<br/>target speakers only"] -.->|reference signal| SpkId
    AsrAnon --> SpkId
    SpkId["chatter speaker-id<br/>(this page)"] --> AsrLabeled["asr-labeled.cha<br/>CHI, INV, MOT, ..."]
    AsrLabeled --> Merge["chatter merge"]
    Ref --> Merge
    Merge --> Aligned["batchalign3 align"]

The speaker-id stage is the single point in the pipeline where “which anonymous speaker corresponds to which CHAT role” is decided. Downstream stages (chatter merge, batchalign3 align, batchalign3 morphotag) all trust that the labels they receive are correct.

Identification modes

Three mutually-exclusive modes, exactly one of which must be selected:

1. Reference mode

The most common case: a separate CHAT file already exists that covers the same media and contains an authoritative speaker (typically the hand-transcribed target speaker). The reference file’s anchor speaker tells us what that speaker’s content looks like; speaker-id finds the matching speaker in the input by text similarity.

The matching algorithm is multiset Jaccard over bags of content tokens, see “Algorithm” below for the full specification. The ASR speaker whose bag-of-words best matches the reference anchor’s bag-of-words is taken as the same speaker, and is marked for drop in the output (because the reference file authoritatively covers them, the downstream chatter merge stage will pull their utterances from the reference, not from this file). The remaining speakers are renamed to the role specified by --inserted-role.

If the Jaccard margin between the winning speaker and the runner-up is below --confidence-threshold, the command refuses to auto-decide. The operator must either lower the threshold (not recommended without spot-checking), supply an explicit mapping (--mapping), or load a previously-adjudicated override (--override-file).

2. Explicit-mapping mode

The operator already knows the mapping (typically because they listened to the audio, or because the contributor’s data sheet documents it). They supply it directly.

chatter speaker-id input.cha \
  --mapping "PAR0=INV:Investigator,PAR1=drop" \
  -o relabeled.cha

The grammar for --mapping:

• One or more comma-separated assignments.
• OLD=CODE:ROLE renames OLD to CODE with role tag ROLE.
• OLD=drop removes OLD’s utterances entirely.
• Every speaker present in the input must be named in the mapping (no defaulting). This is intentional, we want operator decisions to be explicit.

3. Override-file mode

The operator has previously adjudicated this session (perhaps through an interactive review tool) and saved the decision to a shared override file. speaker-id reads the file, finds the entry for this session, and applies it. See “Override file format” below.

chatter speaker-id input.cha \
  --override-file batch-2026-05-27.overrides.toml \
  --session-id NF203-2 \
  -o relabeled.cha

This mode is the production substrate for batch workflows: the orchestrator first runs chatter speaker-id in reference mode for every session; for any session that exits with low-confidence, the operator works through an adjudication tool that writes to the override file; the orchestrator then re-runs chatter speaker-id in override-file mode for those sessions.

CLI contract

chatter speaker-id <INPUT> [OPTIONS]

ARGUMENTS:
  <INPUT>  Path to the CHAT file to relabel.

OPERATION MODES (exactly one required):

  REFERENCE MODE:
    --reference <FILE>
    --anchor <SPEAKER>
    --inserted-role <CODE>:<TAG>[,<CODE>:<TAG>...]

  EXPLICIT-MAPPING MODE:
    --mapping <SPEC>

  OVERRIDE-FILE MODE:
    --override-file <FILE>
    --session-id <ID>

REFERENCE-MODE OPTIONS:
  --confidence-threshold <FLOAT>
      Minimum Jaccard margin (winner_score / loser_score) for the
      command to auto-decide. Below threshold: exit code 4. The
      command prints per-speaker scores to stderr so the operator
      can inspect. Default: 2.0.

  --write-override <FILE>
      When auto-decide succeeds, append the decision to FILE in
      override-file format (creates if missing). Captures the
      audit trail of a batch run.

COMMON OPTIONS:
  -o, --output <PATH>
      Write relabeled CHAT to PATH. Default: stdout.

The operator identity and any free-text note for a session are set
when an operator confirms it through `chatter adjudicate` (see the
merge workflow), not on this command.

Exit codes:

What the output guarantees

These are testable invariants. Every release verifies them against the reference corpus.

Speaker codes match the supplied mapping

For every speaker in the input file:

• If the mapping marks the speaker for drop, none of their utterances appear in the output, AND their @ID row (if any) is removed from the headers, AND their entry is removed from the @Participants header.
• If the mapping marks the speaker for rename, every main-tier line *OLD:\t... becomes *NEW:\t... byte-stable except for the speaker code prefix. The @ID row’s third pipe-separated field (speaker code) and eighth field (role tag) are rewritten; other @ID fields are preserved. The @Participants entry’s code and role-tag tokens are rewritten; any intervening tokens (corpus ID, participant name) are preserved.
• Speakers not in the mapping are passed through unchanged. (In modes 1 and 3, all speakers are assigned automatically; in mode 2, “all speakers must be in the mapping” is a precondition.)

Utterance content is byte-stable except for the speaker prefix

For every retained utterance, every byte EXCEPT the leading *CODE:\t prefix is preserved verbatim. Dependent tiers attached to the utterance are preserved exactly. NAK-delimited time bullets, CHAT markup, special-form annotations, paralinguistic codes, retracing scopes, all untouched.

Headers reconcile per a fixed table

Provenance is captured if --write-override is set

When --write-override <FILE> is supplied AND the command succeeds in reference mode, an entry is appended to FILE recording the session ID (derived from the input filename stem unless overridden), the per-speaker Jaccard scores, the chosen mapping, the operator, and an ISO 8601 timestamp. The format is specified in “Override file format” below. The operator identity and any free-text note are set later, when a session is confirmed via chatter adjudicate.

This is the audit-trail mechanism: a year from now, a researcher who asks “why was PAR0 labeled INV in this session?” can read the override entry and see the scores, the operator, and any notes the operator added.

Algorithm (reference mode)

Token cleaning

Both the reference anchor’s bag of words and each input speaker’s bag of words are built by walking the typed CHAT AST and emitting content tokens. The cleaner strips:

• NAK-delimited time bullets
• bracket-annotated markup [*], [//], [/], [=! ...], etc.
• angle-bracket retracing scope (<...>, unwrap, keep inner text)
• terminator variants +//., +..., +/., +!?, etc.
• filled-pause and phonological-fragment markers &-..., &+...
• unintelligible placeholders xxx, yyy, www
• zero-realization markers 0
• special-form suffixes (word@l → word)
• CHAT compound underscores (Valentine's_Day → Valentine s Day)
• punctuation, then lowercase, then filter to alpha-only tokens of length ≥ 2

Both sides are cleaned identically so the comparison is apples-to-apples. This is the same cleaner specified in the reference corpus under spec/constructs/speaker-id/token-cleaner/.

Multiset Jaccard

For two bags-of-words A and B (counted multisets):

J(A, B) = sum_w min(A[w], B[w])  /  sum_w max(A[w], B[w])

Range [0, 1]. The multiset (rather than set) form rewards speakers who say similar things to the anchor in similar volume, not just speakers whose vocabulary happens to intersect.

Decision

scores  = { speaker: J(anchor_bag, speaker_bag) for speaker in input }
winner  = argmax(scores)
loser   = argmax(scores - {winner})
margin  = scores[winner] / scores[loser]    # ∞ when loser score = 0

• winner is the input speaker whose content matches the reference anchor’s content best → marked for drop (the reference authoritatively covers them).
• loser (and any other lower-scoring speakers, in the multi-speaker case) → renamed to the role given by --inserted-role.

If margin < --confidence-threshold (default 2.0), the command exits with code 4 and prints per-speaker scores to stderr. The operator must inspect, adjudicate, and re-run with --mapping or --override-file.

Why this algorithm

The choice was empirical, not theoretical, and was made against a calibration set of CHAT files paired with their corresponding ASR output. Two earlier candidates were tested first and rejected:

• Raw temporal-overlap (sum of ms of an input speaker’s activity inside the anchor’s bullet windows): too weak on real data. Hand transcripts often place per-utterance time bullets as end-to-end segmentation boundaries covering 95-99% of the session timeline, rather than as tight “speaker active here” windows. Both input speakers fall almost entirely “inside” the anchor’s bullet windows and the signal disappears.
• Speaker purity (fraction of each input speaker’s activity falling inside anchor windows): same root cause, same failure.

Multiset Jaccard over content tokens succeeded on every session of the calibration set. The borderline cases (margin below 2.0x) clustered around tasks where the non-anchor speaker shares vocabulary with the anchor by the structure of the task, e.g. a clinician describing the same scene the child is also describing in a picture-narrative task. These borderline cases are the reason for the conservative threshold and the --mapping/--override-file escape hatches; the algorithm correctly refuses to auto-decide them rather than silently picking wrong.

Override file format

The override file is a UTF-8 TOML document with one [<session_id>] table per decision. A minimal entry:

schema_version = 1

[session-101-t1]
mode = "auto"
inserted_role = { code = "INV", tag = "Investigator" }
mapping = { PAR0 = "rename", PAR1 = "drop" }
scores = { PAR0 = 0.1931, PAR1 = 0.7347 }
margin = 3.81
operator = "alice"
decided_at = 2026-05-27T08:41:00-04:00

The complete schema specification, every field, every type, every mode-semantics rule, the strict refuse-with-clear-error versioning policy, and worked examples for auto/explicit/replay/diarization-mixed cases, is on the dedicated reference page: Merge Override File Format.

Highlights from the reference:

• mode = "auto" | "explicit" | "override" records how the decision was made (informational for audit trail; behavior at apply time is the same).
• inserted_role.code is the CHAT speaker code (INV, MOT, FAT, PAR, …); inserted_role.tag is the CHAT role-tag (Investigator, Mother, …). All renamed speakers in one entry share the same role.
• mapping must cover every speaker in the input, no defaulting.
• scores and margin are optional but the writer always records them when an auto attempt produced them (even when the final decision was operator-supplied).
• flags carries operator-supplied markers like "diarization-mixed" for unusual cases. Unknown strings are preserved verbatim.

Preconditions

chatter speaker-id refuses (exit code 2) if any hold:

Reference mode

• The reference file has no utterances for --anchor
• The reference file fails to parse
• The input file has fewer than 2 distinct speakers (no discrimination problem)

Explicit-mapping mode

• A speaker in the mapping is not present in the input
• A speaker in the input is not covered by the mapping (no defaulting)

Override-file mode

• The override file does not contain a <session-id> entry
• The entry’s mapping references a speaker not in the input
• The entry’s mapping does not cover every speaker in the input

What chatter speaker-id is NOT

• Not voice diarization. Use Batchalign’s ASR pipeline upstream; the labels this command consumes are the labels Batchalign emits.
• Not content correction. If the speaker the command identifies has been mis-transcribed by ASR, this command does not fix that , re-run ASR with a better engine.
• Not a merge. This command operates on a single CHAT file. To combine the relabeled file with the reference, use chatter merge.
• Not interactive. chatter speaker-id is batch-only: it succeeds, refuses, or fails. The interactive review that resolves a low-confidence refusal into an override-file entry is a separate command, chatter adjudicate, run as part of the merge workflow.

Worked example

A typical fully-automated reference-mode call from an orchestrator script:

chatter speaker-id asr-anonymous.cha \
  --reference hand-transcript.cha \
  --anchor CHI \
  --inserted-role INV:Investigator \
  --confidence-threshold 2.0 \
  --write-override batch.overrides.toml \
  -o asr-labeled.cha

For a session this refused (e.g., shared-vocabulary narrative task with margin 1.82x), the orchestrator captures the failure and the operator later resolves it:

# Inspect the scores the command emitted to stderr:
#   PAR0=0.6286  PAR1=0.3457  margin=1.82x  threshold=2.0
# Operator listens to a few seconds of audio and confirms PAR0 is
# the child:

chatter speaker-id asr-anonymous.cha \
  --mapping "PAR0=drop,PAR1=INV:Investigator" \
  --write-override batch.overrides.toml \
  -o asr-labeled.cha

Later, if anyone re-runs the batch, they use override-file mode:

chatter speaker-id asr-anonymous.cha \
  --override-file batch.overrides.toml \
  --session-id NF204-2 \
  -o asr-labeled.cha

The same asr-labeled.cha content is produced; the audit trail remains intact.

Implementation notes (for contributors)

• Source: crates/talkbank-transform/src/speaker_id/ (proposed layout).
• CLI surface: crates/chatter/src/commands/speaker_id/.
• Domain types (SpeakerCode, RoleTag, SpeakerMapping, MergeOverride, JaccardScore, ConfidenceThreshold, Margin) live in talkbank-model and are shared with chatter merge plus any future adjudication UI.
• The Jaccard cleaner walks talkbank-model::ChatFile directly via the existing content walker (talkbank-model::walk_words); it does NOT re-implement CHAT parsing or use regex on raw bytes for tokenization.
• Spec entries for the cleaner and the algorithm live in spec/constructs/speaker-id/. Every invariant on this page has a spec; regenerate them with the current spec/tools commands from Spec Workflow.
• The override-file reader/writer is a typed serde round-trip on a TOML representation; the schema lives in talkbank-model so the format is one shared type across the codebase, not duplicated parsing logic in each consumer.

Merge (chatter merge)

Status: Draft Last modified: 2026-06-11 15:32 EDT

chatter merge combines two CHAT transcripts that cover the same media recording into one. The caller designates which speakers’ utterances are authoritative in which file; the merged output interleaves them by time while byte-preserving every utterance from its designated source.

The command is structural: it does not invent or rewrite utterance content, does not run ASR, does not run forced alignment, does not infer speaker identity. It is the moment in a multi-input CHAT workflow where two parsed transcripts become one.

When to use it

Whenever you have two valid CHAT files of the same recording and you want a single combined CHAT file out, with explicit per-speaker provenance.

Two recurring shapes from real TalkBank workflows:

• Hand-coded target speaker + ASR everyone else. A contributor has hand-transcribed only the target speaker (often the child in child-language research) with rich disfluency and error coding, and separately someone runs ASR on the same media to produce a rough-but-complete transcript with all speakers. chatter merge combines them with the hand-coded target speaker’s utterances byte-preserved and the other speakers spliced in from the ASR file.
• Older hand transcript + later supplementary transcription. A legacy CHAT file covers most of the recording; a newer pass transcribes additional content (an investigator’s turns, a parent’s turns, a second target child). Merge with --retain listing the speakers whose content lives in the legacy file.

In both shapes the speakers are the unit of authority, not the files. chatter merge’s job is to express that mapping cleanly.

Conceptual model

A CHAT file describes utterances on a shared media timeline. Two CHAT files of the same media share the same timeline; their utterance sets may overlap (same speech transcribed twice) or be disjoint (each file covers different speakers). The merged output is a single CHAT file on the same timeline whose utterance set is the disjoint union of:

• the utterances of every speaker listed in --retain from the first input file, and
• the utterances of every speaker NOT listed in --retain from the second input file.

Retained-speaker utterances from the first file are kept byte-for-byte identical, including every dependent tier they own (%wor, %mor, %gra, %com, %pho, …). Inserted-speaker utterances from the second file have their downstream-generated dependent tiers (%wor/%mor/%gra/%pho, anything a later pipeline stage will regenerate) stripped before insertion, so the merged file is in a clean state for batchalign3 align and batchalign3 morphotag to own those tiers authoritatively post-merge.

flowchart LR
    File1["File 1<br/>any CHAT file"] --> Merge
    File2["File 2<br/>any CHAT file<br/>(same media)"] --> Merge
    Retain["--retain CHI[,SPK,…]"] -.-> Merge
    Merge["chatter merge<br/>(structural)"] --> Out["Merged CHAT file<br/>retained speakers: byte-stable from File 1<br/>inserted speakers: from File 2,<br/>derived tiers stripped"]

CLI contract

chatter merge <FILE1> <FILE2> --retain <SPEAKER_LIST> [OPTIONS]

ARGUMENTS:
  <FILE1>  Path to the first CHAT file. Speakers listed in --retain are
           taken from here, byte-preserved.
  <FILE2>  Path to the second CHAT file. All other speakers are taken
           from here.

REQUIRED OPTIONS:
  --retain <SPEAKER>[,<SPEAKER>...]
           Comma-separated list of speaker codes (e.g. CHI, or
           CHI,SI2). These speakers' utterances come from <FILE1>;
           everything else comes from <FILE2>.

OPTIONS:
  -o, --output <PATH>
           Write merged output to PATH. Default: stdout.

  --strip-tiers <TIER>[,<TIER>...]
           Dependent tier names to strip from inserted-speaker
           utterances before merging. Default: wor,mor,gra,pho.
           Use empty list (--strip-tiers '') to preserve all
           dependent tiers as-is.

  --allow-bullet-drift
           Permit small backward-time bullets in either input (where
           one utterance's end_ms is slightly greater than the next
           utterance's start_ms). Default behavior: warn but proceed.
           Set this flag to silence the warning.

Exit codes:

What the merged output guarantees

These are testable invariants. Every release verifies them against the reference corpus.

Retained speakers are byte-stable

For every speaker code in --retain, every main-tier line and every dependent-tier line attached to that speaker in <FILE1> appears byte-for-byte identical in the merged output, in the same relative order they appeared in <FILE1>. CHAT markup, NAK-delimited time bullets, paralinguistic annotations, retracing scope, terminator variants, special-form @l/@n/@c suffixes, all preserved.

This is the core semantic guarantee of merge: if you hand-coded disfluency on the target speaker, the disfluency coding survives the merge without any structural change.

Inserted speakers’ downstream-generated tiers are stripped

For every speaker code in <FILE2> that is NOT in --retain, the utterance is included in the merged output with its main tier preserved verbatim BUT with %wor, %mor, %gra, and %pho removed (configurable via --strip-tiers). Other dependent tiers (%com, %spa, %act, %sit, %add, contributor-specific tiers) are preserved.

The rationale: batchalign3 align and batchalign3 morphotag are the authoritative source stages for these tiers in the post-merge pipeline. Carrying inserted-speaker %wor across the merge would leave the merged file in a half-state, some utterances would have %wor, others would not, and downstream behavior on mixed inputs is undefined. The contract is: enter the post-merge stages in a clean state, exit with the tier present and consistent across every utterance.

Utterance order is timeline order

Utterances in the merged output appear in ascending order by their start time bullet (\\x15START_END\\x15, milliseconds). Where two utterances have identical start times, the first-file utterance comes first.

Time bullets are pass-through

chatter merge does NOT recompute, smooth, or refresh time bullets. The bullets in the merged output are exactly those that appeared in the source files. If <FILE2> had %wor rows whose first/last word times implied a slightly different utterance span than the main-tier bullet, the main-tier bullet wins (it was the contract before merge).

If the merge stage detects an inserted-speaker utterance with no main-tier bullet at all (Batchalign occasionally omits these), it lifts a bullet from the corresponding %wor row’s first-word start and last-word end, appending it to the main tier so the merged file has uniform bullet placement. The original %wor is then stripped (per the per-tier rule above).

Header reconciliation

The merged file’s headers are constructed deterministically from the two inputs:

The @Media modality field (audio vs video) is a known divergence point: when ASR runs against an mp4, it may write video on its input but emit audio on its output. File 1’s modality wins, as with all @Media content; no warning is emitted for modality mismatch.

Overlap markup is NOT injected

When an inserted-speaker utterance temporally overlaps a retained-speaker utterance, chatter merge does NOT inject CHAT [>] / [<] / angle-bracket-scoped overlap markers. The time bullets carry overlap information; markers are a CLAN-era surface convention that the output of chatter merge deliberately omits.

The retained speakers’ existing overlap markers (if File 1 already contains some) are preserved byte-stably under the byte-preservation rule above.

Preconditions

chatter merge refuses (exit code 2) if any of these hold:

• File 1 declares no utterances for any speaker in --retain.
• File 1 has no time-bulleted utterances at all (no shared timeline to merge against).
• The two files’ @Languages headers disagree.
• A speaker code appears in both files but not in --retain (use --retain to disambiguate).
• File 2 is missing or unparseable.

chatter merge does NOT refuse on these (proceeds with warning):

• Small backward-time bullets in either input (one utterance ends slightly after the next starts), common in hand transcripts, not corrupting; downstream batchalign3 align cleans these.
• File 2’s @Media modality disagrees with File 1’s (audio vs video).
• File 1 has fewer utterances than File 2, or vice versa.

Speaker identity in File 2 must already be coherent

chatter merge does NOT identify or rename speakers. If File 2 came from ASR and carries anonymous codes like PAR0, PAR1, run chatter speaker-id first to assign CHAT-conformant codes. The merge step trusts whatever speaker codes appear in its inputs.

What chatter merge is NOT

• Not ASR. Use batchalign3 transcribe.
• Not forced alignment. Use batchalign3 align.
• Not morphological tagging. Use batchalign3 morphotag.
• Not speaker identification. Use chatter speaker-id.
• Not content reconciliation. If two files disagree about what a speaker said at the same time, chatter merge does not adjudicate; it trusts --retain to designate one file as authoritative per speaker.
• Not three-way or n-way merge in this release. The 2-input case composes into the n-input case by chaining (chatter merge a b --retain X -o tmp.cha && chatter merge tmp.cha c --retain Y -o out.cha). A future release may add native n-ary merging if a workflow appears for which chained 2-way merges are awkward.

Worked example

A speech-pathology lab hand-transcribed a child’s spontaneous-speech session, marking disfluency carefully, but did not transcribe the clinician’s turns. They send the media and the child-only transcript; the project runs ASR on the media to produce a full-coverage transcript with anonymous speaker codes; then chatter speaker-id labels the ASR file’s adult speaker as INV; then chatter merge combines.

# After ASR labeling: asr.cha has speakers CHI and INV.
chatter merge child-only.cha asr.cha \
  --retain CHI \
  -o merged.cha

# Then alignment regenerates %wor cleanly across all speakers:
batchalign3 align merged.cha

# Then morphotag regenerates %mor and %gra:
batchalign3 morphotag merged.cha

The merged file contains:

• Every *CHI utterance byte-stable from child-only.cha, including every disfluency marker, every retracing scope, every paralinguistic annotation, every %com session-structural comment.
• Every *INV utterance from asr.cha, in their original time order, interleaved with the *CHI utterances by start time.
• One @Participants row listing CHI and INV; @ID rows for both; the union of @Comment rows including any ASR provenance comments from asr.cha.

Relationship to other commands

flowchart TB
    Media[Media file mp4 / wav] --> Transcribe
    Transcribe["batchalign3 transcribe<br/>ASR"] --> AsrAnon["asr-anonymous.cha<br/>PAR0, PAR1, ..."]
    HandTranscript["hand-transcript.cha<br/>target speakers only"] --> SpeakerId
    AsrAnon --> SpeakerId
    SpeakerId["chatter speaker-id<br/>label anon speakers"] --> AsrLabeled["asr-labeled.cha<br/>CHI, INV, MOT, ..."]
    HandTranscript --> Merge
    AsrLabeled --> Merge
    Merge["chatter merge<br/>(this page)"] --> Merged[merged.cha]
    Merged --> Align
    Align["batchalign3 align"] --> Aligned[aligned.cha]
    Aligned --> Morph
    Morph["batchalign3 morphotag"] --> Final[final.cha]

chatter merge sits between speaker-identity resolution and forced alignment. It assumes its inputs have coherent CHAT-conformant speaker codes (no anonymous PAR0/PAR1) and emits a file ready for batchalign3 align to refresh timing and produce %wor.

Inputs must be valid CHAT (pipeline / batch)

The per-session chatter pipeline shortcut and the directory-level chatter batch driver validate every input as CHAT before doing any speaker-id or merge work. Each donor and the reference it is merged against must pass the same validation chatter validate runs; an input that fails is never merged. Clean invalid transcripts to valid CHAT first (run chatter validate <file> to see the errors), then re-run.

• chatter pipeline refuses (exit 2, no output written) if its donor or reference is invalid CHAT.
• chatter batch is fail-closed and whole-batch: if any input under the donor/reference directories is invalid CHAT, it reports every offending file and aborts the entire run without merging a single session. “All inputs are chatter-valid” is a hard precondition of the batch, not something discovered session-by-session mid-run.

This gate catches validation-only invalidity (files that parse but fail chatter validate, e.g. a malformed @ID), which the lower-level chatter merge parse is otherwise lenient about.

LLM holistic judgment (pending-only)

--judgment holistic is now reachable from pipeline and batch (not just speaker-id). In holistic mode the command is pending-only: it writes an engine = "llm" review-gated entry via --write-pending and produces no merged file. The operator supplies the LLM connection with --llm-endpoint / --llm-model (or the environment variables CHATTER_LLM_ENDPOINT / CHATTER_LLM_MODEL); an optional --session-context <file.json> provides per-session context that the LLM prompt includes to sharpen its judgment.

Session-context JSON (--session-context)

The session-context file is a corpus-agnostic JSON object mapping session IDs (the donor file’s basename stem) to context records. Every record field is optional, and the label fields are free vocabulary: chatter imposes no closed set, the labels are surfaced verbatim into the LLM prompt.

{
  "SESSION-ID": {
    "sample_type": "clinician interview",
    "declared_roles": ["Investigator"],
    "consent_tier": "video+audio",
    "age_months": 52
  }
}

• sample_type: what kind of speech sample the session is (e.g. "narrative retell").
• declared_roles: adult roles declared present in the session.
• consent_tier: media-consent tier governing what may be shared.
• age_months: child age in months at the session.

When --session-context is absent, the CHATTER_SESSION_CONTEXT environment variable supplies the path (empty counts as unset). Per session, each context field resolves in order: the explicit record from the file; for the age only, the donor’s CHAT @ID age header (pure CHAT, no external metadata needed); otherwise unknown. Absent sessions or fields are passed to the judgment as unknown, never guessed. A configured-but-malformed file is a hard error, and labels must contain at least one non-whitespace character. Configuring session context on a non-holistic run prints a warning (the deterministic judgment never consults it).

Conversion from a contributor’s own records format (a spreadsheet, a database export) to this JSON happens outside chatter.

The two-pass operator flow is:

1. batch --judgment holistic --session-context context.json --write-pending P accumulates one engine = "llm" pending entry per session in P.
2. Operator reviews P, accepts or corrects each entry.
3. chatter adjudicate promotes reviewed entries to the override file.
4. batch (deterministic, reading the override file) replays every confirmed mapping and writes the merged files.

Note: the MLU sanity-scan is unreliable for the FluencyBank clinical-interview corpus (children out-narrate the adult, so MLU ratios invert relative to typical child-language recordings). Holistic-pending review via --judgment holistic is the trustworthy alternative there.

Implementation notes (for contributors)

• Source: crates/talkbank-transform/src/transcript_merge/ (proposed layout, see the design plan).
• CLI surface: crates/chatter/src/commands/transcript_merge/.
• Domain types (SpeakerCode, RetainSet, MergeOverride, SpeakerMapping) live in talkbank-model so the override-file format is sharable across the speaker-id stage, the orchestrator, and any future adjudication UI.
• The merge operates on talkbank-model::ChatFile; both inputs are parsed via talkbank-parser. The byte-preservation guarantee on retained-speaker utterances relies on the parser’s existing round-trip serialization.
• Spec entries exercising the merge live in spec/constructs/, every behavioral invariant on this page has a spec; tests are regenerated via the current spec/tools workflow documented in Spec Workflow.
• This page is the user contract; book/src/chatter/reference/ carries the override-file reference for the speaker-id stage that this merge consumes.

The Merge Workflow (pipeline, batch, adjudicate, sanity-scan)

Status: Draft (experimental) Last modified: 2026-06-15 10:39 EDT

The merge workflow combines, at scale, the two structural primitives documented elsewhere, chatter speaker-id (assign CHAT-conformant speaker codes to an anonymous donor) and chatter merge (combine two transcripts of the same recording), and adds the operator loop needed when the automatic speaker decision is not confident enough to trust.

Four commands make up the workflow. They are experimental and in active development; flags and behavior may change.

If you only have one pair of files and one clean answer, reach for pipeline. Everything else here is about doing that safely across a directory of sessions where some answers are not clean.

The big picture: a two-pass loop

The hard part of merging at scale is not the merge; it is deciding, per session, which anonymous ASR speaker is the child the reference already covers. speaker-id’s multiset-Jaccard match (see its page) answers that automatically when the winner clearly beats the runner-up, and refuses (exit code 4) when it does not. The workflow turns that refusal into a reviewable queue.

flowchart TD
    subgraph Pass1["Pass 1: automatic"]
        B1["chatter batch DONOR_DIR REF_DIR\n--write-override audit.toml\n--write-pending pending.toml"]
        B1 --> Clean["confident sessions:\nmerged file written,\ndecision logged to audit.toml"]
        B1 --> Refused["low-confidence sessions:\nNO merge, appended to pending.toml\n(exit code 4)"]
    end
    Refused --> Adj["chatter adjudicate pending.toml\n--override-file audit.toml\n(operator decides)"]
    Adj --> Pass2["Pass 2: chatter batch ... --override-file audit.toml\n(replays the operator's decisions,\nmerges the previously-refused sessions)"]
    Clean --> Done["all sessions merged"]
    Pass2 --> Done

Pass 1 merges everything it is confident about and parks the rest. The operator works the parked queue once. Pass 2 replays their decisions. The same chatter batch (or chatter pipeline) command runs both passes; what changes is whether an override file with entries exists yet.

chatter pipeline (one session)

The per-session shortcut: run speaker-id in reference mode to relabel an anonymous donor, then merge the relabeled donor with the reference, in one command instead of two.

chatter pipeline <DONOR> <REFERENCE> \
  --anchor <SPEAKER> --inserted-role <CODE>:<ROLE> --output <PATH> [OPTIONS]

ARGUMENTS:
  <DONOR>      Donor CHAT file with anonymous speaker codes (the ASR output).
  <REFERENCE>  Reference CHAT file carrying the authoritative anchor speaker
               (typically the hand-coded child transcript).

REQUIRED:
  --anchor <SPEAKER>            Anchor code in the reference (typically CHI).
  --inserted-role <CODE>:<ROLE> Role for the donor's non-anchor speakers
                                (e.g. INV:Investigator).
  -o, --output <PATH>           Output path for the merged CHAT file.

KEY OPTIONS:
  --retain <SPEAKER>            Speaker(s) taken from the reference in the
                               final merge (typically the same as --anchor).
  --confidence-threshold <F>    Minimum winner/runner-up Jaccard margin to
                               auto-decide (default 2.0x).
  --write-override <FILE>       On a confident auto-decision, append a
                               mode = "auto" audit entry for this session.
  --write-pending <FILE>        On a low-confidence refusal, append a pending
                               entry (exit code 4 still fires).
  --override-file <FILE>        If the file has an entry for this session
                               (the donor's basename stem), replay that
                               decision instead of running reference mode.

The same command serves pass 1 (no override entry yet, run reference mode) and pass 2 (entry present, replay it). Validation is a hard precondition: a donor or reference that fails chatter validate is never merged (exit 2, nothing written).

chatter batch (a directory pair)

Loops pipeline over matched files: the reference for DONOR_DIR/X.cha is REFERENCE_DIR/X.cha. Donors without a matching reference are warned and skipped. It is fail-closed and whole-batch on validity: if any input under either directory is invalid CHAT, the batch reports every offending file and aborts without merging a single session.

chatter batch <DONOR_DIR> <REFERENCE_DIR> \
  --anchor <SPEAKER> --inserted-role <CODE>:<ROLE> --output <DIR> [OPTIONS]

PASS-1 AUDIT + QUEUE:
  --write-override <FILE>  Append every confident auto-decision (mode =
                          "auto"). Required if you want --sanity-scan.
  --write-pending <FILE>   Aggregate every low-confidence refusal into one
                          pending file. One `chatter adjudicate` run resolves
                          them all. Refusals do NOT abort the batch.

PASS-2 REPLAY:
  --override-file <FILE>   Threaded to every per-session pipeline call.
                          Sessions with an entry replay it; the rest fall
                          through to reference mode.

POST-MERGE QA:
  --sanity-scan            Run `sanity-scan` after the loop. Requires
                          --write-override (it reads the auto-decisions) and
                          --write-pending (flagged sessions are appended).
                          Exit code 4 fires if it flags any session.
  --sanity-scan-threshold <F>  Heuristic ratio (default 1.5).

OPERATIONAL:
  --skip-existing          Skip donors whose merged output already exists, to
                          resume an interrupted batch.

batch also accepts the same --judgment deterministic|holistic and LLM / --session-context options as pipeline; see Merge, LLM holistic judgment for that mode and the session-context JSON format.

chatter adjudicate (the operator step)

Reads the pending file a pass produced, walks the operator through the unresolved sessions, and appends the resolved decisions to the override file. On success the pending file is rewritten to drop the entries that were resolved, so re-running adjudicate only ever shows what is left.

chatter adjudicate <PENDING> --override-file <FILE> [--interactive | --scripted <TOML>]

ARGUMENTS:
  <PENDING>  The pending-adjudications TOML a pass wrote.

REQUIRED:
  --override-file <FILE>  Override file to append resolved decisions to
                         (created if absent). This is the same file pass 2
                         reads back.

DECISION SOURCE (one of):
  --interactive           Prompt per pending entry on stdin. Currently
                         supports `accept` / `a` (accept the suggested
                         mapping).
  --scripted <TOML>       Pre-canned operator decisions, for replayable /
                         tested runs. Mutually exclusive with --interactive.

  --operator <NAME>       Recorded in each override entry (defaults to $USER).

This is the interactive review tool the speaker-id and merge pages refer to: the audit trail (who decided, the scores, any note) lands in the override file so a later reader can see why a session was labeled the way it was. The decision schema is the same override-file format used everywhere in the workflow; see Merge Override File Format, and the Adjudication Workflow architecture page for the design.

chatter sanity-scan (post-merge QA)

A confident auto-decision can still be wrong, the runner-up was simply even further off. sanity-scan re-reads the merged output and the pass-1 audit file and flags sessions that pass an out-of-band check: the mean utterance word count of the anchor speaker versus the inserted speaker. In a typical child-language recording the adult out-talks the child, so an anchor (child) mean that is much higher than the inserted (adult) mean is suspicious, possibly the two were swapped.

chatter sanity-scan <MERGED_DIR> \
  --override-file <FILE> --anchor <SPEAKER> --write-pending <FILE> [OPTIONS]

REQUIRED:
  --override-file <FILE>  The pass-1 audit file. Only auto-decided sessions
                         are scanned; explicit-mode entries are skipped (the
                         operator already signed off).
  --anchor <SPEAKER>      Anchor code in the merged files (typically CHI).
  --write-pending <FILE>  Flagged sessions are appended here as
                         sanity-scan-misclassification pending entries for
                         `chatter adjudicate`. Required.

  --threshold <F>         Flag when anchor_mean >= inserted_mean * threshold
                         (default 1.5).

A flag is a question, not a verdict: the session goes back into the adjudication queue for an operator to confirm or correct. Whether to run the scan at all is a judgment about the corpus. It assumes the typical “adult out-talks child” shape, and is unreliable where that inverts (e.g. a clinical-interview corpus where children out-narrate the adult); there, prefer the LLM holistic-pending review described on the merge page.

End-to-end worked example

A directory of ASR donors (asr/) and the matching hand-coded child references (ref/), child anchor CHI, adults labeled INV:

# Pass 1: merge what we are sure of; queue the rest; keep an audit trail.
chatter batch asr/ ref/ \
  --anchor CHI --inserted-role INV:Investigator \
  --output merged/ \
  --write-override audit.toml \
  --write-pending pending.toml \
  --sanity-scan

# Exit 0: every session merged confidently and the scan was clean.
# Exit 4: some sessions are pending (low-confidence and/or scan-flagged).

# Operator resolves the queue once (audit trail recorded):
chatter adjudicate pending.toml --override-file audit.toml --interactive --operator alice

# Pass 2: replay the operator's decisions; the previously-pending
# sessions now merge.
chatter batch asr/ ref/ \
  --anchor CHI --inserted-role INV:Investigator \
  --output merged/ \
  --override-file audit.toml \
  --skip-existing

Exit codes

The workflow commands share the convention used across the merge surface:

Exit code 4 is the normal “there is operator work to do” signal, not an error: a batch that parks ten sessions still merged the rest.

See also

• Speaker-ID, the speaker-relabeling primitive and the Jaccard matching algorithm pass 1 uses.
• Merge, the structural merge primitive and the LLM holistic-judgment mode.
• Merge Override File Format, the shared decision schema.
• Adjudication Workflow and Merge Pipeline, Crate Architecture, the developer-facing design.

CHAT Format Overview

Status: Reference Last updated: 2026-05-11 21:51 EDT

CHAT (Codes for the Human Analysis of Transcripts) is a standardized transcription format for spoken language data, developed by MacWhinney as part of the CHILDES and TalkBank projects. It is the most widely used format in child language research and conversational analysis.

File Anatomy

Every CHAT file follows this structure:

@UTF8
@Begin
@Languages:	eng
@Participants:	CHI Target_Child, MOT Mother
@ID:	eng|corpus|CHI|2;6.||||Target_Child|||
@ID:	eng|corpus|MOT|||||Mother|||
*MOT:	what do you want ?
%mor:	ADV|what AUX|do PRON|you VERB|want ?
%gra:	1|4|LINK 2|4|AUX 3|4|SUBJ 4|0|ROOT 5|4|PUNCT
*CHI:	I want cookie .
%mor:	PRON|I VERB|want NOUN|cookie .
%gra:	1|2|SUBJ 2|0|ROOT 3|2|OBJ 4|2|PUNCT
@End

A CHAT file consists of:

1. @UTF8: required first line, declares UTF-8 encoding
2. @Begin: marks the start of the transcript
3. Headers: lines starting with @ that provide metadata (participants, languages, IDs, etc.)
4. Utterances: blocks consisting of:
   • A main tier (line starting with *SPEAKER:) containing the transcribed speech
   • Zero or more dependent tiers (lines starting with %tier:) containing annotations
5. @End: marks the end of the transcript

Key Conventions

• Tab separation: a tab character separates the tier prefix from its content (e.g., *CHI:⟶content)
• Terminators: every utterance ends with a terminator (., ?, !, or special forms like +...)
• Line continuation: long lines wrap with a tab at the start of continuation lines
• Speaker codes: short identifiers; the validator accepts up to seven characters from A-Z, 0-9, _, -, '; three uppercase letters is the convention (e.g., CHI, MOT, FAT, INV)
• Media linking: timestamps link transcripts to audio/video via bullet markers

CHAT vs Other Formats

References

• CHAT Manual: the canonical reference
• TalkBank: the data repository

Headers

Status: Reference Last updated: 2026-05-11 20:30 EDT

Headers are lines beginning with @ that provide metadata about the transcript. They appear between @Begin and the first utterance (though some headers like @Comment can appear anywhere).

Required Headers

@UTF8

Must be the very first line of every CHAT file. Declares UTF-8 encoding.

@UTF8

@Begin / @End

Mark the start and end of the transcript body. Every CHAT file must have exactly one @Begin and one @End.

@Participants

Declares all speakers in the transcript. Format: CODE [Name] Role, comma-separated. The role is required; the name is optional, so each entry is either CODE Role or CODE Name Role.

@Participants:	CHI Target_Child, MOT Mother, FAT Father
@Participants:	CHI Alex Target_Child, MOT Mary Mother

In the first line, Target_Child, Mother, and Father are roles, not names. In the second line, Alex and Mary are optional names sitting between the speaker code and the role.

Speaker codes are short identifiers; the validator accepts up to seven characters from A-Z, 0-9, _, -, and '. The convention is three uppercase letters; the most common codes are:

• CHI: target child
• MOT: mother
• FAT: father
• INV: investigator
• OBS: observer

@ID

Provides detailed metadata for each participant. One @ID line per participant.

@ID:	eng|corpus|CHI|2;6.||||Target_Child|||

Fields (pipe-separated): language, corpus, speaker code, age, sex, group, SES, participant role, education, custom field.

Age format: years;months.days (e.g., 2;6. = 2 years, 6 months).

SES field: ethnicity (White, Black, Asian, Latino, Pacific, Native, Multiple, Unknown), socioeconomic code (UC, MC, WC, LI), or combined with comma separator (e.g., White,MC).

Optional Headers

@Languages

Declares the language(s) used in the transcript.

@Languages:	eng, fra

@Date

Recording date in DD-MON-YYYY format.

@Date:	15-JAN-2024

@Location

Where the recording took place.

@Location:	Boston, MA, USA

@Situation

Description of the recording context.

@Situation:	free play with toys in lab

@Activities

Activities during the recording.

@Activities:	toyplay, reading

@Comment

Free-form comments. Can appear anywhere in the file (before, between, or after utterances).

@Comment:	child was tired during this session

@Media

Links the transcript to an audio or video file.

@Media:	session01, audio

@Transcriber / @Coder

Identifies who created or coded the transcript.

@Transcriber:	JDS
@Coder:	ABC

Header Ordering

Headers should follow this conventional order:

1. @UTF8 (required, first line)
2. @Begin (required)
3. @Languages
4. @Participants (required)
5. @ID lines (one per participant)
6. Other metadata headers (@Date, @Location, etc.)
7. @Comment lines (can also appear later)

Validation

The parser validates header structure including:

• @UTF8 must be the first non-empty line
• @Begin and @End are required and must appear exactly once
• @Participants is required and must declare all speakers used in utterances
• @ID participant codes must match @Participants declarations
• Age format validation in @ID lines

Utterances

Status: Reference Last updated: 2026-05-11 23:22 EDT

An utterance is the fundamental unit of a CHAT transcript. It consists of a main tier (the transcribed speech) followed by zero or more dependent tiers (annotations).

Main Tier

The main tier begins with *SPEAKER: followed by a tab and the utterance content, ending with a terminator.

*CHI:	I want a cookie .

Speaker Codes

Speaker codes are short identifiers (up to seven characters from A-Z, 0-9, _, -, '; three uppercase letters is the convention) matching a code declared in @Participants:

@Participants:	CHI Target_Child, MOT Mother
*MOT:	what do you want ?
*CHI:	cookie .

Terminators

Every utterance must end with a terminator:

Line Continuation

Long utterances wrap to the next line with a leading tab:

*MOT:	well I think that we should probably go to
	the store and get some more cookies .

Content Items

The content between *SPEAKER: and the terminator consists of content items separated by whitespace:

• Words: regular words, potentially with annotations
• Groups: bracketed content like <word word> for overlap, retrace, etc.
• Special forms: pauses (.), events &=laughs, fillers &-uh
• Separators: commas , and other punctuation

Words

Words are the primary content unit. See Word Syntax for full details.

Groups

Angle brackets < > group words for annotations:

*CHI:	<I want> [/] I want cookie .

Common group annotations:

• [/]: partial retrace (speaker repeats the same words)
• [//]: full retrace (speaker restarts with different words)
• [///]: multiple retracing (multiple false starts)
• [/-]: reformulation (speaker rephrases with different structure)
• [?]: uncertain transcription

Special Forms

*CHI:	um (.) I want &-uh cookie .

• (.): short pause
• (..): medium pause
• (...): long pause
• (1.5): timed pause in seconds
• &=laughs: paralinguistic event
• &-uh: filler

Media Linking

Utterances can include media timestamps (bullets) that link to audio/video:

*CHI:	I want cookies . •1234_5678•

The numbers represent start and end times in milliseconds. The bullets delimiting the pair render as • in most editors; on disk they are the NAK control character (U+0015). See grammar/grammar.js rule bullet.

Dependent Tiers

See Dependent Tiers for documentation on %mor, %gra, %pho, %wor, and other annotation tiers that follow the main tier.

Retraces and Repetitions

Status: Current Last updated: 2026-05-11 23:16 EDT

Retraces mark content that the speaker said but then corrected, repeated, or abandoned. They are one of the most consequential constructs in CHAT because they affect how every dependent tier aligns to the main tier.

CHAT Syntax

A retrace has two parts: the retraced content (what the speaker said first) and the correction (what follows). The retraced content is marked with a trailing bracket code:

Single-Word Retraces

When only one word is retraced, no angle brackets are needed:

*CHI: I [/] I want that .
*CHI: ana [//] an .
*MOT: the book [/-] the magazine is here .

Group Retraces

When multiple words are retraced, angle brackets delimit the scope:

*MOT: <the dog> [//] the cat ran .
*CHI: <I want> [/] I need cookie .
*CHI: <I want the> [///] give me that .

Retraces with Replacements

A retraced word often has a replacement [: target] and/or error code [* code]. This is common in aphasia and child language corpora where the speaker produces an incorrect form:

*PAR: tika@u [: kitty] [* p:n] [//] kitty is nice .
%mor: noun|kitty aux|be-Fin-Ind-Pres-S3 adj|nice-S1 .

*PAR: lɛɾɪ@u [: later] [* p:n] [//] later in the day .
%mor: adv|late adp|in det|the-Def-Art noun|day .

*CHI: male [: female] [* s:r] [/] male [: female] [* s:r] .
%mor: adj|female-S1 .

In each case, the retraced word (before the [//] or [/]) is excluded from %mor alignment. Only the correction (after the marker) is counted.

Data Model

Retraces are a first-class variant of UtteranceContent:

flowchart TD
    UC["UtteranceContent"]
    UC --> Word
    UC --> RW["ReplacedWord"]
    UC --> Retrace
    UC --> AG["AnnotatedGroup"]
    UC --> Other["...20 other variants"]

    Retrace --> BC["BracketedContent"]
    Retrace --> RK["RetraceKind"]
    BC --> BIW["BracketedItem::Word"]
    BC --> BIRW["BracketedItem::ReplacedWord"]

    style Retrace fill:#f96,stroke:#333

The Retrace struct wraps the retraced content in a BracketedContent container, which can hold any combination of words, replaced words, and other content items:

// crates/talkbank-model/src/model/content/retrace.rs
pub struct Retrace {
    pub content: BracketedContent,          // the retraced words
    pub kind: RetraceKind,                  // Partial, Full, Multiple, Reformulation
    pub is_group: bool,                     // <word> [/] vs word [/]
    pub annotations: Vec<ContentAnnotation>,// non-retrace annotations after marker
    pub span: Span,
}

Why First-Class?

Before the retrace refactor, retraces were represented as annotations on words or groups. This meant every match on content had to inspect annotation lists to determine whether a word was retraced. This led to a class of bugs where retraced content was accidentally included in alignment counting, word extraction, or retokenization.

Making Retrace a top-level UtteranceContent variant means:

1. The compiler enforces handling. Every match on UtteranceContent must have a Retrace arm. Forgetting to handle retraces is a compile error, not a silent runtime bug.
2. Domain-aware gating is centralized. The content walker checks the Retrace variant once, not at every annotation-inspection site.
3. Alignment counting is simple. The count function returns 0 for Retrace in Mor domain, no annotation inspection needed.

Parser Conversion

The tree-sitter grammar parses retrace markers ([/], [//], etc.) as annotations on word_with_optional_annotations. The Rust parser converts them to structural Retrace nodes in parse_word_content():

flowchart LR
    subgraph "Tree-sitter CST"
        WOA["word_with_optional_annotations"]
        SW["standalone_word"]
        BA["base_annotations"]
        RP["retrace_partial / retrace_complete / ..."]
        WOA --> SW
        WOA --> BA
        BA --> RP
    end

    subgraph "Rust Model"
        RET["UtteranceContent::Retrace"]
        BC2["BracketedContent"]
        W2["Word or ReplacedWord"]
        RET --> BC2
        BC2 --> W2
    end

    WOA -->|"parse_word_content()\n(word.rs)"| RET

Three cases in parse_word_content():

1. Word + retrace (I [/]), wrap Word in BracketedItem::Word inside Retrace
2. Word + replacement + retrace (tika@u [: kitty] [* p:n] [//]), build ReplacedWord, then wrap in BracketedItem::ReplacedWord inside Retrace
3. Word + replacement, no retrace (tika@u [: kitty]), emit bare ReplacedWord

Group retraces (<content> [/]) are handled in group/parser.rs via the same structural wrapping.

Alignment Behavior

Retraces interact differently with each dependent tier domain:

flowchart TD
    RT["Retrace node\n(e.g. 'tika@u [: kitty] [* p:n] [//]')"]

    RT -->|"Mor domain"| SKIP["SKIP\n(return 0)\nNot morphologically analyzed"]
    RT -->|"Pho domain"| COUNT["COUNT\nPhonologically produced"]
    RT -->|"Sin domain"| COUNT2["COUNT\nGesturally produced"]
    RT -->|"Wor domain"| COUNT3["RECURSE\napply retrace-aware %wor leaf rule"]

    style SKIP fill:#faa,stroke:#333
    style COUNT fill:#afa,stroke:#333
    style COUNT2 fill:#afa,stroke:#333
    style COUNT3 fill:#afa,stroke:#333

Why %mor skips retraces: The %mor tier represents the morphological analysis of what the speaker meant to say. Retraced content is a false start or error; it was produced phonologically but is not part of the intended linguistic structure. The correction after the retrace marker carries the morphological analysis.

Why %pho/%sin/%wor include retraces: These tiers document what was actually produced, the sounds, gestures, and timing of the speech as it happened, including false starts. The retrace was physically spoken, so it appears in these tiers.

For %wor, retrace ancestry does not change leaf-level membership:

• spoken word tokens count both inside and outside retrace
• that includes fillers, fragments, nonwords, and untranscribed placeholders
• overlap annotations do not affect %wor membership

Exact corpus-shaped contrast:

*CHI:	<one &+ss> [/] one play ground .
%wor:	one •321008_321148• ss •321148_321368• one •321809_321969• play •322049_322310• ground •322390_322890• .

*CHI:	&+ih <the what> [/] what's letter &+th is this ?
%wor:	ih •49063_49103• the •49103_49163• what •49183_50205• what's •50205_50405• letter •50405_50685• th •50886_50946• is •50946_51046• this •51086_51586• ?

Implementation

Counting: count_alignable_item() in alignment/helpers/count.rs:

UtteranceContent::Retrace(retrace) => {
    if domain == TierDomain::Mor {
        0  // excluded from morphological alignment
    } else {
        count_bracketed_alignable_content(&retrace.content, domain, true)
    }
}

Walking: walk_words() in alignment/helpers/walk/mod.rs:

UtteranceContent::Retrace(retrace) => {
    if !matches!(domain, Some(TierDomain::Mor)) {
        walk_bracketed_content(&retrace.content.content, domain, f);
    }
}

%wor generation and overlap counting still use dedicated recursive helpers, but now for %wor-specific sequencing details like replacement handling rather than for retrace-sensitive membership.

Validation

Cross-Utterance Retrace Validation

The retrace validators in validation/retrace/ check:

• Collection: collection/utterance.rs and collection/bracketed.rs walk the content tree to find all Retrace nodes
• Detection: detection.rs provides utterance_item_has_retrace() for quick retrace presence checks

Alignment Validation (E705)

E705 fires when the main tier has more alignable items than %mor. If retraces are correctly parsed as Retrace nodes; they are excluded from the count and E705 does not fire. If a retrace is accidentally parsed as a bare ReplacedWord (the bug fixed in c90b9bf), it is counted and triggers a false E705.

Regression Tests

tests/retrace_replaced_word_regression.rs contains 6 targeted tests:

Reference corpus entries: corpus/reference/annotation/retrace.cha

See Also

• Alignment Architecture: full alignment system docs
• The %mor Tier: morphological tier format and alignment rules
• CHAT Manual: Retracing

Replacements

Status: Current Last modified: 2026-05-29 17:47 EDT

A replacement is a CHAT annotation [: ...] that pairs a single spoken word on the main tier with one or more “intended” words. It records both what the speaker actually said and what the analysis should treat the utterance as containing.

*CHI:	wanna [: want to] go .
*CHI:	dis [: this] is fun .
*CHI:	rocking+house [: rocking+horse] [*] ?

This page is the canonical reference for what replacements mean in TalkBank, both as a CHAT-manual construct and as a typed AST in this repo. The most important load-bearing fact, which the rest of the page expands on:

Replacements are word-level, not group-level. Each tier domain chooses one side of the pair: %mor analyzes the replacement (right side); %wor, %pho, %sin align to the original (left side). %gra follows %mor.

CHAT Syntax

Word-Level Scope

A replacement attaches to a single standalone_word on the main tier and contains one or more replacement words inside the brackets:

*CHI:	gonna [: going to] eat lunch .
*CHI:	dis [: this] toy .
*CHI:	rocking+house [: rocking+horse] [*] ?

The grammar rules are word_with_optional_annotations and replacement in grammar/grammar.js grep for the rule names rather than line numbers so this stays accurate as the grammar evolves. Replacement words can be separated by whitespace, so [: going to] is a single replacement of gonna with two words.

There Is No Group-Level Replacement

<dat is> [: that is] is not valid CHAT. A replacement does not attach to a group; it attaches to a single word. The grammar enforces this by typing: ReplacedWord.word: Word, never Group. To replace words inside a group, attach the replacement to the inner word:

*CHI:	<dat [: that] is> [/] is broken .

This shape, replacement inside a group inside a retrace, is legal because each annotation operates at its own scope.

There Is No [::] Form

Some literature on CHILDES tooling references a [::] annotation; it does not exist in this repo’s grammar, parser, or model, and is not defined by the current CHAT manual. Only [:] exists. If you encounter [::] in legacy data, treat it as a parse error to investigate, not a construct to support.

The Per-Domain Alignment Rule

This is the rule contributors most often get wrong. Different tier domains align to different sides of a replacement pair:

The mnemonic: the replacement encodes the intended form (what the speaker meant or what a corrected transcript would read). Tiers analyzing intent (%mor/%gra) use the replacement; tiers documenting realization (%wor/%pho/%sin) use the original.

flowchart LR
    spoken["Original word\n(left of [:)\n'dis'"]
    target["Replacement words\n(inside [: ])\n'this'"]

    spoken -->|"%wor (timing)"| wor["%wor: dis"]
    spoken -->|"%pho (phonology)"| pho["%pho: dɪs"]
    spoken -->|"%sin (spelling)"| sin["%sin: dis"]
    target -->|"%mor (UD parse)"| mor["%mor: pron|this"]
    target -->|"%gra (paired with %mor)"| gra["%gra: 1|0|ROOT"]

For multi-word replacements like gonna [: going to], the rule generalizes consistently:

• %wor / %pho / %sin produce one entry, for gonna.
• %mor produces two entries, for going and to.
• %gra produces two entries, paired to the two %mor items.

The alignment-counting code that enforces this is in alignment/units.rs look for the UtteranceContent::ReplacedWord arm. The full table of per-domain rules is in spec/docs/ALIGNMENT_RULES.md.

Rust AST

A replacement is modeled as a first-class UtteranceContent variant, not as a flag on Word:

// crates/talkbank-model/src/model/annotation/replacement.rs
pub struct ReplacedWord {
    pub word: Word,                       // left side: original spoken word
    pub replacement: Replacement,         // right side: 1+ intended words
    pub scoped_annotations: ReplacedWordAnnotations,
}

Two consequences of this shape:

1. A replacement is a wrapper around a Word, not a kind of Word. ReplacedWord lives as its own variant of UtteranceContent (and BracketedItem), holding an inner word: Word plus the replacement payload. Contrast with retraces: Retrace is also a variant of UtteranceContent/BracketedItem, but it wraps a group of content (a single word or a <...> group), not a single Word. Different mechanism, different scope, same top-level slot in the AST.
2. The walk_words() content walker yields WordItem::ReplacedWord as a distinct leaf (defined in crates/talkbank-model/src/alignment/helpers/walk/mod.rs). Domain-aware extraction code branches on this leaf type and chooses original or replacement per the table above.

Validation

Each Replacement Word Is Validated Like a Main-Tier Word

The replacement is a Vec<Word>. Each Word inside it goes through the same validator that runs on main-tier words:

*CHI:	dog [: C-3PO] .

This produces [E220] "C-3PO" is not a legal word in language(s) "eng": numeric digits not allowed, exactly as if C-3PO had appeared on the main tier directly. The replacement does not provide an escape from word-level validation. The implementation is in replacement.rs.

This is critical for any code generating replacements programmatically: do not assume [: ...] lets you smuggle arbitrary text past the word validator. If your producer emits a replacement, both sides must be CHAT-legal under the utterance’s declared language.

Replacement-Specific Error Codes

Three error codes are specific to replacements and do not apply to main-tier words:

The principle: a replacement must be a concrete intended form. Empty, omitted, or unintelligible content defeats that purpose.

Interactions with Other Annotations

Replacements and Retraces Are Orthogonal

A retrace ([/], [//], [///], [/-]) and a replacement ([:]) are distinct annotations operating at different structural levels:

• Retraces wrap content (a single word or a group). They are first- class UtteranceContent variants and represent post-hoc speaker correction.
• Replacements attach inside a Word slot via ReplacedWord. They are editorial metadata about an individual spoken word.

Both can coexist:

*CHI:	<dat [: that] is> [/] is broken .   (replacement inside retrace)

A retrace cannot live inside a replacement (the grammar wraps replacements around standalone_word, not arbitrary content).

Replacements and Error Coding

Error codes follow the replacement and operate on the replaced word as a unit:

*CHI:	rocking+house [: rocking+horse] [*] ?

Here [*] marks rocking+house as containing a phonological/lexical error; the [: rocking+horse] records the intended form. The two annotations cooperate: the replacement encodes what was meant, the error code classifies how it deviates. Implementation: scoped_annotations field on ReplacedWord.

Common Misconceptions

These are bugs we have repeatedly written down then forgotten, recording them here so future contributors don’t reinvent them.

1. “[: ...] lets me put any text I want.” No. Each replacement word is validated. [: C-3PO] fails E220 in English just as C-3PO would.
2. “[:] is the right mechanism for ASR sanitization.” Usually no. ASR-introduced normalization typically wants [% ...] (free- form comment) or [= ...] (free-form explanation), neither of which validates word grammar. Use [:] only when you have a concrete CHAT-legal intended form.
3. “%mor analyzes the original.” No. %mor analyzes the replacement. This is the correction’s morphology, not the error’s.
4. “%wor count must equal %mor count.” No. For gonna [: going to], %wor has 1 entry and %mor has 2. They align to different sides. The validator’s per-domain rule respects this.
5. “<a b> [: c d] is a group-level replacement.” No. Group-level replacements don’t exist. Either replace inside (<a [: c] b [: d]>) or rephrase the transcription.

Source Citations

See Also

• Retraces and Repetitions: the orthogonal post-hoc correction mechanism.
• The %mor Tier: UD-syntax morphosyntactic analysis that aligns to the replacement form.
• Word Syntax: the word grammar that replacement words must satisfy.
• Dependent Tiers: overview of %mor, %wor, etc., with their alignment relationships.

Untranscribed Markers: xxx, yyy, www

Status: Reference Last updated: 2026-06-14 19:57 EDT

CHAT reserves three short word-level markers for material the human transcriber cannot or chose not to render as words on the main tier. Each one has a specific meaning. Tools that emit CHAT, including ASR pipelines, format converters, and editor heuristics, must respect those meanings, because every downstream consumer (researchers, validators, and aggregate-statistics tools like CLAN’s freq, kideval, mlu) reads them at face value.

The shared property: each marker is the human transcriber telling later readers something specific about their experience listening to the audio. None of them mean “tooling could not process this token”.

Why this matters

When a researcher loads a CHAT corpus and counts xxx occurrences, the result is a measure of human listening difficulty: it tells them how much of the audio resisted human transcription. That number feeds into methodology decisions (“can we get reliable MLU from this corpus?”, “what’s the noise floor on this child’s speech?”, “should we re-record in a quieter environment next time?”). It is a load-bearing signal in language-development research.

If an ASR pipeline emits xxx whenever it can’t sanitize a token, for example, substituting xxx for any word that fails CHAT validation under a strict language profile, every xxx count in the corpus becomes a meaningless mixture of “human couldn’t tell” and “pipeline gave up”. Researchers then reading those counts are silently misled. The signal is destroyed for the entire history of that corpus, because the corruption is indistinguishable from real unintelligibility once committed.

The same reasoning applies to yyy and www. A converter or post-processor that emits any of these three markers because the tooling couldn’t handle a token is committing semantic vandalism against the whole field.

Rules for tooling

1. Never emit xxx, yyy, or www from a tool to mean “could not process”. These markers are reserved for human transcriber judgment.
2. When a token cannot be validated as legal CHAT under the declared language, prefer one of:
   • Pass the token through verbatim and let the CHAT validator (or CLAN’s check) flag it for human review. The transcriber listens, decides, and corrects.
   • Fail loud, abort the file rather than emit corrupted output.
   • Apply only purely orthographic, semantically null repairs (e.g., stripping a stray boundary quote mark from "My). These are safe because no information is lost.
3. Never sanitize a token by replacing it with one of the three markers. That is exactly the corrupting behavior this document prohibits.
4. Never delete a token to “fix” a validation failure. Deletion loses data without any flag.

What tools synthesizing CHAT should do instead

Any tool that builds CHAT from an external source (ASR output, an importer, a format converter) should follow the same division of labor:

1. Silently fix only orthographically inarguable problems (for example, stripping a stray boundary quote mark from "My).
2. For tokens that fail language-level validation but are structurally legal CHAT (e.g., C-3PO under English: tree-sitter accepts the digit-hyphen compound but Word::validate fires E220 “numeric digits not allowed”), ship the token verbatim. The full-file validator and check fire E220 on the same word, the file ends up in the human review queue, and the transcriber listens to the audio and decides what was actually said.
3. For tokens that fail structural parsing (tree-sitter rejects), fail loud: emitting malformed CHAT would corrupt the file beyond the validator’s ability to flag it.

The division of labor is: the tool fixes only what is mechanically unambiguous; CHECK and the human transcriber handle everything that requires judgment about what the speaker said.

Related rules

• xxx / yyy / www survive the transcript through all NLP passes (morphotag, utseg, translate, coref) without re-interpretation. Tools that walk the AST treat them as opaque tokens; they have no POS tag, no lemma, no dependency parent, no translation.
• %wor excludes all three (no phoneme sequence to align). %pho may reference yyy directly because the phonetic content is the whole point of the marker.
• See word-syntax.md for grammar; this document is the policy reference for who is allowed to emit them and why.

Postcodes ([+ ...])

Status: Reference Last updated: 2026-05-11 23:01 EDT

A postcode is a tagged annotation token that attaches to an utterance as a whole and appears after the terminator. The canonical CHAT syntax is [+ <text>]. Postcodes carry researcher / analysis tags about the utterance, whether it should be excluded from analysis, how it should be coded, what kind of speech act it represents, without modifying the utterance’s word content.

Syntax and Scope

*CHI:   I want cookie .  [+ exc]
*MOT:   what did you say ?  [+ imp]
*CHI:   no I don't want it !  [+ neg] [+ trn]

Three structural facts to internalize:

1. Postcodes attach to the utterance, not to a word. They sit after the terminator, on the main tier, alongside (but distinct from) any utterance-level bullet. Unlike word-scoped annotations ([: ...] replacement, [% ...] comment, [= ...] explanation, [* ...] error code), a postcode does not modify the interpretation of any single word, it tags the whole utterance.
2. Multiple postcodes may follow a single terminator. They are ordered, but the order is not semantically privileged.
3. The body is free-form text. The CHAT word grammar is not applied to postcode contents. Researchers can write arbitrary tags, codes, descriptions, comments, or analytic notes. The model stores the raw text and leaves interpretation to downstream tooling and conventions.

Common Postcodes, Empirical Survey

The postcode vocabulary is open-ended: the CHAT format imposes no closed set, and an audit of every [+ ...] token across a JSON-mirrored snapshot of the TalkBank corpora (~99k files, 23+ data-repo families) found 488 distinct values in active use.

The findings split into three tiers ranked by repo spread (in how many distinct corpus families the code appears), the more useful ranking than raw count, because high-count codes can be concentrated in a single corpus.

Tier 1, Cross-corpus codes (in 7+ repos)

These are the conventions every CHAT consumer should expect to encounter across collections:

Tier 2, Multi-corpus protocol codes (in 4-6 repos)

Codes deployed across several CHILDES sub-collections, typically encoding picture-narration / story-reading / imitation experimental conditions. Substantial raw counts (often tens of thousands), but their meaning is set by the originating protocol, consult per-corpus documentation rather than assuming a global definition:

Tier 3, Single-corpus and long-tail codes

About 80% of the 488 distinct values appear in one repo only. The single-corpus codes include high-volume protocol vocabularies (e.g. [+ uncued] ~19,500 in one repo, [+ NAC] ~3,500 in one repo, [+ diary] ~2,800 in a Romance/Germanic diary-study collection, [+ noatt] ~2,300 in one repo, [+ inter-utter-switch] ~720 flagging code-switching turns).

The long tail also includes researcher-private notes, typos that survived check, and per-study coding schemes. Tooling MUST treat any unknown postcode value as opaque text, the corpus author may know what it means, the format does not.

Caveats

• Numbers are from a snapshot audit and will drift as corpora are added or revised. Treat the broad shape (open vocabulary, ~4 truly cross-corpus codes, ~10 multi-corpus protocol codes, ~hundreds of single-corpus or long-tail codes) as the load-bearing finding, not the exact counts.
• “Repo spread” counts data-repo families, not individual files. Two corpora curated by the same group inside one data-repo count as one for spread; researchers using the same code in two different family-of-corpora packages count as two.
• The CHAT manual remains the source of truth for standard conventions. The empirical survey above shows what is actually deployed; when ingesting a new corpus, consult its own documentation for the postcodes in use.

What Postcodes Are NOT

Postcodes are easy to confuse with several other CHAT annotation forms because they all use square brackets. The differences are substantive and load-bearing.

Two consequences worth pinning down explicitly:

• A postcode cannot carry per-word semantics. If you want to attach a comment, replacement, or error code to a single word, use the appropriate word-scoped form. Stretching a postcode to mean “this word is X” loses the per-word position downstream tools depend on.
• A word-scoped annotation cannot tag an utterance. If you want to mark an entire utterance for exclusion or translation, use a postcode. A [% exclude this] after a word does not mean “exclude the utterance” to any consumer.

Not Postcodes: Quotation Markers

Quotation marking in CHAT is not a postcode form. The constructs +"/. (quotation end), +"/, and +" (quotation linkers / continuations) are tier-level terminators and linkers, not [+ ...] postcodes, the grammar rule postcode in grammar/grammar.js is strictly [+ <text>], and the quotation forms live under separate grammar rules (quoted_new_line, linker_quotation_follows).

See Utterances → Terminators for the syntactic forms, and the talkbank-model::validation::cross_utterance validator family (gated by ValidationContext::enable_quotation_validation) for the cross-utterance balance checks.

A walker in talkbank-model::validation::utterance::quotation (check_quotation_balance) does scan the postcode list for text "/ and "/., but a sweep over the data-json corpus mirror (101,414 files, 2026-05-11) returned zero such postcodes, that code path is effectively dead, retained presumably as defence against hand-edited oddities. The real quotation-balance work happens in the cross-utterance family above.

Position in the AST

An utterance’s main tier is MainTier, whose content: TierContent field carries the actual tier payload, including postcodes, as a typed list:

pub struct MainTier {
    pub speaker: SpeakerCode,
    pub content: TierContent,
    // spans omitted for brevity
}

pub struct TierContent {
    pub linkers: TierLinkers,                  // utterance-leading +<, ++, etc.
    pub language_code: Option<LanguageCode>,   // [- code]
    pub content: TierContentItems,             // word-level items (newtype over Vec<UtteranceContent>)
    pub terminator: Option<Terminator>,        // ., ?, !, +..., etc.
    pub postcodes: TierPostcodes,              // [+ ...] tokens after the terminator
    pub bullet: Option<Bullet>,                // optional terminal media bullet
    // content_span omitted for brevity
}

(See talkbank-model/src/model/content/main_tier.rs and tier_content.rs for the exact shape; the same TierContent type is shared by dependent tiers, so the postcode slot exists on every tier even though only main-tier postcodes are conventional.)

Because postcodes live at the utterance level, the per-word traversal helpers (walk_words, walk_words_mut) do not visit them. Code that needs to read or rewrite postcodes accesses the list directly.

The model stores postcode text as SmolStr and preserves it verbatim through CHAT roundtrips. Downstream tooling, including CLAN command implementations such as freq, mlu, kideval, is responsible for interpreting individual postcode values per its own conventions.

Tooling Rules

Tools that emit or consume CHAT must respect the scope distinction.

• Emitters: when adding a researcher tag to an utterance, attach a Postcode to the utterance’s MainTierContent, not a ContentAnnotation to a word. Both serialize, but only the former reaches downstream consumers as utterance-level metadata.
• Consumers: when reading utterance-level tags (e.g., implementing an “exclude” filter), iterate main.content.postcodes on each utterance, not the word-level annotations in UtteranceContent. The two lists are populated by different parser branches and have different semantics.
• Round-trip preservers (extract→modify→inject pipelines such as the NLP injection passes in crates/batchalign-*): preserve the postcode list unchanged. None of the standard NLP passes have a reason to add, remove, or reorder postcodes.

References

• CHAT manual: Postcodes
• CHAT manual: Excluded Utterance Postcode
• CHAT manual: Included Utterance Postcode
• Model: talkbank-model/src/model/content/postcode.rs
• Quotation validator: talkbank-model/src/validation/utterance/quotation.rs

Dependent Tiers

Status: Reference Last updated: 2026-06-22 23:33 EDT

Dependent tiers appear on lines beginning with % immediately after an utterance. They provide annotations linked to the main tier content.

CHAT defines four structural categories of dependent tiers:

1. Structured linguistic tiers: parsed into typed AST nodes with word-level alignment
2. Phon phonological tiers: syllabification and segmental alignment from the Phon project
3. Bullet-content tiers: free-form text with optional inline timing markers
4. Text tiers: plain text with no structural alignment

Structured Linguistic Tiers

These tiers have rich, parsed representations in the data model. Each token aligns 1-to-1 with an alignable word on the main tier (excluding retraces, pauses, and events). Terminators (., ?, !) must match the main tier terminator.

%mor, Morphological Analysis

The %mor tier carries part-of-speech tags, lemmas, and morphological features for each word on the main tier. See The %mor Tier for full documentation covering the UD-style format, data model, divergences from Universal Dependencies, and migration from traditional CHAT MOR.

Format: POS|lemma[-Feature]*, with ~ separating post-clitics.

*CHI:	she's eating cookies .
%mor:	PRON|she~AUX|be-Pres-S3 VERB|eat-Prog NOUN|cookie-Plur .

%gra, Grammatical Relations

The %gra tier encodes dependency syntax using Universal Dependencies relation labels. Each entry has the format index|head|relation, where indices are 1-based and head 0 indicates ROOT.

*CHI:	I want cookies .
%mor:	PRON|I VERB|want NOUN|cookie-Plur .
%gra:	1|2|SUBJ 2|0|ROOT 3|2|OBJ 4|2|PUNCT

The %gra tier aligns with %mor chunks (clitics expand into multiple chunks). Validation checks sequential indices (E721), ROOT structure (E722 missing root, E723 multiple roots), and circular dependencies (E724).

%pho / %mod, Phonological Transcription

The %pho tier records actual pronunciation; %mod records target/model pronunciation. Both use the same format: space-separated phonetic tokens aligned 1-to-1 with main tier words.

*CHI:	I want three cookies .
%pho:	aɪ wɑnt fwi kʊkiz .
%mod:	aɪ wɑnt θri kʊkiz .

Phonological tiers support IPA, UNIBET, X-SAMPA, or custom notation systems. They are used for child language, speech disorders, L2 learning, and dialectal variation studies.

Parsing strategy: We deliberately parse only the minimal word/group-level structure in %pho and %mod needed for coarse alignment with the main tier. The full IPA phoneme content is stored as opaque strings, deep phonological analysis is handled by Phon, and we avoid duplicating that work. The Phon extension tiers (%modsyl, %phosyl, %phoaln) follow the same strategy.

%sin, Gesture and Sign Annotation

The %sin tier codes gestures and signs aligned with speech. Each token is either 0 (no gesture) or g:referent:type (e.g., g:ball:dpoint for a deictic point at a ball).

*CHI:	that ball .
%sin:	g:ball:dpoint 0 .

Multiple simultaneous gestures use bracket grouping: 〔g:toy:hold g:toy:shake〕.

%wor, Word Timing

The %wor tier carries word-level timing annotations for media synchronization. Words may include inline bullets with millisecond timestamps. Word text is display-only (“eye candy”); timing data comes from the bullet fields.

⚠ IMPORTANT: %wor word text is the cleaned form, by design. When chatter serializes a %wor word it writes the word’s cleaned text, the spoken form with surface markers removed, NOT the raw main-tier surface form. This is a deliberate convention (see WorTier::write_chat in crates/talkbank-model/src/model/dependent_tier/wor.rs), chosen for human readability and because %wor exists to anchor timing, not to re-state the main tier’s orthography. The generated %wor text and the TextGrid export both use this cleaned form.

Consequence you must know: surface markers carried on a word, prosodic lengthening (wabe:), and similar in-word notation, are not preserved in %wor output. A main-tier word wabe: becomes wabe on %wor. This means a %wor line containing such words does not byte-roundtrip (parse, serialize, reparse changes the surface text), and that is expected, not a bug. %wor is a cleaned, timing-only view; the main tier remains the faithful record of surface forms. Do not “fix” the %wor serializer to emit raw text without an explicit decision to change this convention.

%wor is not a flat “all tokens except punctuation” tier. It follows a word-level alignment rule:

• Regular words count.
• Fillers (&-um, &-uh, &-you_know) count; they are real spoken words with known phoneme sequences.
• Fragments (&+...) do NOT count: incomplete phoneme sequences; the FA engine cannot reliably anchor partial phonological material.
• Nonwords (&~...) do NOT count: interactional/gestural sounds without stable lexical phoneme content for alignment.
• Untranscribed placeholders (xxx, yyy, www) do NOT count: they have no known phoneme sequence; CTC forced alignment cannot produce timings for unknown material.
• Replacements keep the original spoken word slot for %wor; the replacement text matters for %mor, not %wor. If the original slot is untranscribed or a fragment/nonword, it is still excluded.
• Retrace scope does not change %wor membership.
• Overlap markers do not change %wor membership.

%wor is a timing-annotation tier. Its word count equals the number of Wor-domain words and may differ from a naive main-tier word count. There is no downstream positional indexing into %wor; the %wor count is not validated against the main-tier word count.

*CHI:	I want cookies .
%wor:	I want cookies .

Exact corpus-shaped contrast:

*CHI:	<one &+ss> [/] one play ground .
%wor:	one •321809_321969• play •322049_322310• ground •322390_322890• .
# &+ss is a fragment, excluded from %wor regardless of retrace context.

*EXP:	&+ih <the what> [/] what's letter &+th is this ?
%wor:	the •49103_49163• what •49183_50205• what's •50205_50405• letter •50405_50685• is •50946_51046• this •51086_51586• ?
# Fragments &+ih and &+th excluded; regular words remain.

*EXP:	what's is dis [: this] ?
%wor:	what's •37050_37471• is •37491_37631• dis •37631_38131• ?

*CHI:	xxx snack .
%wor:	snack •884668_885168• .
# xxx has no phoneme sequence, excluded from %wor; only snack appears.

*CHI:	&~um a boat .
%wor:	a •1073779_1073799• boat •1076861_1077361• .
# &~um is a nonword, excluded from %wor.

*CHI:	&-mm [<] bananas are good .
%wor:	mm •1949506_1949566• bananas •1949566_1949766• are •1949846_1949987• good •1950067_1950567• .
# &-mm is a filler, included in %wor (real spoken word with alignable phoneme sequence).

flowchart TD
    A["Main-tier word candidate"] --> B{"Timestamp token /\nomission / empty?"}
    B -->|Yes| OUT["Excluded from %wor"]
    B -->|No| C{"Untranscribed?\n(xxx/yyy/www)"}
    C -->|Yes| OUT
    C -->|No| D{"Fragment or nonword?\n(&+ or &~)"}
    D -->|Yes| OUT
    D -->|No| IN["Counts for %wor\n(word or filler &-)"]

    style IN fill:#afa,stroke:#333
    style OUT fill:#faa,stroke:#333

Phon Phonological Tiers

These tiers originate from the Phon project and provide syllable-annotated phonological transcription and segmental alignment. They were originally serialized as %x-prefixed user-defined tiers (%xmodsyl, %xphosyl, %xphoaln) and are being promoted to official CHAT tiers. Phon stores phonological data in its own XML format; the CHAT representation is generated by PhonTalk.

%modsyl / %phosyl, Syllabified Phonology

%modsyl is a syllabified version of %mod (target pronunciation); %phosyl is a syllabified version of %pho (actual pronunciation). Each phoneme is annotated with a syllable position code (N=nucleus, O=onset, C=coda, etc.). Words are space-separated and align 1-to-1 with the corresponding %mod or %pho tier.

*CHI:	the best .
%mod:	ðə bɛst .
%modsyl:	ð:Oə:N b:Oɛ:Ns:Ct:C .
%pho:	ðə bɛs .
%phosyl:	ð:Oə:N b:Oɛ:Ns:C .

Alignment: Content-based, stripping position codes (:N, :O, :C, etc.) and stress markers (ˈ, ˌ) from %modsyl should yield the same phonemes as %mod. Same for %phosyl → %pho.

%phoaln, Phone Alignment

%phoaln provides segmental alignment between target and actual IPA, showing phoneme-by-phoneme correspondence. Each pair uses source↔target notation; ∅ marks insertions or deletions.

*CHI:	the best .
%phoaln:	ð↔ð,ə↔ə b↔b,ɛ↔ɛ,s↔s,t↔∅

Alignment: Positional, word-by-word, word N in %phoaln aligns with word N in both %mod and %pho.

Parsing strategy: Same as %pho/%mod, we parse just enough structure for alignment (word boundaries for %modsyl/%phosyl, alignment pairs for %phoaln). IPA phoneme content is treated as opaque strings.

Validation (E725-E728)

Because these are derived views, word counts must match between each syllabification tier and its parent IPA tier:

These checks are gated on ParseHealth, if either tier in a pair has parse errors, the alignment check is suppressed to avoid false positives.

Known PhonTalk Export Issue

The PhonTalk XML→CHAT converter writes %mod/%pho through a OneToOne alignment path that maps IPA words to orthography words and silently drops extras. The syllabification tiers (%modsyl, %phosyl, %phoaln) bypass this path and include all IPA words. In child phonology data where children produce more IPA words than orthographic targets (~4% of Phon corpus files), this creates tier-to-tier word count mismatches. The mismatches originate in the Phon XML source data (orthography↔IPA word count discrepancies) and are inconsistently handled during CHAT export. This is being investigated in collaboration with the Phon team.

Bullet-Content Tiers

These tiers contain free-form text with optional embedded timing markers (•START_END•) and picture references (•%pic:"file.jpg"•). They do not align word-by-word with the main tier.

%cod is bullet-content in the shared TalkBank AST. In the %cod coding convention, a word selector such as <w4> scopes the code that follows it (it names which main-tier word the code applies to) rather than being a code in its own right.

Example with timing:

*CHI:	gimme that .
%act:	reaches toward shelf
%com:	child is pointing to picture

Text Tiers

These tiers contain plain text with no bullets, timing, or structural alignment:

User-Defined Tiers

Tiers prefixed with %x (e.g., %xcod, %xact) are user-defined dependent tiers. They are preserved during parsing and roundtrip but receive no structural validation beyond basic format checks. Any %x-prefixed tier is always accepted, this is the open extension point for project-specific annotation.

The Supported Set Is Closed

A dependent tier is valid in chatter only if it is one of the standard tiers documented above (the structured, Phon, bullet-content, and text tiers) or a %x-prefixed user-defined tier. Any other %-tier is invalid CHAT, and chatter rejects the file with error E605 (UnsupportedDependentTier). This is a closed set by design: chatter validate is the binding judgment on CHAT validity, so an unrecognized dependent tier is an error, not a warning.

Deliberate Divergence from CLAN: Retired Legacy Tiers

When TalkBank standardized morphology on a single Universal Dependencies %mor tier (plus %gra for relations), several legacy dependent tiers were retired. CLAN’s check still accepts three of them, so on these chatter is intentionally stricter, a deliberate, documented divergence:

The modern UD-%mor workflow has one morphology tier (%mor) plus %gra; the older training/translation/variant tiers are no longer part of the format chatter validates. %umor is rejected by both validators and is listed only for completeness. Note that %xtra (with the %x prefix) is a perfectly valid user-defined tier; only the bare %tra is retired.

This is one instance of a general principle: where chatter intentionally departs from CLAN/CHECK behavior, the divergence is documented rather than left implicit. See CHECK Parity Audit.

The %mor Tier: Morphological Analysis

Status: Reference Last updated: 2026-05-11 20:35 EDT

The %mor (morphological) dependent tier provides word-by-word morphosyntactic annotation aligned with the main tier. Each main-tier word receives a morphological code specifying part of speech, lemma, and grammatical features.

Format Overview

*CHI:	I want cookies .
%mor:	pron|I-Prs-Nom-S1 verb|want-Fin-Ind-Pres-S1 noun|cookie-Plur .

Each %mor item has the structure POS|lemma[-Feature]*, where:

• POS: part-of-speech category (noun, verb, pron, det, aux, etc.)
• |: pipe separator (always present)
• Lemma: base form of the word (cookie, be, I). May contain language-specific compound or derivational boundary markers (see Compound Lemma Boundaries below)
• Features: zero or more morphological features, each preceded by - (-Plur, -Fin-Ind-Pres-S3)

Items are space-separated and terminate with a punctuation marker (., ?, !, etc.).

The UD MOR Format

TalkBank’s %mor tier uses a format inspired by Universal Dependencies (UD) but adapted to CHAT conventions. We call this the UD MOR format to distinguish it from the older CLAN-era MOR format.

The UD MOR format was introduced via batchalign’s Stanza-based morphosyntax pipeline. Stanza produces standard UD analysis (UPOS, lemma, morphological features, dependency relations), and the Rust mapping layer converts this to CHAT %mor and %gra tiers. The new format has been adopted for all new corpus annotation.

Structure: Flat POS|lemma[-Feature]*

Every morphological word is flat, a single POS tag, a single lemma, and a linear chain of features:

POS|lemma[-Feature1][-Feature2][-Feature3]...

There are no compounds, prefixes, subcategories, or nested structures in the UD MOR format. The entire morphological analysis of a word is captured by the POS+lemma+features triple.

Examples:

Multi-Word Tokens (Clitics)

English contractions and similar multi-word tokens (MWTs) are represented using the tilde (~) separator for post-clitics:

*CHI:	it's red .
%mor:	pron|it~aux|be-Fin-Ind-Pres-S3 adj|red .

Here it's is a single main-tier word that expands to two morphological words: pron|it (main) and aux|be-Fin-Ind-Pres-S3 (post-clitic). The ~ indicates the two MOR words are fused into one orthographic token.

Each clitic counts as its own chunk for %gra alignment; pron|it~aux|be-Fin-Ind-Pres-S3 produces 2 chunks, each needing its own grammatical relation.

Terminator

The %mor tier ends with a terminator that matches the main tier’s utterance terminator:

*CHI:	what is that ?
%mor:	pron|what aux|be-Fin-Ind-Pres-S3 det|that ?

The terminator (., ?, !, +..., etc.) counts as one chunk for %gra alignment.

How It Diverges from UD

The UD MOR format is UD-inspired but not UD-compliant. Several deliberate adaptations make it fit CHAT conventions while preserving most UD information. This section catalogs every divergence.

1. POS Tags Are Lowercased UPOS

UD uses uppercase UPOS tags (NOUN, VERB, PRON). CHAT uses lowercase (noun, verb, pron). This is a lossless, trivially reversible surface change.

2. Feature Values Are Flat, Not Key=Value (Currently)

UD represents morphological features as key=value pairs: Number=Plur, Tense=Past, Person=3. The current CHAT convention drops the keys and uses only the values: -Plur, -Past, -S3.

This is the most significant divergence from UD, because:

• Information loss: Plur could in principle be Number=Plur or Degree=Plur (though in practice the UD feature value set has no real ambiguities).
• Collapsed person/number: UD Person=3|Number=Sing becomes -S3, a combined code that cannot be mechanically decomposed back to its UD components.
• Feature ordering: Features appear in a conventional order determined by the generation pipeline, not in UD’s alphabetical order.

The data model now supports key=value features. The MorFeature type has an optional key field, when present, the feature serializes as Key=Value (e.g., -Number=Plur); when absent, it serializes as just the value (e.g., -Plur). This is forward-compatible: existing flat features parse and serialize identically, and if batchalign’s mapper begins emitting Key=Value features, they flow through the parser and model without any format changes.

3. Multi-Value Features: Commas Preserved

UD encodes multi-value features with commas: PronType=Int,Rel (the word is both interrogative and relative). In CHAT %mor, the comma is preserved within the feature value:

-Int,Rel

This is treated as a single feature value "Int,Rel". The grammar accepts commas within feature values, and the model stores them as-is. No decomposition occurs; the model faithfully records the string that appears in the %mor tier.

Historical note: Earlier documentation described a “comma-stripping” convention where PronType=Int,Rel became -IntRel (concatenated without separator). The current grammar and parser preserve the comma. Existing corpus data using the concatenated form (-IntRel) also parses correctly; it’s simply treated as the flat value "IntRel".

4. Dependency Relations Are Uppercase with Dash Subtypes

The %gra tier (not %mor, but closely related) uses uppercase relation names with dashes for subtypes, where UD uses lowercase with colons:

This is lossless; case and separator are trivially reversible.

5. ROOT Head Convention

In UD, the root word has head=0. In %gra, two conventions coexist:

• UD convention: head=0 (e.g., 3|0|ROOT), the standard we now emit
• Legacy TalkBank convention: head=self (e.g., 3|3|ROOT), found in older corpus data

The parser and validator accept both forms. New output uses head=0.

6. No XPOS, No DEPREL Subtypes in %mor

UD provides both UPOS (universal POS) and XPOS (language-specific POS). CHAT %mor uses only UPOS-equivalent tags; there is no XPOS field. Language-specific POS distinctions are not represented.

Similarly, UD’s fine-grained dependency relation subtypes (e.g., nsubj:pass) appear in %gra as NSUBJ-PASS, but the %mor tier itself contains no dependency information.

7. No Morpheme Segmentation

Traditional CHAT MOR formats (CLAN-era) supported morpheme-level segmentation with compound markers (+), prefix markers (#), and suffix chains (-SUFFIX&type). The UD MOR format does not use any of these, each word is analyzed as a flat POS+lemma+features triple.

The grammar still accepts some of these legacy markers for backward compatibility with older corpus data, but the canonical UD MOR format does not produce them.

Compound Lemma Boundaries

Several UD treebanks use special characters inside lemmas to mark morphological boundaries. These are meaningful linguistic annotations preserved in the CHAT %mor lemma field when possible.

Known Markers Across Languages

= and ! pass through the cleaning pipeline because they are not reserved CHAT %mor syntax characters. # is reserved in traditional CHAT MOR for prefix markers (e.g., v|#un#do), so the sanitizer replaces it with _.

Gotcha: = ambiguity with legacy CLAN translation glosses. Legacy CLAN %mor tiers use = for translation glosses (e.g., n|perro=dog), a convention predating UD adoption. The parser treats = identically in both cases; it is preserved as part of the lemma string. This means legacy n|perro=dog parses successfully but the translation semantics are lost: the model stores perro=dog as a single lemma, indistinguishable from an Estonian compound like maja=uks. Since we cannot reliably disambiguate the two uses without language-specific context, legacy translation glosses are silently absorbed into the lemma. Files with legacy =translation syntax still parse and round-trip correctly, but the translation information is not semantically accessible. This affects corpora that predate our UD MOR adoption and lack Stanza coverage for their language.

Multi-Word Expression Lemmas (Stanza _ Convention)

Stanza uses underscores in lemmas to represent multi-word expressions across many languages: New_York, parce_que (French), pick_up (English), a_causa_di (Italian). The current cleaning pipeline strips underscores entirely (New_York → NewYork), which is a known data quality issue and should be treated as an open data-quality limitation of the current mapper.

Multi-Value Features (Commas in Feature Values)

UD encodes multi-value features with commas: PronType=Int,Rel means a word is both interrogative and relative. These commas appear in the CHAT %mor feature suffix and are preserved as-is:

pron|wat-Int,Rel

This is sometimes mistaken for a compound lemma marker, but commas in UD always appear in the feature column (CONLLU column 6), never in the lemma column (CONLLU column 3). In CHAT %mor, they appear after the - feature separator, not inside the lemma. The grammar, both parsers, and the data model all accept commas in feature values. See Section 3: Multi-Value Features above.

Future Direction

The current handling of compound lemma boundaries is inconsistent across languages. A possible future improvement is a unified Unicode separator character that would normalize all compound/derivational boundary markers (=, !, #, and potentially _) into a single convention. This has not been implemented as of 2026-03-02 and requires a design decision on which character to use and whether to preserve the original markers in a structured field.

Data Model

The Rust data model in talkbank-model represents %mor tiers with these types:

MorTier

The top-level tier container:

pub struct MorTier {
    pub tier_type: MorTierType,    // MorTierType::Mor
    pub(crate) items: MorItems,    // Vec<Mor> wrapper; accessed via accessor methods
    pub terminator: Terminator,    // typed terminator (.`, `?`, `!`, `+...`, etc.)
    pub span: Span,                // source location
}

Mor (Item)

One item aligned with one main-tier word:

pub struct Mor {
    pub main: MorWord,                        // required main word
    pub post_clitics: SmallVec<[MorWord; 2]>, // optional ~clitics
}

MorWord

A single morphological word (POS + lemma + features):

pub struct MorWord {
    pub pos: PosCategory,                    // e.g., "noun"
    pub lemma: MorStem,                      // e.g., "dog"
    pub features: SmallVec<[MorFeature; 4]>, // e.g., [Plur]
}

MorFeature

A morphological feature with optional key:

pub struct MorFeature {
    key: Option<Arc<str>>,  // e.g., Some("Number") or None
    value: Arc<str>,        // e.g., "Plur"
}

Construction examples:

// Flat feature (current convention)
MorFeature::new("Plur")         // key=None, value="Plur"
MorFeature::new("S3")           // key=None, value="S3"
MorFeature::new("Int,Rel")      // key=None, value="Int,Rel"

// Keyed feature (UD-standard, forward-compatible)
MorFeature::new("Number=Plur")  // key=Some("Number"), value="Plur"
MorFeature::new("Tense=Past")   // key=Some("Tense"), value="Past"

// Explicit constructors
MorFeature::flat("Plur")
MorFeature::with_key_value("Number", "Plur")

Lossless roundtrip guarantee: MorFeature::new auto-detects the = delimiter. Features without = are flat; features with = split into key+value. Serialization reproduces the original format exactly, flat features stay flat, keyed features keep their key.

PosCategory and MorStem

Both are interned Arc<str> newtypes for memory efficiency:

pub struct PosCategory(pub Arc<str>);  // interned via pos_interner()
pub struct MorStem(pub Arc<str>);      // interned via stem_interner()

Common values (noun, verb, the, a, be, etc.) are pre-populated in the interner. Cloning is O(1), atomic reference count increment.

Memory Layout

The model uses SmallVec for inline storage of common cases:

• Mor.post_clitics: SmallVec<[MorWord; 2]>: most words have 0-1 clitics
• MorWord.features: SmallVec<[MorFeature; 4]>: most words have 0-4 features
• MorFeature key and value are Arc<str>, interned for deduplication

For a typical 30-word utterance with %mor, the model allocates approximately 30 Mor items, each with 1 MorWord and 0-4 MorFeature values. The interning system ensures that repeated POS tags, stems, and feature values share a single allocation across the entire file.

Grammar

The tree-sitter grammar for %mor is defined in grammar.js. The relevant rules:

mor_content → mor_word (mor_post_clitic)*
mor_post_clitic → tilde mor_word
mor_word → mor_pos pipe mor_lemma (mor_feature)*
mor_feature → hyphen mor_feature_value
mor_feature_value → /[^\.\?\|\+~\-\s\r\n]+/

Key design decisions:

• mor_feature_value accepts = and !: The regex [^\.\?\|\+~\-\s\r\n]+ matches any characters except the MOR structural delimiters. This means Number=Plur parses as a single mor_feature_value node. The split on = happens in the model layer, not the grammar, following the “parse, don’t validate” principle.
• mor_feature_value accepts ,: Multi-value features like Int,Rel parse as a single node.
• No compound/prefix rules: The grammar has no rules for + (compounds) or # (prefixes) in the UD MOR format. These are legacy CHAT MOR features not used in UD-style output.

Parser

The tree-sitter parser produces MorTier from CHAT text. It is GLR-based and error-recovering, producing a CST that the Rust talkbank-parser crate walks to construct MorTier. Used by the CLI, LSP, and batchalign. High-frequency values (PosCategory, MorStem) are interned via Arc<str> during construction.

The corpus/reference/ set is the correctness gate for %mor parsing, every file must parse and round-trip cleanly. The file count grows as new constructs are added; run find corpus/reference -name '*.cha' | wc -l to get the live total.

Validation

The %mor tier undergoes several validation checks:

Content Validation (E711)

Every MorWord is checked for:

• Empty POS: |lemma with no POS before the pipe
• Empty lemma: pos| with no lemma after the pipe
• Empty feature: bare - separator with no feature text

Main-tier Alignment (E705 / E706)

The %mor tier must align 1-to-1 with the main tier’s alignable words (excluding pauses, events, and other non-word content). The number of Mor items must equal the number of alignable main-tier words. The validator emits E705 MorCountMismatchTooFew when %mor has fewer items than the main tier and E706 MorCountMismatchTooMany when it has more. Terminator-mismatch errors are emitted separately as E707 (presence) and E716 (value).

GRA Alignment (E720)

When both %mor and %gra tiers are present, the number of %gra relations must equal the number of %mor chunks (including clitics and the terminator). A mismatch emits E720 MorGraCountMismatch. This is computed via MorTier::count_chunks().

(%gra’s own internal validators, E708 malformed relation, E709 invalid index, E712 word-index out of range, E713 head-index out of range, E721 non-sequential index, E722 no ROOT, E723 multiple ROOTs, E724 circular dependency, are documented in Dependent Tiers § %gra.)

JSON Serialization

The MorTier serializes to JSON using serde. MorFeature serializes as a plain string ("Plur" or "Number=Plur"), so the JSON schema is simply "type": "string". Example:

{
  "tier_type": "Mor",
  "items": [
    {
      "main": {
        "pos": "pron",
        "lemma": "I",
        "features": ["Prs", "Nom", "S1"]
      }
    },
    {
      "main": {
        "pos": "verb",
        "lemma": "want",
        "features": ["Fin", "Ind", "Pres", "S1"]
      }
    },
    {
      "main": {
        "pos": "noun",
        "lemma": "cookie",
        "features": ["Plur"]
      }
    }
  ],
  "terminator": "."
}

When key=value features are present, they serialize with the key included:

"features": ["Number=Plur", "Tense=Past"]

The JSON schema for MorFeature is "type": "string" regardless of whether keys are present.

Migration from Traditional CHAT MOR

What Changed

The traditional CHAT MOR format (CLAN-era) used a complex, hierarchically structured notation:

%mor:	pro:sub|I v|want n|cookie-PL .

Key differences from the UD MOR format:

What the Model Removed

The UD MOR redesign (2026) removed the following types from the data model:

• MorSuffix: suffix with type discriminant (fusional, derivational, etc.)
• MorCompound: compound word with + separator
• MorPrefix: prefix with # separator
• MorSubcategory: POS subcategory after colon
• AnnotatedChunk: chunk with optional translation
• Chunk: enum of word/compound/terminator

These were replaced by the flat MorWord { pos, lemma, features } structure. The model went from ~12 types to 4 (MorTier, Mor, MorWord, MorFeature).

Backward Compatibility

The grammar still accepts many traditional CHAT MOR constructs (colons in POS tags, etc.) because the reference corpus contains files in both formats. The parser produces the same flat MorWord regardless; legacy constructs are mapped to the simplified structure during parsing.

What Stays the Same

Despite the format changes, fundamental CHAT conventions remain:

• Pipe (|) separates POS from lemma
• Hyphen (-) introduces features
• Tilde (~) marks post-clitics
• Space separates items
• Terminator ends the tier
• 1-to-1 alignment with main tier words

Toward Full UD Compatibility

The current format is UD-inspired but not UD-compliant. Here is a roadmap of what would be needed for full lossless UD round-tripping:

Already Supported

• POS tags (UPOS equivalents)
• Lemmas
• Feature values (flat and key=value)
• MWT expansions (clitics)
• Dependency relations (via %gra)

Gaps Remaining

1. Feature keys: The model supports Key=Value features, but batchalign’s mapper currently emits flat values only. When the mapper switches to emitting Number=Plur instead of just Plur, the parser, model, and serializer handle it automatically with no code changes.

2. Person+Number composites: UD has separate Person=3 and Number=Sing features. CHAT combines them into -S3 (3rd person singular). Decomposing S3 back to Person=3|Number=Sing would require a lookup table or a convention change.

3. Multi-value feature delimiter: UD uses commas (PronType=Int,Rel). CHAT preserves these commas in the feature value, but the semantic structure (two separate values) is not explicitly modeled. The model treats Int,Rel as an opaque string.

4. XPOS: UD provides language-specific POS tags (XPOS) alongside universal tags (UPOS). CHAT %mor has no XPOS field. This information is simply not represented.

5. Morpheme-level analysis: UD’s MISC field can encode morpheme boundaries and glosses. CHAT’s UD MOR format does not attempt morpheme segmentation, features are abstract grammatical categories, not morphemic decompositions.

The Path Forward

The model is designed so that moving toward UD compliance requires no breaking changes:

• MorFeature already supports Key=Value, just needs the mapper to emit keys
• PosCategory is an opaque string, could hold XPOS in a separate field if needed
• JSON schema uses "type": "string" for features, adding keys doesn’t break consumers
• The grammar already accepts = in feature values, no grammar changes needed

The migration can happen incrementally: the mapper starts emitting key=value features, existing flat data continues to parse identically, and corpus files can be upgraded at their own pace.

Phon Tiers (%xmodsyl, %xphosyl, %xphoaln, %xphoint)

Status: Reference Last updated: 2026-06-23 07:28 EDT

The Phon extension tiers provide syllable-level phonological annotation, segmental alignment between target and actual IPA, and per-phone time intervals. They are produced by the Phon application and exported to CHAT via PhonTalk.

chatter parses and validates all four tiers as first-class CHAT tiers.

The x prefix. Phon emits these tiers with a leading x (%xmodsyl, %xphosyl, %xphoaln, %xphoint) to mark them as extension tiers. The grammar accepts both the x-prefixed names and the historical non-x names (%modsyl, %phosyl, %phoaln, %phoint); the parser and validator key off the tier kind, not the literal prefix. The canonical serialized form is the x-prefixed name.

The four tiers

%xmodsyl, %xphosyl, and %xphoaln are word-aligned to their source tier(s) with single ASCII spaces. %xphoint uses / (space-slash-space) as its word separator because single spaces already separate the phone and bullet tokens inside each word.

Tier formats

%xmodsyl / %xphosyl, syllabification

A word is one or more phone:CODE units concatenated with no internal whitespace; words are separated by single spaces. The phone is one IPA phone (IPA length is written with the modifier letter ː, U+02D0, never an ASCII colon, so the : separator is unambiguous). A leading stress marker (ˈ primary, ˌ secondary) is part of the phone it precedes.

The constituent code is one character. The legal codes are O N C L R E A D U:

The remaining Phon SyllableConstituentType mnemonics, B (boundary), S (stress), W (word boundary), T (tone), are not emitted on these tiers: boundary, stress, and tone need no per-phone marker.

*CHI:	I want three .
%mod:	aɪ wɑnt θri
%xmodsyl:	a:Dɪ:D w:Oɑ:Nn:Ct:C θ:Oɹ:Oi:N
%pho:	aɪ wɑn fwi
%xphosyl:	a:Dɪ:D w:Oɑ:Nn:C f:Ow:Oi:N

%xphoaln, phone alignment

A word is one or more comma-separated pairs; a pair is model↔actual (↔ is U+2194). Either side may be ∅ (U+2205, empty set): ∅ on the left is an epenthesis (a phone produced but not targeted); ∅ on the right is a deletion. Both sides are never ∅ at once.

*CHI:	the best .
%mod:	ðə bɛst
%pho:	ðə bɛs
%xphoaln:	ð↔ð,ə↔ə b↔b,ɛ↔ɛ,s↔s,t↔∅

The alignment lists segments (phones). Suprasegmental stress (ˈ/ˌ) that may appear on the %mod/%pho word is therefore not part of the alignment pairs; the reconstruction checks below compare modulo those stress markers.

%xphoint, per-phone intervals

%xphoint gives the time segmentation of each individual phone on %pho, effectively phone-level bullets analogous to the word-level timing on %wor. Groups (one per %pho word) are separated by /. Within a group, each phone is followed by a CLAN time-alignment bullet: the byte 0x15 (NAK), the interval start_end, then 0x15.

*CHI:	I want . •0_500•
%pho:	aɪ wɑnt
%xphoint:	aɪ •0_250• / w •250_320• ɑ •320_400• n •400_460• t •460_500•

(Bullets are shown as • above; in the file they are the 0x15 byte.)

Validation

These checks run by default. Pass --suppress xphon to silence the entire Phon %x validation surface, or suppress an individual code. (The historical --check-xphon opt-in flag is now a deprecated no-op: the checks it used to gate are on by default.)

Word-count cross-checks (each %x tier has the same number of words as the tier(s) it depends on):

• %xmodsyl ↔ %mod: E725
• %xphosyl ↔ %pho: E726
• %xphoaln ↔ %mod: E727, ↔ %pho: E728

Content checks:

See Alignment Architecture for the word-count implementation.

Parsing strategy

• %xmodsyl / %xphosyl: stored as flat word strings (talkbank-model::dependent_tier::phon::SylTier), consistent with how %pho and %mod store flat phone words. The validator tokenizes each word into typed phone:CODE units (PositionCode) to apply the content rules above; the IPA characters themselves stay verbatim for exact round-trip.
• %xphoaln: each word is parsed into a Vec<AlignmentPair>, where AlignmentPair { source, target } carries one model↔actual mapping (None is ∅).
• %xphoint: parsed into typed groups of (phone, bullet) pairs (XphointTier / XphointGroup / PhoneInterval), reusing the same 0x15 bullet machinery as %wor.

Deep phonological analysis is Phon’s domain; chatter parses the structure that validation needs and keeps the IPA content verbatim.

Phon XML source format

In Phon’s native XML format, phonological data is stored as structured elements:

<ipaTarget>
  <pho>
    <pw>
      <ph scType="onset"><base>θ</base></ph>
      <ph scType="nucleus"><base>ɹ</base></ph>
      <ph scType="nucleus"><base>i</base></ph>
    </pw>
  </pho>
</ipaTarget>

Each <pw> (phonological word) element contains <ph> elements with syllable constituent types (scType). The <alignment> element provides phone-level mappings between target and actual using index-based <pm> (phone map) entries.

Data quality notes

A small percentage of Phon corpus XML records have an orthography↔IPA word-count mismatch: the number of <pw> elements in <ipaTarget> / <ipaActual> differs from the number of <w> elements in <orthography>. This is expected in child phonology data: children may produce extra syllables, partial words, or over-productions relative to the target.

For current counts on a local CHILDES/TalkBank data tree, run:

python3 scripts/analysis/scan_phon_mismatches.py /path/to/data

The PhonTalk CHAT export handles this discrepancy inconsistently:

1. %mod/%pho are written through a OneToOne alignment path that maps IPA words to orthography words; extras are silently dropped.
2. %xmodsyl/%xphosyl/%xphoaln are written directly from the raw IPATranscript; all IPA words are included.

This produces CHAT files where %xmodsyl may have more words than %mod, triggering the E725-E728 word-count errors. This is being investigated in collaboration with the Phon team.

Word Syntax

Status: Reference Last updated: 2026-05-11 23:33 EDT

Words are the primary content unit on the main tier. CHAT defines several word types and annotation mechanisms.

Standalone Words

Most words are simple tokens separated by whitespace:

*CHI:	I want a cookie .

Words can contain Unicode characters for any language:

*CHI:	ich möchte Kekse .

Compounds

Compound words join multiple elements with +:

*CHI:	I want ice+cream .

Special Word Forms

Shortened Forms

Parentheses mark omitted portions of a word:

*CHI:	(be)cause I want it .

The full form is because; the child produced cause.

Replacements

Square brackets with colon mark what the speaker actually meant:

*CHI:	I goed [: went] to the store .

The speaker said “goed” but the intended word was “went”.

Language Markers

The @s: suffix marks a word’s language in multilingual transcripts:

*CHI:	I want a Keks@s:deu .

Other @ markers:

• @l: letter
• @c: child-invented form
• @f: family-specific word
• @n: neologism
• @o: onomatopoeia
• @b: babbling
• @wp: word play
• @si: signed word

Annotations

Words and groups can carry post-positioned annotations in square brackets:

Error Marking

*CHI:	he goed [*] to school .

[*] marks an error. More specific error codes can follow: [* m:+ed].

Explanations

*CHI:	that one [= the red ball] .

[= text] provides an explanation or gloss.

Replacements

*CHI:	I wanna [: want to] go .

[: text] marks the target/intended form.

Best Guess

*CHI:	I want the birfer [?] .

[?] marks uncertain transcription.

Events and Actions

Paralinguistic Events

Events marked with &= describe non-speech sounds:

*CHI:	&=laughs I want cookie .
*CHI:	&=coughs .

Fillers

Fillers are marked with &-:

*CHI:	&-um I want &-uh cookie .

Interposed Speech (Other Speaker)

Brief background speech from a different speaker is marked with the &*SPK:text prefix, it captures the interjection without creating a full turn line:

*CHI:	I want &*MOT:careful a cookie .

This says CHI was speaking and MOT briefly said “careful” mid-turn. If the intervention is substantial enough to constitute its own turn, transcribe it as a separate *MOT: utterance instead. Model: crates/talkbank-model/src/model/content/other_spoken.rs.

(Note: [^ text] is a freecode, a standalone free-form researcher annotation that sits as its own content item on the main tier (variant of UtteranceContent::Freecode, sibling of Word and Group; it is NOT attached to any word). See grammar/grammar.js rule freecode and crates/talkbank-model/src/model/content/utterance_content/. Used for transcriber notes that are independent of any single word; for notes about a single word use [% text] or [= text] instead.)

Pauses

*CHI:	I (.) want (..) a (...) cookie .
*CHI:	I (1.5) want a cookie .

• (.): short pause
• (..): medium pause
• (...): long pause
• (N.N): timed pause in seconds

Overlap

Overlapping speech between speakers uses angle brackets and overlap markers:

*MOT:	do you want <a cookie> [>] ?
*CHI:	<cookie> [<] !

• [>]: follows the overlap (this speaker started first)
• [<]: overlaps the previous speaker

Retrace and Repetition

Groups followed by retrace markers indicate speech disfluencies:

*CHI:	<I want> [/] I want a cookie .
*CHI:	<I want> [//] I need a cookie .
*CHI:	<I want a> [///] give me a cookie .

• [/]: partial retrace (speaker repeats the same words)
• [//]: full retrace (speaker restarts with different words)
• [///]: multiple retracing (multiple false starts)
• [/-]: reformulation (speaker rephrases with different structure)

The CHAT Word

Status: Current Last modified: 2026-05-29 18:43 EDT

“Word” is the most complex and most misunderstood concept in CHAT. This chapter documents what a word actually is, how the grammar parses it, and how the Rust model represents it. If you maintain this codebase, you will encounter word-level bugs. This chapter exists so you can understand them.

The Fundamental Rule

Whitespace delimits words. Contiguous non-whitespace characters form one word token. This applies everywhere on the main tier.

*CHI:   hello world .
        ^^^^^              word: "hello"
              ^^^^^        word: "world"

The grammar uses extras: $ => [] – no implicit whitespace. Whitespace nodes (whitespaces, space) are explicit in the CST. Tree-sitter does not skip whitespace between tokens. This is the foundation of every tokenization decision in the grammar.

There are no exceptions to this rule. Every ambiguity described in this chapter is resolved by applying this rule consistently.

Word Structure

A word in the grammar is standalone_word – a sequence of an optional prefix, a required body, optional suffixes, and an optional POS tag.

The following diagram shows the full decomposition. All named nodes are separate CST children.

flowchart TD
    sw["standalone_word\n(grammar.js, prec.right 6)"]

    zero["zero\n'0' -- omission prefix"]
    wp["word_prefix\n'&-' filler | '&~' nonword | '&+' fragment"]

    wb["word_body\n(required)"]
    fm["form_marker\n@b, @c, @d, @z:label, ..."]
    wls["word_lang_suffix\n@s, @s:eng, @s:eng+fra"]
    pos["pos_tag\n$n, $v, $adj, ..."]

    sw -->|"optional prefix"| zero & wp
    sw -->|"required"| wb
    sw -->|"optional suffix"| fm & wls
    sw -->|"optional"| pos

    ws["word_segment\npure spoken text"]
    short["shortening\n'(text)' omitted sound"]
    sm["stress_marker\nprimary or secondary"]
    len["lengthening\n':' one or more colons"]
    op["overlap_point\none of four brackets"]
    cae["ca_element\nsingle CA marker"]
    cad["ca_delimiter\npaired CA marker"]
    ub["underline_begin\ncontrol char pair"]
    ue["underline_end\ncontrol char pair"]
    cm["'+'\ncompound marker"]

    wb -->|"children (any order)"| ws & short & sm & len & op & cae & cad & ub & ue & cm

In the grammar (search grammar/grammar.js for the standalone_word and word_body rules), the structure is:

standalone_word: $ => prec.right(6, seq(
  optional(choice($.word_prefix, $.zero)),
  $.word_body,
  optional($.form_marker),
  optional($.word_lang_suffix),
  optional($.pos_tag),
)),

word_body: $ => prec.right(choice(
  seq(
    choice($.word_segment, $.shortening, $.stress_marker),
    repeat(choice($.word_segment, $.shortening, $.stress_marker, $._word_marker)),
  ),
  seq(
    choice($.overlap_point, $.ca_element, $.ca_delimiter, $.underline_begin),
    choice($.word_segment, $.shortening, $.stress_marker),
    repeat(choice($.word_segment, $.shortening, $.stress_marker, $._word_marker)),
  ),
)),

word_body has two branches:

1. Standard start: the word begins with word_segment, shortening, or stress_marker, followed by any number of body children.
2. Marker-initial: the word begins with a structural marker (overlap, CA, underline), but that marker must be immediately followed by text content. This prevents degenerate words like a standalone overlap marker from forming a valid standalone_word.

Lengthening and + (compound marker) are excluded from starting a word body. This is how standalone : falls through to separator(colon) – see Section 5 (Tokenization Ambiguities) below.

The word_segment Purity Invariant

word_segment contains ONLY pure spoken text. All structural markers are separate typed children in word_body, never consumed by word_segment.

This is a hard invariant with three consequences:

1. cleaned_text() never scans for markers. It concatenates Text and Shortening elements. No stripping needed.
2. Validation finds ALL markers by type. Overlap markers, CA elements, and underline pairs are always WordContent variants, regardless of position within the word.
3. Editors get typed CST nodes. Syntax highlighting, bracket matching, and hover info work on individual markers, not opaque substrings.

How it works

word_segment is a DFA token at prec(5) with a regex that excludes all structural characters. The exclusions are generated from the symbol registry (grammar/src/generated_symbol_sets.js) – never hand-written.

word_segment: $ => token(prec(5, seq(
  WORD_SEGMENT_FIRST_RE,    // generated: excludes structural chars + '0' at start
  WORD_SEGMENT_REST_RE,     // generated: excludes structural chars
))),

Full exclusion table

Every character in this table is excluded from word_segment and becomes a separate typed node in the CST.

First-character-only exclusion: 0 is excluded from the first character of word_segment (it is the omission prefix). 0 in non-initial positions is valid: 200, h0me, abc0 all parse correctly.

The Rust Data Model

Word struct

The Word struct (crates/talkbank-model/src/model/content/word/word_type.rs) is the canonical typed representation:

pub struct Word {
    pub span: Span,
    pub word_id: Option<SmolStr>,
    pub(crate) raw_text: SmolStr,
    pub content: WordContents,
    pub category: Option<WordCategory>,
    pub form_type: Option<FormType>,
    pub lang: Option<WordLanguageMarker>,
    pub part_of_speech: Option<SmolStr>,
    pub inline_bullet: Option<Bullet>,
}

Key fields:

• raw_text: the exact text from the input, including all markers. Used for roundtrip serialization.
• content: a WordContents (SmallVec-backed sequence of WordContent elements). This is the structured decomposition. Most words have 1-2 elements; SmallVec avoids heap allocation for the common case.
• category: optional prefix (Omission, CAOmission, Filler, Nonword, PhonologicalFragment).
• form_type: optional @ suffix (@c child-invented, @d dialect, @z:label user-defined, etc.).
• lang: optional @s language marker (Shortcut, Explicit, Multiple, Ambiguous).
• part_of_speech: optional $ tag.

WordContent enum

WordContent (crates/talkbank-model/src/model/content/word/content.rs) is the enum of everything that can appear inside a word body. Each variant maps directly to a grammar node.

cleaned_text()

Word::cleaned_text() derives NLP-ready text from content by concatenating only Text and Shortening variants:

pub fn compute_cleaned_text(&self) -> String {
    let mut result = String::new();
    for item in &self.content {
        match item {
            WordContent::Text(t) => result.push_str(t.as_ref()),
            WordContent::Shortening(s) => result.push_str(s.as_ref()),
            _ => {}
        }
    }
    result
}

This works because the purity invariant guarantees that Text elements never contain structural markers. There is nothing to strip.

Examples:

The result is cached via OnceLock on first access.

What is included in cleaned_text vs what is stripped

The following table is the complete inventory of how every word-internal element contributes to (or is excluded from) cleaned_text(). This must match what NLP pipelines (Stanza, etc.) expect as input.

Characters that stay in word_segment (ARE spoken text):

• Letters (all Unicode)
• Digits (in non-initial position; 0 in initial = omission prefix)
• Hyphen (-), part of word text, e.g., ice-cream, self-conscious
• Apostrophe ('), contractions, e.g., don't, it's
• Hash (#), appears in some transcription conventions
• Underscore (_), compound boundary in some conventions

Characters NOT in word_segment (excluded by symbol registry): See the full exclusion table in Precedence Decisions in the grammar docs.

Comparison with batchalign2

batchalign2’s annotation_clean() (60 lines of .replace() calls) strips all the same characters that our grammar excludes from word_segment. Key differences:

1. Parentheses: ba2 COMMENTED OUT the strip. We handle them as Shortening, the content inside parens IS included in cleaned_text.
2. IPA characters (ạ ā ʔ ʕ ʰ): ba2 incorrectly strips them. We correctly keep them; they are real phonetic content.
3. Hyphen (-): ba2 strips it. We keep it in word_segment because hyphen is a valid word character (contractions, compounds, morphological suffixes in %mor tier).

Our design eliminates the need for character-by-character stripping entirely. cleaned_text() is a simple concatenation of Text + Shortening elements, with zero scanning.

The Six Tokenization Ambiguities

CHAT was designed for human readability, not machine parsing. Six characters have context-dependent meanings that the grammar must disambiguate. Full details with proof grammars are in grammar/docs/tokenization-rules.md and grammar/docs/precedence-decisions.md. What follows is a summary for orientation.

1. Overlap markers (⌈⌉⌊⌋)

Adjacent to text = part of the word. Space-separated = standalone overlap_point.

Yeah⌋⌈2 hey      ONE word: "Yeah⌋⌈2"
Yeah ⌋ ⌈2 hey    three tokens: "Yeah", ⌋, ⌈2

Maximal munch at prec(5) makes word_segment consume adjacent overlap characters. Overlap markers are only recognized as overlap_point when space-separated on both sides.

2. Zero/omission prefix (0)

Adjacent to word body = omission prefix. Space-separated = action marker.

0die              ONE word: standalone_word(zero, word_body("die"))
0 die             TWO tokens: nonword(zero), word("die")

standalone_word at prec.right(6) beats nonword at prec(1). The extras: [] setting prevents whitespace from being skipped between zero and word_body. The zero token is inlined directly into standalone_word (not through word_prefix) because tree-sitter’s precedence does not propagate through intermediate rules. This was proven empirically with a minimal test grammar – see grammar/docs/precedence-decisions.md.

3. CA parenthetical vs shortening

In CA mode (@Options: CA), a fully parenthesized word (word) is an uncertain/omitted word (CAOmission), semantically equivalent to 0word. Partially parenthesized hel(lo) is always a shortening.

@Options: CA
*CHI:   (ja) .            CAOmission: uncertain "ja"
*CHI:   hel(lo) .         Shortening: "(lo)" is the shortened part

Distinguishing these requires file-level context (@Options header). The parser sets WordCategory::CAOmission when the word is fully parenthesized in CA mode. Isolated parser.parse_word_fragment() calls cannot determine CA mode – they need a FragmentSemanticContext.

4. Colon – lengthening vs separator

Inside a word (after text): prosodic lengthening. Standalone: separator.

no::              ONE word: Text("no") + Lengthening(2)
hello : world     separator(colon)

The DFA always produces lengthening for : (higher precedence). But word_body rejects lengthening as a first element, so standalone : cannot form a valid word and falls through to separator(colon). This is the “constrain the parser, not the DFA” pattern.

5. Plus (+) – compound vs terminator vs linker

Inside a word: compound marker. At line end: terminator prefix. At line start: linker prefix.

ice+cream         ONE word with compound marker
and then +...     terminator: trailing_off (prec 10 beats prec 5)
+< but I +/.      linker: lazy_overlap, terminator: interruption

Terminators and linkers use prec(10), which beats word_segment at prec(5). No valid CHAT word ends with + – the grammar enforces this by structure.

6. Bracket annotations vs plain brackets

Bracket annotations ([= text], [=! text], [% text]) use prec(8) prefix tokens to beat generic bracket handling.

Historical Context: The Coarsening and Its Reversal

The original structured grammar (pre-coarsening)

The original grammar (preserved in grammar/docs/pre-coarsening-grammar.js.reference) had all word-internal markers as children of word_content:

// Pre-coarsening: every marker was a child of word_content
word_content: $ => choice(
  $.word_segment,
  $.shortening,
  $.stress,
  $.colon,
  $.caret,
  $.tilde,
  $.plus,
  $.overlap_point,
  $.ca_element,
  $.ca_delimiter,
  $.underline_begin,
  $.underline_end,
),

The coarsening decision

At one point, standalone_word was coarsened into an opaque token – a single DFA regex that consumed the entire word as one undifferentiated string. The rationale was:

• Simpler grammar with fewer tree-sitter conflicts.
• A Chumsky-based direct parser in Rust would re-parse the opaque token into structured WordContent elements.

This worked but had costs:

• Two parsers (tree-sitter + Chumsky) with independent bugs.
• Validation could not find structural markers without re-parsing.
• Editors got one opaque node instead of typed children.
• cleaned_text() had to scan for and strip marker characters.

The reversal (Chumsky elimination)

When the Chumsky direct parser was eliminated (making tree-sitter the sole parser), the structured word grammar was restored. The key decisions:

1. All marker characters were re-excluded from word_segment using the symbol registry as the single source of truth.
2. Each marker type became a separate CST child in word_body.
3. The WordContent enum in the Rust model was aligned 1:1 with the grammar nodes.
4. The word_segment purity invariant was established as a TDD gate.

The result is one parser, one source of truth for exclusions, and typed markers from grammar through model.

Testing: The word_segment Purity Gate

The purity invariant, each structural marker produces a separate CST child rather than being consumed by word_segment, is enforced by a group of tree-sitter corpus tests under grammar/test/corpus/word/. Each *_in_word_lint.txt file embeds a structural marker inside a word and asserts the CST splits the word appropriately:

Underline and stress invariants are covered by corpus tests elsewhere in grammar/test/corpus/ and by the parser-equivalence tests in crates/talkbank-parser-tests/. The historical word_segment_purity.txt consolidated 8 named tests in one file; it was retired in commit fdceeac2 when the corresponding constructs were given their own per-construct test files (this is the new layout that the current spec generators produce from the spec sources).

How to add a new purity-style test

If you add a new structural marker to the grammar:

1. Add its characters to the symbol registry (spec/symbols/symbol_registry.json).
2. Run just symbols-gen to regenerate the exclusion sets.
3. Add a spec in spec/constructs/ that embeds the marker inside a word; regenerate the affected grammar/parser fixtures with the current spec/tools commands from Spec Workflow so a per-construct test fixture is created in grammar/test/corpus/word/. Verify the CST output names each marker as its own child.
4. Run the full verification sequence:

   cd grammar && tree-sitter generate && tree-sitter test
   cargo nextest run -p talkbank-parser
   cargo nextest run -p talkbank-parser-tests -E 'test(parser_equivalence)'
   cargo nextest run -p talkbank-parser-tests --test roundtrip_reference_corpus
   

Key Source Files

Symbols

Status: Reference Last modified: 2026-05-29 18:43 EDT

CHAT uses a rich set of symbols for transcription conventions. This page documents the symbol categories and the symbol registry that drives both the grammar and the Rust crates. The symbol registry (spec/symbols/symbol_registry.json) is the source of truth, when this page and the registry disagree, the registry wins.

Symbol Registry

The authoritative symbol definitions live in spec/symbols/symbol_registry.json. This JSON file is the single source of truth, it generates:

• Character sets for the tree-sitter grammar (grammar.js)
• Rust constants for the model and validation crates
• Validation rules for the spec tool

After any change to the symbol registry, run:

just symbols-gen

Symbol Categories

Terminators

Punctuation that ends an utterance:

CA (Conversation Analysis) Symbols

CA notation symbols fall into three parser-distinct categories in spec/symbols/symbol_registry.json. They are not interchangeable, the grammar treats them as different node kinds.

CA element symbols (ca_element_symbols) attach to a word, so book↑ is a single token whose content carries the symbol:

CA arrow separators (in word_segment_forbidden_start_symbols) are own-node separators between words, not word-attachments. The parser splits them as their own nodes:

CA delimiter symbols (ca_delimiter_symbols) bracket annotated prosodic regions:

Confirm the current contents of each category by reading spec/symbols/symbol_registry.json directly, that is the file just symbols-gen derives the grammar and Rust constants from.

Word Segment Characters

Characters that are forbidden at the start of words, forbidden in the rest of words, or forbidden throughout. These define the lexical boundaries of what constitutes a “word” in CHAT.

The grammar uses these sets to construct the word-matching regex patterns. Characters like [, ], <, >, (, ) are structural delimiters and cannot appear inside words.

Event Segment Characters

Characters forbidden in event descriptions (&=event content). Events have slightly different lexical rules than words.

Language Codes

CHAT uses ISO 639-3 three-letter language codes in @Languages headers and @s: word markers:

@Languages:	eng, fra
*CHI:	I want a croissant@s:fra .

Common codes: eng (English), fra (French), deu (German), spa (Spanish), zho (Mandarin), jpn (Japanese).

Special Markers

@ Markers (Word-Level)

The authoritative form-marker set is FormType in crates/talkbank-model/src/model/content/word/form.rs. Current variants:

The second-language qualifier @s:LANG is a separate construct (see the L2 morphotag section of the Batchalign book); it is not part of FormType.

& Markers (Events and Fillers)

Scope Markers

Architecture Overview

Status: Current Last modified: 2026-06-15 15:00 EDT

TalkBank/chatter is the standalone home of the TalkBank CHAT specification, tree-sitter grammar, Rust crates, chatter CLI, LSP server, and desktop app. It is self-contained: the CHAT-format core builds and runs without any external TalkBank repository, so downstream consumers can depend on its crates directly.

Data Flow

Specification is the source of truth. Code is generated downstream from it.

spec/           Source of truth (CHAT specification)
    ↓
grammar.js      Tree-sitter grammar (in grammar/)
    ↓
parser.c        Generated C parser (never hand-edited)
    ↓
Rust crates     Parser → Model → Validation → Transform
    ↓
Applications    chatter CLI, LSP server, desktop app

Two layers

Within this repository, the architecture splits into two layers:

Source-of-truth artifacts. spec/, spec/symbols/, and grammar/ define the CHAT language and generate downstream parser tests, error docs, and shared symbol sets.

Consumer crates and applications. The Rust crates under crates/, the chatter CLI, talkbank-lsp, and the desktop app all consume those source-of-truth artifacts rather than defining CHAT semantics independently.

Crate Dependency Graph

flowchart TD
    derive["talkbank-derive\nProc macros"]
    model["talkbank-model\nData model, validation, alignment, errors"]
    cache["talkbank-cache\nValidation + roundtrip cache"]
    parser["talkbank-parser\nCanonical parser (tree-sitter)"]
    re2c["talkbank-parser-re2c\nAlternate parser (equivalence oracle)"]
    transform["talkbank-transform\nPipelines, CHAT↔JSON, caching"]
    cli["chatter\nCLI: validate, normalize, convert"]
    lsp["talkbank-lsp\nLanguage Server Protocol"]
    s2c["send2clan\nCLAN app bindings"]
    desktop["chatter-desktop\nDesktop validation app (Tauri)"]
    tests["talkbank-parser-tests\nEquivalence tests"]

    derive --> model
    model --> parser & re2c
    parser --> transform
    re2c --> transform
    cache --> transform
    transform --> cli & lsp & desktop
    s2c --> cli & desktop
    parser --> tests
    re2c --> tests

Repository Layout

chatter/
├── grammar/                Tree-sitter grammar
├── spec/                   CHAT specification (source of truth)
│   ├── constructs/         Valid CHAT examples + expected parse trees
│   ├── errors/             Invalid CHAT examples + expected error codes
│   ├── symbols/            Shared symbol registry (JSON)
│   ├── tools/              Core spec generators
│   └── runtime-tools/      Runtime-aware spec bootstrap/validation tools
├── crates/                 Rust crates (model, parser, transform, CLI support, LSP)
├── corpus/                 Reference corpus
├── tests/                  Integration tests and fixtures
├── schema/                 JSON Schema (auto-generated)
├── apps/chatter-desktop/   Desktop validation app (Tauri v2, React)
├── fuzz/                   cargo-fuzz targets (separate workspace)
├── book/                   This documentation
└── docs/                   Strategy, proposals, investigations

Cargo Workspaces

Three separate Cargo workspaces live here:

1. Root workspace (Cargo.toml), all Rust crates for parsing, model, transform, CLI, LSP, and apps/chatter-desktop/src-tauri.
2. Spec workspace (spec/Cargo.toml), spec/tools for core generation, spec/runtime-tools for runtime-aware spec tooling.
3. Fuzz workspace (fuzz/Cargo.toml), cargo-fuzz targets for parser and validation robustness checks.

Use the relevant manifest path for the workspace you mean to operate in:

• spec/tools/Cargo.toml for generators
• spec/runtime-tools/Cargo.toml for bootstrap/mining/runtime validation
• fuzz/Cargo.toml for cargo-fuzz targets

Where to read next

For per-topic detail (sections being consolidated; see SUMMARY for the authoritative current list):

• Spec System, Grammar, Parser Backends, how CHAT becomes typed AST.
• CHAT model: the AST itself, content traversal, wide-struct rule.
• Alignment: tier alignment, DP, sequence alignment.
• Errors and validation: diagnostics, validation gates, and parser/model invariants.
• Editor/runtime integration: talkbank-lsp and application boundaries layered on top of the CHAT core.
• Memory and Ownership, Type-Driven Design (lands during M11 errors-and-validation work).
• XML Emitter: projection.

For per-crate summaries see Crate Reference.

Spec System

Status: Current Last modified: 2026-05-29 17:50 EDT

Specifications in spec/ are the authoritative source of truth for the CHAT format. They drive grammar artifact generation, validation/error docs, and targeted test generation.

Historical note: This system was originally shaped during a dual-parser era. The chumsky-based direct parser was removed in March 2026. Today the canonical parser is tree-sitter (talkbank-parser); a second implementation, talkbank-parser-re2c, exists as a specification oracle and high-throughput batch parser. Fragment specs remain valuable, but synthetic tree-sitter wrapper behavior is audit-only legacy unless a page or test explicitly says otherwise.

Spec Types

Construct Specs (spec/constructs/)

Each construct spec defines a valid CHAT pattern with its expected parse tree:

# example_name

Description of what this example tests.

## Input

\```mor_dependent_tier
%mor:	VERB|eat .
\```

## Expected CST

\```cst
(mor_dependent_tier
  (mor_tier_prefix)
  ...)
\```

## Metadata

- **Level**: tier
- **Category**: tiers

The Input code fence label (e.g., mor_dependent_tier, utterance) selects which template wraps the fragment into a full CHAT file for parsing.

That is an explicit grammar/test templating mechanism. It is useful, but it does not by itself define honest isolated-fragment semantics for the direct parser.

Error Specs (spec/errors/)

Each error spec defines an invalid CHAT pattern with expected error codes:

# Error E301

## Metadata

- Code: E301
- Name: missing_participants
- Severity: Error
- Layer: parser

## Examples

### missing_participants_1

\```chat
@UTF8
@Begin
*CHI: hello .
@End
\```

Key metadata fields:

• Layer: parser: error caught during parsing (returns Err)
• Layer: validation: error caught after successful parse
• Status: not_implemented: generates #[ignore] tests

Symbol Registry (spec/symbols/)

symbol_registry.json defines character sets used by both the grammar and Rust crates. In this repo, just symbols-gen validates the registry and regenerates the checked-in grammar and Rust symbol-set outputs. The generation step produces:

• JavaScript constants for grammar.js
• Rust constants for model validation

Test Generation

The predecessor monorepo used make test-gen as shorthand for three generator classes. That root wrapper is not yet ported into this repo, but the underlying generation responsibilities are still:

1. Tree-sitter Corpus Tests

gen_tree_sitter_tests reads construct specs and error specs, then:

• Wraps each Input in a template to create a full CHAT file
• Parses with tree-sitter and checks for error nodes
• Writes Expected CST to grammar/test/corpus/

For error specs, it captures the actual parse (with ERROR nodes) as the expected tree.

2. Rust Tests

gen_rust_tests generates Rust test functions:

• Construct specs become parse-and-compare tests
• Parser-layer error specs become parser.parse_chat_file() tests expecting Err
• Validation-layer error specs become parse-then-validate tests

Output: crates/talkbank-parser-tests/tests/generated/

The generated suites are useful as grammar/audit support and regression coverage, but they are not the sole authority for parser semantics.

3. Error Documentation

gen_error_docs generates optional local markdown pages for each error code under docs/errors/ when maintainers want a browsable reference set while working on diagnostics. The source of truth remains spec/errors/.

Workflow After Spec Changes

1. Regenerate only the affected spec-driven artifacts using the current commands documented in spec/CLAUDE.md.
2. Run the concrete verification commands from Contributing > Setup.

Never hand-edit generated artifacts, always regenerate from specs.

Post-Bootstrap Doctrine

• spec/tools remains the generator/validator for grammar corpus tests, error docs, and shared symbol artifacts.
• talkbank-parser-tests owns parser equivalence and roundtrip contracts.
• Isolated grammar additions should usually need two things: one grammar corpus example and one full-file fixture. They should not require the old bootstrap ritual unless generated artifacts really changed.

Grammar

Status: Current Last updated: 2026-03-24 00:01 EDT

The CHAT grammar is defined in grammar/grammar.js using the tree-sitter parser generator. It produces a GLR parser that handles the full CHAT format with error recovery.

Design Principles

Explicit Whitespace

Unlike most tree-sitter grammars, CHAT does not use extras for whitespace. All whitespace is grammar-visible because CHAT’s structure is whitespace-sensitive:

• Tab separates tier prefix from content
• Newline ends tiers
• Line continuation uses tab-at-start-of-line
• Space separates words and annotations

Two-Level Structure

The grammar has two structural levels:

1. Document level: headers, utterances, @Begin/@End
2. Tier level: main tier content, dependent tier content (each with distinct rules)

Opaque Lemmas

In the %mor tier rules, lemmas are parsed as opaque Unicode strings. The grammar does not attempt to decompose lemma content, that happens in the model layer. This follows the “parse, don’t validate” principle.

Key Grammar Rules

Document Structure

document → utf8_header, begin_header, lines..., end_header
line → header | utterance
utterance → main_tier, dependent_tiers...

Main Tier

main_tier → star, speaker, colon, tab, tier_body
tier_body → contents, utterance_end
contents → content_item, (whitespace, content_item)...

MOR Tier (UD-style)

mor_contents → mor_content, (whitespace, mor_content)..., terminator
mor_content → mor_word, mor_post_clitic*
mor_word → mor_pos, pipe, mor_lemma, mor_feature*
mor_post_clitic → tilde, mor_word
mor_feature → hyphen, mor_feature_value

POS tags are simple identifiers (no subcategories). Lemmas are opaque strings. Features are hyphen-separated values that may contain = for Key=Value pairs and , for multi-value features.

Grammar Change Workflow

parser.c is generated from grammar.js, never edit it directly.

After any change to grammar.js:

1. cd grammar && tree-sitter generate
2. tree-sitter test (160 tests)
3. cargo test -p talkbank-parser
4. cargo nextest run -p talkbank-parser-tests (reference corpus equivalence, per-file)
5. Verify the 78-file reference corpus passes at 100%

Conflict Resolution

The grammar uses tree-sitter’s precedence and conflict mechanisms to handle ambiguities:

• Word tokens use prec(5) to win over separators
• Inline bullets use prec(10) for their delimiters
• CA (conversation analysis) symbols use prec(3) for colon disambiguation

Generated Artifacts

Running tree-sitter generate produces:

• src/parser.c: the C parser
• src/node-types.json: node type metadata

The Rust crate talkbank-parser references node-types.json to generate node_types.rs (a generated constants file).

Parsing

Status: Current Last updated: 2026-05-19 16:54 EDT

The parsing pipeline converts CHAT text into a typed ChatFile AST. The default and canonical parser is the tree-sitter parser (talkbank-parser). A second implementation, talkbank-parser-re2c, exists alongside it as a specification oracle and high-throughput batch parser; it produces the same ChatFile model and is opt-in via chatter validate --parser re2c. The LSP and all production paths use the tree-sitter parser.

Tree-Sitter Parser

The talkbank-parser crate wraps the tree-sitter C parser and converts its concrete syntax tree (CST) into the ChatFile model.

Full-file parsing is the canonical entry point. TreeSitterParser also provides fragment methods (parse_word_fragment(), parse_main_tier_fragment(), parse_chat_file_fragment(), etc.) for parsing isolated CHAT fragments directly.

CST → AST Pipeline

flowchart LR
    chat["CHAT text\n(.cha file)"]
    grammar["tree-sitter grammar\n(grammar.js → parser.c)"]
    cst["Concrete Syntax Tree\n(all whitespace preserved)"]
    walker["TreeSitterParser\n(CST traversal)"]
    ast["ChatFile AST\n(semantic model)"]

    chat --> grammar --> cst --> walker --> ast

Source text
    ↓ tree-sitter parse
Concrete Syntax Tree (CST), green tree with all tokens
    ↓ tree_parsing (Rust)
ChatFile AST, typed model with validation-ready data

The CST preserves every character of the source (whitespace, punctuation, comments). The Rust tree-parsing modules walk the CST and extract semantic information into the typed model.

Error Recovery

Tree-sitter’s GLR algorithm provides automatic error recovery. When the parser encounters unexpected input, it:

1. Inserts ERROR nodes in the CST
2. Continues parsing the rest of the file
3. Reports parse errors via the ErrorSink trait

This means the parser always produces a result, even for malformed files, it extracts as much structure as possible.

ParseOutcome

Individual parse functions return ParseOutcome<T>:

• ParseOutcome::parsed(value): successfully parsed
• ParseOutcome::rejected(): could not parse this node (error already reported)

This allows the parser to skip individual malformed elements while continuing to parse the rest of the file.

Parser Equivalence

The 78-file reference corpus is the primary correctness guarantee:

cargo nextest run -p talkbank-parser-tests -E 'test(parser_equivalence)'

Each .cha file is its own test, nextest runs them in parallel and reports individual failures.

TreeSitterParser API

TreeSitterParser is the sole API handle for parsing. Callers create one instance and pass &TreeSitterParser to all parsing call sites. There is no trait abstraction, TreeSitterParser is a concrete type in the talkbank-parser crate.

use talkbank_parser::TreeSitterParser;

let parser = TreeSitterParser::new()?;

// Full-file parsing (methods on TreeSitterParser).
// parse_chat_file returns ParseResult<ChatFile> with the diagnostic list
// embedded in the result envelope.
let chat_file = parser.parse_chat_file(&source)?;
// parse_chat_file_streaming pushes diagnostics into an ErrorSink as it
// goes, useful for very large files or LSP-style incremental flows.
let chat_file = parser.parse_chat_file_streaming(&source, &errors);

// Fragment parsing (methods on TreeSitterParser), used when synthesizing
// CHAT from non-CHAT sources (ASR output, UD annotations).
let word = parser.parse_word_fragment(word_text, &errors);
let main_tier = parser.parse_main_tier_fragment(tier_text, &errors);

AST Structure

The resulting ChatFile AST has a recursive content structure:

flowchart TD
    cf["ChatFile"]
    hdr["Headers\n@Languages, @Participants,\n@ID, @Options"]
    utts["Utterances[]"]
    mt["MainTier\nspeaker + content"]
    dt["DependentTiers[]\n%mor, %gra, %pho, %sin, %wor"]
    uc["UtteranceContent\n24 variants"]
    leaf["Leaves\nWord | ReplacedWord | Separator"]
    group["Groups\nGroup | AnnotatedGroup |\nRetrace | PhoGroup | SinGroup | Quotation"]

    cf --> hdr & utts
    utts --> mt & dt
    mt --> uc
    uc --> leaf & group
    group -->|recurse| uc

Parser String Handling

The tree-sitter parser constructs owned model types (e.g., MorWord, GrammaticalRelation) directly from CST text. String-heavy types like PosCategory and MorStem use Arc<str> interning to avoid redundant allocations for repeated values. Short strings in model newtypes use SmolStr for inline storage up to 23 bytes.

CHAT Data Model

Status: Current Last updated: 2026-06-14 19:57 EDT

The talkbank-model crate defines the typed AST for CHAT files. Every other crate, parser, transform, CLAN, CLI, LSP, and the entire batchalign runtime, depends on it. This page describes the model itself, the three-level content hierarchy, the content-walker primitives, and the extract → infer → inject pattern that all NLP tasks follow.

ChatFile

The root type is ChatFile, representing a complete CHAT transcript:

pub struct ChatFile {
    pub lines: Vec<Line>,
    pub participants: IndexMap<SpeakerCode, Participant>,
    pub languages: LanguageCodes,
    pub options: ChatOptionFlags,
}

Each Line is either a Header or an Utterance. The full ownership tree:

flowchart TD
    chatfile["ChatFile\n(talkbank-model/src/model/file/chat_file/core.rs)"]
    chatfile --> lines["lines: Vec<Line>"]
    chatfile --> participants["participants:\nIndexMap<SpeakerCode, Participant>"]
    chatfile --> languages["languages: LanguageCodes"]
    chatfile --> options["options: ChatOptionFlags"]

    lines --> header_line["Line::Header (Header)"]
    lines --> utt_line["Line::Utterance (Utterance)"]

    utt_line --> preceding["preceding_headers:\nSmallVec<Header>"]
    utt_line --> main["main: MainTier"]
    utt_line --> deptiers["dependent_tiers:\nVec<DependentTier>"]
    utt_line --> health["parse_health: ParseHealthState"]

    main --> speaker["speaker: SpeakerCode"]
    main --> tiercontent["content: TierContent"]
    tiercontent --> linkers["linkers: Vec<Linker>"]
    tiercontent --> uttcontent["utterance_content:\nVec<UtteranceContent>\n(24 variants)"]
    tiercontent --> terminator["terminator: Option<Terminator>"]
    tiercontent --> bullet["bullet: Option<Bullet>"]

The DependentTier enum has 25 variants: structured linguistic (Mor/Gra/Pho/Mod/Sin/Act/Cod/Wor), with-inline-bullets (Add/Com/Exp/Gpx/Int/Sit/Spa), text-only (Alt/Coh/Def/Eng/Err/Fac/Flo/Gls/Ort/Par/Tim), Phon-project (Modsyl/Phosyl/Phoaln), and UserDefined / Unsupported.

Three-Level Content Hierarchy

CHAT main-tier content is a tree with three nesting levels. Every content traversal must understand all three.

ChatFile
└── Line::Utterance
    └── MainTier
        └── TierContent
            ├── content: Vec<UtteranceContent>     ← Level 1
            │   ├── Word(Box<Word>)
            │   │   └── content: Vec<WordContent>  ← Level 3
            │   ├── OverlapPoint(OverlapPoint)
            │   ├── Group(Group)
            │   │   └── BracketedContent
            │   │       └── Vec<BracketedItem>     ← Level 2
            │   ├── PhoGroup, SinGroup, Quotation
            │   │   └── (same BracketedContent)
            │   ├── Retrace(Box<Retrace>)
            │   ├── Pause, Event, Separator, ...
            │   └── AnnotatedWord, AnnotatedGroup, ...
            ├── bullet: Option<Bullet>
            ├── linkers: Linkers
            └── terminator: Terminator

Level 1, UtteranceContent (24 variants)

What you iterate when walking utterance.main.content.content.0:

Critical rule: every match on UtteranceContent must explicitly list all 24 variants. No _ => catch-alls. Project policy: silent data loss when new variants are added is unacceptable.

Level 2, BracketedItem (22 variants)

Content inside groups (<...>, ‹...›, 〔...〕, "..."). Accessed via group.content.content.0 (the double .content.content.0 is not a typo, Group.content is BracketedContent, which has .content: BracketedItems, which has .0: Vec<BracketedItem>).

BracketedItem mirrors UtteranceContent closely. Retrace content (<word word> [/], word [//]) is a dedicated Retrace variant at both levels, not hidden inside AnnotatedGroup. Groups can nest arbitrarily deep.

Level 3, WordContent (11 variants)

Content inside a single word token, accessed via word.content:

Key insight: overlap markers can appear at all three levels, as standalone UtteranceContent::OverlapPoint (space-separated: ⌈ word ⌉), as BracketedItem::OverlapPoint (inside groups), or as WordContent::OverlapPoint (intra-word: butt⌈er⌉). Any traversal looking for overlap markers must check all three levels.

Annotated Wrappers and Replaced Words

Annotated<T>

Adds scoped annotations ([/], [* m], [= explanation], etc.) to any annotatable inner type:

pub struct Annotated<T> {
    pub inner: T,
    pub annotations: Vec<ContentAnnotation>,
    pub span: Span,
}

At Level 1: AnnotatedWord(Box<Annotated<Word>>), AnnotatedGroup(Annotated<Group>), AnnotatedEvent(Annotated<Event>), AnnotatedAction(Annotated<Action>). Same variants exist at Level 2.

ReplacedWord

Represents word [: replacement], a surface form with a replacement:

pub struct ReplacedWord {
    pub word: Word,
    pub replacement: Replacement,
}
pub struct Replacement {
    pub words: Vec<Word>,
}

Convention when extracting words for NLP: use replacement words if non-empty, else the surface form (Wor and Mor domains both follow this, each with its own counts_for_tier filter).

Tier Domains

Different NLP tasks need different views of the same content. The TierDomain enum controls which words count for each tier and how groups are traversed:

The content walker takes Option<TierDomain>: Some(domain) for domain-aware gating, None to recurse everything unconditionally.

Content Walkers

talkbank-model exports closure-based walkers. Two layers:

• walk_content: generic, visits all content items (custom traversals).
• walk_words / walk_words_mut, filtered to words / replaced words / separators, with domain-aware gating. The primary primitive.

use talkbank_model::alignment::helpers::{
    walk_words, walk_words_mut,
    WordItem, WordItemMut,
    TierDomain,
};

walk_words(content, Some(TierDomain::Wor), &mut |leaf| {
    match leaf {
        WordItem::Word(word) => { /* ... */ }
        WordItem::ReplacedWord(replaced) => { /* ... */ }
        WordItem::Separator(sep) => { /* ... */ }
    }
});

flowchart TD
    input["&[UtteranceContent]\n+ domain: Option<TierDomain>"]
    dispatch["Match variant\n(24 UtteranceContent variants)"]
    word["Word → emit WordItem::Word"]
    rw["ReplacedWord → emit WordItem::ReplacedWord"]
    sep["Separator → emit WordItem::Separator"]
    group["Group / AnnotatedGroup /\nPhoGroup / SinGroup / Quotation"]
    gate{"Domain\ngating"}
    skip["Skip\n(atomic unit)"]
    recurse["Recurse into\ngroup.content"]

    input --> dispatch
    dispatch --> word & rw & sep & group
    group --> gate
    gate -->|"Mor: skip retraces"| skip
    gate -->|"Pho/Sin: skip groups"| skip
    gate -->|"None: recurse all"| recurse
    recurse -->|back| dispatch

What walk_words does NOT visit

Only words and separators. Not OverlapPoint (any level), not CAElement within words, not events / pauses / actions, not internal bullets. For these, write a custom traversal, see talkbank-model/validation/utterance/overlap.rs for the reference pattern.

walk_overlap_points, overlap marker iterator

walk_overlap_points(content, &mut |visit| {
    // visit.point.kind, visit.point.index, visit.word_position
});

Visits every OverlapPoint at all three content levels with its word-position context. Used by the alignment pipeline (onset estimation) and the validator (pairing checks). For region-level analysis (pairing ⌈ with ⌉ by index), use extract_overlap_info() which builds OverlapRegion structs. For whole-file analysis, analyze_file_overlaps() matches top regions (⌈) with bottom regions (⌊) across utterances with 1:N support (used by E347 and chatter debug overlap-audit).

Validation

Beyond what the grammar enforces, validate_with_alignment() checks semantic constraints:

• %mor alignment: number of MOR items matches alignable main-tier words.
• %gra structure: sequential indices, ROOT checks, circular dependency.
• Header consistency: @ID codes match @Participants.
• Speaker references: all *SPEAKER: codes declared.

Five parallel alignment flows are computed against the main tier:

flowchart TD
    main["MainTier content"]
    walker["walk_words()\ncount alignable words"]

    subgraph "5 Parallel Alignment Flows"
        mor["%mor\ncustom logic\n(clitic handling)"]
        pho["%pho\npositional_align()\n(skip PhoGroup)"]
        sin["%sin\npositional_align()\n(skip SinGroup)"]
        wor["%wor\npositional_align()\n(LCS diff format)"]
        gra["%gra\nalign to %mor chunks\n(not main tier)"]
    end

    main --> walker
    walker --> mor & pho & sin & wor
    mor --> gra

For the alignment algorithms themselves, see Alignment.

Common Pitfalls

1. “Consecutive” means in-order traversal, not adjacent array indices. When CHAT tools speak of “consecutive” or “sequential” items on the main tier, this always means document order via recursive traversal, accounting for groups (<...>), retrace groups (<...> [/]), quotations ("..."), and all other bracketed structures. Never check adjacency in the flat Vec<UtteranceContent>, use walk_words or equivalent in-order traversal.
2. Missing intra-word content. Overlap markers, CA elements, and other markers can appear inside Word.content. Checking only UtteranceContent::OverlapPoint misses WordContent::OverlapPoint (e.g., butt⌈er⌉, a⌈nd).
3. Missing annotated variants. UtteranceContent::AnnotatedWord and AnnotatedGroup wrap inner types in Annotated<T> and are easy to forget.
4. BracketedContent access. Group.content → BracketedContent, with .content: BracketedItems, with .0: Vec<BracketedItem>.
5. Separator counter sync (Mor domain). Tag-marker separators (, „ ‡) count as NLP words because they have %mor items. Any code counting words in the Mor domain must count these separators too.

Serialization

• CHAT: WriteChat trait writes any model type back to CHAT format.
• JSON: all model types implement Serialize/Deserialize. Format per the JSON Schema.
• JSON Schema: derived via JsonSchema. Run cargo test --test generate_schema to regenerate schema/chat-file.schema.json.

Memory and Interning

String-heavy types (PosCategory, MorStem, MorFeature) use Arc<str> with a global interner, significant memory savings on large corpora where the same POS tags and lemmas appear thousands of times.

Collections that are typically small use SmallVec for inline storage:

• SmallVec<[MorFeature; 4]>: features per word (usually 0-4).
• SmallVec<[MorWord; 2]>: post-clitics (usually 0-1).

Transform Pipeline

Status: Current Last updated: 2026-06-14 19:57 EDT

The talkbank-transform crate provides high-level pipelines that compose parsing, validation, and serialization into reusable workflows.

Core Pipelines

Parse + Validate

The most common pipeline: parse a CHAT file and validate it.

use talkbank_transform::parse_and_validate;

let result = parse_and_validate(source, &parser, &error_collector);

This:

1. Parses the source text into a ChatFile AST
2. Runs validation (alignment checks, header consistency, etc.)
3. Collects all errors and warnings into the ErrorSink

CHAT → JSON

Convert a CHAT file to its JSON representation:

use talkbank_transform::chat_to_json;

let json = chat_to_json(source, &parser)?;

The JSON follows the schema at schema/chat-file.schema.json.

JSON → CHAT

The JSON produced by chat_to_json is schema-conformant and round-trips. Deserialize it back into a ChatFile with serde_json (the model derives Deserialize), then serialize through WriteChat to reproduce CHAT text:

let chat_file: talkbank_model::ChatFile = serde_json::from_str(json_str)?;
let chat_text = chat_file.to_chat_string();

The chatter from-json command wraps this path (crates/chatter/src/commands/json.rs, json_to_chat).

CHAT → CHAT (Normalize)

Parse and reserialize to normalize formatting:

use talkbank_transform::normalize_chat;

let normalized = normalize_chat(source, &parser)?;

normalize_chat lives in crates/talkbank-transform/src/pipeline/convert.rs.

Validation + Roundtrip Cache Lifecycle

The following diagram shows the full validation and roundtrip pipeline, including the cache layer:

flowchart TD
    file["CHAT file"]
    cache{"Cache\nhit?"}
    parse["Parse\n(tree-sitter → AST)"]
    validate["Validate\n(per-file → per-utterance →\nmain tier → dependent tiers)"]
    rt{"Roundtrip\nflag?"}
    ser1["Serialize → CHAT text"]
    reparse["Reparse CHAT text"]
    ser2["Serialize again"]
    cmp{"Two\nserializations\nmatch?"}
    store["Store in cache\n(SQLite)"]
    pass["Pass"]
    fail["Fail"]
    cached["Return cached result"]

    file --> cache
    cache -->|miss| parse --> validate --> rt
    cache -->|hit| cached
    rt -->|yes| ser1 --> reparse --> ser2 --> cmp
    rt -->|no| store --> pass
    cmp -->|yes| store
    cmp -->|no| fail

Streaming Parse

For large files or interactive use, the transform crate supports streaming parse where utterances are processed incrementally rather than loading the entire AST into memory.

Caching

The transform layer integrates with a file-system cache. Validation results are keyed by content hash, so unchanged files skip re-validation. Cache location is platform-specific: ~/Library/Caches/talkbank-chat/ (macOS), ~/.cache/talkbank-chat/ (Linux), %LocalAppData%\talkbank-chat\ (Windows).

Use --force to bypass the cache for specific paths.

Error Collection

Pipelines use the ErrorSink trait for error reporting. Callers can provide:

• A collecting sink (gathers all diagnostics for batch output)
• A printing sink (writes diagnostics to stderr in real-time)
• A custom sink (for LSP diagnostics, JSON output, etc.)

Merge Pipeline, Domain Types

Status: Draft Last modified: 2026-05-29 18:43 EDT

This page specifies the typed Rust vocabulary shared by chatter merge, chatter speaker-id, the override-file reader/writer, and any future adjudication tooling (CLI, VS Code, web). Documenting these types before writing the implementing code is deliberate: the types are the spec, and they need to be designed against the user contract in chatter merge and chatter speaker-id without being inferred from prototype code.

The design follows the cross-cutting rules in this repo’s root CLAUDE.md: newtypes over primitives at every stable boundary; no boolean blindness; no tuple-packed seams; typed errors via thiserror; deterministic BTreeMap/BTreeSet over hash maps for serialized state.

Where the types live

All new types live in talkbank-model::merge. Rationale:

• Existing CHAT-domain types (SpeakerCode, ParticipantRole, ParticipantEntry, IDHeader, ChatFile) already live in talkbank-model; the new merge-pipeline types reference them pervasively and benefit from being co-located.
• Consumers outside talkbank-transform (a future override-file reader in a small CLI, an adjudication UI, an orchestrator script’s Rust port) want the types without pulling in the tree-sitter parser, the DP-aligner, etc. talkbank-model is the lightweight type-and-validation crate that fits.
• If talkbank-model::merge grows past the file-size budget (≤400 lines per file, ≤800 hard) we split into submodules (merge::override_file, merge::scoring, etc.), same crate. Hoisting to a separate talkbank-merge-types crate is a future option but not pre-emptively warranted.

Existing types reused (not redefined)

None of these are redefined; the merge module imports and references them.

New types (specification)

JaccardScore

A multiset-Jaccard similarity value, by construction in the closed range [0.0, 1.0].

/// Multiset Jaccard similarity between two bags of tokens.
///
/// By construction in [0.0, 1.0]. `JaccardScore::zero()` is the
/// no-overlap point; `JaccardScore::one()` is identical-bag.
///
/// Used by the speaker-id stage to score how well each donor
/// speaker matches a reference anchor's content.
#[derive(Clone, Copy, Debug, PartialEq, PartialOrd, Serialize, Deserialize, JsonSchema)]
#[serde(try_from = "f64", into = "f64")]
pub struct JaccardScore(f64);

impl JaccardScore {
    pub fn new(v: f64) -> Result<Self, JaccardScoreError>;
    pub fn zero() -> Self;
    pub fn one() -> Self;
    pub fn value(self) -> f64;
}

impl Display for JaccardScore { /* "0.735" three-digit */ }
impl TryFrom<f64> for JaccardScore { /* validates range */ }
impl From<JaccardScore> for f64 { /* infallible widen */ }

Construction is fallible: JaccardScore::new(1.5) returns Err(JaccardScoreError::OutOfRange(1.5)). NaN is also rejected. Internal computation that’s guaranteed in-range by construction (the multiset formula) uses an internal from_unchecked private constructor; public API is fallible.

ConfidenceThreshold

The minimum Jaccard margin (winner / loser) the speaker-id stage will auto-accept. By construction in [1.0, ∞), a threshold of < 1.0 makes no sense (means the loser scores higher than the winner, which can’t happen). Default 2.0 per the empirical calibration recorded in chatter speaker-id.

#[derive(Clone, Copy, Debug, PartialEq, PartialOrd, Serialize, Deserialize, JsonSchema)]
#[serde(try_from = "f64", into = "f64")]
pub struct ConfidenceThreshold(f64);

impl ConfidenceThreshold {
    pub const DEFAULT: Self = Self(2.0);
    pub fn new(v: f64) -> Result<Self, ConfidenceThresholdError>;
    pub fn value(self) -> f64;
}

impl Default for ConfidenceThreshold {
    fn default() -> Self { Self::DEFAULT }
}

Margin

The decisive ratio between the highest-scoring speaker and the runner-up. Distinguished from ConfidenceThreshold by intent (this is observed; the threshold is configured) and from JaccardScore by range (margin is ≥ 1.0; score is ≤ 1.0).

Uses an enum rather than a bare float to model the divide-by-zero case (runner-up has zero Jaccard) cleanly. Avoids the f64::INFINITY sentinel that doesn’t round-trip through all serializers.

/// Ratio of winning speaker's score to runner-up's score.
///
/// `Finite(r)` for `r >= 1.0`. `Unbounded` when the runner-up
/// has zero score (winner scored anything, runner-up scored
/// nothing). Compares meaningfully against `ConfidenceThreshold`
/// regardless of variant.
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(untagged)]
pub enum Margin {
    Finite(f64),
    /// Serialized as the JSON/TOML string "unbounded"; never as
    /// f64::INFINITY (which round-trips inconsistently).
    Unbounded,
}

impl Margin {
    pub fn from_scores(winner: JaccardScore, loser: JaccardScore) -> Self;
    pub fn meets(self, threshold: ConfidenceThreshold) -> bool;
}

impl Display for Margin { /* "3.81x" or "∞" */ }

RetainSet

The set of speaker codes specified by --retain on chatter merge. A BTreeSet<SpeakerCode> wrapped in a newtype so the type signatures of merge functions communicate intent. Empty is allowed (means “no speakers come from File 1; File 1 contributes only headers”, a degenerate but legal case).

/// Speakers whose utterances come from the first input to
/// `chatter merge`. All other speakers come from the second
/// input.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
pub struct RetainSet(BTreeSet<SpeakerCode>);

impl RetainSet {
    pub fn new() -> Self;
    pub fn from_iter<I: IntoIterator<Item = SpeakerCode>>(it: I) -> Self;
    pub fn contains(&self, code: &SpeakerCode) -> bool;
    pub fn iter(&self) -> impl Iterator<Item = &SpeakerCode>;
    pub fn is_empty(&self) -> bool;
}

impl FromStr for RetainSet {
    type Err = RetainSetParseError;
    /// Parses `"CHI,SI2"` → `{CHI, SI2}`. Empty entries rejected.
    fn from_str(s: &str) -> Result<Self, Self::Err>;
}

InsertedRole

The CHAT code + role-tag pair to assign to renamed speakers in the speaker-id stage. A struct rather than two function arguments because the pair is meaningful as a unit (in TOML override files it serializes as a nested table; in CLI it parses as CODE:TAG).

/// The CHAT identity to assign to non-anchor speakers in the
/// speaker-id stage. Example: `INV:Investigator`, `MOT:Mother`.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
pub struct InsertedRole {
    pub code: SpeakerCode,
    pub tag: ParticipantRole,
}

impl InsertedRole {
    pub fn investigator() -> Self;    // INV:Investigator
    pub fn mother() -> Self;          // MOT:Mother
    pub fn father() -> Self;          // FAT:Father
    pub fn adult() -> Self;           // PAR:Adult
}

impl FromStr for InsertedRole {
    type Err = InsertedRoleParseError;
    /// Parses `"INV:Investigator"`. Both halves required.
    fn from_str(s: &str) -> Result<Self, Self::Err>;
}

impl Display for InsertedRole { /* "INV:Investigator" */ }

The convenience constructors (investigator(), mother(), etc.) are the closed-set anchor points; arbitrary InsertedRole { code, tag } is also allowed for contributor-specific roles.

MappingAction

What happens to a particular speaker in the input under a SpeakerMapping. Enum (not boolean) to avoid blindness and to leave room for future variants (e.g. RenameTo { code, tag } when multi-role renaming becomes a need).

/// Action to apply to one speaker in a SpeakerMapping.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "lowercase")]
pub enum MappingAction {
    /// Remove this speaker's utterances and its @Participants /
    /// @ID rows entirely.
    Drop,
    /// Rename this speaker to the mapping's `inserted_role.code`.
    /// Rewrites speaker codes on every utterance and the
    /// corresponding @Participants and @ID entries.
    Rename,
}

The TOML serialization uses "drop" / "rename" lowercase strings, matching the override-file format documented in speaker-id.md.

SpeakerMapping

The decision record produced by the speaker-id stage and consumed by the speaker-id apply step. Carries enough information to apply deterministically to a ChatFile.

/// A decision about how to relabel a ChatFile's speakers.
///
/// Produced by `identify_mapping` (reference mode, auto), by the
/// `--mapping` flag parser (explicit mode), or by reading an
/// override-file entry (override mode). Consumed by `apply_mapping`,
/// which rewrites a ChatFile per the assignments.
///
/// All speakers in the input must appear as keys in `assignments`
///, no defaulting. This is a precondition checked at apply time
/// and is intentional (we want every decision to be explicit).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SpeakerMapping {
    /// The CHAT identity assigned to every speaker whose action
    /// is `MappingAction::Rename`. All renamed speakers go to
    /// the same role in v1 of this schema.
    pub inserted_role: InsertedRole,

    /// Per-speaker action. Use BTreeMap for deterministic
    /// serialization order.
    pub assignments: BTreeMap<SpeakerCode, MappingAction>,
}

The “single inserted_role across all renamed speakers” constraint matches the doc and keeps the most-common case clean. Future multi-role-rename use cases (a 3-speaker file where two get different roles) extend MappingAction with a RenameTo variant rather than changing this struct’s shape.

DecisionMode

How a MergeOverride entry came to exist. Three variants matching the three speaker-id operation modes.

#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "lowercase")]
pub enum DecisionMode {
    /// Reference-mode auto-decide with Jaccard above threshold.
    Auto,
    /// Operator supplied --mapping directly on a one-off run.
    Explicit,
    /// Read from a prior override-file entry; this is a replay.
    Override,
}

MergeFlag

Extensible operator-supplied flags on an override entry. Closed variants for known cases plus a Custom(String) escape hatch.

#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "kebab-case")]
pub enum MergeFlag {
    /// ASR diarization mixed multiple real-world roles into one
    /// speaker label. The rename may still be the best available
    /// approximation but the output is imperfect.
    DiarizationMixed,
    /// The operator could not confidently determine which speaker
    /// is which; mapping is best-guess.
    BestGuess,
    /// Open variant for contributor-specific flag vocabulary.
    /// Serializes as the inner string verbatim.
    #[serde(untagged)]
    Custom(String),
}

OperatorId

Who made the decision. String newtype.

string_newtype!(
    /// Identifier of the operator who created an override entry.
    /// Free-form; typically a username or initials. Recorded as
    /// audit trail.
    pub struct OperatorId;
);

SessionId

Identifies an entry within an override file. Typically the basename stem of the input CHAT file, but the override-file schema doesn’t constrain its shape, contributors may use any stable identifier they like (<participant>-<timepoint>, <recording-id>, etc.).

string_newtype!(
    /// Identifies a session within an override file. Free-form
    /// stable string; typically the CHAT-file basename stem.
    pub struct SessionId;
);

MergeOverride