Parsing

Status: Current Last updated: 2026-10-27 (commit c0e233d0)

The parsing pipeline converts CHAT text into a typed ChatFile AST. Retaining a model does not establish that its input is valid: recovery may produce both a model and diagnostics, and some inputs cannot be lowered. Validation is a separate stage; see Validation.

For a first reading, start with the CST → AST pipeline, then error recovery and the public API. For runnable integration examples, use the library recipes. Backend compatibility defines the independent parser comparison contract; editor source revisions explains incremental reuse. The detailed contracts below document individual ownership and recovery boundaries, not additional entry points a caller must assemble.

Detailed admission contracts

ParseError::internal constructs E001 producer-failure evidence; it must not classify a tool fault as invalid CHAT, even after presentation severity changes. Diagnostic display maps own their text and keep the zero-byte origin implicit. Only post-origin breakpoints are stored; an empty table is an identity map, so no caller can omit a required origin sentinel. Display events consume source bytes monotonically, preserving UTF-8 byte-coordinate mapping. Display-span normalization bounds incoming source coordinates before offset arithmetic, so even extreme out-of-source spans clamp safely to display EOF. That low-level map policy is not diagnostic source admission: SourceDiagnostic rejects unreadable primary or label ranges and retains the original evidence, instead of repairing those ranges into display coordinates. Source-span overlap uses half-open byte ranges: an empty insertion span never overlaps another span, even when its point is inside it. Containment of an insertion point is a separate operation. Recovery deduplication retains its explicit same-point/same-code rule rather than treating a point as covered bytes. Line-end lookup also checks index arithmetic before accessing its line table: even an extreme out-of-range line request uses the documented source-EOF clamp. Header lowering rejects an unreadable comment body or participant name/role word after reporting the internal source-binding failure. It does not invent an unknown comment or reinterpret a shortened word sequence as a participant. This is distinct from structural CHAT recovery, whose diagnostics remain intact. Unsupported-tier lowering likewise reports E001 if a generated anonymous body span cannot be read from its source. It does not report E330 or substitute empty content: an unreadable span is a tool failure, whereas authored empty content remains represented in the model for validation. The shared transitional raw-node text reader also reports unreadable ranges as E001 through SourceBindingError::InvalidRange. Freecode, event and grammatical-relation fields use that one boundary rather than attributing a failed source read to malformed CHAT. This reader proves readable coordinates, not tree/source identity; generated source-bound admission remains the stronger contract. Structural recovery and lexical validation retain their own findings. Postcode decoding requires SourceBound<PostcodeNode>: final-code groups project their generated source fields and admit the selected token range before decoding. The decoder takes no independent source string and retains the token’s exact span. First and repeated groups share one leaf-slot decoder; displaced diagnostics retain their existing per-group order. Error/Absent recovery still contributes no postcode, while source-field failures propagate as CstFailure rather than silently dropping a code. The ownership test rejects a node from an independent equal-text parse; shared source admission owns range refusal. User-tier parse-health classification consumes a source-bound tier and admits its prefix before comparison; a failed read cannot become successful “no domain” classification. Childless separator recovery rejects unreadable text through the source-bound entry boundary, then reads admitted text without a second decode. Its internal structural extraction still retains all recovery states. Fragment admission and recovery likewise distinguish binding/range failures (E001) from missing or malformed CHAT, retaining fragment coordinates in the reported context. Main-tier word/replacement wrappers, annotated angle groups, quotations and phonology/sign groups consume the same compiled canonical-grammar admission. Their composite word, replacement, annotation, quotation and contents slots cannot themselves be Missing. The shared contents walker still handles lexical and mixed-choice recovery; delimiters, absent required content, ERROR nodes, source failures and empty-group refusal retain their established policies. Carrier-owned recovery reporting accepts the corresponding admitted group carriers, preserving body-region diagnostics for displaced children. This is not a clean-tree proof and does not permit dropping recovery inside a composite.

Standalone-word and word-body extraction also carry canonical grammar admission through prefix, form-marker and segment/overlap-initial piece choices. The required composite word-body slot cannot itself be Missing. Its contents remain a mixed choice, so Missing, Error and Absent handling still applies there and to lexical pieces. Prefix, form, repeated-form and POS projections select the producer’s source-bound choice and retain it through conversion. No token adapter takes an independently chosen source. Form-registry policy, opaque POS labels and source-read failures remain separate; admission does not certify a recovered word as valid. Repeated and undeclared form payloads remain verbatim with E203, not rewritten into a declared marker. The standalone-word root consumes admit_ranges() before decoding its five positions. The kind proof survives this transition, and present prefix/body/ form/language/POS fields retain readable slices instead of five independent fallible reads. A range failure rejects the word with an internal diagnostic; it cannot turn an unreadable optional field into an omitted marker. Displaced recovery remains reported on failure and after successful field lowering. Word-body and shortening producers now consume that same transition. The six piece choices retain readable leaves; decoding no longer performs a fallible source read per piece. Lexical/nonempty admission and recovery remain separate. The word/replacement, nonword and quotation annotation hosts, angle/phonology/ sign groups, scoped lists, code-switch wrappers and tier bodies likewise admit their selected ranges at the producer. Kind and required-presence proofs survive; the range phase does not certify nested payload validity. The four bracketed cores’ required-presence proof excludes delimiter ERROR/absence, so their dead delimiter reporters and the angle-group extent fallback are removed. Lexical MISSING delimiters still supply their actual zero-width positions. The precode decoder needs only its syntax outcome after its enclosing tier-body admission; source refusal cannot be mistaken for an absent or malformed precode. These are selected-carrier transitions, not a whole-tree prepass or a claim of faster parsing; transient carrier layout and counted admission work are separate from throughput measurements. The language suffix is deliberately one immediate token, not a structured code-list CST. Its adapter takes only the source-bound WordLangSuffixNode; spelling admission retains that owner through atomic candidate conversion. Every authored component survives into registry validation or the conversion is refused with a diagnostic on that token. There is no candidate filtering or invented und fallback. Shortcut, explicit, mixed and ambiguous model/JSON forms are unchanged, including the existing interpretation of mixed separators; this admission change does not settle a new syntax or validity policy. Replacement extraction uses the same proof for its first and repeated word slots. No word-position counter is needed to describe an impossible composite Missing state. Zero-width words, unreadable source, delimiter recovery, Error/Absent slots and refusal of an empty replacement remain separate checks.

Main-tier linkers, endings, final-code groups and bullet timestamp fields also consume selected-range admission at their producers. Linker decoding accepts only its source-owned node, not a node plus independently chosen text; its failure prevents tier-body construction after sibling recovery is reported. Ending decoding uses one readable carrier rather than zipping raw and associated iterations or reprojecting the bullet group. Linker source order and lexical placeholders are retained. Bullet recovery refusal, numeric overflow precedence and start-before-end leading-zero reports stay separate from source readability. The existing postcode token decoder and shared terminator mapping remain the policy owners; this migration adds no grammar, validity or wire-format rule.

MOR item collection has a private one-way admission state. Items and failure health cannot be supplied independently to finalization. Any item, separator or terminator failure rejects the collection; subsequent traversal still reports diagnostics but cannot revive it. Finalization handles that rejection before checking for an absent terminator, preserving diagnostic precedence. Source failures and lexical/structural recovery remain explicit.

Within each MOR word, the generated selected-range transition retains the compiled kind proof before POS, lemma and repeated features are decoded. Each feature separately admits its hyphen/value positions. Lexical decoding therefore receives checked slices rather than performing fallible source reads. Displaced children are reported before each consuming transition, preserving their diagnostic order without cloning recovery. Range failure still propagates as internal failure to the item boundary. Missing POS/lemma, the pipe’s local missing report, the hyphen’s recovery-backstop policy and the feature value’s shared Present/Missing empty-text refusal remain separate. This is neither a nonempty-token proof nor a whole-tree prepass. Delimiter checks and larger transient carriers are additional costs, not a parsing-speed claim.

The low-level @ID header entrypoint likewise takes a SourceBound<IdHeaderNode> rather than an independently supplied node and text. Header, contents and required/optional field projections retain that source association. The contents consume generated selected-range admission once, including structural positions, before required or optional payload decoding. Those readers accept only readable slots and no longer return a source-read failure: text comes from their checked fields. Admission failure flows as an internal producer error rather than an empty field or invalid CHAT. Required-field rejection, omitted optional values, displaced-node reporting and the whole-tree recovery backstop retain their policies. Required field policy has one speaker/role owner; optional reading returns only Option<String>. Language components retain their own source-bound lowering. This is not a new grammar, a whole-tree range prepass, or a change to the public ChatParser trait’s ID-fragment API. Checking the wider selected carrier is not a parsing-speed claim.

All six dedicated structured-header dispatch paths (Languages, Participants, ID, Media, Situation, Types) retain source-bound nodes through one shared diagnostic-forwarding wrapper. No path in this family accepts a separate source argument. Situation text and the three Types fields are read through source-bound projections; unreadable ranges are internal failures, not missing authored text. Missing-field recovery, Types field order and its first-missing- field short-circuit are unchanged. This does not assert readable child ranges before their checked reads or certify recovered headers as valid.

The low-level %mor tier entrypoint takes SourceBound<MorDependentTierNode> from the existing parse owner, with no independent source argument. Dependent-tier dispatch retains this capability through tier/contents extraction, choice and repeat projection, and item admission. Each tier, contents, item and post-clitic carrier consumes generated range admission before lowering. Present leaves retain checked source slices; there is no parallel selected-field read path or private read-admission wrapper. Displaced nodes still use the shared recovery collector in their existing order. Word, post-clitic and feature decoding also retain source-bound nodes, and POS/lemma/feature text comes from admitted fields rather than a second UTF-8 decode. Source-binding and grammar/kind-admission faults propagate as CstFailure to the owning item boundary, without becoming missing POS/lemma diagnostics. String-based fragment APIs are unchanged. Tier-body, main morphology-word, post-clitic, contents-item and repeated feature boundaries consume canonical-grammar admission: their composite-node Missing state is uninhabited in the generated slot type. This does not by itself prove required presence or nonempty lexical payloads. The ordinary tier envelope separately consumes required-presence admission; nested positions retain the recovery states allowed by their own types. Displaced-node and lexical missing-placeholder recovery remain. Structural delimiter recovery uses the retained slot evidence with the owner’s source. POS, lemma and feature-value tokens still permit Missing; the enclosing composite feature does not. Source ownership alone would not justify this narrowing. The contents alternative remains a mixed choice with its own Missing/Error/Absent handling. Both admitted terminator choices use the shared exhaustive terminator conversion; admission does not invent a new terminator or whitespace policy. One exhaustive variant table supports the actual raw main/WOR and ranged MOR terminator carriers, deriving spans from their concrete leaves. Checked source projection is emitted only for the ranged lexical MOR choice’s missing slot; unused carrier phases and speculative projection implementations are absent. Admitting prefix, separator, newline and clitic-tilde ranges adds checks and enlarges transient carriers; it does not add lexical policy or establish a speed improvement.

The low-level %gra entry likewise requires a source-bound node. Dispatch, tier/contents extraction, repeated pairs, relations and field text retain the same source owner. A single root reading derives separator provenance before consuming selected range admission; its private range phase retains both until the tier/separator result is consumed into document attachment. The contents transition also admits its inline repeat pairs; relation roles project only their matching field and diagnostic parent from the whole admitted relation. There is no separately pairable field/context input. Index, head and label admission retains its existing order and numeric/empty checks; source faults propagate as CstFailure to dispatch instead of dropping a relation and misclassifying a producer failure as CHAT invalidity. GraLowering records each declared relation’s actual outcome and consumes collection with finish(location) -> ParseOutcome<GraTier>. Any rejection irreversibly rejects the graph. Root syntax recovery also rejects, including text outside declared relation slots. Diagnostics continue to be collected; neither parser constructs a partial graph or a completeness flag. File parsing retains the original tier as UnparsedGra, including through JSON; fragment parsing returns rejection. Builders and JSON import do not certify source origin or CHAT validity. Tier admission is total (tier_parsers::admission::TierAdmission, since 2026-10-19): a source-bound tier is either a consumed TierRecovery, which only the whole-tier reporter accepts, or a HealthyTier, the proof that no descendant is ERROR or MISSING. The typed %mor parser takes the proof, so a Missing or Error slot inside it is a producer fault (CstFailure::RecoveryInsideHealthyTier), never a CHAT diagnostic; the other tier parsers still take the bare node the proof wraps. The whole-tier reporter preserves concrete recovery diagnostics; if the native flag has no individually reportable ERROR/MISSING child, it reports E316 at the actual tier range. Thus a zero-width recovered GRA index cannot silently become a clean placeholder or a successful strict fragment. Root/body displacement order and lexical recovery remain explicit. Range failure refuses the owner as CstFailure/E001, not empty CHAT. Native APIs explicitly consume only the tier; document attachment consumes the shared ParsedDependentTier. String-based grammar-tier and relation fragment APIs are unchanged. Compiled-grammar admission narrows the tier-body and repeated relation carriers: composite gra_contents and gra_relation cannot be Missing. The ordinary tier envelope separately proves required body presence; its body cannot be Error or Absent. Nested recovery retains its own policy; lexical whitespace, index, head and label recovery remain. Declared-versus-lowered relation accounting still comes from the same traversal and is not inferred from this composite-node proof.

The low-level %sin entry carries source ownership through group choices, repeats, tokens and whole-group fallback. Its root reading derives separator metadata once and consumes selected range admission before lowering; document attachment consumes the shared ParsedDependentTier rather than re-extracting and independently pairing a separator. Groups, the flat/bracketed choice with its inline sequence, and grouped contents with token repeats admit their whole selected carriers before decoding. Whole-group fallback receives that admitted group, whose retained parent supplies text; no unrelated group/context can be paired with a choice. The explicit EmptyText constructor verdict retains empty-token omission without discarding an unrelated error. Internal source failures propagate to dispatch as CstFailure; they cannot become an empty token list or trigger recovery fallback. Existing empty-token and whole-group recovery policies remain. Document has_error still reports and drops Sin; it does not attach a recovery placeholder. The native tier envelope consumes required-presence admission, so a missing required body is not an empty-tier recovery branch. These source/range facts do not prove CHAT validity, universal recovery reachability or a speed improvement. This does not alter %phoaln/%phoint or the string fragment APIs.

The shared %pho/%mod decoder also retains source ownership through tier selection, repeated groups, compound words and whole-group fallback. Its typed tier variant determines both the extractor and model tag. Low-level entry points accept SourceBound and return CstFailure; internal faults propagate to dispatch, never into empty-content or recovery branches. Compound spelling, empty-content omission and string fragment APIs retain their policies; recovery handling is bounded by each generated position’s admission, as described next.

Both families consume compiled canonical-grammar admission at their tier, group-list, group-choice and grouped-content boundaries. Generated KindSlot carriers exclude Missing for the admitted composite bodies, groups, grouped content, pho_words and sin_word. The ordinary tier envelope separately excludes Error/Absent at required positions. Mixed choices and nested positions retain the recovery states their types permit, including whole-group fallback; displaced nodes and lexical whitespace Missing remain explicit. Empty-token omission and string fragment APIs are unchanged. Grammar/source admission failures propagate as internal failures rather than taking any CHAT recovery branch.

%wor marker construction retains its already-admitted source-bound comma, tag and vocative nodes, so it has no second fallible byte decode that could silently drop a separator. Its source-bound language-token and dependent-tier body readers use checked range admission and report internal failure before lexical or empty-content policy is applied. The tier, body and word-item extractors also consume compiled canonical-grammar admission: body, language-code and standalone-word composite slots exclude Missing. Required-presence admission separately narrows the ordinary envelope. Mixed word/bullet/marker choices, lexical whitespace and terminators still retain their recovery states; Error, Absent and displaced children remain explicit. The admitted terminator choice uses the same exhaustive mapping as other tiers. No clean-tier assumption substitutes for those types, and source admission does not prove timing completeness or CHAT validity. The free-text dependent tiers (the nine bullet-payload tiers, the fifteen simple text-like tiers and %x user-defined tiers) also extract under compiled canonical-grammar admission. Each optional-present body uses SelectedNonMissingKindSlot: selection excludes absence and the exact-kind proof excludes composite Missing. An ERROR body remains recovery, not text. The separator reader consumes only a present tier_sep, keeping its recovery payload producer-owned. Error, an absent optional body and displaced children keep their diagnostics. Header and field readers whose callers hold both narrowed and plain kind slots stay generic over their recovery payloads (AnyKindSlot); a placeholder there only refuses, so one body serves every reading.

The fifteen simple text-like appliers retain their concrete SourceBound through dispatch rather than reducing it to a node and an independent source string. They consume the generated selected carrier’s range admission before decoding any body. Their common optional-body reader accepts the whole RangeAdmittedChildren, so the parent and body cannot be paired independently. Its body projection is emitted beside each concrete applier, not maintained as a second tier catalogue. The same admission covers %x prefix/body fields and unsupported-tier anonymous bodies. It checks selected prefix, separator, newline, recovery and extra ranges as well as the body; no speed or peak-memory improvement follows from moving those checks. Kind/source/reconstruction failure refuses the owner as E001 before lexical decoding, never as empty CHAT content. Separator metadata is derived from the same selected carrier before consuming it, and displaced recovery is reported first. Optional absence, empty-content validation, recovery-body diagnostic order and the existing timing/Phon content parsers keep their separate policies. These root transitions do not certify every parser field or prove CHAT validity.

Dependent-tier entries also retain optional, producer-owned content-range provenance. Tree-sitter selects the interval between the source-bound separator end and newline start (nested inside the %wor body); re2c retains the same boundaries from its lexer/token/span owner. The range includes original body whitespace and continuations, excludes the actual line ending, and can be known empty. It is never inferred from label length, rendered text or newline width. Unknown boundaries, programmatic construction and JSON imports return None from content_span(). Recorded geometry participates in span shifting but not JSON, schema or semantic equality. It locates original source and does not certify valid recovery content or the spelling of a subsequently modified model. Re2c token exhaustion alone does not prove physical EOF: an early lexical stop leaves ending geometry unknown instead of inventing the unconsumed buffer end.

The nine bullet-payload tier parsers likewise consume selected root range admission before lowering the body. Their common reader accepts a whole RangeAdmittedChildren, with the body projection emitted from the existing per-tier policy catalogue, including %com’s picture-capable carrier. The same selected extraction supplies separator metadata and displaced recovery before admission; document dispatch no longer repeats a broad extraction for the separator. ParsedDependentTier retains tier and separator until attachment, while the existing low-level public APIs explicitly request only the tier. These api::tiers entry points require a concrete SourceBound from the native parse owner, unlike ChatParser’s string-based content fragments. The existing dependent reference contract exercises both paths and compares actual models, source spans and diagnostics; a fragment adapter alone does not test the CST API. Optional empty bodies, malformed-body reporting, label policy and wire formats remain separate. The nested bullet/inline-picture parsers retain their existing source-bound reads; selected root admission is not a claim that every nested field is range-admitted or that recovered content is valid. Larger admitted carriers and counted selected checks are costs, not a speed or peak-memory claim. Misplaced-linker diagnostics likewise retain source-bound token text instead of substituting a grammar name. Recovery collectors and main-tier language precodes report unreadable coordinates as internal failures; an unreadable word-recovery range is not an invalid control character and must not advise editing the CHAT. The same distinction applies to generic, dependent-tier and utterance recovery admission. An unreadable range produces an internal-failure diagnostic before the readable-recovery classifier can run; utterance admission still taints every potentially affected alignment domain. This does not change the diagnostics for readable malformed CHAT or turn recovery into validity. Structured-tier recovery keeps its generated SourceBound through source_slice() and source-associated child traversal. The recursive reporter accepts no independent source string and passes admitted recovery text directly to the classifier. It retains ERROR/MISSING traversal boundaries, tier context, diagnostic order and explicit internal failure on an unreadable child range. Utterance recovery slots likewise retain their generated SourceField until read admission. Their reporter accepts no independent node/text pair; equal source bytes from another parse do not establish ownership. Unidentified recovery still taints every potentially affected alignment domain, and failed range admission remains an internal failure rather than CHAT invalidity. Gem-label and option-name accumulation also rejects a failed source read rather than returning a shortened label or incomplete flag set. Their outcome types distinguish rejected admission from a genuinely absent label or empty options. Participant entry lowering consumes the generated range-admitted carrier: speaker/name/role slices are checked before decoding and read without another fallible source operation. Missing/Error/Absent states, name order, role admission and displaced recovery diagnostics remain. Admission failure reports internal failure at the entry boundary and rejects the entry; it cannot expose a partial name whose final word would be mistaken for a role. Only displaced-node evidence is copied across the consuming transition, then released on success. This opt-in costs storage: reference code+role and code+name+role entries perform 3/5 checks; repeated reads add none. Inline carrier storage grows 224 to 288 bytes, with 200 to 264 occupied bytes per repeated word group on the measured 64-bit build. The admitted owner occupies 360 bytes. These are transient layout figures, not peak-memory or throughput claims; malformed/recovery entries may also need storage for retained displaced evidence. Public APIs and grammar are unchanged. Gem headers retain SourceBound through their optional separator/text group and free-text choices. Label lowering accepts no independent source string; admitted text pieces need no second byte decode. Bare markers, missing/error positions, continuation spacing and displaced-child reporting retain their existing semantics. A node from an independent parse cannot enter this lowering through another parse’s owner, even when both inputs have identical bytes. Unknown-header recovery preserves the authored header text, never a grammar-node name substituted after a failed read. HeaderSite retains its admitted text; transitional raw-node construction returns SourceBindingError and its callers propagate internal failure. A rejected header fragment does not construct a throwaway unknown-header model before returning its diagnostics. The shared bullet/text-tier reader similarly propagates CstFailure through its typed adapters to dependent-tier dispatch. A failed source-bound read cannot become a successful empty payload. Authored empty optional bodies still lower as empty content for validation, and malformed slots retain recovery diagnostics. Nested segment reads use the same fallible transition: text, leaf and missing-node source admission must succeed before accumulation can return BulletContent. A failed read rejects that content instead of silently omitting a segment. Inline bullet lowering likewise propagates source-binding failure separately from malformed timestamp rejection; ordinary bullet recovery is unchanged. The %wor timing adapter preserves its existing alignment-owned rejection policy for malformed timestamps, but explicitly reports admission faults as internal failures instead of silently discarding them with .ok(). Main-tier body lowering preserves its source-bound ending through extraction. Failure to read the body content or ending propagates as CstFailure; it cannot produce a successfully constructed empty body. Structural recovery remains separate from these producer failures. All structured timestamp consumers pass SourceBound<BulletNode> to the shared reader. Start/end fields retain that association through generated field projection and checked reads. There is no independent source-string argument to mismatch with a bullet, and unreadable fields report producer failure rather than masquerading as missing timestamps. This changes neither overflow nor leading-zero policy. The Tree-sitter backend (talkbank-parser) and re2c backend (talkbank-parser-re2c) have equal standing in the shared model’s design. Tree-sitter was prototyped first and is currently the CLI default and incremental LSP backend. Select re2c with chatter validate --parser re2c. Those implementation facts do not grant either parser ownership of the model or authority to define CHAT validity. Both can motivate shared-model changes, including explicit malformed or recovery states.

The finite reference workflow checks editor-facing source ownership after lowering: byte offsets at main/dependent-tier starts and interiors select the owning utterance, while headers, EOF and a turn’s exclusive end do not select that finished turn. These are byte coordinates, including multiline tiers, not character indices. This does not prove that source-free reconstructed ASTs have usable spans.

Model access is also distinct from validity. The missing-ID E522 spec still exposes both declared speakers through declared_speakers(), but the speaker without an ID has no id_metadata(). The participant-map join contains only the materialized ID record. Diagnostics remain present; a UI may display the declaration without fabricating metadata or certifying the file as valid.

Tree-Sitter Parser

Generated recovery capabilities should survive through diagnostic helpers. For example, the empty-colon reporter requires KindMissing<ColonNode>, not an arbitrary colon node followed by a zero-width test. The producer already distinguishes a present token from a missing placeholder. Retaining that state does not establish that CHAT fixtures reach it: deleting a speaker colon still produces generic E316 recovery, and E322 remains deferred.

An ERROR child in main-tier contents is recovery evidence, not a word suffix. The parser reports its source-associated location without appending guessed text to a preceding word, annotated word or replacement. An @ prefix or a marker-like character sequence cannot establish lexical ownership. The whole-tree recovery backstop remains active; removing that text heuristic does not remove recovery diagnostics or certify a recovered model as valid.

The canonical corpus contract checks the main-tier word traversal against both reference files and error-spec fixtures: each retained word’s raw spelling equals its own UTF-8 source span, including in documents with parse errors. This verifies source ownership, not validity of those recovered documents or exhaustive coverage of all possible recovery trees.

The talkbank-parser crate wraps the tree-sitter C parser and converts its concrete syntax tree (CST) into the ChatFile model.

Full-file parsing is the canonical entry point. TreeSitterParser also provides fragment methods (parse_word_fragment(), parse_main_tier_fragment(), parse_chat_file_fragment(), etc.) for parsing isolated CHAT fragments directly.

CST → AST Pipeline

flowchart LR
    chat["CHAT text\n(.cha file)"]
    grammar["tree-sitter grammar\n(grammar.js → parser.c)"]
    cst["CST associated with\nits original source bytes"]
    walker["Source-bound typed traversal\nand Rust lowering"]
    ast["Model, when buildable,\nand diagnostics"]

    chat --> grammar --> cst --> walker --> ast

The parsed-source owner retains the original bytes alongside the CST. This does not mean every whitespace byte has its own tree node. Rust lowering extracts semantic information through the generated typed traversal layer, described next. Source association, successful lowering and semantic validity are separate contracts; the AST is not a byte-for-byte copy of the source.

Source-preserving document and participant lowering

Document classification retains generated source-bound capabilities for both complete and recovered documents. Associated child projections carry the producer’s source through lines and header dispatch without repeated root-membership searches. Header fragments use the same dispatcher with their already admitted wrapped-source slice.

Source association is not a proof that all runtime ranges are readable. The incremental API still accepts a raw old tree, whose caller must apply every source edit before reuse. The finite-corpus boundary test deliberately reuses an unedited reference tree after deleting its input: parsing completes, but root admission rejects InvalidRange; a cold parse of the same empty input has a readable root. This is an API-contract violation, not a CHAT syntax rule. Keep range refusals until the producer models the edit/source relationship; neither a bound parent nor unreached fixture branches establishes that proof.

The editor uses TreeSitterParser::parse_chat_file_revision and its opaque ParsedRevision cache. Only the parser constructs that cache from the source it actually parsed. The next transition accepts new text and a previous revision, derives a UTF-8-safe edit from the retained baseline, edits a cloned tree, and parses it. The LSP owns no independent edit calculator and stores no separately replaceable source/tree pair. Detached trees remain available for structural queries but cannot be installed back into a revision. This closes stale pairing in the revision API, not the raw-tree compatibility APIs described above, and does not certify recovered CHAT as valid. New editor integrations should use the revision API. The raw CST, strict-model, and streaming incremental methods remain compatibility entry points whose callers must edit the old tree correctly; they do not accept a proof of that transition. Their continued availability prevents treating source-bound child range failures as impossible across the entire public parser API. The edit calculator counts shared complete Unicode scalar values, not shared bytes requiring later UTF-8 repair. Equal characters have equal byte widths; the resulting prefix is a boundary in both sources. The suffix is counted only within the remainder, so it cannot overlap the prefix. This removes intermediate split-code-point states without removing any CST recovery or range admission.

Utterance lowering also takes SourceBound<UtteranceNode> directly from the document projection. Its entry point accepts no independently chosen source string. Its generated dependent-tier repeat retains SourceField association until attachment admits the choice once as SourceBound<Choice>. Parse-health classification and exhaustive dependent-tier dispatch select from the generated ChoiceBoundView: each variant retains the same admitted range, so dispatch does not repeat range checks in every arm. This capability exists only for single-node choices; composite choices still require individual child admission. Concrete main/dependent-tier entry points retain this source-bound capability through their generated projections. Association alone does not prove recovery or decoding branches unreachable: kind, required-presence and range admission are separate proofs. Remaining recovery states stay distinct, with conservative parse-health taint preserved.

User-defined %x* lowering keeps the admitted concrete tier and projects its optional body through associated slots. A present body retains its own range admission; its composite Missing state is excluded by exact-kind proof. Text comes from the resulting bound node, not a second node/source decoding attempt. Empty tiers remain in the model, and ERROR bodies and optional omission retain their existing policies. Prefix fields also consume generated range and required-presence admission.

@Participants lowering keeps this association through contents, list groups, speaker codes, names and roles. Generated SourceSlotView keeps genuine Missing, Error and Absent outcomes distinct, with impossible payloads remaining uninhabited. Leaf reads admit their canonical byte ranges at the shared boundary; there is no independent source argument that can be paired with a participant node. This removes transitional decoding from that family, not recovery. @Media keeps the same association through its body, filename, media type and optional status group. Body extraction consumes the selected carrier through admit_ranges() before lowering: typed leaves retain checked slices and the payload reader can refuse only structural recovery, not source admission. No independent source argument or temporary payload strings are needed. This opt-in transition checks all selected ranges, even delimiters whose text lowering does not use. It trades larger transient carriers and eager checks for a single explicit producer-failure boundary; it is not a throughput claim. The two reference media headers admit four and seven ranges respectively. On the measured 64-bit build, their inline body carrier is 888 bytes versus 632 before admission (960 bytes including its owner). These are transient layout figures, not peak-memory measurements or a reason to migrate every consumer. Missing/Error/Absent slots, displaced nodes, range refusals and the validated MediaFilename constructor remain. Header-level recovery still uses the shared diagnostic site. The thirteen scalar/text headers and the single-value @Number, @Recording Quality and @Transcription headers use the same source-associated reader. Their entry points cannot take an independently selected source, and model constructors receive borrowed checked payloads. HeaderSite::bound derives diagnostic identity from that same capability. The two-slot @Birth of, @Birthplace of and @L1 of headers do likewise: their participant slot requires generated SpeakerNode identity, and the admitted result distinguishes SpeakerCode from borrowed value text. A failed speaker read still prevents value decoding. The old owned-string slot reader has no callers and is removed. Comment headers retain association through body admission too; the bullet-text adapter receives that same source-bound node, with no separate source argument. HeaderSite has only the producer-bound constructor, so diagnostic sites cannot be built from independently paired nodes and input. Options enters through a source-bound envelope too; its internal option-flag decoder remains transitional. Source ownership does not by itself exclude recovery states or certify semantic validity of source-admitted values.

The generated typed traversal (generated_traversal)

The bridge between the tree-sitter CST and the typed model is a single generated module, crates/talkbank-parser/src/generated_traversal.rs, produced by the tree-sitter-grammar-utils generator from the grammar’s own machine-readable description (grammar/src/grammar.json plus node-types.json). It contains one extract_* function per grammar rule, each returning a typed view of that rule’s children, so consumer code dispatches on generated types rather than on node.kind() strings.

Every child position a grammar rule models is exposed as a NodeSlot with five states, of which each position’s type admits only the ones that position can produce:

The generator names the position’s kind in the slot’s type. A ChildSlot (a child taken by kind) is never Unexpected; a SeqSlot (an inline sequence) is never Missing or Unexpected; a selected ChoiceSlot is never Unexpected; a ClassifiedSlot (a supertype rule’s own node) is never Absent. A ChoiceSlot also names its Missing payload: the generator’s <Position>Missing enum, one variant per kind of the choice’s single-node alternatives (a composite alternative matches its own leading MISSING token at its first position), Never when every alternative is composite, or the raw node beside a raw partial alternative (none in this grammar). The enum still names nonterminal kinds, which tree-sitter never inserts MISSING (its runtime inserts only terminal leaves), but each such variant carries a NarrowedMissing<K, _>: under the broad proof that is the placeholder, under KindAdmitted it is the uninhabited Never. A consumer that extracts under the admitted proof can handle a nonterminal head with an empty match (separator, long_feature and nonvocal do). A consumer that only reports missing syntax needs no conversion of the missing head. A tree contradicting the proof never reaches such a consumer: extraction refuses it, and the refusal is reported once as a producer fault. A consumer that extracts broadly still receives the placeholder. A consumer that interprets the placeholder matches it by kind, exhaustively; one that only reports it reads its node. Neither re-reads a raw MISSING node. A MISSING node of a kind the choice does not name leaves the position Absent and reaches the displaced-node sink. The impossible states have the uninhabited Never as their payload, so an arm that reads a node out of one does not compile, and a match by value may omit it. Every generated accessor hands out a reference, and a match through a reference must still name every variant, so a consumer matches slot.view(), which copies the recovery states out by value and borrows only the present payload. The parser’s shared verbs (expect_present, expect_structure, expect_delimiter, present) are generic over the slot’s recovery payload types. Repeat elements and present optional elements use SelectedKindSlot, SelectedChildSlot, SelectedSeqSlot or SelectedChoiceSlot: their Absent payload is Never. Nested fixed positions retain their own slot types; no missing/error recovery is removed by selecting an outer element.

This design makes recovery evidence explicit at modeled positions, but an exhaustive match alone cannot ensure that a consumer reports it. Consumers and the whole-tree backstop still own diagnostic emission. An uninhabited payload prevents lowering a node from that impossible state. The generic backstop uses E342 for MISSING placeholders and E316 for ERROR regions (“content could not be parsed”); context-specific lowering may supply a narrower diagnostic.

That asymmetry matters when reading a diagnostic. E342 names a specific fact the parser knows. E316 names the absence of one, so an E316 on input a human can read is a standing invitation to ask whether the parser, rather than the file, is at fault. Generic rejection alone is not a specificity defect; a narrower diagnosis requires structural evidence. Generated traversal owns grammar structure; consumers must not invent a parallel kind catalogue or reconstruct CHAT structure from ERROR text prefixes. Recovery classifiers may use admitted lexical evidence under their explicit diagnostic policy, without treating readable recovery as a successfully parsed construct.

What to write instead, when the question really is “which alternative is this node?” A grammar rule whose alternatives are each a single named kind lowers to a <Rule>Choice enum, and the generator emits that enum’s own classifier, <Rule>Choice::from_node. It returns Option<Self>: None says the node is not one of the alternatives, which is a fact about the input, not a default to paper over. Matching the result is exhaustive, so adding an alternative to the grammar breaks compilation at every site that decides on it, which a chain of kind() string comparisons never does.

The classifier is emitted exactly when a kind can identify an alternative. A choice with a sequence, repeat or optional alternative is told apart by STRUCTURE, so it deliberately gets none: there, the shape is the question, and a kind-keyed answer would be a guess. When no classifier exists, extract the rule and match on the carrier rather than reaching for kind().

Bullet-capable free text follows that structural path: BulletTextNode retains the two grammar carriers, and lowering consumes their generated first/repeated choices and nested bullet/picture groups. The groups own trailing spaces; recovery positions and displaced sinks remain explicit. Structured timestamp parsing requires SourceBound<BulletNode> and derives start/end fields from its generated carrier, rather than accepting arbitrary nodes and looking up string field names. Checked child-field reads retain the producer-owned source association. The internal bullet-text sum type exposes only source-bound carrier conversions, not an independent raw-node classifier or raw projection. All nine bullet-text dependent-tier adapters preserve the dispatcher’s binding, and their optional body slots retain source identity through generated projections. Present bodies and their selected fields undergo fallible range admission; composite MISSING is excluded by exact-kind proof. ERROR and absent optional bodies retain their separate diagnostic policies. Generated choice and nested-group projections carry source identity through the inner segment sink too; it owns no separately supplied source. Text, inline-picture and inline-bullet leaves accept admitted source-bound nodes. Picture delimiter/filename admission and the inline all-zero timestamp policy remain separate from source admission. The shared timestamp decoder also retains source ownership; this does not prove its lexical recovery guards redundant. Displaced children still use the shared recovery collector. Reference tests obtain carriers through the producer-bound descendant API rather than pairing raw nodes with text.

Leading-zero timestamp diagnostics require an admitted LeadingZeroTime spelling and stream directly to the sink. Both numeric fields must first fit the timestamp representation; overflow takes precedence over spelling reports. The two-component case reports start before end, retaining the bullet’s source span and numeric value. The E748 spec family and its corpus contract check that multiplicity, ordering and precedence; the observation snapshot alone records code sets and cannot establish diagnostic counts.

The legacy single-utterance helper probes through an admitted wrapper and the checked parse producer, rather than calling tree-sitter directly and treating failure as fragment shape. Its private classified-input state retains either the complete document envelope or the exact input requiring scaffolding. A valid no-transcription document may contain no utterance; the helper then reports MissingMainTier without reclassifying the document as a fragment.

Both backends’ single-utterance APIs admit their parsed population through SingleUtterance. Zero or multiple utterances cannot produce this owner; its consuming projection releases the sole utterance without another count check. Headers and attached dependent tiers remain supported, including a complete document containing one utterance. A document containing several utterances is rejected instead of silently returning its first one. Use parse_chat_file when the caller needs a whole document. Cardinality admission does not certify CHAT validity or remove parser recovery diagnostics.

The native word-body scanner admits ClosedShortening only after consuming an observed closing parenthesis and nonempty content. Its consuming projection constructs the body item without an end-of-input substitute for the delimiter. WordBodyError distinguishes unclosed shortenings from empty lexical segments; the rich-word grammar propagates either failure as rejection. This boundary proves delimiter presence and non-emptiness, not full CHAT validity.

Inline pictures additionally admit their decoded text into a private nonempty filename value before conversion to an owned string. Its constructor checks both delimiters and the nonempty payload. Tests retain real picture nodes and exercise incompatible source and delimiter boundaries; those are API admission witnesses, not evidence that the grammar produces malformed picture tokens.

Inline bullet times likewise pass a private admission value that excludes the all-zero pair. This is an inline-tier rule, not a time-ordering proof. Parse-backed tests produce separate trees for zero/positive timestamp combinations; source incompatibility is tested separately without treating it as malformed CHAT.

The generated classifiers provide the alternative-selection API without a second handwritten kind catalogue, and are shared by consuming grammars.

Recovery handling is two-layered by design: the per-position NodeSlot states cover every position the grammar models, and a whole-tree recovery backstop (see the recovery discussion below) surfaces recovery nodes that land where no grammar rule models a slot, such as top-level junk. The layers are complementary, and both are load-bearing: removing the backstop demonstrably regresses the CHECK-parity and recovery-is-not-validity test suites.

Main-tier body sinks are owned by a sealed set of generated carriers. Their reporter derives the displaced nodes from the carrier and the diagnostic context from its owning rule’s generated NamedKind; callers cannot supply a different sink, region or label. ERROR children use the shared body classifier, while other displaced nodes use the ordinary reporter. This does not prove node/source identity or remove recovery states. Raw slot-level errors still retain their explicit body-versus-outside-body policy.

Annotated angle-bracket, phonology and sign groups belong to this sealed body-carrier set. Recovery can place malformed word material beside a group’s contents slot; that material must remain rejected, without requiring a particular diagnostic. The E202 spec pairs a valid suffix with a one-character deletion inside a retraced group after multibyte text, exercising the normal document path. Parallel valid/deleted-suffix pairs exercise phonology and sign groups; their displaced word faults are rejected with generic E316. The parser does not rescan recovery text for a dangling @ or an unclosed replacement opener. E202 applies at the model-validation boundary; E311 has no live producer.

Fragment-corpus tests also extract dependent tiers through model-owned spans and compare public full-line and content-only APIs against file parsing. Timing-tier payload states remain distinct during rebasing: their text and clock values are unchanged, but the source span in every state shifts with the owning fragment. Invalid E603/E756 fixtures cover unsupported and empty timing content without inventing model values.

Linkers likewise use generated first/repeated groups and exhaustive concrete alternatives, not a second kind-name catalogue. A source-ordered accumulator keeps each token’s kind and span together and reinserts displaced recovery in source order. Normal traversal appends directly; the recovery boundary uses the generated supertype classifier to retain any concrete linkers found in a sink.

Main-tier speaker admission retains SpeakerNode until checked text decoding. Only that admission can construct the private nonempty speaker/span value, which conversion consumes after reporting the rest of the tier’s diagnostics. Missing placeholders retain their MissingSpeaker diagnostic; unreadable source ranges reject with internal E001. The speaker is projected from the source-bound main tier, not paired with an independent input string.

Unsupported and user-defined tier lowering consume generated range admission while retaining the dispatcher’s concrete SourceBound. Prefix decoders read checked token slices instead of accepting independent nodes and text. The user-defined envelope excludes required Error/Absent prefix states; the unsupported-tier carrier retains its broader recovery policy. Lexical Missing, marker and nonempty-label policies remain. Unsupported bodies retain the generated ReadableLeafSpan: its parent, bounds and UTF-8 slice are checked together, and the body read is infallible. No failed slice becomes an empty tier. Parent source admission alone is not this child-range proof, and readable recovery is not validity.

Prefix decoding retains kind-proven placeholders for star, speaker, colon and tab through the generated SourceSlotView projection. Speaker and colon diagnostics receive their respective typed nodes, rather than a raw-node projection. Required-presence admission excludes ERROR and absent prefix positions; lexical MISSING remains explicit. A present colon needs no width heuristic, and a typed Missing colon follows its own diagnostic branch.

Malformed dependent-tier labels use a private nonempty admission value after the leading percent sign. Only colon, tab, space, carriage return and newline terminate a label; unknown labels remain unknown rather than matching a known prefix. This recovery classification does not certify CHAT validity. Both document and utterance recovery share its alignment-domain policy.

The independent re2c text-tier path admits its source-located tokens before building recovered content. All-zero inline media bullets produce E360 at their lexer spans and are omitted, while surrounding tokens remain. Full-file text tiers and fragment entry points share this admission; fragment sinks rebase diagnostic locations through the admitted fragment source. This does not change main-tier bullet timing or establish complete backend parity.

Top-level recovery binds its node to a producer-owned SourceSlice, deriving text and location from the same input. Binding refusal reports internal E001. Admitted recovery reaches structural delimiter evidence or generic rejection; no headers or tier labels are reconstructed from its text. Generic file-error analysis consumes that same binding instead of re-reading a raw node against a separately supplied string. Line-slot ERROR handling also binds before entering this analyzer; source-binding refusal is an internal error, not a diagnostic constructed from a mismatched source. Both routes share the same binding/refusal function. Its regression test uses real ERROR nodes from a retained spec fixture and a separately parsed identical source: the producing owner admits them, while the other owner must refuse. That is source-identity boundary evidence, not a production recovery-slot witness.

Delimiter-specific diagnoses require grammar evidence. A wrapper proving readable text does not establish an unclosed construct, so no text classifier produces E312 or E313.

No classifier infers an annotation from text prefixes or missing whitespace from the preceding character. Such malformed inputs are rejected by grammar recovery, with generic E316 where no structural evidence supports a narrower fault. Canonical valid controls remain clean; a diagnostic-specific state is not itself proof of parser structure.

Non-ASCII speaker IDs remain invalid. Parsed speaker fields use the model’s E307 assessment; a main-tier ERROR without an admitted speaker field receives E316 rather than reconstructing a speaker ID by splitting text. Supported ASCII forms remain unchanged.

Recovery never reconstructs headers or dependent tiers by scanning an ERROR prefix. Unidentified top-level recovery conservatively taints the preceding utterance’s dependent alignments; unidentified utterance recovery taints the main tier and dependent alignments. A typed dependent-tier choice can still narrow taint to its own domain. Recovery text is retained for diagnostics, not promoted into a fabricated header or reparsed to select an alignment domain.

Postcodes retain opaque labels in Postcode; their text is not reparsed as quotation syntax during validation. Actual quotation delimiters, typed linkers and terminators have their own checks. A postcode whose label resembles a quotation marker cannot create a quotation-balance obligation.

Bracket-to-word spacing validation projects each content item’s following-word and closing-code evidence together with its enclosed sequence. One exhaustive mapping serves both top-level and bracketed content. A private code-end value exists only for a located closing code, so an unlocated constructed node is never tested for adjacency. Each recursive sequence starts with no predecessor: flattening an outer wrapper and its children would create false neighbors. E757 controls and single-space deletions cover annotated events, actions, quotations, retraces and phonology/sign groups; the diagnostic contract checks exact byte offsets and serialization back to the paired control. This is source-location evidence, not a replacement for the parser’s tree/source ownership boundary.

Replacement-word producers also retain the whole typed CST wrapper’s span, including trailing scoped annotations. The shared spacing projection admits that wrapper’s code end while retaining the spoken word’s own start. Canonical replacement pairs exercise missing separators at either closing bracket and nested use.

Wrapped-header selection starts at ParsedSource::root() and carries bound descendants through SourceSlice::children. Admission takes only the owning parsed wrapper and header ordinal, so it cannot pair a selected node with another wrapper’s input mapping. Child traversal resets its cursor to the bound parent and avoids repeated root-wide lineage searches. Header-counting policy and complete-input checks remain separate from that source-ownership proof. The lookup result is a privately constructed SelectedHeader: generated header/pre-begin choices and anchor wrappers admit its kind, with the deliberate Thumbnail exclusion retained. The grammar’s invisible supertypes do not add concrete wrapper nodes, so lookup traverses the real line wrapper but never tries to unwrap an abstract header or pre_begin_header. A selected header cannot be a generic ERROR node. MISSING placeholders and other lowering failures remain recovery states, not proof that the fragment is valid. The old handwritten header-kind predicates and their membership-only test are removed with their last consumer. Generated choices own subtype membership; reference/spec fragment tests continue to own behavior and recovery policy.

User-defined dependent-tier taint routing reads the generated Present prefix slot, not a positional raw child. %xmod identifies the model-alignment domain; other labels and recovered prefixes do not identify one. Source-admission failure propagates as an internal fault rather than an unrecognized label. When attachment reports errors without a specific domain, all alignment dependents remain conservatively tainted.

The module is regenerated whenever the grammar changes; the command and its preconditions are in Grammar Workflow. It is never edited by hand: generator defects are fixed in tree-sitter-grammar-utils and regenerated.

The generated cursor owns its child slice and next-child frontier. Lookahead memoizes answers and construction uses the same typed decision functions, including choice selections that carry their selected terminal child. This removes the separately interpreted plan arena and its self-disagreement failures. Broad extraction returns a carrier directly; only compiled-grammar, source-range and kind/presence-proof admission can fail. A selected choice has an uninhabited Unexpected payload. Missing, Error, Absent and displaced-node recovery remain intact, as does Unexpected for supertype classification. These are producer-construction invariants, not inferences from corpus coverage; extraction does not certify CHAT validity. Real admission failures remain E001 internal failures, never missing-field or invalid-source diagnostics.

The fold also retains selected presence for repeat elements and the Some payload of optional positions. Their extraction consumes a retained element, so it cannot yield Absent. Empty repeats and optional None remain valid outcomes. This does not narrow independent fixed positions or prove that a selected element is free of MISSING/ERROR recovery.

Document, utterance, participant/language header and main-tier/body/ending reconstruction use compiled-language admission. Complete document and ERROR-root extraction share this contract; neither route may fabricate a complete document. The grammar crate compiles TSGU’s generated C metadata bridge against its own parser header. A cached, fallible CanonicalLanguage capability verifies the generated narrowing requirements once; extraction checks actual language identity and retains the proof in the existing selection memo. There is no second parser, rematching pass or consumer assertion based on a kind name. Kind identity includes both the name and tree-sitter’s named/anonymous flag. Consumers use the full identity for classification; KIND.name() is only a diagnostic label. Per-kind admission allows unrelated composite kinds to narrow even when the grammar contains aliases. Lexical kinds and contextual alias targets retain Missing; raw nonterminal extraction remains broad too. Compiled-language admission is not a blanket nonterminal non-missing proof: only a generated narrowed slot carrying that exact proof may omit its Missing arm. Carriers declare a kind-proof axis only when they actually reach such a slot; an Admitted* mirror name does not exist when there is no distinct narrowed reading. Error, Absent, displaced recovery and source-read failures remain independent. Raw extraction stays broad. The unsafe bridge boundary requires the reviewed Tree-sitter 0.27.1 engine, not merely ABI 15. TSGU’s bounded actual-C metadata proofs support admission; they do not prove the whole engine, Rust lifetimes or Chatter’s behavior. The producer revision in tsgu-revision owns both generated artifacts. A failed metadata admission is an internal tool failure, not invalid CHAT. A separate per-rule RequiredRule<R> capability admits the ordinary eligible flat header and dependent-tier envelopes, main_tier, and flat inner nodes: morphology features/clitics, grammatical relations, content wrappers, feature/event markers and separators. The MOR word and content parents also consume required-prefix evidence: mor_pos pipe mor_lemma before repeated features, and mor_word before repeated post-clitics. Their sealed descriptors use a separate native prefix predicate, checking every complete path into every selected reduction at every width. Suffix traversal cannot erase or reorder the visible required prefix under the reviewed engine premises. Repeated-item Error, lexical Missing, nonempty text, nested parsing and displaced sinks remain independent; the composite mor_contents body is not admitted by this fragment. Each rule owns a separate cached admission, including refusal; descriptor eligibility alone cannot narrow a consumer. The compiled predicate checks exact complete child words, including optional positions, or required prefixes over complete paths, against the owned grammar. A nonmissing-kind proof alone cannot provide this guarantee. Required extraction retains the presence proof in the existing carrier: fixed fields exclude Error and Absent; lexical Missing, optional recovery, source-read failures and unexpected sinks remain. Nested payloads without their own admitted rule and ERROR-root recovery retain broad presence. Text-bearing inner readers retain source-bound fields; only the two separator types expose geometry-only extraction without source decoding. The main-tier body is therefore read directly from its proven position, without a sink search or an invented empty body. Recovery outside that position remains diagnosed. Range-admitted ReadableSlot::view preserves source ownership and the same uninhabited states without rechecking leaf ranges. Nested terminator recovery and lexical prefix diagnostics remain unchanged. These proofs do not certify CHAT validity or qualify unreviewed runtime engines or Wasm. Selected kind choices retain Missing wherever their kind proof permits it, even though selection excludes absence. Document recovery ownership failures use E001 and cannot certify source invalidity. The low-level pre-begin/dependent-tier adapters accept the generated admitted choice types; their concrete header/tier policies remain unchanged. The generated broad/raw APIs remain available for other producers.

Generator upgrades can flatten nested choices and normalize whitespace/field sequences into typed repeats without changing the grammar. Word pieces lower every flattened alternative exhaustively. ID decoding selects the concrete field alternative and ignores only structural whitespace; pseudonymization travels the same source-bound projections and refuses recovery before offering edits. Regenerate the conformance inventory alongside the traversal, then check reference parsing, backend equivalence, diagnostics and roundtrips. Do not hand-edit generated outputs or update observations merely to obtain a pass.

Wrapper and supertype child positions use KindSlot: extraction matches the concrete kind before constructing KindMissing<T>. The kind-preserving known_or_placeholder projection therefore has no unclassified-placeholder case. Choice slots are kind-proven too: their Missing payload is the generated missing enum (above), so their consumers have no foreign-kind fallback either. Raw partial slots alone retain the fallible typed_or_placeholder projection. Generic conformance observes every Missing payload through RecoveryNode.

Generic, word and dependent-tier recovery cross a shared ReadableRecovery admission boundary before examining text. The admitted value retains its node, source and checked slice together; the classifier consumes that value instead of accepting independent text. Its fields remain private and consumers cannot construct it without checked slicing. This proves range compatibility, not tree/source identity. Incompatible ranges retain each boundary’s diagnostic family instead of panicking in tree-sitter’s byte indexing. Dependent-tier classification derives both coordinates and text from that carrier; nonempty admitted text therefore needs no second positive-span check. Delimiter findings remain priority-sensitive: a leading space can bypass the first-character delimiter finding, so a later trim-based bracket fallback is not generally redundant.

Empty-POS morphology findings retain both the recognized token and its relative byte range. Both dependent-tier analysis and whole-tree recovery use that same occurrence; searching for the token again could select an identical split tail that the classifier deliberately excluded. UTF-8 and Unicode whitespace retain byte-correct spans, and there is no invented whole-node fallback range.

Language-list admission consumes the generated language-code slot through one policy for first and repeated entries. A position enum preserves their distinct diagnostic wording; only Present enters the fallible model constructor. Missing, ERROR and absence retain their recovery behavior. Sequence-level Missing is eliminated only through its generated uninhabited payload, not corpus absence.

Word-language resolution keeps explicit, mixed and ambiguous candidate sets distinct. Each explicit resolution is consumed into an outcome carrying its registry diagnostics, using one shared transition over every candidate code. An ambiguous candidate is not exempt from ISO validation, and an unresolved shortcut never invents a fallback language.

Participant-list slots use the same admission pattern, with a position type that carries the enclosing header only for the first slot. Absence there still reports an empty header; repeated-slot absence does not. Both positions share entry conversion and Missing reporting while preserving their ERROR wording.

Scoped overlap indices are admitted as 1-9 at construction and JSON decoding; the schema carries the same bounds. Unindexed markers use None, never a numeric sentinel. Atomic token decoding distinguishes an absent index from a malformed one, so malformed content cannot become an accepted unindexed marker. The experimental backend carries the admitted index with its original lexer slice into model conversion. CA overlap-point indices retain their separate recovery representation and 2-9 validation policy.

CA element and delimiter nodes retain their generated kinds into a private decoder trait. Each kind fixes its registry, diagnostic label and model output; callers cannot mix those policies. Source admission uses checked bounds and UTF-8 decoding before character classification. Missing placeholders remain rejected for whole-tree recovery reporting, and unknown symbols still diagnose TreeParsingError. Source compatibility is not proof of original-tree identity.

Utterance construction and main-tier conversion consume the generated SourceBound<MainTierNode>. This replaces the handwritten ReadableMainTier range wrapper: the immutable parse owner and its field projections supply both node identity and checked text. Failed child-field admission reports an internal producer failure without constructing an utterance. Fragment conversion retains the same capability; fragment-owned context handles caller coordinates without an independent source argument to conversion. Inner word/content adapters have their own migration boundaries; this parent capability alone does not prove that every leaf API is source-bound.

Speaker-prefix admission also consumes a SourceBound<SpeakerNode>, projected from that main tier’s associated children. It cannot accept a separately chosen source string; even identical text from a different parse owner is not the same capability. Present text is range-checked at the generated leaf boundary before nonempty speaker admission. Lexical Missing retains its recovery handling; required-presence admission excludes Error/Absent at this prefix position. A failed source binding is not CHAT invalidity.

Main-tier extraction consumes required-presence admission and projects its body directly from the generated required slot. There is no absent-body branch or search of displaced nodes for a substitute body. Reading that field remains fallible: source-range refusal is an internal producer failure, not a missing terminator. Body decoding consumes a bound TierBodyNode and derives its source and carrier range from that node. Direct ERROR children and the generated unexpected sink still supply recovery diagnostics, with duplicate reports suppressed; required presence does not certify the body as valid CHAT.

The recursive contents/group cycle retains generated source association: body contents, first/repeated content choices, content-item choices, annotated angle groups, quotations, phonology groups, and sign groups all pass bound nodes or associated children into the same walker. Bound choice views preserve the admitted wrapper without another kind classification. Exact-kind admission excludes Missing for the narrowed composite group contents; mixed choices and lexical positions keep their own source-associated recovery policies. The utterance-body versus inside-brackets distinction still owns E759 behavior, and angle-edge whitespace still owns E750. Range refusals and recovery sinks remain visible.

Only present contents choices become content leaves. Missing whitespace or overlap markers are reported as missing syntax, with no attempt to decode a present token from their empty text. The shared walker returns a producer failure for an absent required first selection or an unreadable selected range, rather than a successful partial vector. Its displaced-node sink is still surfaced on failure. The bracketed adapter reports that failure once and returns a rejected outcome; all four enclosure callers retain the rejection while reporting their remaining delimiters and displaced recovery. An absent required content_item selection is likewise a producer fault, not silent rejection. These policies do not change the separate separator placeholder policy described below.

Standalone-word conversion also requires a producer-bound word. Base-content choice dispatch retains that association through annotated words and replacement sequences; fragment lowering never detaches its admitted word. The independent %wor route carries its bound tier through body, repeated item choices and word items to the same converter. No leaf reconstructs ownership from a detached node and arbitrary source. Word-level source text uses the admitted slice; missing and empty words retain their separate refusals. Document attachment still drops malformed %wor tiers, while the public bound-node adapter retains recovery handling because source ownership does not prove syntax validity.

Word bodies retain ownership through both generated sequence shapes and every piece choice. Segments, stress, lengthening and shortening content consume admitted text; standalone and word-internal overlap markers share one source-bound decoder. Missing/error/absent positions and displaced children retain their existing recovery handling. Source admission does not prove nonempty lexical content or valid marker semantics, so those checks remain.

Inline CA element and delimiter conversion likewise takes a source-bound token. Its private kind-to-registry association selects the correct model; callers cannot supply another source, decoder or label. Foreign tree membership is refused at source admission even when the two parses have identical bytes. Lexical and registry checks remain because source ownership is not validity. This does not assert that every low-level helper has migrated. Model constructors and mutable annotations are not globally restricted by this parser boundary.

Utterance-level recovery consumes ReadableRecovery too, forwarding that admitted value directly into dependent-tier classification when appropriate. Unreadable input cannot select a tier: it reports internal failure and taints Main plus all alignment dependents. Valid-source recovery retains the existing label-based taint policy; boundary tests do not imply production slot witnesses.

The generated repeat producer uses the shared memoized decision functions described above, not a retained plan arena or separate continuity checker. Selected elements exclude Absent; exact-kind proofs narrow only their named positions. Remaining recovery and element-owned extras stay explicit. Real grammar/source/presence admission faults propagate through E001, never as empty successful carriers.

Word and main-tier fragment admission stores SourceBound<GeneratedNode> from ParsedSource::bind_typed. The binding’s sealed wrapper trait prevents external adapters from claiming a mutable node identity. Lowering obtains both the node and source from that one value; there is no separate node field to pair with the wrong bound slice. Existing completeness, wrapper-offset and recovery policy checks remain independent and unchanged.

Phonology fallback accepts SourceBound<PhoGroupNode> rather than an arbitrary raw node. Its retained text is already readable: nonempty group text becomes one fallback word, and empty text adds no item. Source-admission failure propagates before fallback rather than becoming an empty list. Boundary tests do not prove that a valid corpus fixture reaches fallback through production extraction; the recovery branches remain until the producer justifies narrowing them.

The staleness guard proves less than its name suggests. generated_traversal_is_current recomputes the digests of grammar.json and node-types.json, so it catches a forgotten regeneration after a grammar change and nothing else. It cannot see which generator produced the file, so a module emitted by an older backend passes indefinitely. The generator’s name and version are stamped in the file’s own header comment; read that when the question is which backend built it.

The compiled conformance walk checks real CSTs from the reference corpus. Its slot-state census admits the complete reference and error populations before reporting; discovery, read or parse failures cannot silently shrink the measurement. Positions carry the generated carrier type and field name, so a nested group’s child_0 cannot merge with its outer carrier’s child_0. A recovery observation is a reachability witness. A state absent from this finite population is not proof that the producer cannot emit it, and is not grounds for deleting its handling.

Error Recovery

Tree-sitter can recover from unexpected input using ERROR nodes and missing token placeholders. Rust lowering interprets that recovery evidence and reports diagnostics through ErrorSink; a whole-tree backstop also reports recovery regions. The sink is part of the Rust API, not tree-sitter’s API.

Recovery can retain useful surrounding structure, but does not guarantee a complete or valid model. Lowering may reject individual elements, and a full-file product may be unbuildable. Even an empty returned ChatFile must be interpreted together with its diagnostics.

ParseOutcome

Fragment and many lowering functions use ParseOutcome<T>:

• ParseOutcome::parsed(value): retains a value, potentially with recovery diagnostics.
• ParseOutcome::rejected(): retains no value; callers are responsible for reporting rejection. The enum alone does not prove that a diagnostic was emitted.

This allows the parser to skip individual malformed elements while continuing to parse the rest of the file.

Parser Equivalence

The reference corpus supports distinct regression checks. This check reports parse cleanliness separately for each .cha file; it does not compare backends:

cargo test -p talkbank-parser-tests --test integration reference_corpus_parses

Semantic equivalence on the reference corpus has its own test:

cargo test -p talkbank-parser-re2c --test integration equivalence_tests::equivalence_reference_corpus -- --exact

Neither check alone establishes validation, roundtrip fidelity or completeness over all CHAT input. Error reporting has a separate backend compatibility mandate.

TreeSitterParser API

TreeSitterParser is the concrete Tree-sitter backend handle. Reuse an instance across calls. The shared talkbank_model::ChatParser trait also supports generic callers and backend parity tests; both tree-sitter and re2c implement it. Its sink methods are generic, so the trait is not a dyn trait object.

Use the source-included recipes for parsing and validation and custom diagnostic handling. For editor reuse, see source revisions: the revision-aware owner binds edits to its own prior text instead of asking a caller to coordinate an independent old tree and source snapshot.

Diagnostic coordinates

Fragment parsing adds the caller’s offset to model spans and diagnostic document locations. FragmentSource admits the complete input range before parsing and owns this translation for both backends. Admission checks origin + input.len() without overflowing; ranges beyond u32::MAX are rejected with E310 and an unknown location, since no representable location exists. Origins above i32::MAX remain supported: the existing signed edit-shift interface receives bounded positive steps, never a wrapped negative origin. A diagnostic’s ErrorContext owns its own source text, so its highlight remains relative to that text. Wrapper removal is a separate operation owned by WrappedFragment; it projects the synthetic source before applying any document origin.

Three coordinate spaces meet in a wrapped parse, and two of them are typed. FragmentSource is the witness for the caller’s DOCUMENT space (its input and admitted origin). Synthetic<T> is the witness for the wrapper’s own space: every model or span lowered from a wrapper’s parse is minted inside it (WrappedFragment::parse_file, ParsedFragment::lowered, which the two lowerers call with the value they built from the tree, and node_span), can be narrowed but neither read bare nor used as a model, and leaves only through the wrapper’s rebase, rebase_tier (clamped to the caller’s input), rebase_main_tier, or its span queries (covers_whole_source, input_span). The wrapper takes the caller’s admitted FragmentSource rather than an (input, offset) pair, so the document range is admitted once, and whole_input_error is the one owner of a diagnostic about the caller’s whole input, placed in document coordinates at any origin. Forgetting the move, or moving twice, does not compile; coverage checks (require_complete_input, require_complete_tier_input) take Synthetic<Span> for the same reason. The third space, INPUT-relative snippet coordinates, arises only inside the diagnostic sink. Introducing the type on 2026-10-18 surfaced one omission: parse_header returned the lowered header without the move, so payload spans (the @Media comma whitespace) came back in wrapper coordinates. The participant-entry and MOR-word fragment APIs, which used to hand their synthetic source to another public fragment API as if it were a document (a second wrapper around the first), lower inside one wrapper now.

The same ownership rule applies to SpanShift on ParseError: document locations and secondary labels shift, but the retained ErrorContext text and its relative highlight do not. Rebasing a diagnostic cannot rewrite an independent context snapshot. Corpus fragment contracts check this public operation against parser-produced rebasing, including the inverse shift.

Word and main-tier fragments use the multi-root grammar directly, so there is no synthetic prefix to subtract. MainTierFragment admits a typed main-tier node from a ParsedFragment, deriving the original input from the same owner that assembled and parsed the newline-extended source. Capacity admission precedes allocation; callers cannot supply an independent original input. The node is admitted only when it covers the complete parse source and the root has no extra or unexpected content. Lowering consumes that proof together with the original input, clipping the aggregate tier/content spans to exclude an appended line terminator. LF and CRLF supplied by the caller remain part of those spans. Trailing garbage or another tier cannot be silently ignored. Separator decoding maps a present alternative and a MISSING head of the same kind through one semantic mapping: the generated missing enum is converted variant by variant into the choice value its kind names (a measured CHECK parity case, *CHI:\t., depends on this). There is no unclassified placeholder: a MISSING node of a kind the choice does not name is not the choice’s.

Marked-token and typed-text decoding share checked node-text admission. Its typed result distinguishes a range outside the supplied source from a range that cuts a UTF-8 code point. Marker stripping receives only the admitted text and still rejects the wrong marker. This boundary proves readable bytes, not identity with the source that produced the tree; producer-bound slices remain the stronger API.

Language-list recovery carries its offending node in a LanguageListFault variant: an unexpected first/subsequent code or an unparsable repeated group. Comma and post-comma whitespace faults have their own variants as well. The consuming reporter owns the corresponding wording and shared diagnostic location, rather than accepting arbitrary message text at separate call sites. Participant recovery follows the same pattern with ParticipantFault: list comma/whitespace/entry expectations retain E506 and list context, while malformed entry structure retains E316 and entry context. Sharing the reporter does not merge those policies or remove unobserved recovery states. Sign-tier token decoding retains either a generated word or the whole admitted recovery-group carrier in SinTokenSource, which retains its checked source text. Whole-group fallback derives complete text from that carrier’s own parent, preserving one token; an incompatible source is refused before lowering. Its direct boundary test does not claim that the fixture itself enters recovery during normal parsing. Morphology feature-value decoding similarly accepts only its generated node type, including kind-proven MISSING placeholders, and uses shared checked text admission. Empty-value and unreadable-source refusals retain their existing diagnostics; source compatibility is not mistaken for tree/source identity. ReadableRecovery::fragment_diagnostic owns the pairing of a source-located span with fragment-local display context. Word-error classifications supply only their code and message; they cannot accidentally use another recovery node’s location or another fragment’s context through this constructor. The generic recovery classifier uses the same constructor for fragment-local diagnostics. Classifications that need a subspan or full-source context remain separate.

Main-tier conversion returns either the model or a private-constructor ReportedMainTierError issued when the producer reports its rejection diagnostic. Speaker admission retains that evidence through conversion, so the fragment consumer needs no synthetic “failed to build” fallback. Recovery diagnostics still precede rejection, and successful models may still carry diagnostics; the result does not certify validity. Header, utterance, participant-entry and dependent-tier adapters, including the standalone parse_header and parse_tiers entry points, use an owned WrappedFragment: its constructor records the actual input boundary as it assembles the source, and both the model projection and diagnostic sink use that boundary. Its diagnostic sink removes that prefix from both primary and secondary spans. Context is projected only when its text exactly matches the owned synthetic source; an independent context retains its own coordinates regardless of length. cargo test -p talkbank-parser --lib api::fragment::tests checks long inputs, related labels and independent context text. These adapters do not use the legacy sink’s length heuristic. Header lowering consumes a HeaderFragment that owns the located node together with its wrapped source. Admission requires the node to account for all caller text: only surrounding whitespace may lie outside it or extend into the synthetic line terminator. A start-only check would accept the first of two headers and discard the second. The public regression covers that refusal boundary, while controls retain folded header content and caller-supplied LF/CRLF. Raw ordinal lookup and document-root navigation remain separate traversal improvements. Header lookup failures carry tree facts in HeaderNotFound, rather than constructing a parse error with invented empty context. The fragment caller attaches the real input and its full span; public fragment rebasing then adds the document origin to the location while leaving that context local. The context_public_api::unlocated_header_reports_the_callers_source_and_origin regression exercises this failure through the public API at origins zero and 200. Complete documents passed to the utterance adapter are recognized through generated typed CST traversal and receive no extra document wrapper. Main-tier recovery collection is a method of the admitted MainTierFragment. The generated SourceBound::descendants() iterator owns its private cursor, starts at the admitted main tier and cannot leave that subtree. Each descendant retains its canonical source slice, including ERROR/MISSING nodes; a runtime range refusal remains a diagnostic rather than a skipped node. Bound error analysis consumes that slice without an independent source argument or another UTF-8 admission. No caller supplies a separate recovery node, source or offset. Missing and error nodes remain reported in source order. Diagnostic ranges are clipped to caller input, excluding an appended newline, and error text uses checked UTF-8 slices.

cargo test -p talkbank-parser --test integration context_public_api reproduces the fragment regression checks: UTF-8 word spans, caller offsets, rejected fragments, and utterances with or without a trailing newline or document headers. These guard against prefix subtraction that collapses valid word spans to zero, subtraction of caller origins from errors, and unremoved synthetic prefixes on rejected utterances and participant entries.

The E326 boundary test exercises both parsers with LF and CRLF, UTF-8 content, and offsets zero and 200. Unsupported-line recovery must identify each skipped line, retain following utterances, and preserve the diagnostic’s local source highlight. The fragment_range_tests public-API controls exercise both backends above 2 GiB, past the former 32-bit coordinate limit, and with overflowing ranges. Document offsets are usize; only the input length is bounded, by tree-sitter’s MAX_PARSER_INPUT_BYTES. They also verify that diagnostic context stays snippet-relative. Synthetic terminators for morphology, phonology and grammatical-relation fragments are parsed at local origin zero; only the extracted caller result is moved into document coordinates. Wrapper allocation separately admits its complete synthetic source size, and trimming a caller newline cannot bypass admission of the original input range. Spans store usize offsets, so no producer narrows an offset; the admitted parser paths use no raw origin casts. The legacy OffsetAdjustingErrorSink is exported by the model crate for compatibility; the tree-sitter parser has no callers of it.

Missing encoding declaration

The document grammar admits an absent @UTF8 anchor as an explicit optional slot. Lowering consumes that generated slot and retains the present headers and utterances without inventing an encoding declaration. Shared validation still rejects the file with E503. The Tree-sitter backend retains the present headers rather than discarding the document and reporting them as missing. The authored E503 example and its declaration-present control exercise this recovery; CLAN CHECK reports the corresponding CHECK (69).

The re2c file parser carries each header’s syntax, admitted source slice and separator together in LocatedHeader. Lowering derives the original full-buffer source and lexer extent from that owner, without accepting a replacement source, so shared missing-header diagnostics derive a real EOF from the final header. The cross-backend missing_encoding_keeps_the_document_and_locates_the_single_refusal test checks retained utterances, exact diagnostics, and EOF locations at zero, nonzero and maximum representable document origins.

Recovery-wrapper suppression requires a private DocumentRecoveryWrapper proof at the document position: every direct child must be a generated document construct or separately reported recovery. A recognizable header beside malformed raw tokens, or a stray header beside a complete document, cannot suppress E316.

DocumentRoot retains the parse owner and the selected document state. It derives the syntax root from that owner and a complete document’s raw node from its generated typed wrapper; neither identity is stored a second time. Lowering finds a complete document even after a recovery sibling, while the diagnostic backstop covers the entire source. This prevents trailing text after @End from validating clean and avoids missing-header cascades when a leading error precedes an otherwise complete document. Private fields prevent callers from combining a document with an unrelated diagnostic scope.

Lowering consumes source-ordered DocumentPart values, rather than extracting document children and discarding outer recovery. ERROR siblings of a concrete document pass through the same producer-bound diagnostic route as errors inside it. A duplicate @End fixture therefore retains E316 when omission of the final newline moves recovery outside the document node. Reconstructed ERROR wrappers do not establish that outer-document boundary: their siblings remain covered by the whole-input backstop, so a lone End after a malformed Begin is not mislabeled as a duplicate. The E501 spec supplies both final-newline regression variants.

Its input is ParsedSource, created by TreeSitterParser::parse_source_incremental. The generated owner retains the exact input supplied to tree-sitter and has no independent tree/string constructor. Document lowering consumes the classification and derives source and capacity from it; there is no separate source parameter. into_tree consumes the association for incremental editing, and reparsing produces a new association. This consuming raw-tree route is distinct from the revision API, which edits a cloned tree while retaining the previous revision’s baseline.

Recovery text uses the generated SourceSlice boundary. Raw node binding checks both tree membership and canonical coordinates: a copied tree-sitter node can be edited without editing its tree. Foreign and independently edited nodes cannot yield a source slice. Required-slot Absent remains a legitimate reconstruction result for incomplete ERROR nodes; lack of a corpus witness does not make it impossible. Standalone word and main-tier admission retain checked source slices as well, with word selection driven by the generated source-file union. Wrapped headers follow WrappedFragment -> ParsedFragment -> HeaderFragment -> Synthetic<Header>: the middle phase parses the wrapper’s own source, header admission binds the selected node through that owner before checking complete input coverage (whole line, or label supplied by the wrapper), and lowering yields a value the wrapper must rebase. Deeper token APIs still retain independent source parameters and remain migration work.

The transitional raw-node text helper returns ParseOutcome<&str>, rejecting out-of-bounds or non-boundary ranges without manufacturing empty text. Its consumers only construct successful model values after a successful read. This is a checked range boundary, not a substitute for source ownership. Source-bound readers instead retain the parse owner and admit child fields through it. In either route, unreadable source is an internal failure, not malformed lexical content attributed to the author. Empty-feature and missing-marker recovery remain separate structural diagnostics. Other-speaker event lowering uses generated required slots; the generator, not a parallel list of child positions and kind strings, owns its CST shape.

Header fragments and full documents share the exhaustive pre-@Begin header decoder. Its generated choice includes @PID, @Window, @Color words, and @Font; routing only @PID through that decoder would let the other three fall through to successful Unknown values. Finite reference-corpus tests compare header, main-tier and utterance fragment models with full-file parsing, carrying the file’s CA semantic context explicitly. The shared decoder consumes a source-bound choice. Document lowering projects it from the associated repeat; fragment lowering classifies its already-bound source slice. Header spans derive from that same choice, not a separate caller argument. PID lowering projects its generated free-text field and admits its range before reading the value. A failed binding propagates as an internal failure, distinct from missing/empty PID recovery. Window, font and color-word decoding use the same generated source-associated leaf admission: their shared reader takes no independent source string and cannot turn a binding failure into a malformed-header fallback. Required-presence admission excludes Error/Absent at the admitted pre-begin envelope positions; lexical Missing and empty-value policies remain. The @Languages decoder likewise retains source ownership through the contents and repeated list fields. Present language codes become model values only after source-bound leaf admission; missing codes and malformed separators retain their existing recovery diagnostics. A producer fault is an internal failure, not an empty language code or a successful partial validation. The @ID language field has its own typed component list: compact and spaced commas are supported without relaxing the whitespace rule for @Languages. Its decoder uses the same language-code admission for strict code leaves and preserves opaque nonempty components for model validation. Empty components retain their existing omission policy; no comma-splitting text parser mirrors the grammar. An omitted entire field builds an empty model list (E510), while recovery and source-binding failures cannot become successful omissions. Date, birth-date and time headers likewise use optional grammar payloads. Their reader distinguishes omission, a recovered slot and a source-read failure. An omission uses the existing model constructor and retains the empty spelling, including the established empty-time representation on the JSON boundary. Validation still reports empty Date as E516 and permits the existing empty Birth/Time forms. Strict date/time tokens win equal-length ties; a longer malformed value stays whole for validation. Participant-header contents are required by the admitted ordinary envelope; their source read remains fallible. A read failure propagates internally and rejects header construction, without adding an empty-participants diagnostic. The source-associated content reader for simple headers and participant metadata carries distinct structural-recovery and producer-failure variants. Only structural recovery can construct an Unknown header; a source-binding failure is reported once at dispatch and rejects construction. Media-body range admission propagates the producer failure before its infallible text reads, instead of returning an unreadable body as a successful unknown header. Nested slots retain the recovery states allowed by their individual proofs; ordinary required-envelope admission is not a blanket recovery-free guarantee. Participant entries own their WriteChat implementation, which the enclosing header writer also uses. Reference-corpus entries exercise the standalone participant fragment API through this canonical wire boundary, without a second formatting implementation or fabricated participant models. Error-spec documents also supply retained main-tier, header and dependent-tier source spans to the standalone fragment APIs. Recovery tests compare local and rebased admission, models and diagnostics, requiring evidence for rejection and caller-owned UTF-8 ranges for diagnostic spans and labels, while leaving diagnostic context relative to the original snippet. Full-document spec tests independently enforce the authored diagnostic claims. Main-tier inputs are also exercised after removing their terminal newline, so the coordinate contract covers the adapter’s synthetic newline boundary. The legacy utterance adapter’s full-document input mode is checked over the same spec corpus, with and without terminal newlines, including documents that cannot supply an admissible utterance. These are coordinate and recovery contracts, not new claims about which whole-file validation rules should fire.

Nested choices receive a generated FromNodeKind implementation when their leaf kind sets are disjoint. The generator retains each leaf’s complete constructor path and refuses ambiguous or composite alternatives. Content recovery uses this classifier; it maintains no parallel list of alternatives. Present content items retain their typed wrapper through dispatch, eliminating the raw-node conversion and impossible second kind refusal.

Syntax completeness does not establish semantic validity; shared validation still owns required headers and other CHAT rules. The LSP owns source-bound analysis snapshots rather than a second parser-level cache-admission API.

At EOF, lowering binds a generated MainTierNode stranded outside its line wrapper to the original parse owner before reusing the normal utterance builder and parse-health transition. For the flattened simple terminal sequence without a final newline, TerminalMainTier pairs the generated grammar tokens with the original source range. Lowering binds the retained speaker, contents and terminator nodes to their original parse owner, then uses the ordinary contents and terminator decoders directly. It does not reparse a source fragment. The diagnostic backstop uses the same structural admission. The E502 example checks retained speech and diagnostics in both newline forms, including maximum representable source origins; leading and trailing recovery-region regressions remain separate.

AST Structure

The resulting ChatFile AST has a recursive content structure:

flowchart TD
    cf["ChatFile"]
    hdr["Headers\n@Languages, @Participants,\n@ID, @Options"]
    utts["Utterances[]"]
    mt["MainTier\nspeaker + content"]
    dt["DependentTiers[]\n%mor, %gra, %pho, %sin, %wor"]
    uc["UtteranceContent"]
    bi["BracketedItem"]
    leaf["Leaves\nWord | ReplacedWord | Separator"]
    group["Groups\nGroup | AnnotatedGroup |\nRetrace | PhoGroup | SinGroup | Quotation"]

    cf --> hdr & utts
    utts --> mt & dt
    mt --> uc
    uc --> leaf & group
    group -->|nested content| bi
    bi --> leaf & group

Parser String Handling

The tree-sitter parser constructs owned model types (e.g., MorWord, GrammaticalRelation) directly from CST text. String-heavy types like PosCategory and MorStem use Arc<str> interning to avoid redundant allocations for repeated values. Short strings in model newtypes use SmolStr for inline storage; its capacity is an implementation detail, not a parser contract.

Editor source revisions

The LSP stores one DocumentAnalysis owning exact source bytes, a tree-sitter CST, the lowered model, and diagnostics. Its constructor is the only route to those artifacts. Reusing the CST first applies an InputEdit computed from its own prior source, so debounced intermediate edits cannot substitute the wrong baseline. Both the edit boundaries and tree-sitter columns use UTF-8 bytes; LSP wire positions remain UTF-16.

Each changed analysis lowers the model and calls ChatFile::validate_with_alignment in full. Previous header errors or absolute AST spans are not copied into a new revision. This removes the independent cache maps and custom validation sequence that missed deleted headers and file-level checks. Tree-sitter incrementality and whole-analysis reuse for identical source remain. More selective semantic reuse needs an explicit dependency and span-identity design plus measurements.

Feature requests during debounce admit cached models/trees only for identical source, otherwise parsing the requested text transiently. Pull diagnostics use the same analysis constructor as pushed diagnostics. A replaced or closed revision cannot commit its analysis, and push results carry the editor version. No cache guard crosses asynchronous publication.

The existing stdio integration binary checks fresh-open/edit parity, skipped revisions and requests during debounce. Its process owner handles shutdown and cleanup; the message inbox preserves interleaved notifications while awaiting responses. This catches production orchestration errors that isolated tree splicing helpers could not.

For a local computation measurement, run the ignored measure_analysis_latency library test with --ignored --nocapture. Optionally set TALKBANK_LSP_BENCH_SOURCE to an existing transcript; it is read without changes. Record build profile and distinguish computation from the 250 ms debounce. The benchmark is intentionally excluded from CI and sets no timing threshold.

This page last changed: 2026-10-27 (commit c0e233d0). The whole book last changed: 2026-10-27 (commit 5bfee2f9).