Grammar
Status: Current Last updated: 2026-10-20 (commit f7e32e22)
The CHAT grammar is defined in grammar/grammar.js using the Tree-sitter
parser generator. It recognizes document and fragment syntax with error
recovery. Recognition is not proof of complete CHAT support or semantic validity.
Design Principles
Explicit Whitespace
Unlike most tree-sitter grammars, CHAT does not use extras for whitespace. All whitespace is grammar-visible because CHAT’s structure is whitespace-sensitive:
- Tab separates tier prefix from content
- Newline ends tiers
- Line continuation uses tab-at-start-of-line
- Space separates words and annotations
Two-Level Structure
The grammar has two structural levels:
- Document level: headers, utterances,
@Begin/@End - Tier level: main tier content, dependent tier content (each with distinct rules)
Opaque Lemmas
In the %mor tier rules, lemmas are constrained lexical strings retained as
opaque model values. Neither the grammar nor the model decomposes them into
typed compound boundaries or translation glosses. POS, lemma and feature
boundaries are separate syntax. See MOR.
Key Grammar Rules
Document Structure
source_file → full_document | utterance | main_tier | dependent_tier | header | pre_begin_header | standalone_word
full_document → utf8_header?, pre_begin_header*, begin_header, line*, end_header
line → header | utterance
utterance → main_tier, dependent_tiers...
These are schematic summaries, not a second grammar. The actual root is
source_file. Missing required headers may be represented through recovery
and diagnosed later; this does not make their absence valid.
Main Tier
main_tier → star, speaker, colon, tab, tier_body
tier_body → contents, utterance_end
contents → content_item, (whitespace, content_item)...
MOR Tier (UD-style)
mor_contents → mor_content, (whitespace, mor_content)..., terminator
mor_content → mor_word, mor_post_clitic*
mor_word → mor_pos, pipe, mor_lemma, mor_feature*
mor_post_clitic → tilde, mor_word
mor_feature → hyphen, mor_feature_value
POS tags may contain colon-separated refinements such as pro:sub. Lemmas
remain opaque strings. Features are hyphen-separated values that may contain
= for key/value pairs and , within a multi-value spelling; the model does
not infer a UD feature set from the latter.
Grammar Change Workflow
parser.c is generated from grammar.js, never edit it directly.
Use just regen for the consolidated generation batch, review generated
changes and diagnostic observations, then run affected grammar/parser/model
contracts. The Grammar Workflow owns the
details, including the pinned external traversal generator and its current
public-bootstrap limitations. A finite reference sweep is not completeness
evidence; do not copy old test or corpus counts into acceptance criteria.
Conflict Resolution
The grammar uses lexical/syntactic precedence and explicit conflicts. Inspect the owning rule rather than relying on copied numeric precedence values. A change can alter which token the compiled lexer selects, especially for a strict prefix versus a longer malformed value. See token governance.
Generated Artifacts
Running tree-sitter generate produces:
src/parser.c: the C parsersrc/node-types.json: node type metadata
crates/talkbank-parser/src/node_types.rs is GENERATED from it, by
scripts/generate-node-types.js, which also reads scripts/node-type-docs.json
for the per-constant doc comments:
just node-types-check # generate to staging and compare; do not rewrite the tracked file
The doc file is the hand-written half and the only part a human edits; the kind
list comes from the grammar. Both are gated, because both have drifted: the
script was left behind in the repository chatter was extracted from, so for
three months node_types.rs carried a “DO NOT EDIT, auto-generated by
scripts/generate-node-types.js” banner naming a script this repo did not have,
and it drifted to 8 missing kinds and 6 constants for kinds the grammar no
longer had.
This page last changed: 2026-10-20 (commit f7e32e22). The whole book last changed: 2026-10-27 (commit 5bfee2f9).