The CHAT Word

Status: Current Last modified: 2026-10-27 (commit c0e233d0)

“Word” is the most complex and most misunderstood concept in CHAT. This chapter documents what a word actually is, how the grammar parses it, and how the Rust model represents it. If you maintain this codebase, you will encounter word-level bugs. This chapter exists so you can understand them.

Whitespace and structured content

Whitespace is explicit in the grammar and usually separates main-tier content items. It is not a universal tokenizer: a contiguous sequence can contain several grammar tokens, and annotations or groups have their own structure. Do not derive CHAT words by splitting text on whitespace.

*CHI:   hello world .
        ^^^^^              word: "hello"
              ^^^^^        word: "world"

The grammar uses extras: $ => [] – no implicit whitespace. Whitespace nodes (whitespaces, space) are explicit in the CST. Tree-sitter does not skip whitespace between tokens. This is the foundation of every tokenization decision in the grammar.

Adjacency matters, but it works together with the grammar’s alternatives and precedence rules. A recognized structure can still require a validation error.

Word Structure

A word in the grammar is standalone_word – a sequence of an optional prefix, a required body, optional suffixes, and an optional POS tag.

The following diagram is an orientation to the structure, not an exhaustive grammar or a claim that its children may appear in any order.

flowchart TD
    sw["standalone_word\n(grammar.js, prec.right 6)"]

    zero["zero\n'0' -- omission prefix"]
    wp["word_prefix\n'&-' filler | '&~' nonword | '&+' fragment"]

    wb["word_body\n(required)"]
    fm["form_marker\n@b, @c, @d, @z:label, ..."]
    wls["word_lang_suffix\n@s, @s:eng, @s:eng+fra"]
    pos["pos_tag\n$n, $v, $adj, ..."]

    sw -->|"optional prefix"| zero & wp
    sw -->|"required"| wb
    sw -->|"optional suffix"| fm & wls
    sw -->|"optional"| pos

    ws["word_segment\npure spoken text"]
    short["shortening\n'(text)' omitted sound"]
    sm["stress_marker\nprimary or secondary"]
    len["lengthening\n':' one or more colons"]
    op["overlap_point\none of four brackets"]
    cae["ca_element\nsingle CA marker"]
    cad["ca_delimiter\npaired CA marker"]
    ub["underline_begin\ncontrol char pair"]
    ue["underline_end\ncontrol char pair"]
    cm["'+'\ncompound marker"]

    wb -->|"examples of ordered children"| ws & short & sm & len & op & cae & cad & ub & ue & cm

The exact alternatives are owned by standalone_word, word_body and _word_marker in grammar/grammar.js. In particular, the form position also admits repeated_form_marker for diagnosis; parsing that shape does not certify a valid form combination.

word_body has two branches:

1. Standard start: the word begins with word_segment, shortening, or stress_marker, followed by any number of body children.
2. Marker-initial: one or more leading overlap, CA, underline or syllable-pause markers precede a word segment, shortening or stress marker. Repetition permits stacked notation such as °↑hello°; the leading prefix alone does not satisfy this branch. A leading syllable pause is represented for diagnosis, not silently accepted as valid placement.

Lengthening and + (compound marker) are excluded from starting a word body. This is how standalone : falls through to separator(colon) – see Tokenization Ambiguities below.

The word_segment Purity Invariant

word_segment excludes the structural characters assigned separate grammar roles. Word-internal markers are separate children in word_body, not hidden inside a segment. This is a structural invariant, not proof that every parsed segment is valid lexical material or an accurate transcription.

This is a hard invariant with three consequences:

1. Lexical projection uses typed leaves. cleaned_text() consumes lexical_parts(), including its segment-repetition state, rather than stripping characters from raw source.
2. Ordinary word markers have typed model variants. Validation can inspect them without another text parser. The @u phonetic folding described below is a distinct conversion path, not a one-to-one marker mapping.
3. Editors can use structured CST nodes. The availability of a node is not proof that every editor feature has been implemented for it.

How it works

word_segment is a DFA token at prec(5) with a regex that excludes all structural characters. The exclusions are generated from the symbol registry (grammar/src/generated_symbol_sets.js) – never hand-written.

word_segment: $ => token(prec(5, seq(
  WORD_SEGMENT_FIRST_RE,    // generated: excludes structural chars + '0' at start
  WORD_SEGMENT_REST_RE,     // generated: excludes structural chars
))),

Examples of excluded character classes

The following classes illustrate why a segment is not an opaque word token. Some start annotations or groups instead of becoming word-internal nodes. The generated exclusion sets, not this overview, own the complete inventory.

One first-character-only exclusion: 0 is excluded from the first character of word_segment (it is the omission prefix). 0 in non-initial positions can be represented: 200, h0me, abc0 can parse. Their lexical validity still depends on the applicable language and word rules.

The Rust Data Model

Word struct

The Word struct (crates/talkbank-model/src/model/content/word/word_type.rs) is the canonical typed representation. This field sketch omits attributes and the private derived-text cache; consult the source for construction APIs:

pub struct Word {
    pub span: Span,
    pub word_id: Option<SmolStr>,
    content: WordContents,
    pub category: Option<WordCategory>,
    pub form_type: Option<FormType>,
    pub lang: Option<WordLanguageMarker>,
    pub part_of_speech: Option<SmolStr>,
    pub inline_bullet: Option<Bullet>,
}

Key fields:

• raw_text(): a derived spelling, including typed markers, also emitted as the JSON raw_text field. It is not stored independently and is not an exact source slice. See the computed-field contract.
• content: a WordContents (SmallVec-backed sequence of WordContent elements). This is the structured decomposition. Most words have 1-2 elements; SmallVec avoids heap allocation for the common case. Access it with content() and typed mutation methods, which invalidate cached cleaned text.
• category: optional prefix (Omission, CAOmission, Filler, Nonword, PhonologicalFragment).
• form_type: optional @ suffix (@c child-invented, @d dialect, @z:label user-defined, etc.). The user-defined form requires the colon and a label (@z:label); a colon-less marker such as @zzz is not a valid form and is rejected with E203 (matching CLAN CHECK 147).
• lang: optional @s language marker (Shortcut, Explicit, Multiple, Ambiguous).
• part_of_speech: optional $ tag.

WordContent enum

WordContent (crates/talkbank-model/src/model/content/word/content.rs) is the enum of everything that can appear inside a word body. Each variant records typed content. Most ordinary marker variants correspond to grammar children; phonetic folding is a context-dependent conversion rather than a one-to-one node mapping.

cleaned_text()

Word::cleaned_text() concatenates the Spoken contributions produced by lexical_parts(). That iterator classifies Text, Phonetic and Shortening leaves in their segment-repetition context: text inside a ↫...↫ pair is Repeated, not Spoken, and is excluded from the cleaned projection. Other structural leaves contribute no text. For example, ↫b-b-b↫boy projects to boy, while ∆fast∆ retains fast.

The implementation is owned by Word::compute_cleaned_text in word_type.rs and WordLexicalParts in word/lexical.rs. Consumers needing repeated sounds or compound/clitic boundaries should retain those typed leaves rather than try to reconstruct them from cleaned text. This projection is not a universal input contract for every NLP engine and is not byte-preserving serialization.

@u phonetic forms are typed phonetic content

A @u word is a phonetic transcription (UNIBET or, more usually, IPA) standing in a word slot: used when the orthographic word is unknown, unintelligible, or a paraphasia, frequently the spoken side of a [: target] replacement in aphasia data (rɛmbə˞@u [: remember]). Because its content obeys phonetic conventions rather than orthographic word conventions, the parsers fold a @u word’s body into a single WordContent::Phonetic(WordPhonetic) node instead of Text. This makes “orthographic rules apply to orthographic words only” a property of the model: word-hygiene rules structurally cannot reach phonetic content. The phonetic string itself is deliberately lenient (IPA, ASCII UNIBET, and X-SAMPA all pass; only non-emptiness is enforced), matching the %pho tier’s PhoWord stance. In to-json output the node appears as {"type": "phonetic", "content": "..."}, and cleaned_text remains the phonetic string verbatim. The scope is @u only: sibling special forms (@b, @o, …) remain orthographic words.

Examples:

The result is cached via OnceLock on first access.

What is included in cleaned_text vs what is stripped

The following table describes lexical contributions. Text-bearing leaves are included only outside segment-repetition spans; the iterator, not a second character-filtering implementation, owns that context.

Examples of characters that can remain in a segment:

• Letters (all Unicode)
• Digits (in non-initial position; 0 in initial = omission prefix)
• Hyphen (-), part of word text, e.g., ice-cream, self-conscious
• Apostrophe ('), contractions, e.g., don't, it's
• Hash (#), appears in some transcription conventions
• Underscore (_), compound boundary in some conventions

Characters NOT in word_segment (excluded by symbol registry): See the full exclusion table in Precedence Decisions in the grammar docs.

Consumer boundary

Do not replace the typed projection with a downstream list of characters to strip. Such a list loses segment-repetition context and can remove legitimate lexical or phonetic content. Check a downstream consumer’s actual input contract separately; similarity to an older tool’s cleanup function does not establish equivalent behavior.

The Six Tokenization Ambiguities

CHAT was designed for human readability, not machine parsing. Six characters have context-dependent meanings that the grammar must disambiguate. Full details with proof grammars are in grammar/docs/tokenization-rules.md and grammar/docs/precedence-decisions.md. What follows is a summary for orientation.

1. Overlap markers (⌈⌉⌊⌋)

Adjacent to text = part of the word. Space-separated = standalone overlap_point. A historical alternative would put edge markers at the top level while keeping interior markers in-word. Its design history and experimental observations, not an approved implementation plan, are documented in Overlap Marker Binding.

Yeah⌋⌈2 hey      two words: "Yeah⌋⌈2", "hey"
Yeah ⌋ ⌈2 hey    four content items: "Yeah", ⌋, ⌈2, "hey"

The enclosing word_body owns adjacent markers as separate overlap_point children. word_segment never consumes those characters. A space-separated marker can instead be a top-level content item; CST kind and structural ownership are different questions.

2. Zero/omission prefix (0)

Adjacent to word body = omission prefix. Space-separated = action marker.

0die              ONE word: standalone_word(zero, word_body("die"))
0 die             TWO tokens: nonword(zero), word("die")

standalone_word at prec.right(6) beats nonword at prec(1). The extras: [] setting prevents whitespace from being skipped between zero and word_body. The zero token is inlined directly into standalone_word (not through word_prefix) because tree-sitter’s precedence does not propagate through intermediate rules. This was proven empirically with a minimal test grammar – see grammar/docs/precedence-decisions.md.

3. CA parenthetical vs shortening

In CA mode (@Options: CA), a fully parenthesized word (word) is an uncertain/omitted word (CAOmission), semantically equivalent to 0word. Partially parenthesized hel(lo) is always a shortening.

@Options: CA
*CHI:   (ja) .            CAOmission: uncertain "ja"
*CHI:   hel(lo) .         Shortening: "(lo)" is the shortened part

Distinguishing these requires file-level context (@Options header). The parser sets WordCategory::CAOmission when the word is fully parenthesized in CA mode. Isolated parser.parse_word_fragment() calls cannot determine CA mode – they need a FragmentSemanticContext.

4. Colon – lengthening vs separator

Inside a word (after text): prosodic lengthening. Standalone: separator.

no::              ONE word: Text("no") + Lengthening(2)
hello : world     separator(colon)

The DFA always produces lengthening for : (higher precedence). But word_body rejects lengthening as a first element, so standalone : cannot form a valid word and falls through to separator(colon). This is the “constrain the parser, not the DFA” pattern.

5. Plus (+) – compound vs terminator vs linker

Inside a word: compound marker. At line end: terminator prefix. At line start: linker prefix.

ice+cream         ONE word with compound marker
and then +...     terminator: trailing_off (prec 10 beats prec 5)
+< but I +/.      linker: lazy_overlap, terminator: interruption

Terminator and linker alternatives compete with word structure. An internal compound + is its own child, not part of a longer word_segment token. Do not infer compound validity from grammar recognition alone: marker-position validation remains a separate obligation.

6. Bracket annotations vs plain brackets

Bracket annotations ([= text], [=! text], [% text]) use prec(8) prefix tokens to beat generic bracket handling.

Design: Structured Word Content

standalone_word is a structured grammar, not an opaque token. Every word-internal marker is a separate child of word_body. A historical precursor is kept in grammar/docs/pre-coarsening-grammar.js.reference; the following is that historical vocabulary, not a replacement for the current grammar rules:

word_content: $ => choice(
  $.word_segment,
  $.shortening,
  $.stress,
  $.colon,
  $.caret,
  $.tilde,
  $.plus,
  $.overlap_point,
  $.ca_element,
  $.ca_delimiter,
  $.underline_begin,
  $.underline_end,
),

The design decisions that follow from this:

1. All marker characters are excluded from word_segment, using the symbol registry as the single source of truth for the exclusion sets.
2. Each marker type is a separate CST child in word_body, so editors get typed nodes and validation finds structural markers without re-parsing.
3. The model preserves typed word content, with explicit conversion rules such as phonetic folding. cleaned_text() uses the lexical-contribution iterator instead of scanning raw spelling for markers.
4. The word_segment purity invariant is a gate: a structural marker is never consumed by word_segment.

The result is one parser, one source of truth for exclusions, and typed markers from grammar through model.

Testing: The word_segment Purity Gate

The purity invariant, each structural marker produces a separate CST child rather than being consumed by word_segment, is enforced by a group of tree-sitter corpus tests under grammar/test/corpus/generated/word/. Each *_in_word_lint.txt file embeds a structural marker inside a word and asserts the CST splits the word appropriately:

Underline and stress invariants are covered by corpus tests elsewhere in grammar/test/corpus/ and by the parser-equivalence tests in crates/talkbank-parser-tests/. Each construct has its own test file, as the spec generators produce from the spec sources.

How to add a new purity-style test

If you add a new structural marker to the grammar:

1. Add its characters to the symbol registry (spec/symbols/symbol_registry.json).
2. Run just symbols-gen to regenerate the exclusion sets.
3. Add a spec in spec/constructs/ that embeds the marker inside a word; regenerate the affected grammar/parser fixtures with the current spec/tools commands from Spec Workflow so a per-construct test fixture is created in grammar/test/corpus/generated/word/. Verify the CST output names each marker as its own child.
4. Follow the current Grammar Workflow and Spec Workflow for regeneration and affected parser, equivalence and roundtrip checks. Review authored CST expectations separately: generated fixture currency alone does not prove the authored fragment agrees with the implementation.

Key Source Files

This page last changed: 2026-10-27 (commit c0e233d0). The whole book last changed: 2026-10-27 (commit 5bfee2f9).