Token Catalog & Symbol Dispatch

Gram represents all syntactic units through two central constructs:

• Token (Enum): The formal, immutable enumeration of all recognized lexical token types.
• TokenType (class): The concrete lexeme container emitted by Lexer, encapsulating the formal token type, source value, coordinates, and lexical flags.

1. The Token Enumeration

All standard tokens reside in gram.core.lexer.tokens.Token. They are organized into 11 logical categories:

1. Structure and Flow

2. Literals

3. Arithmetic Operators

4. Assignment Operators

5. Comparison Operators

6. Logical and Bitwise Operators

7. Delimiters

8. Punctuation and Special Symbols

9. Arrows

10. Unicode (Logic & Mathematics)

11. Special States

• Token.ERROR: Emitted when a character or construct cannot be tokenized.
• Token.UNKNOWN: Unclassified token state.
• Token.EMPTY_LINE: Diagnostic marker for completely blank source lines.

Helper Classmethods on Token

from gram.core.lexer import Token

# 1. Lookup by string name (case-insensitive)
tok1 = Token.from_string("plus")       # Token.PLUS
tok2 = Token.from_string("IDENT")      # Token.IDENT

# 2. Lookup by integer ordinal code
tok3 = Token.from_code(0)             # Token.EOF

# 3. Complete immutable tuple of all tokens
all_tokens = Token.get_tokens()
print(f"Total defined tokens: {len(all_tokens)}")

2. The TokenType Container

Every lexeme emitted by the lexer is an instance of TokenType:

class TokenType:
    __slots__ = ("col", "description", "flags", "line", "token", "value")

Attributes

• token (Token | CustomToken): The formal token type enum member or dynamic token.
• value (Any): The scanned payload from the source (e.g., 'my_variable', 42, 'string text').
• line (int): Physical source line number (1-based).
• col (int): Column offset (0-based raw index).
• description (str): Contextual tag (e.g., string prefix such as "L" or "u8").
• flags (TokenFlags): Bitwise flags attached during tokenization (e.g., SIMPLE_QUOTE, CHAR).

Key Methods

long() -> int

Computes the ending column coordinate of the token: $$\text{long} = \text{col} + \text{len}(\text{str}(\text{value}))$$

formatter(msg: str) -> str

Interpolates token placeholders into diagnostic strings:

token = TokenType(Token.IDENT, "counter", line=11, col=4)
print(token.formatter("Error at $name '$value' ($line:$col, end: $long)"))
# "Error at IDENT 'counter' (12:4, end: 11)"

Structural Equality & Hash Contract

• Equality (__eq__): Two TokenType instances are equal if both their token and value match. In addition, comparing a TokenType directly against a Token or CustomToken checks if self.token == other.
• Hashing (__hash__): Hashes hash(self.token). This ensures agreement between set/dict lookups across both TokenType instances and bare Token members.

3. Dynamic Tokens: CustomToken

CustomToken enables language developers to synthesize dynamic token types at runtime without modifying Gram's immutable Token enum:

from gram.core.lexer import CustomToken, Token

# Create a dynamic token for regex literals or domain-specific markers:
REGEX_TOKEN = CustomToken("REGEX", description="Regular expression literal")

# Equality works across strings, other CustomTokens, and Token types:
print(REGEX_TOKEN == "REGEX")         # True
print(repr(REGEX_TOKEN))              # "Token.REGEX"

Combinators like Tokenize and Item seamlessly match both native Token members and CustomToken instances.

4. Lexical Flags: TokenFlags

Attached to token.flags via gram.core.lexer.flags:

class TokenFlags(IntFlag):
    NONE = 0
    SIMPLE_QUOTE = 1 << 0
    CHAR = 1 << 1

• flags.SIMPLE_QUOTE: Set on string literals delimited by single quotes ('...') or double quotes ("...").
• flags.CHAR: Set when a string literal's payload has an exact length of 1 (e.g., 'a' or "x").

from gram.core.lexer import Lexer, flags, words
words.add_keyword("dummy", allow_override=True)

tokens = Lexer("'c' \"long string\"").process()
char_tok, str_tok = tokens[0], tokens[1]

print(bool(char_tok.flags & flags.CHAR))          # True
print(bool(char_tok.flags & flags.SIMPLE_QUOTE))  # True
print(bool(str_tok.flags & flags.CHAR))           # False

5. The Operator Engine & MAP_SYMBOLS

Gram resolves operators through a specialized bidirectional table, MAP_SYMBOLS (_MapSymbolsDict):

• Forward lookup: MAP_SYMBOLS["+"] -> Token.PLUS
• Reverse lookup by Token: MAP_SYMBOLS[Token.PLUS] -> "+"
• Reverse lookup by Name: MAP_SYMBOLS["PLUS"] -> "+"

Maximal Munch Resolution

Gram enforces maximal munch (longest-match): operators are matched using tokens.get_sorted_symbols(), which sorts registered symbols by length descending:

# '<<=' (len 3) is checked before '<<' (len 2) and '<' (len 1)
# '**' (len 2) is checked before '*' (len 1)
# '->' (len 2) is checked before '-' (len 1)

Custom Symbol Registration: insert_symbol & disabled_symbol

You can extend or modify the operator table at runtime:

from gram.core.lexer import Token, disabled_symbol, insert_symbol

# 1. Register a new custom operator for an existing Token member
insert_symbol("<>", Token.NOT_EQUAL)

# 2. Disable an operator to avoid collision with custom syntax
old_token = disabled_symbol("§")

Validation Invariants in insert_symbol:

• symbol must be a non-empty str with no whitespace characters.
• token must be an existing member of Token.
• If symbol is already mapped to a different token, insert_symbol() raises ValueError to prevent accidental overwrites (use disabled_symbol() first).
• Calling insert_symbol() or disabled_symbol() automatically rebuilds the reverse lookup index and the length-sorted maximal-munch list in place via _sync_symbol_indexes().