Configuration & Diagnostics Reference

Gram provides a rich, unified telemetry and configuration framework governing the lexical analysis subsystem.

1. Exhaustive Configuration Options

All lexer flags reside in gram.config. When running inside a Session, the lexer reads overrides from session.config first before falling back to gram.config.

2. Diagnostic Telemetry & Execution Tracing

LexerState inherits directly from Diagnostics (gram.utilities.Diagnostics), providing deep, structured telemetry for language frontends.

Hierarchical Diagnostic Stack

When LEXER_ADD_INFO=True, the lexer automatically creates nested telemetry nodes:

• LEXER: Initialized at lexer creation, tracking total lines and logfile configuration.
• LexingProcess: Active throughout process(), tracking step-by-step token production.
• VISITOR-STRING / VISITOR-NUMBER / VISITOR-WORD / VISITOR-SYMBOL: Nested under LexingProcess for each dispatched character.

from gram import config
from gram.core.lexer import Lexer, words
words.add_keyword("let", allow_override=True)

# Enable detailed logging to a diagnostic file:
config.INFO_GENERATE_LOGFILE = True
lexer = Lexer("let x = 42;", logfile="my_compiler.log")
tokens = lexer.process()

# Access the active diagnostic stack:
stack = lexer.stack
print(f"Recorded diagnostic nodes: {len(stack.main.nodes)}")

Disabling Telemetry for High Performance

In production compilers, test suites, or performance benchmarks, set LEXER_ADD_INFO = False:

• Skips creation of StackInfo nodes.
• Disables note string formatting and sender calls.
• Executes the scanning loop at raw Python speed with zero telemetry overhead.

3. Error Handling & Exit Policies

When the lexer encounters invalid syntax or malformed tokens, it invokes LexerState.report_error(err) or LexerState.fail(name, code, *caution).

def report_error(self, err: errors.Error) -> errors.Error:
    res = self.record_failure(err)
    if (
        res is not None
        and not getattr(err, "_suppress_raise", False)
        and self.get_config("ERROR_EXIT_ON_ERROR", True)
    ):
        err.raise_error()
    return err

Collecting Errors Without Process Exit

Since version 2.0.0b1, config.ERROR_EXIT_ON_ERROR defaults to True. To collect errors programmatically (for example, in Language Server diagnostics, IDEs, or unit tests):

from gram import config, errors
from gram.core.lexer import Lexer, words
words.add_keyword("dummy", allow_override=True)

# 1. Disable exit on error:
config.ERROR_EXIT_ON_ERROR = False
config.ERROR_HIDE_CONSOLE = True

# 2. Tokenize malformed source:
lexer = Lexer('int x = "unclosed string')
tokens = lexer.process()

# 3. Inspect collected diagnostic errors:
for failure in lexer.failures:
    print(f"Error [{failure.code}]: {failure.message}")
    for hint in failure.caution:
        print(f"  Hint: {hint}")

Registering Error Hooks (ErrorAction)

You can intercept and modify error handling dynamically using error hooks:

from gram.utilities import Diagnostics, ErrorAction

def my_error_hook(err):
    # Suppress terminal exit while recording telemetry:
    return ErrorAction.SUPPRESS_RAISE

Diagnostics.register_error_hook(my_error_hook)

4. OSGDC Lexer Error Codes Matrix

Gram adheres to the OSGDC error code standard across all failure conditions in gram.errors:

5. Multi-Language Sessions & Lexicon Scoping

When building multi-language frontends, embedded DSLs, or parallel test suites, use gram.sessions.Session to prevent cross-contamination of keywords, symbols, and lexer configurations.

from gram.sessions import Session
from gram.core.lexer import Token

# 1. Create independent sessions for two distinct DSLs:
session_sql = Session("SQL_Dialect", "Many")
session_c = Session("C_Dialect", "Many")

# 2. Configure SQL session:
session_sql.init()
session_sql.config["LEXER_COMMENT_TOKEN"] = "--"
session_sql.add_keyword("select")
session_sql.add_keyword("from")
session_sql.migrate()

# 3. Configure C session:
session_c.init()
session_c.config["LEXER_COMMENT_TOKEN"] = "//"
session_c.add_keyword("int")
session_c.add_keyword("return")
session_c.migrate()

# 4. Tokenize with session isolation:
tokens_sql = session_sql.tokenize("select * from users; -- SQL query")
tokens_c = session_c.tokenize("int main() { return 0; } // C code")

# SQL session only recognizes SQL keywords:
print([t.value for t in tokens_sql if t.token == Token.KEYWORD])
# ['select', 'from']

# C session only recognizes C keywords:
print([t.value for t in tokens_c if t.token == Token.KEYWORD])
# ['int', 'return']

Inside Session.init(), Gram calls words.bind_session(session), routing all loose calls to words.add_keyword() and words.add_group() directly to the session's internal catalogs until session.migrate() is called.