Skip to content

Concepts and boundaries ​

Malloy owns parsing, types, aggregate semantics, and SQL generation. PyMalloy builds Malloy syntax and supplies execution with DuckDB, an embedded SQL database.

From definitions to results ​

text
Expr → Source + View → Draft
                        │
Connection.model(draft) → Model.query(...) → Query.run() → Result
                            └─ source() → ModelSource
Connection.validate(draft, checks) → Validation → bundle(...) → SourceBundle
ValueWhat it holds
ExprA scalar operation, such as a field reference, comparison, or sum
SourceA relation with fields, measures, joins, saved views, or a query pipeline
ViewA source-relative query stage defining grouping, projection, aggregation, or nesting
DraftAn immutable model revision with source identity, optional captured imports, and inputs
ConnectionThe native database owner and shared execution lock for its models
ModelCompiled source, resolved imports, discovered schemas, and query descriptors
QueryA selection that retains its model and binds givens when preparing SQL
ResultMaterialized data, column descriptions, and executed SQL
ModelSourceA closed source graph and retained captured input owners
ValidationDiagnostics and supplied data-check outcomes for one draft revision
InputAn immutable capture of Python data, including its identity and verified file representation

Sources, views, expressions, and drafts build syntax before compilation. A model retains schemas while queries read current data. Recompile after a source or schema change. Running queries covers selection and lifetime.

What gets captured ​

A ModelSource retains the root URL, document kind, text, imported text keyed by URL, and captured inputs in inputs. Imports resolve from that mapping. A missing import fails compilation. Construction checks the record's shape. Compilation or bundling checks import closure.

pm.data(frame) copies materialized Python values into a retained Arrow, a columnar data format, snapshot. The first compilation or widget request materializes verified Parquet, a columnar file format, bytes. Drafts, snapshots, validations, and execution evidence keep those captures alive. External files and database tables remain live dependencies.

A source bundle copies captured inputs and explicitly bound files. Its manifest records source and file hashes, query selection, givens, and validation outcomes. Review catalog tables, SQL readers, remote services, and connection settings before treating it as self-contained. See bundle replay.

What checks establish ​

CheckEvidence
pm.parseThe installed Malloy parser accepts the source
connection.checkNames, types, and schemas resolve with the configured connection
Documentation policySelected objects have nonempty annotations on accepted routes
connection.validate(draft, checks)Each supplied counterexample query returned zero rows
Replay comparisonResults match recorded expectations under an explicit comparison policy

Validation with an empty check mapping establishes compilation and any requested documentation checks. Grain, units, denominators, coverage, and null meaning need explicit definitions and review. A passing assertion describes the observed data at validation time. It cannot establish future data quality or whether a metric matches the intended question.

Descriptions use Malloy's native annotation routes. Dataset-specific intent, review records, and visualization requirements belong to the producing project.

Results and lifetime ​

A native Python connection owns or borrows a DuckDB connection. Each native model owns an embedded compiler on a dedicated thread. Closing a model releases that compiler. Closing the connection closes its child models and owned database. Borrowed databases and transactions remain caller-owned.

Operations serialize across a Python connection's models. A SQL timeout interrupts the statement and leaves a healthy model reusable. A compiler request timeout closes the affected model. Results retain their materialized data after closure.

Browser Python uses asynchronous browser.Connection owners over DuckDB WebAssembly workers. Models retain their connection and file snapshot. Results remain readable after closure. Node and browser JavaScript callers use Session for asynchronous work.

A widget created from source owns a browser session. A widget created from a compiled model borrows its connection. Both publish revision-bound results to Python. Widgets describes their asynchronous state.