Five-minute quickstart#

This example creates a small real holdout and a generated table with slightly shifted distributions. In your own evaluation, supply an unseen real holdout and the records produced by your generator.

Before running this example, follow the installation guide to prepare the CPU prerequisites for Linux or Apple Silicon macOS.

Run the example#

The complete example below uses native CPU execution and needs no GPU or external dataset. Native CPU operators compile on first use and are cached for future re-use.

"""Compute a small offline report using native CPU execution."""

import numpy as np
import pandas as pd

from tabgen_eval import EvalData, Schema, evaluate

# prepare dataset
rng = np.random.default_rng(42)
holdout = pd.DataFrame(
    {
        "age": rng.normal(40, 10, 400),
        "income": rng.lognormal(10, 0.5, 400),
        "region": rng.choice(["north", "south", "west"], 400),
    }
)
generated = pd.DataFrame(
    {
        "age": rng.normal(43, 10, 200),
        "income": rng.lognormal(10.1, 0.5, 200),
        "region": rng.choice(["north", "south", "west"], 200),
    }
)
eval_data = EvalData(
    test=holdout,
    generated=generated,
    schema=Schema(numeric=["age", "income"], categorical=["region"]),
)

# get a report with captured column and pair scores.
report = evaluate(
    eval_data,
    metrics=["shape", "trend"],
    device="cpu",
    seed=42,
    uncertainty={"mode": "both", "n_replicates": 10},
    details=True,
)
print(report.summary())

From a source checkout, run it with:

python examples/report_walkthrough.py

Read the first scores#

This table is generated from the example when the documentation is built:

metric stat value ci_low ci_high ideal ideal_ci_low ideal_ci_high n_eff
shape shape 0.8543 0.8219 0.8733 0.8973 0.8619 0.9303 100
trend trend 0.7898 0.7640 0.8142 0.7935 0.7635 0.8417 100
  • value is the mean generated-real score across the repetitions.

  • ideal is the mean real-real score at the same resolution.

  • ci_low / ci_high and ideal_ci_low / ideal_ci_high are the synthetic-vs.-real and real-vs.-real percentile bands, respectively.

  • n_eff is the number of rows per side used in each comparison.

Here the holdout has 400 rows and the generated table has 200. With uncertainty on both sides, the default resolution is 100 rows per role (50% of the generated table size). Each repetition draws two disjoint real samples and one synthetic sample of that size. Shape and Trend are higher-is-better similarity scores; a real-real score can be below one because finite samples differ.

The example requests 10 repeated comparisons on both sides using uncertainty={"mode": "both", "n_replicates": 10}. The bands span the 2.5th to 97.5th percentiles of the repeated scores; they describe score variability. Continue with Read your first report for the full printed output.

Plot your report#

The example retains column and pair scores, so these figures use the stored results directly:

report.fidelity.plot(kind="scores").show()
report.fidelity.plot(kind="columns", metric="shape").show()
report.fidelity.plot(kind="pairs", metric="trend").show()

The score plot from the same quickstart report shows its repetition means and uncertainty bands for both comparisons:

The dots and bands summarize the same repetitions; no additional point comparison is computed. On narrow screens, scroll the two panels horizontally.

The visual walkthrough shows interactive examples of a weak column, a broken association and excess concentration, plus saving figures as offline HTML.

Use your own data#

import pandas as pd
from tabgen_eval import EvalData, Schema, evaluate

holdout = pd.read_parquet("real_holdout.parquet")
generated = pd.read_csv("generated.csv")
schema = Schema(numeric=["age", "income"], categorical=["region"])

report = evaluate(
    EvalData(test=holdout, generated=generated, schema=schema),
    metrics=["shape", "trend"],
    uncertainty={"mode": "both", "n_replicates": 10},
)
print(report.summary())
report.to_json("evaluation.json")

Both frames must have the same complete set of uniquely named columns, including columns omitted from the schema. Their column order and row counts may differ. A schema is required. Schema.infer(holdout) explicitly proposes roles; inspect or edit them before passing the result to evaluation. See column roles and the metric guide.

This example filters to Shape and Trend. Omitting both filters selects every metric supported by the input roles, including MLE and nonlinear C2ST. That default needs schema.target for MLE; XGBoost is included in the base installation. Missing prerequisites raise. Use the registered metric table to choose a subset.