Run an evaluation#
evaluate computes the metrics you select and returns a Report. First,
prepare your data as an EvalData object.
Choose metrics#
Start with the measurements you need:
from tabgen_eval import evaluate
report = evaluate(data, metrics=["shape", "trend"])
print(report.summary())
Use group to select a registered metric group:
report = evaluate(data, group=["fidelity"]) # Shape and Trend
A nonempty metrics list overrides group. Names in both lists must be
registered, and the selected metrics must support your data roles.
With both filters empty, evaluate(data) computes every metric supported by
the supplied roles. For test-based evaluation, this includes MLE and nonlinear
C2ST, so the default requires Schema.target for MLE. XGBoost is included in the
base installation. Select a subset to run
only the metrics you need.
Registered names#
See the pillar-to-group mapping for registered groups, metric names and required data pools.
Discover names and data requirements programmatically:
from tabgen_eval import list_groups, list_metrics
print(list_groups())
print(list_metrics())
An explicit group selects all its members. For example, the complete
memorization group requires train, test, and generated data. See the
metric guides to choose measurements and interpret them.
Use test data alone#
To evaluate a real-data reference without generated data:
from tabgen_eval import EvalData
reference = evaluate(
EvalData(test=holdout, schema=schema),
metrics=["shape", "trend"],
uncertainty={"mode": "reference", "n_replicates": 10},
)
samples = reference["shape"].reference_distribution.samples
This retains ten ordered real-real scores in reference_distribution. Its
.mean summarizes the finite scores; .band_low and .band_high give their
central percentile band. Both point fields are marked absent for this repeated
test-only evaluation. Omit uncertainty to retain one real-real score in .value
without a band. See uncertainty options
and Methodology for the reference and band interpretation.
Test-only MLE follows the same reporting convention for real-trained scores; see its
training and scoring protocol.
Read your results#
Access a metric through its registered name:
shape = report["shape"]
print(shape.value)
For a worked explanation of the output, read your first report. Use the Report API to look up methods, fields and export behavior. For uncertainty, reproducibility, and hardware choices, see uncertainty and execution. For rejected inputs or undefined results, see troubleshooting.