Metrics#
TabGenEval consists of three metric pillars, each targeting different questions. Fidelity metrics compare generated records with unseen real observations. Memorization metrics measure proximity to the generator’s training records. Utility metrics ask how effective the generated data is relative to real data for downstream machine learning prediction.
Question |
Metric guides |
|---|---|
Are individual columns and their associations preserved? |
|
Is local geometric structure preserved? |
|
Can a classifier distinguish the sources? |
|
Does the generator over-concentrate its observations? |
|
Does it reproduce training records? |
|
How do generated-trained and real-trained predictors perform and agree? |
Each metric guide explains its definition, outputs, missing-data handling, and interpretation.
Pillars and selector groups#
The three pillars describe what you are evaluating. Selector groups choose
which metrics to compute. The Fidelity pillar spans five selector groups;
group=["fidelity"] selects only Shape and Trend.
Pillar |
Selector group |
Metrics selected |
Data for evaluating generated output |
|---|---|---|---|
Fidelity |
|
|
|
Fidelity |
|
|
|
Fidelity |
|
|
|
Fidelity |
|
|
|
Fidelity |
|
|
|
Memorization |
|
|
|
Utility |
|
|
|
Fidelity and utility also support real-only evaluation with test; MLE still
requires Schema.target. When selected individually, near_match_rate and
replication_ratio need only train and generated. Selecting the entire
memorization group also selects dcr_ratio, which requires test.
See data roles for preparing these pools.
To evaluate the full Fidelity pillar:
from tabgen_eval import evaluate
report = evaluate(
data,
group=[
"fidelity",
"conditional_fidelity",
"geometric_fidelity",
"detection",
"concentration",
],
)
report.fidelity presents the fidelity-related results already computed in your
report. Accessing it does not select or calculate additional metrics. See
family views for the report interface and
metric selection for selection rules.