Fidelity
Fidelity metrics compare generated data with unseen real observations:
Shape compares individual column distributions.
Trend compares relationships between column pairs.
Conditional Fidelity compares distributions, relationships,
and frequencies within real-defined subpopulations.
Use them together when good marginal scores could hide lost associations or
subpopulation differences. The matched real-real reference
puts each score in context. For training-record overlap, see
memorization.
Prepare your data, choose metrics,
and request details=True to inspect columns, pairs, or slices. Shared
uncertainty controls belong to uncertainty and execution;
retention behavior is documented under component tables.
Shared example
The three guides use the same customer example as the
visual walkthrough: 600 real holdout
rows and 300 generated rows, with numeric visits, spend, and tenure and
categorical segment. The toy generator shifts tenure, permutes spend,
and repeats one record. Run this setup once to create data for their snippets.
import numpy as np
import pandas as pd
from tabgen_eval import EvalData, Schema, evaluate
def make_data(*, perturb=True):
"""Return mixed-type samples, optionally adding three generator defects."""
rng = np.random.default_rng(23)
def sample(size):
visits = rng.normal(12, 3, size)
return pd.DataFrame(
{
"visits": visits,
"spend": 20 * visits + rng.normal(0, 15, size),
"tenure": rng.uniform(1, 12, size),
"segment": rng.choice(
["local", "regional", "online"],
size,
p=[0.55, 0.30, 0.15],
),
}
)
holdout, generated = sample(600), sample(300)
if perturb:
generated["tenure"] += 5
generated["spend"] = rng.permutation(generated["spend"].to_numpy())
generated.iloc[:75] = generated.iloc[150].to_numpy()
return holdout, generated
holdout, generated = make_data()
data = EvalData(
test=holdout,
generated=generated,
schema=Schema(numeric=["visits", "spend", "tenure"], categorical=["segment"]),
)