Fidelity

Contents

Fidelity#

Fidelity metrics compare generated data with unseen real observations:

  • Shape compares individual column distributions.

  • Trend compares relationships between column pairs.

  • Conditional Fidelity compares distributions, relationships, and frequencies within real-defined subpopulations.

Use them together when good marginal scores could hide lost associations or subpopulation differences. The matched real-real reference puts each score in context. For training-record overlap, see memorization.

Prepare your data, choose metrics, and request details=True to inspect columns, pairs, or slices. Shared uncertainty controls belong to uncertainty and execution; retention behavior is documented under component tables.

Shared example#

The three guides use the same customer example as the visual walkthrough: 600 real holdout rows and 300 generated rows, with numeric visits, spend, and tenure and categorical segment. The toy generator shifts tenure, permutes spend, and repeats one record. Run this setup once to create data for their snippets.

Create the example data
import numpy as np
import pandas as pd

from tabgen_eval import EvalData, Schema, evaluate

def make_data(*, perturb=True):
    """Return mixed-type samples, optionally adding three generator defects."""
    rng = np.random.default_rng(23)

    def sample(size):
        visits = rng.normal(12, 3, size)
        return pd.DataFrame(
            {
                "visits": visits,
                "spend": 20 * visits + rng.normal(0, 15, size),
                "tenure": rng.uniform(1, 12, size),
                "segment": rng.choice(
                    ["local", "regional", "online"],
                    size,
                    p=[0.55, 0.30, 0.15],
                ),
            }
        )

    holdout, generated = sample(600), sample(300)
    if perturb:
        generated["tenure"] += 5
        generated["spend"] = rng.permutation(generated["spend"].to_numpy())
        generated.iloc[:75] = generated.iloc[150].to_numpy()
    return holdout, generated
holdout, generated = make_data()
data = EvalData(
    test=holdout,
    generated=generated,
    schema=Schema(numeric=["visits", "spend", "tenure"], categorical=["segment"]),
)