Visualizing reports#

A report can show where a generator differs from real data. The interactive figures use Plotly: hover to inspect values or use the toolbar to save an image. Where a legend is shown, click an entry to show or hide that series.

Start with the real reference#

figure = report.fidelity.plot(kind="scores", metrics=["shape", "trend"])
figure.show()

Dots show the reported estimates: repetition means when uncertainty is enabled for that side, or a single score otherwise. Hover text identifies whether a dot is a point or a mean. Plotting reads retained results without resampling.

In the example below, the generated–real points are shown alongside real–real means and bands, which provide the yardstick. Ten repeated real-real comparisons form the reference percentile bands.

View the source values
metric stat value ideal ideal_ci_low ideal_ci_high
shape shape 0.7042 0.9319 0.9097 0.9516
trend trend 0.6967 0.8937 0.8758 0.9138

These bands describe variation in aggregate scores across sampled comparisons. They are not confidence intervals for the population expected scores. See statistical methodology for interpretation. Component plots below show descriptive point scores, without column, pair or slice uncertainty estimates.

Find a weak column#

Retain components with details=True when evaluating, then plot the stored Shape scores:

report.fidelity.plot(kind="columns", metric="shape").show()
View the source values
column score
visits 0.8233
spend 0.7233
tenure 0.4367
segment 0.8333

The tenure column has the lowest marginal similarity. This locates a discrepancy; it does not establish its cause. To inspect the data themselves, pass the input tables explicitly:

from tabgen_eval.visualization import plot_column

plot_column(holdout, generated, "tenure", schema=schema).show()

Numeric columns use empirical cumulative distributions; categorical columns use frequency bars. They show the supplied tables using predefined sampling for display, so they need not show the exact row subset used for a report’s draw.

Inspect relationships#

report.fidelity.plot(kind="pairs", metric="trend").show()
View the source values
column_1 column_2 score
visits spend 0.4381
visits tenure 0.9957
visits segment 0.7100
spend tenure 0.8664
spend segment 0.7100
tenure segment 0.4600

Trend compares each column pair. A heatmap makes weak pairs visible even when the aggregate hides them. An undefined entry is distinct from a low score. Compare the actual numeric relationship with a bounded scatter plot:

from tabgen_eval.visualization import plot_column_pair

plot_column_pair(holdout, generated, "visits", "spend", schema=schema).show()
View the source values

Real input: 600 rows. Generated input: 300 rows. The generated spend values were permuted before repeating 75 records.

The inputs show a strong real association and a weaker generated association. The generated point pile also reflects the repeated record in this example. This view displays all 600 holdout and 300 generated rows; the report’s metric comparisons use matched 300-row samples instead. Scatter plots use at most max_points=3000 rows per table by default; set the sampling seed to reproduce a view of larger inputs.

Inspect a conditional slice#

Choose a conditioning column and one retained slice statistic:

report.fidelity.plot(kind="slices", metric="cond_shape", column="segment", stat="shape").show()
View the source values
column label reference_count candidate_count reference_total candidate_total eligible rare shape lower upper left_closed right_closed
segment regional 94 144 300 300 True False 0.4679 NaN NaN False False
segment online 47 29 300 300 True False 0.6356 NaN NaN False False
segment local 159 127 300 300 True False 0.7803 NaN NaN False False

These scores compare the other columns within each real-defined segment. Hover for counts and eligibility. They summarize the fixed generated–real comparison; real–real slice scores are not retained. To inspect frequency recovery instead, use metric="recovery", stat="recovery".

Compare concentration#

report.fidelity.plot(kind="concentration").show()
View the source values
radius_id radius reference_mean candidate_mean mean_ratio reference_upper candidate_upper upper_ratio reference_n candidate_n
radii_1e-3 0.0010 0.0000 0.0635 6354516.0502 0.0000 0.2508 2.508361e+07 300 300
q05 0.1784 0.0503 0.0810 1.6113 0.1070 0.2508 2.343700e+00 300 300
q25 0.4789 0.2449 0.2414 0.9858 0.4348 0.3779 8.692000e-01 300 300

The fixed small radius exposes the repeated generated record: generated local mass is positive while real local mass is zero. The upper profile is a quantile of local masses, not an uncertainty bound. Larger real-derived radii probe other scales. This figure uses the stored point profiles and actual radii; the scalar concentration gaps have their own real–real comparisons in the report.

PRDC’s four geometric-fidelity scores use the same overview interface:

report.fidelity.plot(kind="scores", metrics=["precision", "recall", "density", "coverage"]).show()

See the PRDC guide for this example’s result.

Report families and available views#

All specialized reports inherit lookup, tables and exports from Report. Select a family from a mixed report without recomputing its results:

fidelity = report.fidelity
memorization = report.memorization
utility = report.utility

View

Plot kinds

FidelityReport

scores, Shape/Conditional columns, Trend pairs, Conditional slices, concentration

MemorizationReport

scores for retained DCR, near-match and replication-ratio summaries

UtilityReport

scores, utility for MLE raw MAE/AUROC, generated-to-real score ratios and shared-test prediction statistics

PRDC belongs to fidelity because it measures local geometric fidelity. Missing component tables require a new evaluation with details=True; plotting cannot recover data that were not retained. Memorization plots show available scalar summaries, rather than reconstructing distance distributions from quantiles.

Customize, save and reproduce#

Every plotting function returns a Plotly Figure, so normal Plotly controls apply:

figure = report.fidelity.plot(kind="columns", metric="shape", top_n=10)
figure.update_layout(title="Marginal fidelity by column")
figure.write_html("shape.html", include_plotlyjs=True)

That HTML file includes its plotting library and works offline. Numerical tables and report JSON remain available through the report API.

Run the complete CPU-only example from a source checkout:

python examples/visualization_walkthrough.py --output visualization-demo

It writes seven interactive figures and a JSON report. The files share one local Plotly JavaScript bundle, and require no network access to open. Download the example report.

def make_report(holdout, generated):
    """Evaluate once, retaining the component scores used by the figures."""
    return evaluate(
        EvalData(
            test=holdout,
            generated=generated,
            schema=Schema(numeric=["visits", "spend", "tenure"], categorical=["segment"]),
        ),
        metrics=[
            "shape",
            "trend",
            "cond_shape",
            "cond_trend",
            "recovery",
            "precision",
            "recall",
            "density",
            "coverage",
            "concentration",
        ],
        details=True,
        device="cpu",
        seed=23,
        uncertainty={"mode": "reference", "n_replicates": 10},
    )