Holmes-inspecting-tables

TabGenEval is a comprehensive, high-performance package for evaluating synthetic tabular data generated by tabular generative models, using real data as the reference. Its evaluation framework is organized around three core pillars: Fidelity, Memorization, and Utility.

TabGenEval offers four key features: (1) comprehensive population- and subpopulation-level evaluations, (2) real-data references, (3) uncertainty quantification, and (4) high-performance evaluation.

Comprehensive evaluation#

Fidelity

Synthetic data should preserve the statistical structure of real data. The fidelity pillar measures how closely synthetic data match an independent real holdout, from individual columns and their correlations to subpopulations and local geometry.

Shape · Trend · Conditional Fidelity · PRDC · C2ST · Concentration

Metrics
Memorization

Memorizing training data can inflate fidelity scores, indicate overfitting, and increase privacy risks. The memorization pillar measures the extent to which synthetic data reproduce training records.

DCR · Near-match Rate · Replication Ratio

Memorization
Utility

Synthetic data should maintain predictive power. The utility pillar measures how models trained on synthetic data compare with models trained on real data, using an independent real test set to assess predictive performance.

MLE · Prediction Agreement

Predictive utility

Give each score a reference#

Finite-sample variability introduces uncertainty into evaluation scores. TabGenEval provides real-vs.-real references and uncertainty quantification for all metrics at the same sample size as the synthetic-vs.-real comparison, enabling meaningful interpretation of the observed scores.

The real-vs.-real comparison provides a reference for natural variation between finite samples. See Methodology for the procedure, resolution and interpretation of the bands.

Fast evaluation on CPU and GPU#

Some evaluation metrics are computationally expensive, with quadratic complexity in sample size. To enable efficient evaluation of large, high-dimensional tables, TabGenEval leverages optimized CPU operators (compiled C++/ATen) and GPU kernels (compiled CUDA/Triton). Compared to equivalent Python CPU implementations, these optimized CPU and GPU routines achieve 33.4x and 90.8x speedups, respectively, reducing evaluation time to a few seconds.

Across five datasets and four sample sources, average evaluation latency was 233.35 seconds for Python CPU, 6.99 seconds for optimized CPU, and 2.57 seconds for GPU per computation – approximately a 33.4x speedup on optimized CPU and a 90.8x speedup on GPU relative to Python CPU in this benchmark.

Benchmark values, datasets, and evaluation details

Each bar averages 220 recorded calls: 5 datasets x 4 evaluation sources x 11 unique computation groups, covering all 16 metrics. The four sources comprise one real-data reference and three model-generated samples. Shared outputs count once; the reported latency is per computation.

Average evaluation latency

Execution Mean seconds per computation Completed calls
Python CPU 233.346 220
Optimized CPU 6.991 220
GPU 2.569 220

Datasets in this benchmark

Dataset Training rows Test / sample rows Columns
Video Transcoding Time Prediction 55,028 6,878 20
Hotel Booking Demand 65,134 8,142 32
Allstate Claims Severity 150,653 18,832 131
Covertype 410,100 51,262 15
Mercari Price Suggestion 1,000,000 125,000 5

Test/sample rows give the size of the real test split and of each generated sample file. These are input sizes; each metric follows its own sampling protocol.

The linear axis starts at zero; preparation and remaining first-use work are included. File loading, imports and synthetic warmup are excluded. Individual metrics and workloads can favor different devices. Download the recorded timings and context or see execution options.

Get started#