TabGenEval is a comprehensive, high-performance package for evaluating synthetic tabular data generated by tabular generative models, using real data as the reference. Its evaluation framework is organized around three core pillars: Fidelity, Memorization, and Utility.
TabGenEval offers four key features: (1) comprehensive population- and subpopulation-level evaluations, (2) real-data references, (3) uncertainty quantification, and (4) high-performance evaluation.
Comprehensive evaluation#
Synthetic data should preserve the statistical structure of real data. The fidelity pillar measures how closely synthetic data match an independent real holdout, from individual columns and their correlations to subpopulations and local geometry.
Shape · Trend · Conditional Fidelity · PRDC · C2ST · Concentration
Memorizing training data can inflate fidelity scores, indicate overfitting, and increase privacy risks. The memorization pillar measures the extent to which synthetic data reproduce training records.
DCR · Near-match Rate · Replication Ratio
Synthetic data should maintain predictive power. The utility pillar measures how models trained on synthetic data compare with models trained on real data, using an independent real test set to assess predictive performance.
MLE · Prediction Agreement
Give each score a reference#
Finite-sample variability introduces uncertainty into evaluation scores. TabGenEval provides real-vs.-real references and uncertainty quantification for all metrics at the same sample size as the synthetic-vs.-real comparison, enabling meaningful interpretation of the observed scores.
The real-vs.-real comparison provides a reference for natural variation between finite samples. See Methodology for the procedure, resolution and interpretation of the bands.
Fast evaluation on CPU and GPU#
Some evaluation metrics are computationally expensive, with quadratic complexity in sample size. To enable efficient evaluation of large, high-dimensional tables, TabGenEval leverages optimized CPU operators (compiled C++/ATen) and GPU kernels (compiled CUDA/Triton). Compared to equivalent Python CPU implementations, these optimized CPU and GPU routines achieve 33.4x and 90.8x speedups, respectively, reducing evaluation time to a few seconds.
Across five datasets and four sample sources, average evaluation latency was 233.35 seconds for Python CPU, 6.99 seconds for optimized CPU, and 2.57 seconds for GPU per computation – approximately a 33.4x speedup on optimized CPU and a 90.8x speedup on GPU relative to Python CPU in this benchmark.
Benchmark values, datasets, and evaluation details
Each bar averages 220 recorded calls: 5 datasets x 4 evaluation sources x 11 unique computation groups, covering all 16 metrics. The four sources comprise one real-data reference and three model-generated samples. Shared outputs count once; the reported latency is per computation.
Average evaluation latency
| Execution | Mean seconds per computation | Completed calls |
|---|---|---|
| Python CPU | 233.346 | 220 |
| Optimized CPU | 6.991 | 220 |
| GPU | 2.569 | 220 |
Datasets in this benchmark
| Dataset | Training rows | Test / sample rows | Columns |
|---|---|---|---|
| Video Transcoding Time Prediction | 55,028 | 6,878 | 20 |
| Hotel Booking Demand | 65,134 | 8,142 | 32 |
| Allstate Claims Severity | 150,653 | 18,832 | 131 |
| Covertype | 410,100 | 51,262 | 15 |
| Mercari Price Suggestion | 1,000,000 | 125,000 | 5 |
Test/sample rows give the size of the real test split and of each generated sample file. These are input sizes; each metric follows its own sampling protocol.
The linear axis starts at zero; preparation and remaining first-use work are included. File loading,
imports and synthetic warmup are excluded. Individual metrics and workloads
can favor different devices. Download the
recorded timings and context
or see execution options.
Get started#
Install TabGenEval and choose your execution environment.
Run the five-minute example using CPU.
Choose metrics for your question and available data.
Read The Sharp Bits for interpretation nuances and limitations.