Command-line evaluation

Command-line evaluation#

Use tabgen-eval to evaluate CSV or Parquet files. Supply the input roles and save your column declarations in a schema JSON file, for example schema.json:

{"numeric": ["age", "income"], "categorical": ["region"], "target": null}

See Prepare evaluation data for column meanings and input roles.

Evaluate files#

Compute fixed fidelity scores from CSV files:

tabgen-eval --test real_test.csv --generated generated.csv --schema schema.json \
  --metrics shape trend

Request repeated scores for both comparisons from Parquet files and save them:

tabgen-eval --test real_test.parquet --generated generated.parquet --schema schema.json \
  --metrics shape trend --uncertainty both --n-replicates 25 \
  --include-replicates --out evaluation.json

Parquet input uses pandas with PyArrow, included in the base installation.

--metrics selects named metrics; --group selects whole families. A metric selection overrides the group selection. Without either filter, all compatible metrics are selected. With test data, these include MLE, which requires a declared target. See Evaluate metrics for metric names and required data roles.

Add --train training.csv to include the generator’s actual training data. Test-only evaluation supplies only --test. --uncertainty accepts none (the default), reference, candidate, or both. --n-replicates sets the number of repeated comparisons and defaults to 10; it must be a positive integer. Each comparison samples rows without replacement. The resulting percentile bands describe repeated score variability and are not confidence intervals for the population expected score. MLE follows its training and scoring protocol. See uncertainty options for sampling behavior. Use --uncertainty candidate for generated-real intervals without reference intervals.

Read and save results#

The default output is a printed summary. Use --format markdown or --format json to send those formats to stdout. --out evaluation.json writes JSON; --out evaluation.md writes Markdown. Add --include-replicates to retain each distribution’s ordered scores in JSON output. Failure diagnostics remain in provenance when available.

Run tabgen-eval --help for the full option list, including execution controls.