Uncertainty and execution#
Pass these options directly to evaluate; the defaults are enough for a first
run. Prepare your EvalData and choose metrics first.
Choose uncertainty#
For ordinary paired comparisons, uncertainty selects the repeated comparisons:
Setting |
What it requests |
|---|---|
|
Fixed generated-real and real-real scores |
|
Repeated real-real comparisons |
|
Repeated generated-real comparisons only; retain the real-real point without a reference interval |
|
Repeated generated-real and real-real comparisons |
Enabled uncertainty accepts one optional repetition count:
Key |
Default |
Meaning |
|---|---|---|
|
|
Number of repeated score comparisons |
For example, request 25 repeated comparisons:
from tabgen_eval import evaluate
report = evaluate(
data,
metrics=["shape", "trend"],
uncertainty={
"mode": "both",
"n_replicates": 25,
},
seed=42,
)
mode is required, and n_replicates must be a positive integer.
Invalid dictionaries raise with the accepted keys and an example. Use None
to turn uncertainty off.
"candidate" requires generated data. Its generated-real draws, sample resolution,
and band match "both" for the same seed and repetition count, while skipping
repeated real-real comparisons. Read the candidate band through
report["shape"].ci_low and .ci_high; .reference_distribution is None.
Memorization metrics remain point-only in every mode.
For each repeated side, the reported score (value or ideal) is the mean of
the finite repeated scores. No additional point comparison is computed for that
side. An unrepeated side keeps a single score. Within each repetition, rows are selected without replacement and the
two real samples are disjoint. All eligible rows become available again for the
next repetition. The finite repeated scores are summarized by their mean and
their 2.5th and 97.5th percentiles.
These percentile bands describe score variability across the sampled comparisons. They are not confidence intervals for the population expected score, including the real-real mean \(\mu_n(P)\). See Methodology for interpretation.
MLE draws disjoint real-training and scoring samples for each comparison. Its model-fitting and scoring protocol explains generated-model reuse in reference mode.
The uncertainty mode can change the
comparison size and reported scores. Changing
n_replicates can change the reported means and percentile bands. More
repetitions stabilize these summaries for the supplied data and require more
computation. See
The Sharp Bits and
Methodology for interpreting the real-data reference and
score bands.
Set execution options#
Argument |
Default |
What to use it for |
|---|---|---|
|
|
Reproducible sampling and fitting; accepts a nonnegative integer or sequence of nonnegative integers |
|
|
Automatic placement, or explicitly |
|
|
Tensor numerical precision; |
|
|
Upper bound on CPU parallelism; actual concurrency may be lower. Use |
|
|
Retain component tables for a generated/test point comparison; see component tables |
For example, add device="cpu", details=True to the call above to retain
component results using CPU execution. The
installation guide covers
GPU setup.