Troubleshooting#
Use this page to diagnose rejected inputs, missing results and execution problems. For interpretation of successful results, see The Sharp Bits.
My first CPU evaluation fails to compile#
Without a compatible precompiled library, the first CPU evaluation builds native operators on Linux or Apple Silicon macOS. Follow the Linux or macOS installation instructions to prepare the compiler and OpenMP. CPU compilation does not require a CUDA toolkit or GPU.
The exception identifies the failed step and its log path. Inspect the reported
check-*.log or build.log:
Failed step |
What to check |
|---|---|
|
Install a compiler and standard-library headers supporting the standard selected for Torch; set |
|
Read the log for incompatibilities with the installed Torch headers. Update the compiler or Apple’s Command Line Tools if needed. |
|
Install OpenMP headers and runtime libraries for the selected compiler, following the installation instructions above. |
|
Ensure the compiled program can find its runtime libraries and the cache filesystem permits execution. |
Cache creation |
Set |
|
Read |
Fix the reported problem and repeat the evaluation. Failed builds are retried.
macOS compiler, OpenMP and library-loading issues#
On Apple Silicon, Python and Torch need to run natively as arm64. This command
checks the Python process:
python -c 'import platform; print(platform.machine())'
It must report arm64. An x86_64 process requires reinstalling native arm64
Python and its dependencies; see the
macOS installation instructions.
The required C++ standard is selected automatically. If check-cxx20 or
check-torch-headers fails, inspect its log and update Apple’s Command Line
Tools if needed. Set CXX=/usr/bin/clang++ to select Apple Clang.
OpenMP is found in Torch or Homebrew. For a custom installation, set its prefix:
export TABGEN_EVAL_OPENMP_ROOT=/path/to/libomp
That prefix must contain include/omp.h and, when Torch does not bundle the
runtime, lib/libomp.dylib.
CPU compilation is slow or repeats#
Once compiled, the CPU operators are reused across Python processes without
repeating the build checks. Updating Torch, changing the source or selecting
another compiler with CXX can trigger a new build. The default cache is beneath
Torch’s extension-cache directory; TABGEN_EVAL_CPU_CACHE selects another
location. TORCH_EXTENSIONS_DIR has no effect on this builder.
export TABGEN_EVAL_CPU_CACHE=/path/to/writable-executable-cache
The cache filesystem must permit writing, file locking and execution. A full build has a 600-second timeout; check memory and host load if a compiler is killed or times out.
My evaluation unexpectedly requires a target#
With no metric filters, evaluate(data) selects every metric supported by your
data roles. When test is supplied, that includes MLE, which requires
Schema.target.
Declare the prediction target, or select the metrics you intend to evaluate:
report = evaluate(data, metrics=["shape", "trend"])
See the metric selection guide for selection rules.
My input is rejected#
Check the exception against these requirements:
Supply pandas DataFrames and an explicit
Schema.Input tables must have the same complete set of uniquely named columns.
Declared numeric columns must already have numeric storage and finite observed values. Missing values follow each metric’s rules.
Convert numeric text and choose units for temporal measurements before evaluation.
Supply the data roles required by your selected metrics.
Schema declarations assign roles; they do not convert values. See data preparation for the input contract.
My score is undefined or a point is missing#
The flags and point status help explain why a result is unavailable:
stat = report["shape"]
print(stat.value, stat.flag_codes)
print(stat.point_status, stat.ideal_status)
An undefined computed score can indicate insufficient usable data, an inapplicable statistic or an undefined ratio. Check the metric’s definition and diagnostics.
A point marked "absent" was intentionally not computed. Fixed test-only
evaluation stores one reference score in .value and marks .ideal absent.
Repeated test-only evaluation reports the finite mean in .value, leaves
.ideal absent, and retains the scores in reference_distribution. The same convention applies to MLE’s real-trained
scores.
Distance metrics and MLE also report how many rows were excluded:
print(report.row_usage())
Retained rows do not guarantee that every statistic is computable. MLE classification scores, for example, also depend on the classes present in the sampled training and scoring rows.
My report has no bands#
The default uncertainty=None requests fixed scores.
For ordinary paired metrics, the requested bands depend on the mode:
Setting |
Requested bands |
|---|---|
|
None |
|
Real-reference band |
|
Generated-real band |
|
Both bands |
Memorization metrics remain point-only. MLE follows its own repetition protocol; its generated scores can vary in reference mode because scoring rows change.
Repeated test-only evaluation supports "reference" and "both" modes. It
reports the mean of the finite scores in .value and retains the individual
scores in reference_distribution. The reference band is available in
.ideal_ci_low and .ideal_ci_high. "candidate" mode requires generated data.
Each distribution records its finite and unavailable score counts, along with any discarded reasons:
distribution = report["shape"].reference_distribution
if distribution is not None:
print(distribution.finite_count, distribution.unavailable_count)
print(distribution.discarded_reasons)
Use generated_distribution to inspect generated-real scores. An undefined or
infinite score is retained in the samples and excluded from finite summaries.
Other valid statistics remain usable. The metric diagnostics retain available
failure reasons.
Read the reported interval from ci_low/ci_high or
ideal_ci_low/ideal_ci_high. These fields contain the same raw 2.5th and 97.5th
percentiles as the corresponding distribution’s .band_low and .band_high.
They describe score variability across sampled comparisons and are not
confidence intervals for the population expected score.
Equal band endpoints indicate a zero-width interval. This can occur with a
single finite score or identical finite scores and is recorded
by DEGENERATE_CI.
My uncertainty settings are rejected#
Pass None, or a dictionary with:
Required
mode:"reference","candidate"or"both".Optional
n_replicates: a positive integer repetition count, defaulting to 10.
For example:
uncertainty={
"mode": "both",
"n_replicates": 25,
}
Empty dictionaries, unknown keys, mode="none", and nonpositive or noninteger
repetition counts are rejected. Use uncertainty=None to turn repetition off.
A classifier did not converge#
A finite Linear C2ST score is still reported when fitting finishes without meeting its stopping criterion. The diagnostics record the solver, termination reason and fold information:
metric = next(
result for result in report.results
if result.name == "linear_c2st"
)
print(metric.diagnostics)
Solver limits are part of the metric’s fixed definition.
For XGBoost metrics, converged records completion of the boosting budget. See
convergence interpretation
for what these diagnostics establish.
Evaluation is slow or runs out of memory#
Reduce the requested work where appropriate:
Select only the metrics you need.
Start with fixed scores before requesting repeated evaluation.
Reduce
n_replicatesto request fewer comparisons.Lower
n_jobsto reduce concurrent CPU work.Use hardware with sufficient available memory.
Exact nearest-neighbor searches can make large DCR evaluations slow. Memory planning limits temporary work, though individual allocations can still fail. Reducing the input size changes which data are evaluated and may change the comparison size. The execution options and DCR execution guides describe the relevant controls.