Teaching a small model to read tables, and prove every step

The problem

Ask a small language model a precise question about a table ("which line item has the lowest 2019 value while 2018 stays at or above 12.7?"), and two things can go wrong. It can misread the table, or it can read it correctly and then botch the arithmetic: a comparison flipped, a dominance check fumbled, a number hallucinated into a citation.

For anything you'd actually trust, the answer isn't enough: you want the reasoning, and you want each step pinned to the exact cells it used, so a checker (or a person) can verify it. That is a grounded trace. The goal of this project was a small, local model that produces them.

The central question turned out to be which of those two failures dominates, because the fix is completely different. If the model misreads, you need better perception or more data. If it reads fine but miscomputes, you don't need a bigger model at all; you need to stop trusting its arithmetic.

The task & the benchmark

Tables come from TAT-QA (Zhu et al., ACL 2021; CC BY 4.0): real financial tables from company filings. TAT-QA's own questions are ~64% free-form arithmetic (change, %, sums) and only a sliver match clean, checkable predicates, so we keep the real tables and generate three families of constrained, programmatically-verifiable questions over them:

Every example ships with a machine-checked grounded trace; an independent rule-based validator recomputes the answer from a stored spec and verifies each cited cell. The dataset is 1,266 validated examples (100% pass the validator), with 177 used as silver training records and a leakage-free, table-level split: dev (190) / locked test (254) / a separate 12-example out-of-distribution anchor.

Why a table-level split, and how leakage was ruled out. The same source table appears in multiple examples (two orientations × several question types), so a naïve per-example split leaks tables across train/test. We re-split by source table (stripping orientation suffixes); 166/224 tables had straddled a per-example split. Disjointness is asserted at both the id and content-hash level on the exact file the model trained on. Total/subtotal rows are excluded from the entity universe by a single shared rule, so the inference-time universe matches the gold spec.

The approach

We fine-tuned Qwen3 (QLoRA, 4-bit, a single RTX 3090). Supervised fine-tuning alone got the model writing well-formed traces, but its answers were still only ~59% right. The error analysis was decisive: the remaining mistakes were arithmetic-execution errors, not misreadings. So:

The executor: answer by construction

Instead of trusting the model's arithmetic, the model emits a structured operation (the op type, the column names, thresholds, and directions it read from the question), and a small, deterministic, independently-tested engine computes the answer. The model's job becomes comprehension; the arithmetic is done by proven code. This is the move that takes answers from 59.4% to 95.7%.

Grounding: citations by construction

The model's hand-typed citations had the same disease as its arithmetic (wrong/derived numbers), so they got the same cure: the trace cites the cells the engine actually read, which are grounded by construction: they always exist and always match. The prose stays model-authored (the explanation); only the cited evidence becomes system-constructed (the proof). Grounded-trace rate: 71.3% → 96.9%.

The safety gate

The engine is a specialist: it only knows the operations it was trained on. If the model emits an operation the question doesn't actually support (no threshold language for a threshold op, no trade-off language for a trade-off op), the gate treats it as fabricated and falls back to the model's own answer (do no harm).

Is 95.7% real, or circular?

Honest worry: the test gold is computed by code that shares the engine's arithmetic, so on its own that number is consistency, not correctness. To break the circularity we ran two independent checks that never see the spec or the gold.

Blind LLM cross-check. 36 stratified examples, shown as raw table + question only, answered by independent annotators reasoning from scratch: 36/36 agreement with the stored gold (confirming the engine on every one of its sampled correct cases). Two independent reasoning paths agreeing on every case ⇒ the answers are corroborated, not merely self-consistent.

Blind human anchor. A real person, given a self-contained annotation UI, independently answered the same items on the two natural question types (n=24): 22/24 blind agreement with the stored gold: a second, human reasoning path reaching the same answers.

The limit: where it fails, and how it's caught

The one non-circular slice is an out-of-distribution task type (extremum) that is not in training and not in the engine's vocabulary. Here the model's own answer is right 100.0%, but it emits a trained operation instead, so the executor computes the wrong answer 0.0%. The engine isn't broken; the model just doesn't know the operation exists, and the executor overrides its correct intuition and fails confidently. This failure mode is invisible to any in-distribution (circular) test.

The safety gate is exactly for this. Re-scored on saved outputs: out-of-distribution 0.0% → 100.0%; in-distribution 95.7% → 96.1% (it fires on only 9/254, and never on a case the engine got right). Net: never worse than the model alone, and it degrades gracefully. Here is the gate firing on a real out-of-distribution example: the engine produces nothing usable, so the system keeps the model's correct answer.

Caveat: the gate is a keyword heuristic validated on a small out-of-distribution set.

Honest limitations

• In-distribution gold is spec-derived. Now corroborated by two independent blind oracles, but not proven: n is small, trade-off has only the LLM check, the human is one annotator.
• The trade-off questions are non-natural. A non-specialist couldn't assess "Pareto-optimal" (verifiable, but not how a person would ask).
• Specialization. The engine helps only inside its trained operation vocabulary; the gate makes that safe, but it is not a general table engine.
• The trace prose stays model-authored. The system grounds the answer and the cited evidence by construction; an unaided 4B model's own citations remain ~71.3% (near its ceiling).
• Coverage & domain. Only high-confidence ingested tables (~97%); selection / threshold / frontier reasoning, not TAT-QA's dominant free-form arithmetic; TAT-QA is financial, and transfer to scientific/ML tables is unverified.
• Provenance. The 12-example anchor is TAT-QA-gold-anchored (their annotations, cross-checked by our independent recompute), not verified by anyone on this project.

See it run

Each card below replays a held-out example through the full path: the model emits an operation, the engine computes the answer and returns the exact cells it read (highlighted), the gate decides whether to trust it. No model is called; these are saved outputs re-assembled on CPU.

Generate the full set yourself: python scripts/demo.py (writes results/demo/index.html), python scripts/demo.py --list, or --id <example_id>.

Reproduce

# build dataset / leakage-free split / SFT data
python scripts/build_dataset.py
python scripts/freeze_splits.py
python scripts/build_sft.py

# train + score (WSL .venv-train, RTX 3090; see docs/training_env.md)
python scripts/train_sft.py --model Qwen/Qwen3-4B --out models/qwen3-4b-sft-exec
python scripts/eval_executor.py --dataset data/processed/eval_test.v0_1_0.jsonl \
       --adapter models/qwen3-4b-sft-exec --out results/p3_4b_exec_TEST.json

# re-score on CPU (no GPU): grounding, safety gate, this write-up, the demo
python scripts/eval_grounded.py
python scripts/eval_gate.py
python scripts/build_writeup.py
python scripts/demo.py

ML methodology reference: docs/methodology.html. Consolidated findings: RESULTS.md. Interactive demo: results/demo/index.html.