Grounded Table Reasoning Traces — Methodology

1. Problem & approach

Small language models comprehend table questions well but cannot reliably execute the arithmetic. The goal is grounded reasoning traces: every step tied to specific, verifiable cells, with every numeric claim provably true.

The design is neuro-symbolic. The model's job is comprehension: it emits a structured operation (which op, which columns, which thresholds and directions, read from the question). A small deterministic engine computes the answer and returns the exact cells it read. So the model contributes understanding and explanation; proven code contributes the arithmetic and the grounding. A safety gate falls back to the model when the emitted operation is not supported by the question, keeping the system never worse than the model alone.

2. Data & task

Tables come from TAT-QA (Zhu et al., ACL 2021; CC BY 4.0): real financial tables from company filings. TAT-QA's native questions are ~64% free-form arithmetic and only ~8% match clean selection/threshold predicates, so we keep the real tables and generate three families of constrained, programmatically-verifiable questions over them (both row- and column-orientations, via transpose; explicit-direction phrasing so the gold is a mechanical computation):

Independent validator. A rule-based validator recomputes the answer from a stored spec and verifies each cited cell (existence, value match, numeric-comparison and threshold correctness, relevance). It is deliberately kept independent of the answer engine (its own predicate re-implementations, no shared helpers) so that "engine output passes validator" is not circular. Total/subtotal rows are excluded from the entity universe by a single shared rule, so the inference-time universe matches the gold spec.

Splits & sizes. 1,266 validated examples (100% pass the validator); 177 silver-rendered training records. The eval split is leakage-free at the table level: re-split by source table (orientation suffixes stripped; 166/224 tables had straddled a per-example split), asserted disjoint at both id and content-hash level — dev 190 / locked test 254, plus a separate 12-example out-of-distribution anchor (an extremum task type not in training).

3. Output schema

The model emits one JSON object per example: trace_steps[] (each a short natural-language description + a structured cites list), a structured final_answer, and the operation the engine consumes. Citations carry both coordinates and semantics for exact, reorder-robust validation:

4. Model & training

QLoRA SFT, fully local on a single RTX 3090 (~11 min/run). Base Qwen3-1.7B first (to rehearse the converter → SFT → eval loop), then promoted to the Qwen3-4B target on identical data.

• Quantization: 4-bit NF4, bfloat16 compute.
• LoRA: r=16, α=32, dropout=0.05, target_modules="all-linear", causal-LM.
• Optimization: 3 epochs, lr 2e-4, batch 2 (+ gradient accumulation), bf16.
• Loss: completion-only — the prompt is masked to -100 with a deterministic, verified assistant-only mask (this was the decisive fix over full-sequence loss: valid-trace 0%→17% at 1.7B).

Silver training labels. Each question is rendered N=3 times (headless Claude); only answers that agree and pass the Tier-1 groundedness + numeric-consistency checks are kept (high-precision, ~56% yield). The held-out eval set keeps the silver set honest.

5. The executor — answer by construction

SFT alone left answers at ~59%; error analysis showed the residual was arithmetic-execution errors (flipped comparisons, fumbled dominance), not misreadings. So instead of trusting the model's arithmetic, the model emits the structured operation and a deterministic engine (src/executor.py) computes the answer: row filtering, argmax/min, multi-condition thresholds, and Pareto dominance, with explicit epsilon tolerance, tie handling, and total-row exclusion. Operations reference column names, so they are robust to reordering and are exactly what a model can read off the question.

6. Grounding & the safety gate

Grounding by construction

The model's hand-typed citations have the same disease as its arithmetic (wrong/derived numbers), so they get the same cure: the trace cites the cells the engine actually read (executor.evidence_for) — grounded by construction (read straight from the table, so they always exist and match). The trace prose stays model-authored (the explanation); the cited evidence becomes system-constructed (the proof).

Out-of-vocabulary safety gate

The engine only helps inside its trained operation vocabulary. The gate checks the question carries the signal the emitted op type requires — threshold language for best/threshold ops, trade-off/Pareto language for tradeoff. If the signal is absent, the operation is fabricated, so the system falls back to the model's own answer. Deployable from question text + op type alone (no gold); a "do no harm" check, not merely an OOD patch.

7. Evaluation methodology

Metrics (over all examples — an unparseable output scores 0 on every metric): engine answer accuracy vs. the model's own answer, operation present / whole-op exact-match, trace groundedness, full valid-trace, and a per-type breakdown.

Protocol. A dev slice (190) for iteration; the locked test (254) scored exactly once under a pre-registered frozen config (Qwen3-4B + the SFT-exec adapter, greedy/deterministic, strict validator).

Circularity and the anchors. The in-distribution gold is spec-derived and the engine re-implements the same semantics, so an in-distribution score measures operation-comprehension consistency on unseen tables, not correctness against an outside oracle. Two blind anchors break the circularity: independent LLM annotators answering from the raw table + question alone agreed with stored gold 36/36; a blind human oracle agreed 24/24 on the two natural types (0 gold errors). The OOD anchor (12 extremum examples, a task type outside the operation vocabulary) is the only fully non-circular slice — and it is what exposed the executor's specialization limit (below).

8. Results & key findings

Progression (trained-model stages on dev 190; the two system layers re-scored on the locked test 254):

On the locked test (254): engine answer 95.7% [92.4–97.6] vs. the model's own arithmetic 59.4% — a paired +93 / −1 (exact-McNemar p ≈ 10⁻²⁶); by type 97.6 / 96.6 / 92.9%. Grounding lifts the trace-grounded rate 71.3% → 96.9% (valid 68.9% → 94.9%). The gate takes OOD 0% → 100% and in-distribution 95.7% → 96.1% (fires on 9/254, never on a case the engine got right). Full numbers and CIs: RESULTS.md.

Key findings

• Capacity-bound, not data-bound. The 1.7B→4B jump on identical data (valid 17%→44%, answer 20%→52%) located the bottleneck in model size, not training data.
• The residual is execution, not perception. Grounded-but-wrong answers were arithmetic-execution errors — which is what motivated routing the arithmetic through code.
• The model's own citations are near a ceiling (~71%), so grounding is solved by construction rather than by retraining a 4B model to cite better.
• The executor is a specialization, not a general engine. On the OOD anchor the model's own answer is 100% but it emits a trained op type, so the executor scores 0/12 — hence the gate.

9. Limitations

• In-distribution gold is spec-derived — corroborated by two blind oracles, but not proven: n is small, tradeoff has only the LLM check, the human is one annotator.
• Specialization — the engine helps only inside its trained op vocabulary; the gate makes that safe but it is not a general table engine.
• Trace prose stays model-authored — the system grounds the answer and the cited evidence; an unaided 4B model's own citations remain ~71%.
• Coverage & domain — only high-confidence ingested tables (~97%); selection / threshold / frontier reasoning, not TAT-QA's dominant free-form arithmetic; TAT-QA is financial, and transfer to scientific/ML tables is unverified.
• Non-natural trade-off questions — "Pareto-optimal" is technical; a non-specialist human couldn't assess it (12/12 "I don't know").
• Provenance — the 12-example OOD anchor is TAT-QA-gold-anchored (their annotations, cross-checked 16/16 by our recompute), not verified by anyone on this project.