It is a vanity, and a particularly modern one at that, to assume that simply exposing a Large Language Model to a problem of logic is sufficient to produce reasoning. In our initial excursions into the ARC benchmark—a domain that ruthlessly punishes the approximation of intelligence in favor of its actual, hard-won exercise—we found our sovereign agent behaving less like a geometer and more like a spectacularly confident somnambulist. It would spit out grids that were violently mismatched in proportion to reality, and when confronted by the quiet, inexorable logic of the heuristics, it offered nothing but a babel of discordant guesses.

This writeup is an account of the bracing corrections we were forced to make: not merely to the prompting of the machine, but to the very architecture by which its epistemological claims are weighed, judged, and—quite frequently—found wanting.

I. The Pathology of the Misshapen Grid

Before one can debate the finer points of algorithmic epistemology, one must first ensure that the participants know what a rectangle is. The baseline runs of the DeepSeek generator revealed an embarrassing, almost petulant refusal to adhere to the rigid dimensional constraints of the ARC format. The models would hallucinate 5×2 arrays when explicitly commanded to produce 5×5 grids, leading to an immediate and unceremonious rejection by the validation parser. A staggering eighty percent of failures in early test flights were born of this simple, spatial illiteracy.

The solution was two-fold, and gratefully unsentimental. First, the introduction of a visually explicit “Hard Constraint” template into the prompt machinery, forcing the model to acknowledge the grid’s topology before populating it. Second, the deployment of a shape-clamp rescue operative—a post-generation shim that crops or pads near-miss grids (using the dominant cell value as filler) rather than discarding them to the void. The result was a stark drop in shape-mismatch errors from eight out of ten to zero. The LLM was finally compelled to arrive at the courtroom suitably attired.

II. The Tyranny of Maximum Entropy

Yet, securing valid output only exposed a more profound malaise within the tribunal itself. As we deployed our three-generator quorum—the deterministic LLM, the heuristic ‘greedy’, and its perturbed sibling ‘diverse’—we encountered an intellectual stasis.

Task after task disappeared into the void of the RESAMPLE condition. A ninety percent abstention rate is not a mark of discerning judgment; it is a symptom of institutional paralysis. The telemetry revealed the culprit: pure, unadulterated disagreement. With all three generators producing distinct hypotheses, the entropy metric maxed out, the coalition mass fractured precisely into thirds, and the tribunal—functioning flawlessly according to its overly cautious charter—simply folded its arms and refused to endorse any of them. The machine was trapped in a solipsistic deadlock.

III. The Interventions (M0 vs. M1)

Faced with this impasse, one has two choices: lower the standards of the court, or change the composition of the jury. We ran both experiments simultaneously, allowing reality to arbitrate the dispute.

Experiment M0 (The Capitulation): We stripped away the margin guardrails and lowered the selection threshold. The tribunal, effectively gelded, was forced to pronounce a victor on ninety percent of the tasks. The coverage expanding was notable; the intellectual rigor, however, collapsed. It produced eight incorrect selections. It turns out that forcing a consensus from a chorus of equally confident, disagreeing voices does not yield truth, but fiat.

Experiment M1 (The Dialectic): Rather than lowering the bar, we widened the pool. We introduced a fourth generator—a “warm”, stochastic iteration of the LLM (Temperature 0.7)—designed to explore the latent space of the prompt and occasionally find common ground with the heuristic engines. We disabled the margin guardrail (which inherently punishes crowded fields) but retained a sensible coalition floor.

The results of M1 demonstrated the utility of the epistemic method. While coverage settled at a more sober 50%, the resolved accuracy doubled to 20.0%, with a drop in erroneous picks. The warm LLM occasionally formed a coalition with the deterministic LLM, or validated the greedy heuristic, pulling candidate quality and coalition diversity from the noise. It proved that the tribunal’s architecture functions reasonably well; it had simply been starved of the necessary diversity of thought to form a meaningful consensus.

IV. The Forward March

What, then, is the verdict on the Epistemic ARC v2 benchmark as it stands today? The pipeline is hardened, the telemetry is transparent, and the tribunal’s adjudication machinery is demonstrably capable of selecting truth over falsehood—provided it is fed a properly varied diet of hypotheses.

But we must not mistake a successful skirmish for the end of the campaign. The roadmap to our ultimate goal demands the following immediate actions:

  1. True Intellectual Plurality within the LLM: The ‘greedy’ and ‘diverse’ heuristics are ultimately derived from the same ancestral logic tree. To break the remaining single-type coalition deadlocks, we must extract genuinely orthogonal reasoning pathways from the LLM itself. Instead of falling back on deterministic DSL solvers (which defeats the purpose of evaluating LLM sovereignty), we must prompt the LLMs into entirely distinct cognitive architectures—for instance, pitting a Code-Generating LLM against a Visual-Reasoning LLM to force the models into constructing fundamentally untethered hypotheses.
  2. Chain-of-Thought (CoT) Mandates: We have disciplined the LLM’s spatial output, but its internal reasoning remains opaque. Future iterations must enforce rigorous CoT generation prior to array synthesis, giving the tribunal a secondary vector (semantic logic) by which to judge the candidate trace, rather than relying solely on the final matrix.
  3. Execution at Scale: With the architecture evaluated on the ten-task crucible, the next step is clear. We must deploy the multi-generator swarm across the full Kaggle evaluation set, utilizing parallelization to ensure we are not old men by the time the results compile.

We have built a system that refuses to take the machine at its mere word, demanding evidence, coalition, and invariant proof. It is a necessary friction. Let us proceed.