# ESF v1.0 Independent Validation Report

Irregular Evaluation-Environment Test Case #2 | September 19, 2026

## The finding

An independent evaluator was given the locked Evidentiary Sufficiency Framework (ESF) v1.0, the locked Irregular evaluation-environment decision question and recipe, and the same frozen evidence package used for the primary evaluation. The primary evaluation concluded that the affirmative adequacy-and-independence conclusion was not sufficiently established by the submitted record and that the record also did not establish the opposite proposition that the post-incident process was actually inadequate, biased, ineffective, or improperly closed.

The independent evaluation reached the same controlling result. It concluded that the affirmative adequacy-and-independence conclusion could not be established from the locked record, while also declining to convert evidentiary insufficiency into a positive finding that the investigation or remediation was actually inadequate or not independent.

The two evaluations therefore reproduced the substantive outcome. They also converged on the major evidentiary dependencies: incomplete Gemini-specific causal substantiation, lack of public effectiveness verification for Irregular's remediation, no criteria-based closure package, unresolved applicability of binding external requirements, and insufficient evidence showing what impartiality safeguards or independent assessment supported reliance on the resolution conclusions.

**Validation disposition:** substantive convergence; controlling outcome reproduced; no material evaluator deviation identified. Minor differences in phrasing and the strength assigned to individual sub-objectives did not alter the final evidentiary disposition.

This run provides a second case-specific validation of ESF's ability to preserve the distinction between *the submitted evidence does not establish an affirmative conclusion* and *the opposite proposition has been affirmatively proved*. It does not establish a general reliability rate for ESF or prove that every evaluator will apply the framework identically.

## Test design and scope

The locked question asked whether, based solely on the sufficiently reliable evidence available in the submitted case package, the post-incident investigation, root-cause determination, remediation, recurrence-prevention process, and resulting resolution conclusions following the Irregular evaluation-environment containment failure were adequately supported and sufficiently independent under the applicable governing requirements.

This was a test of evidentiary sufficiency under a defined submission. It was not a determination that Irregular, Google, or any other party acted improperly; it was not a technical re-investigation of the incident; and it was not a legal finding that any particular external standard governed the parties.

The frozen package contained the ESF v1.0 framework, the locked case-specific recipe, governing-standards research, incident evidence, conflict-of-interest / role-overlap research, and root-cause / corrective-action / authority-gap research. Both evaluators were instructed to work only from the supplied package, preserve source limitations, avoid inventing governing requirements, and identify gaps without treating missing evidence as proof that the underlying evidence or activity did not exist.

The independent evaluator states that it excluded outside web research, prior evaluations, and inferred facts not contained in the supplied files. The present validation report compares the two written outputs. It does not independently recreate either evaluator's execution environment.

## Comparison point

| Comparison point | Primary evaluation: Codex | Independent evaluation: Grok |
| --- | --- | --- |
| Final evidentiary result | Affirmative adequacy / independence not sufficiently established; opposite proposition also not established | Affirmative adequacy / independence not established; record also insufficient for a fully evidenced positive finding of inadequacy |
| Eligibility | Sufficiently established | Established |
| Subject | Sufficiently established | Established |
| Evidentiary basis | Partially established; enough for limited findings, not enough for the complete conclusion | Established (limited); evaluable record exists, incomplete for affirming adequacy |
| Root-cause support | Core occurrence supported; completeness of RCA not established | High-level dual-factor account partially established; technically complete Gemini RCA not established |
| Corrective action | Responsive corrective approach supported; adequacy / completeness not established | Corrective-action categories partially established; control completeness not verified |
| Effectiveness | Not sufficiently established | Not established |
| Closure / resolution basis | Closure-style assertions established; sufficiency of closure conclusion not established | Not established; no criteria-based closure package |
| Independence / impartiality | Role overlap relevant; sufficient impartiality / independent assessment not affirmatively established; actual bias not established | Structural independence risk documented; sufficient impartiality / independent assessment not established; improper influence not found |
| Governing requirements | Relevant standards identified, but binding applicability largely unestablished | Same; no binding investigation-quality regime established from the locked record |
| Final boundary | Missing evidence does not prove failure; it limits what can be defended | Same; insufficiency for "yes" does not fully establish a positive "no" |

## What reproduced

Both reports identified the same case structure and the same decisive evidentiary break.

First, both evaluators found that the incident was sufficiently established to enter evaluation. They accepted that a containment failure occurred during an Irregular-hosted Gemini cybersecurity evaluation and that investigation, notification, corrective activity, and public resolution claims followed.

Second, both evaluators found that the record contained a plausible and attributable high-level causal account. Irregular's public explanation centered on unintended internet access and a fictional target name overlapping a real domain, with related lab materials supporting the broader class of evaluation-environment failure. Neither evaluator treated that shared pattern as a substitute for a complete Gemini-specific technical reconstruction.

Third, both evaluators found that the described corrective actions were directionally responsive to the reported failure conditions. Disabling the affected evaluation, changing internet-access controls, revalidating scenario names, strengthening monitoring, improving communication, and adding challenge functions all bore a rational relationship to the reported causes. Neither evaluator concluded that the existence of responsive actions, by itself, demonstrated implementation completeness or effectiveness.

Fourth, both reports identified effectiveness verification as a controlling gap. The locked record did not contain published regression testing, defined acceptance criteria, containment-validation results, or independent verification demonstrating that Irregular's remediation controlled the relevant recurrence pathways. Repeated assurances that the issue was "resolved" or that safeguards had been added were treated as assertions, not independent proof of effectiveness.

Fifth, both evaluators identified the same closure problem. The record established public statements that known issues were resolved or no longer active, but it did not establish the criteria used for closure, the evidentiary package supporting that determination, the treatment of residual uncertainty, or a traceable decision authority sufficient to defend the broader resolution conclusion.

Finally, both evaluators treated structural role overlap as relevant to reliance without converting it into a finding of misconduct. Irregular's roles as operator, investigator, remediator, public characterizer, and commercial vendor were material to the independence analysis. Google's position as customer, model owner, investigator, and public characterizer was likewise relevant. Both reports expressly stopped short of finding bias, improper influence, or false findings. Both also recognized that the record did not establish a universal requirement for an external independent incident investigator.

The central result therefore reproduced: the submitted record supports the occurrence of the failure, an attributable cause account, and responsive post-incident activity, but does not sufficiently establish the full affirmative conclusion that the resulting investigation, remediation, recurrence-prevention process, and resolution conclusions were adequately supported and sufficiently independent.

## Minor differences in treatment

The reports were not identical in wording or sub-objective labeling.

Codex described the Evidentiary Basis objective as **partially established**, emphasizing that the record was substantive enough for limited findings but not complete enough to support the full adequacy-and-independence conclusion. Grok labeled the same objective **established (limited)**, reasoning that an evaluable record existed even though it was insufficient for later affirmative findings. This is a classification-language difference at the intermediate-objective level, not a difference in the controlling result.

Grok also used somewhat firmer language in the Governing Requirements section. It explicitly marked effectiveness, closure, and impartiality / independent assessment as **not established**, then classified the governing-requirements objective as not established overall. Codex used more qualified language in several places, including "partially supported" and "not affirmatively established," but reached the same practical conclusion: the required evidentiary chain was incomplete.

The independence analysis likewise differed slightly in emphasis. Grok foregrounded the Conflict-of-Interest Brief's "documented structural conflict" classification for Irregular's multi-role position. Codex treated the same role overlap as relevant to impartiality but was more cautious about adopting the brief's classification language as an evaluative conclusion. Both reports expressly stated that role overlap did not establish actual bias, improper influence, or distorted findings.

These differences are retained for transparency because they show normal evaluator variation in how intermediate objectives are labeled and explained. They did not create a material contradiction or alter the final sufficiency disposition.

## What this validates, and what it does not

This validation run demonstrates substantive reproducibility on the controlling evidentiary question across two independent model evaluations using the same locked framework, recipe, and evidence package.

Both evaluators preserved several ESF disciplines that mattered to the result:

- party assertions were not treated as independent verification;
- repeated reporting from a single source lineage was not multiplied into independent corroboration;
- evidence from related incidents was not silently substituted for Gemini-specific proof;
- recognized standards were not elevated into binding requirements without an applicability hook;
- structural conflicts and role overlaps were treated as relevant risk evidence rather than proof of wrongdoing;
- missing evidence was treated as a limitation on what could be established, not as affirmative proof of the opposite proposition.

Those reproduced behaviors are meaningful because they produced the same narrow conclusion despite different wording and intermediate classification choices.

The run does **not** establish that ESF will always produce identical language, identical sub-objective labels, or identical weighting across evaluators. It does not establish that the underlying investigation or remediation was actually inadequate. It does not establish legal noncompliance with any standard whose applicability was not shown. It also does not prove that the publicly unavailable evidence, if produced, would necessarily change the result.

What the run does show is that, on this locked record, two independent evaluators located the same decisive evidentiary dependencies and refused to convert a polished public resolution narrative into proof of adequacy where the supporting evidentiary chain was not available.

## Validation conclusion

Test Case #2 achieved substantive outcome reproduction.

The primary and independent evaluations both found that the available evidence supports the occurrence of the Irregular evaluation-environment containment failure, an attributable high-level causal account, and responsive investigation and corrective activity. Both also found that the record does not sufficiently establish causal completeness, remediation effectiveness, criteria-based closure, or a sufficient impartiality / independence basis to support the full affirmative decision question.

Neither evaluator converted those deficiencies into a positive finding that the post-incident process was actually inadequate, biased, ineffective, or improperly closed. That distinction is central to the framework and reproduced across both runs.

No material evaluator deviation affecting the controlling conclusion was identified. The differences that remain are limited to intermediate labeling, phrasing, and emphasis. They are documented here for transparency but do not alter the validation disposition.

**Validation disposition: substantive convergence; controlling outcome reproduced; no material evaluator deviation identified.**

ESF v1.0 and the case inputs remain locked. This run should be retained with the unedited primary evaluator output, the raw independent-evaluator output, and this validation report. Together they show not merely that both models arrived at a similar sentence, but that they independently identified the same evidentiary dependencies and preserved the same boundary between insufficiency of proof and proof of failure.

## Source record

- **Primary evaluation:** `STU-002-Primary-ESF-Evaluation-Codex-2026-09-19.md`.
- **Independent evaluator output:** `Irregular-ESF-Evidentiary-validation-2026-09-19.md`.
- **Locked substantive inputs identified by both reports:** ESF v1.0; the locked STU-002 Irregular decision question and UP-based recipe; Governing Standards Research; Incident Evidence; Conflict-of-Interest Evidence; Root Cause / Corrective Action / Authority Gaps research.
- **Validation scope:** This report compares the evaluator outputs and does not add new case evidence.