Smoke 30170202121 left one drift unit: a build declaring its extractor
schema as a raw object (not a JSON string) produced no field contract, so
validation silently skipped. Read both forms; also state the Data Table
id type in the prompt and forbid item-dropping in the retry correction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pinned fixtures drifted from the field names their nodes declare
(invoice_amount where the extractor schema says total_amount; email
where the Data Table column is contact_email), breaking correctly-built
downstream expressions and polluting failure attribution.
- Embed the information extractor's own declared schema (fromAttributes/
fromJson/manual) in the generation prompt — it never reached the model.
- Read the REAL Data Table columns off the builder-created table and
render them as the authoritative pinned-row shape.
- Validate generated pins against those contracts; regenerate once with
explicit corrections, fail loud if drift persists (no silent renames).
- Wrap flat extractor pins in their real { output } envelope.
- Rubric: pinned Data Table rows mirror the real table schema — reading
a nonexistent column is a builder_issue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The real OpenAI envelopes carry the model's answer as a STRING even when it
is JSON (json_object/json_schema formats return a JSON-encoded string); the
HTTP mock sometimes embeds the parsed object at output[].content[].text or
choices[].message.content instead, and downstream consumers crash or read
"[object Object]". Deterministic normalizer + submit_response reject nudge
for both /v1/responses and chat/completions, object-text coercion in the
wire-intercepted Responses envelope rebuild, and the matching quirk rule.
Supersedes the validated half of #34866; its completion-mock conformance
half (fence enforcement, pseudo-JSON schema synthesis, fillMissingRequired)
targeted symptoms of the chain-parser product bug fixed in #34928 and is
intentionally dropped — fillMissingRequired in particular could fabricate
scenario-relevant values and mask real mock content gaps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first smoke run proved the prompt+parse fix produced correct [] pins,
which the Phase-1.5 guard then converted back into [{json:{}}] — its
length===0 branch conflated "missing entry" (real-execution risk) with
"intentionally empty" (the engine honors [] as pinned-zero-items via a
presence check). Backfill only missing entries, and with [] instead of a
phantom item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a scenario premise requires a pinned node to emit 0 items ("no new
postings", "empty contact list"), the pin-data prompt's "generate 1-2 items
per node" rule cornered the model into a phantom single empty item [{}],
which flows downstream and crashes or corrupts the run. Steer the prompt
(zero-item premises MUST pin as []) and add a deterministic backstop in
parsePinDataResponse that drops fieldless items — the execution engine
honors an empty pinned array as "pinned, zero items".
Judges also had no home for pinned-fixture faults and split them between
mock_issue and framework_issue; the rubric now assigns harness-generated
pin data faults to mock_issue explicitly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two judge-evidence/mock-fidelity gaps found while grading the TRUST-344
validation run:
- The HTTP-path ledger stored the served body BY REFERENCE. Nodes that
mutate the response in place (the OpenAI node's json_schema output mode
parses output[].content[].text into an object) retroactively rewrite the
ledger, so judges see a body the mock never sent and misattribute
builder-side failures to mock_issue. Ledger entries are now JSON-round-trip
snapshots (Buffers pass by reference).
- Gmail messages.list answered as a bare array (a recurring model slip)
yields zero messages in the node. Deterministic backstop wraps a bare
array into the { messages, resultSizeEstimate } envelope, with a matching
submit_response violation nudge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three mock facets crashed real downstream decoders:
- OpenAI image generations carried the truncated `iVBORw0KGgo` placeholder;
Edit Image (GraphicsMagick) fails on it. Substitute every b64_json with a
real 512x512 canvas PNG built programmatically in eval-mock-fixtures.
- Gmail raw messages embedded model-invented base64 for PDF attachment parts;
Extract PDF Text (pdfjs) rejects them. The model now authors the document's
plaintext and the harness builds a valid PDF carrying exactly that text
(buildPdfWithText, xref offsets computed); image parts get the standard
decodable fixture.
- Google Drive resumable-upload initiations need the session URI in the
`location` response HEADER, which a model-authored body can't express.
Serve that turn deterministically; the follow-up chunk PUT stays mocked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The wire-server ledger stored the completion mock's internal shorthand
({content}/{tool_calls}) as mockResponse, so judges reading the verification
artifact concluded the node received a malformed envelope and misattributed
node-side failures to mock_issue. Record the post-translation wire body the
SDK actually received (equivalent non-streamed envelope for SSE turns), and
flag wire-server generation/translation failures in the judge pre-analysis
alongside HTTP-mock failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
