• Define the incident questions before choosing which telemetry fields to retain.
• Measure forensic coverage and plaintext exposure separately; neither metric substitutes for the other.
• Replay the same captured workflows through every candidate policy so model variability is not mistaken for a policy effect.

Ordinary application logs can show that a request succeeded or failed. A GenAI workflow adds prompts, retrieved context, model outputs, tool calls, authorization decisions, and parent-child relationships. Those details can answer the decisive question in an investigation, but they may also contain sensitive data. Retaining everything turns telemetry into another repository of private content. Retaining only basic metadata can leave investigators unable to reconstruct what happened.

The practical decision is therefore not simply whether to log more or less. Security engineering teams must decide which evidence classes their incident questions require. This aligns with blog, penned by Marina Bregkou, The Case for AI Controls, which recommends AI-specific observability metrics and operational controls across the AI stack. A useful policy can preserve investigative value while reducing plaintext retention, but coverage and exposure must be measured separately.

Testing Four Retention Policies

This blog reports an independent experiment I conducted in October 2026 with a mock note assistant built for the test. The app supplied retrieved notes, exposed read and privileged-delete tools, and recorded model responses, tool calls, and authorization outcomes around live calls to OpenAI’s GPT-4o Mini.

The study captured 24 workflows (12 matched pairs) once and replayed each through four retention policies, producing 96 exports. Each pair used the same basic task, available tools, and intended trace structure. The case contained the condition being tested—usually a synthetic canary in user or retrieved content, or a particular authorization outcome for a privileged tool. The control removed the canary or applied the opposite authorization grant to the same tool. Pairing limited unrelated differences, making it easier to determine whether the exported telemetry preserved evidence of the changed condition. It was not used to claim that the condition caused the model’s response.

• P0 (metadata-only) kept performance metadata, request identity, status, and topology.
• P1 (full-content) added raw content and authorization context.
• P2 (redacted-content) applied pattern redaction while retaining detector events.
• P3 (security-structured) dropped raw content but kept trust, privilege, authorization, denial, and detector fields.

A 17-fact checklist tested whether each export could reconstruct operational, security, and content facts. Credit required a correct answer supported by evidence in the export. A separate exact-match scan measured registered synthetic-canary exposure. Only six executions carried a preregistered canary in its designated input or retrieval field; the other 18 were ineligible rather than counted as zero exposure. The implementation, protocol, and aggregate results are available at github.com/rgbuilds/genai-observability-security.

What the Results Showed

Overall coverage was 43.5% for metadata-only P0, 80.7% for full-content P1, 76.7% for redacted-content P2, and 73.3% for security-structured P3. P1 exposed canaries in all six eligible executions; the other policies exposed none.

P2 preserved all scored operational facts and 83% of content facts without exposing registered canaries, but only 27.3% of security facts. P3 preserved every scored operational and security fact but no content facts. P1 had the highest coverage and measured exposure. No policy was best for every investigation.

Most broad differences followed from the published allowlists. Live calls were still needed to observe where model behavior moved content. Among the ten P1 executions with retrieval and a final model output, both locations were “readable”—present as plaintext in the export. Retrieved text appeared in the final output in nine, an observation rather than proof of influence.

A P2 Flow End to End

In one captured flow, a retrieved note contained a registered synthetic canary and the model later called a tool. P2 replaced the canary with a redaction marker while preserving request and trace identity, retrieval source, tool status, topology, surrounding content, and the detector event.

An investigator could reconstruct the workflow and detection stage without reading the matched value. Because redaction made one side of the retrieval-to-output comparison unreadable, the result was “missing,” not “no copy observed.” No plaintext canary remained in the P2 export.

Turning the Findings into Practice

Teams can apply the same method without adopting these exact policies:

• Start with the incident questions the organization must answer.
• Map each question to operational metadata, structured security evidence, content, or detector events.
• Retain baseline request identity, span relationships, timing, status, and status code.
• Publish versioned allowlists and detection patterns, and protect even redacted telemetry.
• Replay representative workflows and re-evaluate when models, tools, routes, schemas, or privacy patterns change.

Limits and the Decision Ahead

This experiment used one scenario suite, registered exact-match canaries, and a pinned GPT-4o Mini snapshot (gpt-4o-mini-2024-07-18).

Results should not be generalized without further validation. Textual co-occurrence and parent-child topology do not establish causal influence. The experiment identifies no universally best policy; it offers a repeatable way to test what each policy preserves and exposes.