Exploring AI Data Security · Part 1 — Protecting sensitive information before the first prompt

I Start with the information that must stay confidential

Consider a fictional supplier contract containing customer names, negotiated prices and an unreleased product plan. The requirement is concrete: keep those details within an authorised purpose and audience. That covers publication on the open web, disclosure to another customer, and access by a colleague or external service that was never approved to receive them. Confidential business information extends beyond personal data.^[1]

This first part of Exploring AI Data Security examines three basic mechanisms: training, which changes what the model has learned; inference, which processes the context supplied for a request; and storage, which preserves copies or derived state. Each creates a different route by which sensitive information may reach an unintended recipient.

These risks can be established before an employee opens the tool: through training-data selection, service permissions, integrations and retention defaults. Training and pre-release testing may already have processed real information. Other weaknesses become disclosures only when later requests use the affected path. NCSC guidance places responsibility across design, development and deployment, including secure defaults and protection of model and data access.^[2]

II Training: control what enters the learned model

Training and fine-tuning can incorporate examples into a model's parameters. A confidentiality risk arises when a model later reproduces sensitive material to someone outside its intended audience. Carlini and colleagues demonstrated extraction of training examples from GPT-2, including publicly available personal information. Their study establishes the mechanism in that tested model; it provides no disclosure rate for a particular current service.^[3]

In the contract example, uploading real contracts to a fine-tuning dataset creates a decision about their future reuse before the assistant's first production conversation. For a purchased model, the supplier's training history is an inherited part of the assessment.

Before release: document the provenance and permitted purpose of training material; remove unnecessary confidential details; restrict datasets and model artifacts; and test the approved model version for unintended disclosure. NCSC recommends recording training and fine-tuning sources and managing access to AI assets.^[4] Treat dataset deletion and any required removal from a trained model as separate verification tasks.

III Inference: control what enters and leaves the context

During inference, sensitive information can arrive in an attachment, a tool result or a shared instruction. The application may include material the employee never typed. The security questions are which identity may read it, which service may process it, and which audience may receive the answer. OWASP recommends least-privilege access and restrictions on the sources available to the application.^[1]

For our contract, an internal summary might be authorised while sending the same text to an unapproved integration or publishing the summary on a public page would cross the agreed boundary. An approved processor receiving necessary context is authorised processing; public sharing is a separate action requiring its own check.

Before release: scope service accounts and connectors, enforce the requesting user's access before adding documents to context, and check the destination before sending or publishing an output. In an isolated test, try the same request as an authorised colleague and an unauthorised colleague. A refusal after restricted content has already been delivered to the wrong processor is too late for that transfer.

IV Storage: control the copies that remain

A request can leave copies in conversation history, application traces, provider monitoring, support exports and backups. Each needs its own purpose, readers and retention policy. NCSC explicitly treats logs as sensitive assets and calls for protection of generated content according to its sensitivity.^[4]

Provider terms make the distinction concrete. OpenAI says API data is used for model training only with explicit opt-in. Its data controls separately describe abuse-monitoring logs and application state. Default monitoring retention is up to 30 days, with documented exceptions; application-state retention depends on the endpoint and feature. Additional retention controls have eligibility requirements and limitations.^[6]

Before release: minimise logged content, restrict operational access, and set retention and deletion rules for every store. Test a synthetic contract through a normal request and an error path, then inspect the logs and support export. Deleting the visible conversation should trigger checks for independently retained copies. Verify the stores you operate directly; use the applicable commitments and configuration evidence for provider-managed stores.

V Findings: three decisions before rollout

These are proposed acceptance checks, not results from an audit of a particular product. One synthetic contract can connect them: trace whether it enters training material, which identities and services receive its contents, and where copies remain after the test. Keep the test results tied to the model, application and configuration versions reviewed.

End-user guidance builds on these decisions. The service owner must first make the permitted data, recipients and retained copies explicit. This opening article establishes that foundation for examining access, sharing and DLP in the rest of the series.