Training data, inference data, and logs: three different risks

1. Opener: one fear word, three pipes

AI Security conversations often collapse into one sentence: the model saw our data. That sentence is too blunt to help.

Sometimes “saw” means material that shaped weights during training or fine-tuning. Sometimes it means a prompt, a retrieved chunk, or a tool payload in a live session. Sometimes it means a transcript, a provider log, a ticket trail, or a monitoring export that outlives the chat.

Those are not the same pipe. They fail differently. They need different questions. They answer to different owners.

Early foundations exist so later pieces share a vocabulary. This primer names the three surfaces and keeps them apart. It does not sell a suite. It does not invent retention numbers or exposure rates. Status stays draft / PROMISE.

If you already know the A1 intake — what can the system see, do, and remember — treat this piece as a zoom on see and remember when the path is AI. Models, tools, and agents (A2) still matter: generation, reach, and sequenced action sit on top of these data surfaces. Mixing layers and mixing data pipes is how debates get loud without getting useful.

2. Three surfaces at a glance

Surface	Plain question	Typical failure	First control instinct
Training	What entered the model?	Sensitive material baked into weights or fine-tunes; hard to “unsee”	Provenance, exclusion, licensed corpora, careful fine-tune scope
Inference	What enters this session?	Over-paste, over-retrieval, tool payloads in the context window	Minimum paste, bounded RAG, least-privilege tools
Logs / telemetry	What persists after?	Transcripts, caches, tickets, exports that outlive the answer	Retention, access, redaction, deletion paths

One table is not a scorecard. It is a separation tool. When someone says “AI data risk,” ask which row they mean. If they mean all three at once without naming them, stop and split the sentence.

3. Training data: what entered the model

Training is the long past of a model. Pretraining corpora, continued pretraining, supervised fine-tunes, preference data, and any proprietary fine-tune you attach to a base model all belong here.

The risk is not that the model “remembers” like a database. The risk is that material that should never have been eligible shaped parameters — or that a fine-tune on internal tickets, mail, or customer files creates a system that can regurgitate fragments under pressure. Extraction attacks and memorization research exist; this primer does not invent hit rates. It states the operator truth: once sensitive material is in the training or fine-tune path, your remediation options shrink.

Practical questions for training / fine-tune decisions:

1. Whose data? Customer, employee, partner, public web, licensed set — name the subject class.
2. Was it eligible? Contract, license, policy, and consent are the gate — not convenience.
3. What is the purpose? A narrow task fine-tune is not the same as “throw everything at the model so it knows our company.”
4. Can we exclude? Can you prove a class of records never entered the corpus or the fine-tune set?
5. What is the exit? If the fine-tune was a mistake, do you retire the checkpoint, or only “ask it nicely” in a system prompt?

System prompts are not a delete button for training mistakes. Policy language on top of a contaminated fine-tune is theater. Training risk is mostly decided before the first user session.

Foundation note: training risk is about the model layer (A2). Tools and agents do not undo a bad fine-tune; they can only widen what that model can reach next.

4. Inference data: what enters the session

Inference is the live present. Everything that enters the context for this run counts: user prompts, system instructions, uploaded files, retrieved RAG chunks, tool outputs, memory features that re-inject prior turns, and any connector payload the harness stuffs into the window.

This is where most day-to-day “we put business data into AI” stories live. The model weights may never have trained on your CRM. The session still saw the export you pasted.

Failures look ordinary:

• Over-paste. Dumping a spreadsheet “for context” when the task needed one column or one row.
• Unbounded retrieval. RAG that fetches neighboring secrets because the corpus, query, or top-k was loose.
• Tool over-reach. A connector returns more than the task needs, and the whole payload lands in context.
• Shared threads. A workspace chat that reopens yesterday’s paste for a colleague who did not need it.

Inference risk is session-shaped. It is closer to a shipping label than to a training corpus. You can often shrink it immediately: redacted example instead of full dump; one row instead of the file; retrieval with identity-aware bounds instead of “index everything.” Later S3 field notes deepen the paste checklist and the RAG bounds. Here the foundation claim is enough: inference data is not training data, even when both feel like “the AI saw it.”

Map to A1: inference is mostly see in the moment — and if tools can act, do rides the same session. Map to A2: the model generates from what entered; tools and agents decide how much more enters next.

5. Logs and telemetry: what persists after

Logs are the quiet future. The answer appeared. The user closed the tab. Copies may still exist.

Persistence shows up in many clothes:

• Provider or platform conversation history
• Application logs that capture prompts and completions
• Support tickets that paste the “bad output” plus the secret input
• Screenshots in chat tools
• Analytics and quality pipelines that sample transcripts
• Caches, exports, and “debug packages” sent to vendors
• SIEM or monitoring streams that treat AI traffic like any other API payload

The risk is not mystical. It is classic data security with an AI costume: access, retention, purpose, and deletion. Who can reopen the thread? How long does the default keep it? Was the sample anonymized or only hoped to be? Can you delete on request, or only “hide in the UI”?

Conflating logs with training is a common error. “We don’t train on your data” (when true and Learn-backed for a given product) does not mean “we don’t log your prompts.” Those are different sentences. Conflating logs with inference is another error. The session ended; the residue did not.

Foundation rule: mark unknown retention as promise until a docs-backed fact exists for that product and plan. Do not invent days, percentages, or “industry averages.” Unknown is an honest operator state. Invented certainty is not.

A1’s remember question lives here. Memory features, fine-tunes, ticket trails, and exported transcripts are different mechanisms — but they share the persistence problem: something outlived the chat.

6. Why controls differ (and why one dashboard is not enough)

Controls follow the surface.

Training / fine-tune controls sit upstream: data eligibility, corpus governance, exclusion lists, license review, environment separation, and the hard decision to retire a checkpoint. Prompt hygiene does not fix a contaminated fine-tune. A DLP rule on the chat box does not rewrite weights.

Inference controls sit on the path in: minimum necessary paste, classification before send, redaction, bounded retrieval (corpus, query, chunks), least-privilege tools, and human gates on agents that would otherwise keep fetching. You optimize for what enters this window, not for what shaped last year’s pretraining run.

Log / telemetry controls sit on the residue: retention defaults, access ACLs, purpose limitation for quality sampling, redaction before store, deletion and export procedures, and vendor debug rules. You optimize for what survives and who can reopen it.

When teams buy or build “AI Security” as one blob, they often get one of three things well and the other two by accident. A retrieval firewall does not answer retention. A retention policy does not answer fine-tune eligibility. A “don’t paste secrets” poster does not bound an agent’s tool scope.

Govern, protect, and manage still apply — as verbs for data in the era of AI, not as a product opener. Name the surface first. Then pick the control family. Named platforms and portals appear later only when a Learn-backed fact is required. This site explains. It does not sell a suite.

7. How conflation breaks the conversation

Watch for these collapses in meetings:

1. “The model knows our customers.”

Training? Inference paste? RAG over CRM? Log sampling? Four different remediation paths.

1. “We turned off training, so we’re safe.”

Inference and logs may still be wide open. Safety was never one switch.

1. “It’s just chat.”

Chat with tools is a data path. Chat with retention is a store. Chat with a fine-tune is a model change. “Just” is doing too much work.

1. “RAG means we didn’t put data in the model.”

Correct about training weights; incomplete about inference and about indexes that become shadow search. Useful, leaky if unbounded — a later primer expands that. The foundation point: retrieval is inference-time exposure, not a training substitute.

1. “We’ll fix it in the prompt.”

Prompts steer generation. They are not policy engines for training eligibility or log retention. Intent is not control.

Each collapse mixes A1 questions (see / do / remember) or A2 layers (model / tool / agent) with the wrong data surface. Separating the three risks is how you make the rest of the reading path usable.

8. A short intake (not a scorecard)

Before a pilot, write three blunt sentences. No scores. No traffic lights.

Training: “For this system, eligible training or fine-tune inputs are: ___ . Excluded are: ___ . Exit if wrong: ___ .”

Inference: “For a normal task, the session may receive: ___ . It must not receive: ___ . Retrieval/tools may add: ___ .”

Logs: “After the session, copies may exist in: ___ . Access: ___ . Retention: known / unknown (promise). Deletion path: ___ .”

If any sentence is empty, you do not yet have a foundation — you have a demo. Fill the blanks in plain language. Bring vendors and builders back to the blanks. Refuse invented metrics; prefer “unknown” over a fake number.

This intake is enough for early series. Data-security pieces later deepen inventory (subject, store, path, time). Intersection pieces later treat prompts as shipping labels and agents as blast radius. The vocabulary starts here.

9. What this piece is not

• Not a vendor checklist or feature matrix
• Not a claim about any product’s default retention or training policy
• Not a suite pitch; Purview and other platforms stay out of the opener and out of the CTA
• Not legal advice and not a threat model with invented likelihoods
• Not a substitute for A1 (what AI Security means) or A2 (models, tools, agents) — it assumes those separations and sharpens the data pipes underneath them

10. Close: three risks, three sentences

Training data is what entered the model. Inference data is what enters the session. Logs are what persist after.

Keep those sentences separate and AI Security talk gets quieter and more useful. Mix them and every control debate starts one layer off.