The diagram was clean, confident, and made up

Asked for a diagram of four SLA tiers, a project whose corpus says 24 hours / 3 days / 14 days / 90 days got back a tidy PNG labelled "1 hour / 4 hour / 12 hour / 24 hour". content came back empty. guard_flags was [].

Nothing on the response hinted that the numbers were invented. That is the part that makes this worth a whole post rather than a warning box.

Why it happens, and why it is not a bug

Media generation is exclusive for the turn. The agent produces the asset rather than calling tools.

Follow that through. No tools means no retrieval. So the picture is drawn from the model's own knowledge of what SLA tiers usually look like — and 1/4/12/24 hours is an extremely plausible set of SLA tiers, which is exactly why it is dangerous. It is not nonsense. It is a confident, well-formed, professional-looking answer to a question nobody asked.

And because no context was retrieved, the output groundedness check has nothing to judge the image against, even when it is switched on. A guardrail that compares an answer to its sources cannot fire when there are no sources.

So every safety mechanism in the product behaves correctly and the result is a picture of your data with none of your data in it.

The rule

Never generate an image of your own data in one step.

Retrieve the figures on one turn. Check them. Then ask for a diagram of those figures on the next, passing the values in — because the values you pass are the only thing keeping the picture honest.

const facts = await ai.chat.send({ message: "What are our four SLA tiers?", tags: ["policies"] });
// read facts.citations, confirm the numbers are right

await ai.chat.send({
  session_id: facts.session_id,
  message: `Draw those four tiers as a diagram: ${theNumbersYouJustVerified}`,
  output_modalities: ["image"],
});

Treat a generated image the way you would treat an uncited sentence. It might be right. Nothing in the system is asserting that it is.

The wider shape: file is about access, the rest are about senses

While we are here, the modality model is the thing people most often get backwards, and getting it backwards produces its own quiet failures.

So input:file alone is enough for a photo upload. The picture is accepted and read with your OCR engine, and the agent works from the text. Add input:image when you want the model to actually see it — the difference between reading the words on a receipt and being able to say the logo is crooked.

And output:file alone is enough for the agent to send you a screenshot it took or a chart it rendered in a sandbox. Those are files it produced, not modalities it emitted.

That distinction is why the modality capabilities are the one thing you do not restate when minting a token. They describe what the agent you built can perceive and hand back — a property of the project rather than a decision about one user. Reading their absence as a refusal meant an app that carefully listed its tool permissions ended up with an agent that could write a document and not give it to anybody.

A self-hosted model's senses are guessed from its name

This one costs people an afternoon.

"What this model can perceive" comes from the provider's catalog where there is one. Where there is not — a self-hosted deployment, a private fine-tune — it comes from the model's name, which you chose, and which tells us nothing.

So a vision deployment called hukuk-db is recorded as text-only. The project's ceiling follows. Granting input:image is narrowed straight back out. And the refusal your end-user sees names a capability whose switch appears not to work.

Correct the record on the provider credential and the grant holds:

curl -sX PATCH "$CP/api/projects/$PID/providers/$CRED_ID" \
  -H "X-API-Key: $KEY" -H 'content-type: application/json' \
  -d '{"modelFacts":{"hukuk-db":{"input":["text","image"],"output":["text"],"mode":"chat"}}}'

It is a merge, and it is recorded as your answer — refreshing the provider re-derives everything else from the catalog and leaves this alone. The capabilities screen says so too when it narrows a grant away: it names the model whose facts were guessed and the request that fixes it.

The general principle, which we arrived at the hard way: when a system infers something about your configuration and the inference is wrong, the error has to name the inference. "Capability not granted" sends someone to the capability screen, where everything looks correct.

Match on id, not on url

Last one, and it is the kind of bug that is nearly impossible to get reported.

Every response re-signs a file's URL. An S3 signature is deterministic given the same key, expiry and timestamp — and the timestamp is granular to the second. So the stream event and the response that follows it usually carry a byte-identical URL, and carry different ones when they happen to land either side of a second tick.

Keying your UI on the URL works nearly every time, then shows one file twice. Intermittently. Under load. In a way nobody can reproduce.

The id is stable for the life of the file and is what the tools take. A URL is a handle with a lifetime: show it or download it, never persist it and never compare two.