Files, images & generated media

Modalities are capabilities, granted per token: input:image, input:audio, input:video, input:file and output:image, output:audio, output:video, output:file. Anything not granted is ignored rather than trusted.

You don't list them when you mint: a token gets whatever the project's toggles allow. Pass an output:<kind> only to narrow that side — to stop the agent sending images to a client that can't render them, say — and the other side is untouched either way.

file is about access. The rest are about the model's senses.

This is the whole model, and getting it backwards is the one mistake worth avoiding.

	file	image · audio · video
Input	a user may attach anything, of any type. What the model can't perceive is read as text — extracted where there is a text layer, OCR'd where there is not, a transcript for a recording.	the model may look at that kind with its own senses, raw.
Output	the agent may hand back any file it produced — a document it wrote, a sandbox export, a screenshot.	the model may generate that kind.

So:

• input:file alone is enough for a photo upload. The picture is accepted and read with your project's OCR engine (see Documents), and the agent works from the text. Add input:image when you want the model to actually see it — the difference between reading the words on a receipt and being able to say the logo is crooked.
• output:file alone is enough for the agent to send you a screenshot it took or a chart it rendered in a sandbox. Those are files it produced, not modalities it emitted. output:image is for the model drawing a picture itself.

file therefore needs no multimodal model on either side — extraction and delivery are plain machinery — which is why it's the one modality every project can grant, and why a new project starts with input:file and output:file. See Files for what the agent does with them once they exist.

The media kinds are filled in the moment you register a model that can handle them: provider add seeds the project's input modalities from what the models can perceive, so a vision model arrives with input:image rather than leaving a project that will not accept a photo.

A self-hosted model's senses are guessed from its name

"What this model can perceive" comes from the provider's catalog where there is one, and otherwise from the model's name — which is chosen by you and tells us nothing. So a self-hosted vision deployment called hukuk-db is recorded as text-only, the project's ceiling follows, and granting input:image is narrowed straight back out. The refusal an end-user sees then names a capability whose switch appears not to work.

Correct the record on the provider credential and the grant holds:

curl -sX PATCH "$CP/api/projects/$PID/providers/$CRED_ID" \
  -H "X-API-Key: $KEY" -H 'content-type: application/json' \
  -d '{"modelFacts":{"hukuk-db":{"input":["text","image"],"output":["text"],"mode":"chat"}}}'

It is a merge, so sending only input is enough, and it is recorded as your answer — provider refresh re-derives everything else from the catalog and leaves this alone. capabilities set says so too when it narrows a grant away: it names the model whose facts were guessed and the request that fixes it. ::: Output media is never granted for you — generating a picture is a spend decision, and adding a provider is not making it. Clearing a list is respected: an explicit empty list is a project saying "no attachments", and nothing fills it back in.

The media modalities are bounded by your models. The dashboard reads what each registered model can take and produce, from the provider's own catalog, and one none of them supports cannot be switched on. A project whose only model answers in text will find output:image struck through: granting it would mint a capability on every token the agent then refuses to honour, with nothing on screen explaining why. input:image struck through does not stop your users attaching pictures — that's input:file, and it stays available.

Self-hosted models

Most catalogs answer "what models are there" and nothing else — OpenAI's /v1/models publishes four fields, and every server speaking its shape (vLLM, LM Studio, LocalAI, your own gateway) publishes the same four. So a model is identified in as many ways as there are:

1. what the provider says — modalities, where it publishes them;
2. the real model behind the name — vLLM reports root alongside a --served-model-name, so a model registered as house-model is still recognised as the Qwen/Qwen2.5-VL-… it actually is;
3. the published model index, matched on the model's own name rather than on who is selling it — the same open-weights model has an entry under every provider that hosts it, and none under a server you run yourself.

When none of them knows, the dashboard says so and asks you. A private fine-tune is not described anywhere, and no amount of inference will change that. The model keeps a text-in/text-out floor so it works, the provider form flags it, and you tick what it actually takes and returns — one row per unidentified model, with what it is for (chat, embedding, rerank) beside it and a line saying what that choice will do. Your answer is kept as an answer: re-listing the catalog will not derive over it.

Every model stays editable, not only the unidentified ones. A served model name is a label, and the model behind it can be swapped out from under it — so a catalog that was right in March is not evidence about today. Models the catalog described are one click away under Adjust, in the same provider form.

That is what bounds the modalities your project can grant, so an unticked box is a capability your users won't get.

Sending things in

Attach a file when it's context for this question, rather than something to index permanently.

await ai.chat.send({
  message: "What changed between these two invoices?",
  attachments: [
    { kind: "file", name: "march.pdf", url: `data:application/pdf;base64,${b64(march)}` },
    { kind: "file", name: "april.pdf", url: `data:application/pdf;base64,${b64(april)}` },
  ],
});

An attachment is { kind, url, name?, mime_type? } where url is a data: URI or an https:// URL. Images go to the model as images; documents a model can't read natively are extracted with the same layout-aware pipeline used for uploads. Nothing is added to the knowledge base — use Documents for that.

Prefer an https:// URL for anything large, and to_sandbox: true when the file is data for the agent to process rather than read (see Sandboxed compute).

attachments: [{ kind: "image", url: signedUrl, name: "chart.png" }]

To attach something already uploaded, hand over a signed URL for it rather than re-sending the bytes:

const { url } = await ai.documents.downloadUrl(doc.id);
await ai.chat.send({ message: "Read the chart on page 3", attachments: [{ kind: "file", url }] });

Though for indexed documents the usual answer is simply to let retrieval find them — tags / document_ids on the request, no attachment at all.

Getting media back

Ask for a modality and the answer comes back as an attachment:

const res = await ai.chat.send({
  message: "Draw a clean diagram of this pipeline",
  output_modalities: ["image"],
});

res.attachments;   // [{ id, kind: "image", url, name, mime_type, size }]

Pass model to choose a generation model; the token's model allow-list still applies. Media generation is exclusive for the turn — the agent produces the asset rather than calling tools, so do retrieval on one turn and generation on the next if you need both.

A generated image is not grounded in anything

That exclusivity is not only a sequencing inconvenience. Because no tools run, the turn does no retrieval — so the picture is drawn from the model's own knowledge, and the output groundedness check has no context to judge it against even when it is switched on.

Asked for a diagram of four SLA tiers, a project whose corpus says 24 hours / 3 days / 14 days / 90 days got a clean, confident PNG labelled "1 hour / 4 hour / 12 hour / 24 hour". content came back "" and guard_flags was []: nothing on the response hints that the numbers were invented.

Never generate an image of your own data in one step. Retrieve the figures on one turn, check them, then ask for a diagram of those figures on the next — the values you pass in are the only thing keeping the picture honest. Treat a generated image the way you would treat an uncited sentence.

While streaming, files arrive on the attachments event as soon as they are produced — right after the tool call that made one — and again before done with the full list. Every frame carries the whole list so far, so render from it rather than appending and a missed frame cannot leave a gap.

Match on id, not on url. A URL is signed with an expiry, so it is a handle with a lifetime rather than an identity: do not persist one, and do not compare two. The id is stable for the life of the file and is what the tools take.

Why that is not just tidiness, since the reason used to be given wrongly here: every response re-signs the URL, but an S3 signature is deterministic given the same key, expiry and timestamp — and the timestamp is granular to the second. So the stream event and the response that follows it usually carry a byte-identical URL, and carry different ones when they happen to land either side of a second tick. Keying on the URL works nearly every time, then shows one file twice — the version of this bug that is hardest to reproduce and so hardest to ever get reported.

Reusing what was generated

Generated media is stored, not discarded, and a URL is signed with an expiry rather than kept. That's what makes two expected things work.

Follow-up edits. The previous image is put back in front of the model on the next turn, so revision prompts have the original to work from:

const first = await ai.chat.send({
  message: "A wide banner, dark background, violet accent",
  output_modalities: ["image"],
});

await ai.chat.send({
  session_id: first.session_id,
  message: "Same, but make the accent teal",
  output_modalities: ["image"],
});

Piping into the sandbox. With the computer capability the agent copies an image it generated into the workspace (computer_fetch_attachment) and processes it there — crop it, convert it, drop it into a report — then exports the result as a download.

Files the agent produced

Anything exported from a sandbox rides the same channel — res.attachments, or the attachments stream event — with a kind matching its type. So a chart the agent plotted is an image you can render inline, while a report or a zip stays a file:

// [{ id: "9a50d2717488", kind: "image", url, name: "chart.png", mime_type: "image/png", size: 41022 },
//  { id: "c17f0e42aa10", kind: "file",  url, name: "summary.csv", mime_type: "text/csv", size: 8213 }]

Rendered images also come back into the next turn's context, so "same chart, log scale" doesn't start from scratch. See Sandboxed compute.

Signed URLs expire. Show them or download them; don't persist them in your own database — store the id if you need to refer to a file later, and it is also what the file-taking tools accept.

A note on PII

By default the model receives attachment content verbatim — on a "talk to your own data" product, the user is deliberately sharing their own information. If your compliance posture requires otherwise, PII redaction is available per project under Guardrails.