Tools

The agent decides what to call; you decide what exists. Three kinds, all through the same gate: project ceiling ∩ token capabilities ∩ request flags ∩ allowed_tools.

allowed_tools narrows the tools the SERVER offers. It does not touch what you declare in the request itself — your client tools and your UI components are bound whichever way it is set, because declaring one is already the strongest statement there is about whether you want it. So allowed_tools: [] means "no built-in tools this turn", not "none of my own either".

Discover what's available

const { tools } = await ai.tools.list();
// [{ name: "rag_search", source: "builtin", description: "…" }, …]

The listing is filtered exactly the way a turn is, so it answers "what could the agent actually be handed on my next request?" — use it to build an allowed_tools list or to render a capability UI.

That includes infrastructure, not only permissions. A family your token holds and the project cannot run — today, anything that needs an embedding model — is absent from tools and reported in withheld with the reason:

const { tools, withheld } = await ai.tools.list();
// withheld: [{ name: "remember", source: "builtin",
//              unavailable: "this project has no embedding model, and the platform's
//                            default is not usable on this deployment (…). Nothing can be
//                            remembered, recalled or written to the wiki: configure one
//                            under Retrieval (LLM & limits) …" }]

Both halves matter if you render controls from this. Leaving them in tools put a "Remember this" button in front of users on a project whose next turn refuses it; leaving them out entirely would fix the lie and leave you unable to explain a control that has quietly disappeared.

Server-side tools

These run inside Oberik. Nothing to implement.

Family	Tools	Capability
Retrieval	rag_search	documents:read
Web	web_search, browse_url	web_search
Browser	browser_goto, browser_read, browser_html, browser_viewport, browser_click, browser_type, browser_scroll, browser_wait, browser_screenshot, browser_close	browser
Browser hand-off	browser_handoff, browser_handoff_end	browser_handoff
Memory	remember, recall, wiki_write, wiki_read, search_history	memory
Scheduling	schedule_reminder, cancel_reminder, list_reminders, current_time, wait	tasks:write
Files	list_files, read_file, write_file	— (see Files)
Sending files	send_file, write_file	output:<kind>
Capture a page	screenshot_url	web_search and output:<kind> — it fetches a URL: see below
Plans	todo_write, todo_update, todo_remove, todo_clear	todo
Asking	ask_user	ask_user
Approvals	request_approval	approvals
Skills	load_skill, read_skill_file	— (see Skills)
Subagents	subagent_start, subagent_check, subagent_stop, subagent_answer	subagents
Tool search	tool_search	action_space
Sandbox	computer_* (12 tools)	computer

This table is a map, not the contract. tools.list() is the reference — it is generated from what your token would actually be handed, so it accounts for the project's ceiling, this token's capabilities, allowed_tools, and any MCP or webhook tools you added. Where the two disagree, the listing is right and this table is stale.

Your own tools (client-side)

When a tool has to run in your application — read the signed-in user's cart, call an internal service, drive your UI — declare it per call. The agent pauses, hands the call back, you run it, and the turn continues.

The SDK can do the whole loop for you. Register a handler and use chat.run or chat.stream:

const ai = createClient({
  getToken,
  tools: [{
    name: "cancel_order",
    description: "Cancel an order belonging to the signed-in customer.",
    parameters: {
      type: "object",
      properties: { order_id: { type: "string" } },
      required: ["order_id"],
    },
    handler: async ({ order_id }) => {
      // Authorize HERE: the arguments are model-generated.
      await assertOwnedBySignedInUser(order_id as string);
      return await orders.cancel(order_id as string);
    },
  }],
});

const res = await ai.chat.run({ message: "Cancel my most recent order" });

chat.run dispatches tool calls, submits results, and repeats until the agent is done (maxToolRounds defaults to 10). Streaming works the same way — pass tools in the handlers and handle.done only resolves once the agent has finished, never with requires_action.

Doing it manually

If you'd rather control the loop — or your tools run somewhere else entirely — use chat.send:

let res = await ai.chat.send({ message, client_tools: schemas });

while (res.requires_action && res.tool_calls.length) {
  const tool_results = await Promise.all(
    res.tool_calls.map(async (call) => ({
      tool_call_id: call.id,
      content: JSON.stringify(await runLocally(call.name, call.args)),
    })),
  );
  res = await ai.chat.send({ session_id: res.session_id, tool_results, client_tools: schemas });
}

client_tools are OpenAI-format tool schemas ({ type: "function", function: {…} }). Two rules worth internalising:

• Authorize inside your handler. Arguments come from the model. Check the order_id really belongs to the signed-in user before acting.
• Keep handlers idempotent where you can. Retries and reconnects are normal.

Tools that need a person to say yes

Add requiresApproval: true (or "requires_approval": true beside function on the raw schema) and the call is never handed to your handler until somebody approves it:

{
  name: "cancel_order",
  description: "Cancel an order. Irreversible: the customer is refunded and the slot is released.",
  parameters: { /* … */ },
  requiresApproval: true,
  handler: async ({ order_id }) => orders.cancel(order_id as string),
}

The call comes back in approvals — carrying tool and arguments, and not in tool_calls — so auto-dispatch cannot reach it. Say yes and the identical call arrives in tool_calls on the next round; say no and the agent is told the refusal is the point.

This is not the same as the agent's own request_approval, where the model decides to ask. Here the gate fires because the tool was called, so there is nothing for the model to skip — which for money moving, a message sent as the user, or anything deleted is the difference between a rule and a habit. It needs no capability: you are declaring a property of your own tool.

A webhook tool can be gated too

requiresApproval works on a webhook tool as well, and means the same thing from your side: the turn pauses, the call comes back in approvals, and nothing happens until somebody decides. The difference is who runs it afterwards — there, we do.

Tools you host (webhook tools)

webhook_tools · per-turn flag enable_webhook_tools

A client-side tool runs in your app's process, which is the right place for it — until there is no app process. A scheduled run, a webhook trigger and a subagent all execute server-side with nobody connected, so a client tool is never offered to them: an agent that can cancel an order in a chat window cannot do it at 3am.

A webhook tool closes that. You publish an HTTPS endpoint; Oberik calls it during the turn and hands your handler the end-user's identity.

curl -sX POST "https://oberik.com/api/projects/$ID/webhook-tools" \
  -H "X-API-Key: $KEY" -H "content-type: application/json" -d '{
    "name": "cancel_order",
    "description": "Cancel an order belonging to the signed-in customer.",
    "url": "https://acme.com/agent/orders",
    "parameters": {
      "type": "object",
      "properties": { "order_id": { "type": "string" } },
      "required": ["order_id"]
    }
  }'
# -> { "id": "…", "name": "cancel_order", "secret": "…" }   the secret is shown once

One secret per tool, not per project

Every webhook tool gets its own signing secret, returned once at creation and never listed again. Three tools means three secrets, captured when you create them — if you lose one, rotate it rather than re-creating the tool.

The single OBERIK_TOOL_SECRET in the verifier below is a one-endpoint example. If several tools point at one handler, key the lookup by the tool field in the body; if each has its own route, give each its own secret in the environment.

Gating one behind a decision

Set requiresApproval: true and the tool is not bound at all: the agent can call for it, the turn stops, and your endpoint is called only after somebody approves.

curl -sX PATCH "https://oberik.com/api/projects/$ID/webhook-tools/$TOOL_ID" \
  -H "X-API-Key: $KEY" -H "content-type: application/json" \
  -d '{"requiresApproval": true}'
# -> { …, "requiresApproval": true }

Omit the field to leave the gate alone — do not send null

requiresApproval takes true or false and nothing else. null, 0 and the string "false" are all refused with a 400 that changes nothing, and the first of those is the one worth knowing about: most typed HTTP clients serialise an absent optional as null, so {"requiresApproval": null} is the body a client sends to mean "I am not touching this". It used to be read as false and answered 200 — turning the gate off on an irreversible tool and reporting success.

A partial PATCH is honoured: send only the fields you are changing and the gate is left exactly as it was.

Note also that the write surface spells it requiresApproval while GET /tools reports requires_approval. Sending the underscored name to the PATCH is a 400 rather than a silent no-op, which is what keeps the two spellings from being a trap.

From a client's side this is the same pause a gated client tool produces — approvals carries tool and arguments, tool_calls stays empty — with one difference: approving produces no tool_calls, because we make the call rather than handing it to you. The next round simply continues with its result. There is nothing for your app to execute.

It pauses on an unattended run too — a scheduled task, a webhook trigger, a subagent — and that is deliberate rather than an oversight. An unanswered question is safe to proceed without: the agent uses its judgement and the answer would only have made it better, which is why ask_user is not offered to an unattended run at all. An unanswered permission is not safe to proceed without, so the turn waits, and the decision can be given out of band from your turn.stopped handler with approval_decisions. The two alternatives are worse: withholding the tool takes away the capability you configured on purpose, and skipping the gate where nobody is watching removes a safety control exactly where it matters most.

GET /tools reports requires_approval per webhook tool, so a caller deciding what the agent may do this turn can see which of them stop and wait.

What your endpoint receives

POST /agent/orders
X-Oberik-Signature: sha256=…            // HMAC-SHA256 of the raw body, with your secret
{
  "tool": "cancel_order",
  "arguments": { "order_id": "o_1029" },  // model-generated — validate it
  "subject": "acme:fin:ana",              // who is asking. From the token, never the model
  "scope": "acme:fin",                    // what they may see
  "roles": ["member"],
  "groups": ["finance"],
  "tenant_id": "…",
  "session_id": "…"
}

Answer with anything — JSON or text. The body becomes the tool result the agent reads, capped at 20,000 characters. A non-2xx status comes back to the agent as a result too ('cancel_order' returned 409: order already shipped), so a handler that explains its refusal is telling the model something useful rather than just failing.

Verify the signature

Until you do, anyone who learns your URL can claim to be any of your users.

Mind the header name. A webhook tool delivery is signed with that tool's own secret in X-Oberik-Signature. Everything Oberik POSTs at you for its own reasons — turn.stopped and scheduled_task.fired — is signed with the project's webhook secret in X-Signature. Two schemes, two header names, two secrets; wiring one verifier to the other's header fails every call and looks like a broken signature. See Verifying a delivery for the other one.

import { createHmac, timingSafeEqual } from "node:crypto";

app.post("/agent/orders", async (req, res) => {
  const raw = req.rawBody;                       // the exact bytes, not the parsed body
  const expected = "sha256=" + createHmac("sha256", process.env.OBERIK_TOOL_SECRET!)
    .update(raw).digest("hex");
  const got = req.header("X-Oberik-Signature") ?? "";
  if (got.length !== expected.length ||
      !timingSafeEqual(Buffer.from(got), Buffer.from(expected))) {
    return res.status(401).end();
  }

  const { arguments: args, subject } = JSON.parse(raw.toString());
  const order = await orders.find(args.order_id);
  if (order.customer !== subject) return res.status(403).json({ error: "not your order" });
  await orders.cancel(order.id);
  res.json({ ok: true });
});

Rotating the secret

curl -sX POST "https://oberik.com/api/projects/$ID/webhook-tools/$TOOL_ID/rotate" \
  -H "X-API-Key: $KEY"
# -> { "secret": "…", "previousSecretUntil": 1786… }

For 24 hours X-Oberik-Signature carries two comma-separated signatures, one per live secret. Accept the call if either matches and rotating costs you nothing — you deploy the new secret whenever it suits, instead of during the switch:

const sigs = (req.header("X-Oberik-Signature") ?? "").split(",");
const ok = [process.env.SECRET_NEW, process.env.SECRET_OLD].filter(Boolean).some((s) =>
  sigs.includes("sha256=" + createHmac("sha256", s!).update(raw).digest("hex")),
);

Without the window, rotating would fail every call between issuing the new secret and redeploying — which is why, in most products, nobody ever rotates.

Watching them work

The dashboard's Webhook tools tab lists recent deliveries: which tool, what came back, how long it took, and the error if there was one. Worth checking after a deploy on your side — a handler that starts returning 500 is otherwise only visible inside one turn's tool result, so the first sign is somebody asking why the agent stopped being able to do something.

The same list, without the dashboard — with your project key, against the control plane:

curl -s "$CP/api/projects/$OBERIK_PROJECT_ID/webhook-tools/deliveries" \
  -H "X-API-Key: $OBERIK_PROJECT_KEY"

The data plane publishes GET /tools/deliveries too, but that one wants an admin credential: a project key is a control-plane credential and the data plane does not accept one, so reach it with a token minted with roles: ["admin"] (see the warning about that field) or use the control-plane route above. Prefer the control-plane route — it needs nothing you do not already have.

How it compares

	needs a connected client	knows which end-user	you host
Client-side tool	yes	yes	nothing — it runs in your app
MCP server	no	no — headers are static per project	a server
Webhook tool	no	yes, signed	an endpoint

Reach for a client tool when the work belongs in the browser or needs something only the signed-in session has. Reach for a webhook tool when the agent should be able to do it whether or not anyone is watching.

Publishing and granting are separate

Defining a tool doesn't bind it. The webhook_tools capability decides whether a given token may reach any of them, so you can ship a tool that cancels an order and still mint a read-only token that cannot. enable_webhook_tools: false declines them for one turn, and allowed_tools names them individually like any other tool.

The arguments are model-generated

subject is ours and is signed; everything in arguments is text the model wrote. Authorize inside your handler against subject, exactly as you would against a session — the example above refuses an order that isn't the caller's.

MCP servers

Connect an MCP server in the dashboard and its tools join the project's catalog (source: "mcp"), gated like any other tool. That's how you reach a system Oberik has no built-in tool for.

An MCP answer is cited like any other

Each result the turn used gets a numbered entry in citations with kind: "tool" and the tool's name, so a sentence built on your MCP server carries a marker and renders as a pill — and renderCited puts it under the sentence that used it.

It reads kind: "tool" rather than pretending to be a document, which is the distinction that used to make this impossible: a citation meant "a passage with a document_id and a chunk_index you can open", an MCP result has neither, so MCP answers came back with citations: [], claims: [], attribution: "none" and the [n] in the reply really was the model's invention. What was uncheckable was never the result — it is the tool call in the transcript, which is at least as followable as a 280-character quote — it was that nothing had numbered it.

Groundedness covers it too: the output guardrail judges the answer against what the turn's tools returned as well as against retrieved passages, so an answer built from MCP data is checked like any other.

What the document index still gives you that MCP does not is a page and a chunk to open, and staleness detection when the source changes. Upload it as documents when you want those.

Separately, mcp:manage controls whether your end-users may attach their own servers at request time:

await ai.chat.send({
  message,
  mcp_servers: [{ name: "acme", url: "https://mcp.acme.com/sse", transport: "sse" }],
});

Off by default — a user-supplied server is code you didn't review. Your project's own servers always load either way.

Driving a browser

browse_url fetches a page and closes it, so the agent only ever sees a site's first paint. With the browser capability it gets a page that stays open across tool calls and works it like a person would:

browser_goto("https://shop.example/login", width=390, height=844)   # mobile layout
browser_type(3, "user@example.com")            # by number, from the observation
browser_type(selector="#password", "hunter2", submit=true)   # or by CSS selector
browser_wait(text="Your orders")
browser_html("#order-42")                      # raw markup: attributes, hidden fields
browser_scroll("bottom")
browser_screenshot()                           # look at it; sending is a separate step

Every observation returns the page text plus a numbered list of what is actually clickable or typeable, and actions refer to those numbers by default. Asking a model to invent a CSS selector for a page it has only read as text is asking it to guess, and a bad guess matches nothing and fails silently. The numbering is recomputed after every action, so a page that changed underneath can't leave the agent clicking something that moved.

A selector works anywhere a number does, and is the right choice once the agent has read the markup with browser_html — which is also how it gets at attributes, data- values and hidden fields the rendered text doesn't show. browser_viewport sets the window size, so checking a mobile layout is a normal thing to ask for.

browser_wait(text: "…") returns the moment the text appears rather than sleeping for a guessed duration — pages that load late are most of why this exists.

One page per conversation, reaped when idle (10 min), on age (1 hour) and when too many are open — a page being used is still a page holding memory and a login, so an idle timer alone cannot bound it. A reaped page is reopened transparently on the next call (blank, and the agent is told so).

Granting browser withholds screenshot_url: browser_goto plus browser_screenshot does everything it did, and two ways to capture a page is one more decision than the model needs to make.

screenshot_url reaches the internet — it needs web_search and a delivery kind

It is filed under Sending files because what it produces is a file. But to make that file it fetches a URL of the model's choosing, and the page's text comes back into the turn — so it is browse_url answering in pixels, and it is gated the same way.

A token without web_search does not have it. That was not always true: it used to ride on output:file / output:image alone, so the capability set Authentication prints as "a read-only research token" (["chat", "documents:read"]) came out able to reach the public internet, and the only way to withhold it at mint time was to decline the output modalities — which also takes away send_file. "May hand back a file" and "may fetch a URL" are not the same decision and were being made together.

Still worth knowing:

• Granting browser withholds it, per the paragraph above — browser_goto plus browser_screenshot covers it. So web_search alone gets you this one; browser gets you the better ones instead.
• Whatever the agent fetches is untrusted input in the same way sandbox output is — and a message can ask for it, so a prompt injection can drive it. That is true of web_search and browse_url too: it is what granting the web means.
• allowed_tools on a request still narrows further, per-turn, if you want the capability granted and this one tool withheld.

Handing the page to the user

Some things only a person can do: a bot check, a login the agent has no credentials for, a consent dialog, a 2FA prompt. With browser_handoff the agent gives the user live control of that part of the page instead of guessing or giving up.

The SDK ships the viewer, so this is an <img> and about ten lines:

let view: HandoffController | null = null;

ai.chat.stream({ message: "check my order status" }, {
  onBrowserHandoff: (handoff) => {
    // Also fires when the agent CLOSES an open hand-off, and when it re-announces one
    // it is still waiting on — `update` handles both without blanking the picture.
    if (view) return view.update(handoff);
    view = ai.chat.sessions.handoff(sessionId, handoff, {
      onView: () => render(),                       // a frame, a crop, a check
      onEnded: ({ resume }) => {
        view = null;
        if (resume) ai.chat.handoffDone(sessionId); // a blocking hand-off stopped the turn
      },
    });
  },
});

// Once you have the element — in a React ref callback, an effect, or plain DOM:
const detach = view.attach(img);   // paints into it and relays what the user does

That is the whole integration. attach owns the parts that are easy to get subtly wrong and impossible to notice:

• the live stream, opened per region and left alone while the agent re-announces the same one, with a single polled frame as a fallback if it never paints;
• the crop — frames are the whole viewport, so the image is blown up and offset inside a window the size of the region, which is what lets the challenge overlay that appears after the first click re-aim the view with no re-encoding and no round trip;
• the coordinates — the picture is CSS-scaled to fit, so displayed pixels are converted back to page pixels before they are sent. A click a centimetre from where the user aimed is, on a CAPTCHA grid, simply a wrong answer;
• the gestures — press, move, release relayed as a real drag, throttled to ~60/s because a pointermove handler fires far faster than any network will carry;
• the keyboard — typing, Tab, Enter, Backspace and the arrows go to the page, and so does a paste. Pointing at the image focuses it, so there is no separate click-to-type step. A printable character is sent as text rather than as a key press, which is the only way anything a layout produces with a modifier — an accent, a ı, anything behind AltGr — arrives as what the person typed. Paste is read from the real clipboard event: Ctrl-V inside the remote page would paste the server's clipboard, which is empty, and a password out of a manager reaches a login form no other way;
• the finished-check — after each release, once the page has settled.

What it deliberately leaves alone is the viewer's own browser: Cmd-R, Cmd-T and the address bar keep working, because a picture of a page should not swallow the shortcuts of the window it is in. Ctrl/Cmd-A and Ctrl/Cmd-Z do go to the page — select-all before retyping a field, and undo. Copy and cut do not, and cannot usefully: inside the remote page they act on a clipboard the person has no way to read.

onView gives you everything needed to render the rest — the URL, whether it is live, whether the crop is the agent's selector or the whole page, whether a check is in flight, and whether the page currently holds the keyboard (focused) — so the card around the image is yours and the geometry is not. Render focused: a picture of a login form gives a person no way to tell whether their keystrokes are reaching it, and the failure is silent in the worst way, which is someone typing a password into nothing.

Nothing here needs a browser until attach, and attach takes anything shaped like an image rather than an HTMLImageElement. Importing the SDK server-side is unaffected, and its published types need no DOM lib.

Building your own viewer

The primitives are all public if you want to render it differently — a canvas, a native app, a different framework's binding:

const stream = ai.chat.sessions.browserStream(sessionId, {
  onFrame: (f) => paint(f.image),          // the whole viewport, already encoded
  onClip: (c) => setCrop(c),               // where to crop it, when that MOVES
  onError: (m) => show(m),
}, selector);

await ai.chat.sessions.browserInput(sessionId, {
  type: "pointer_down", x, y, origin: clip.origin, selector, want_frame: false,
});

stream.close();
await ai.chat.handoffDone(sessionId);

Stream, don't poll. browserFrame renders a fresh screenshot per call and gives a viewer roughly one frame a second — enough to watch a page settle, nowhere near enough to work one. Keep it for a still or as a fallback; use browserStream for anything interactive.

What the user can do to the page

Everything a mouse can, not just tapping — which matters because the common bot check is now a slider you drag, and a viewer that only relays clicks hands someone a puzzle they physically cannot finish.

type	For
click	A tap. button for right/middle, clicks: 2 for a double.
pointer_down / pointer_move / pointer_up	A drag. Stream pointer_move from a real pointermove handler and the page sees the gesture the person actually made, path and all.
drag	The same gesture described in one call, for the agent: to, steps, hold_ms. Interpolated, because a straight teleport from A to B is a motion no hand makes.
wheel	Scrolling a pane that isn't the document — a map, a listbox, a scrollable challenge.
type / key	Text and keys. modifiers: ["Control"] to hold one.
scroll	The window itself.

Coordinates are always image space plus the origin of the frame they came from. Echo the origin back; never add it yourself. The server does that arithmetic, so a crop that moved between paint and release cannot displace the gesture.

Only the part that matters

The agent passes a selector for the widget and only that region is shown — along with any challenge overlay currently visible. That second part matters: every widget CAPTCHA renders its puzzle in a separate absolutely-positioned overlay that appears after the checkbox is clicked, so clipping to the named element alone would show a tick box and hide the thing to solve. The region is recomputed continuously for exactly that reason.

A selector that matches nothing no longer means the whole page. The agent has to guess a selector from markup it may never have read, and one wrong guess used to throw away all the aiming silently. Oberik already recognises these widgets well enough to tell the agent a page is a bot check, so it points the camera at one itself. Each frame says which happened:

region	Meaning
selector	The agent's own selector produced this crop (matched: true).
detected	It matched nothing, and the widget was found anyway.
viewport	Neither — this is the whole page, and your UI should say so.

The agent is told when its selector missed, so it can pick a better one rather than believing it handed over what it named.

Ending it without asking them to say so

Someone who has just passed a bot check should not then have to report that they passed it, to a system looking straight at the result. So after each release the region is shown to a separate, small model — separate because the turn's own context is large and this runs repeatedly — which answers one question: is this finished?

chat.sessions.handoff() does this for you — after each release, once the page has settled. Driving it yourself is two calls:

// Call this after each gesture ends, once the page has settled.
async function finishIfDone() {
  const r = await ai.chat.sessions.handoffCheck(sessionId);
  if (r.done) return ai.chat.handoffDone(sessionId);        // press Done for them
  if (!r.will_check_again) autoDone = false;                // and stop asking
}

The page is asked before the model is. Passing a bot check is normally a navigation: the widget goes away and you land on the content, on a redirect, or for a moment on nothing at all. So two of the answers cost nothing and are facts rather than opinions.

state	Meaning	Costs a model call
gone	A bot check that was on screen earlier in this hand-off is not any more, and something has rendered. Done.	no
loading	Mid-navigation — blank, or not finished loading. Decides nothing; ask again shortly.	no
solved	Visibly complete. Done.	yes
unsolved	Still waiting on them.	yes
unclear	Genuinely cannot tell.	yes

Landing somewhere unexpected counts as finished: the person is done with what they were handed, and the agent reads the page the moment it resumes, so it finds out where it ended up. Leaving someone staring at a page they have finished with, because it isn't the one we predicted, is the worse failure. A blank page is the one case that decides nothing — mid-navigation looks identical to failure, so it waits.

gone needs something to have gone

"Nothing recognisable is on screen" is only evidence of an ending if something recognisable was there to begin with — so gone requires having seen a bot check during this hand-off, and everything else goes to the model.

Without that condition it fired on the first check of every hand-off whose subject nothing can recognise. A login form is not a bot check and a hand-off need not carry a selector, so both signals were absent before the person had done anything — and since the check runs on their first release, clicking into the email field ended the hand-off and resumed the turn while they were still reading the page. A hand-off for a consent dialog, a 2FA prompt or a payment form ended the same way.

A selector that stops matching does not count either. A two-step login takes the named form off the page halfway through, and a selector is the agent's guess at a region worth showing rather than a definition of done.

Of the three the model does answer, only those words can ever come back. That is a security property rather than brevity: the checker is looking at a CAPTCHA, and a closed vocabulary is what stops it being a way to read one out. It reports a state; it never describes the picture, and it never touches the page.

checked: false means no call was made, and will_check_again is the field to branch on: it is the difference between waiting and giving up. Calling on every mouse-up is fine; the extras come back unchecked.

reason	will_check_again	What to do
too_soon	true	Nothing. The last check was moments ago; the next gesture's will land.
checked_enough	false	Stop polling. One hand-off gets a finite number of checks, and someone still going after that is not being helped by another one.
autodetect_disabled	false	Stop polling. This deployment has the checker switched off.
nothing_to_check	false	Stop polling. Not a hand-off that can end itself.

Stop polling means show the Done button and leave it to them — which is the fallback either way, since every failure means "not finished".

The agent arms this with auto_done (default on) and it needs a reason to judge against — "pass the captcha" works, "help me" does not. It stays off for open hand-offs, which have no button to press, and off when enable_handoff_autodetect=false, because with no checker running there is nothing for a client to poll and auto_done: true would be an invitation to wait for ever.

auto_done is where a hand-off starts, not a guarantee for its lifetime

A hand-off that begins with auto_done: true can still run out of checks while the person is working — so a client that reads it once and polls on the strength of it will, on a long hand-off, keep asking a question that has stopped being answered. That is what will_check_again: false is for, and chat.sessions.handoff() already does it: the view's autoDone goes false and the Done button becomes the way out.

Three modes, because "here, deal with this" and "here, have a look" are different asks:

mode / flag	What happens
blocking (default)	The turn stops, exactly like a pending client tool call. The response carries handoff.blocking; render a Done button and send handoff_done: true to resume. The agent is told not to ask a question — the button is the answer.
open	The agent carries on. It closes the view itself with browser_handoff_end when it no longer needs you looking.
interactive: false	Show it, don't let them touch it. The input endpoint refuses for these, so a read-only hand-off is read-only on the server rather than merely looking it in the UI.

On CAPTCHAs specifically: the agent is told when a page is a bot check — otherwise it reads a near-empty page, decides the site is broken and reports something untrue — and is told not to work through it. That check exists to establish a person is present, and the agent isn't one. Relaying a real person's clicks is a different thing, and it's what this is. With nobody to ask — a scheduled run, a webhook — it says the page is gated and stops.

It browses as your end-user

Anything reachable behind a login the agent can complete is reachable by the agent. Grant it where you would be comfortable with that.

Large tool catalogs

With action_space, only the tools relevant to the message are bound for that turn, instead of every tool in the catalog. It keeps accuracy up and cost down once you have dozens of tools. Decline it per request with enable_action_space: false.

Two things worth knowing before you turn it on, because neither is guessable:

• It needs an embedding model. Relevance is measured by embedding your message and each tool's description, which is the same machinery retrieval and memory use — so a project with no embedding model configured gets nothing from this capability. GET /capabilities says so: the flag comes back effective: false with blocked: "needs an embedding model", and the turn behaves as though the capability were off (every tool bound, no tool_search) rather than failing. The turn says so too, on warnings, because this is the one capability that degrades to everything rather than to nothing — there is no missing tool to notice, so a caller who set the flag once and never re-read capabilities() would otherwise see a feature that appears to work:

  { "warnings": ["this project has no embedding model, and the platform's default is not
     usable on this deployment (…). enable_action_space was set and could not narrow
     anything, so every tool is bound on every turn instead of the relevant few — the cost
     this capability exists to avoid: configure one under Retrieval (LLM & limits) …"] }

• Nothing is narrowed below about 8 tools. Under that, the whole catalog is smaller than the subset would be, so narrowing is skipped and tool_search is not bound either — a tool whose description is "there are more tools than you can see" must not appear on a turn where that is false. Past it, roughly the 6 most relevant are bound, plus anything pinned or already in use.

  Both numbers are deployment defaults rather than a contract, which is why they are written as "about" and "roughly" — but they are written as digits so that grepping for one finds it. Somebody looking for the threshold searched for a number, found nothing, and reported that it was documented nowhere; it was, four lines from where they looked.

Crucially, the narrowing is reversible by the agent. It always holds tool_search, so when it needs something it can't see it searches for it and whatever matches becomes callable immediately — for the rest of the turn, not just the next step. Tool families that only work whole (the todo tools, scheduling, the browser) are never split, and anything the agent is already mid-way through using is kept regardless of how the current message happens to rank.

That matters because the alternative is worse than a large catalog: an agent that silently lost a tool it used a moment ago, with no way to ask for it back, reasons about its own capabilities from a false picture.

Restricting a single turn

await ai.chat.send({ message, allowed_tools: ["rag_search"] });   // retrieval only
await ai.chat.send({ message, allowed_tools: [] });                // no server-side tools
await ai.chat.send({ message, enable_web_search: false });         // keep it internal

allowed_tools covers server-side and MCP tools; your own client_tools are always available, since you chose to pass them.

A project-wide restriction is also available (Capabilities → allowed tools), applied before any request-level list — and unlike the per-request one, an end-user token cannot widen it. That makes it the answer to "withhold this tool from every token this project mints": set it once on the project rather than remembering to send allowed_tools on every request.

GET /capabilities reports it as allowed_tools (null = no restriction), so a client can tell a tool it may not call from a tool that does not exist. GET /tools already applies it, as it applies every other gate.

UI components — tools that draw

ui_tools · per-turn field ui_tools

A client tool is work handed back: the turn stops, you run it, the result returns as a tool message. That shape is wrong for showing a chart, dropping a booking card into the thread, or highlighting a row — there is no result. Waiting for one turns a one-way instruction into a round trip, and a client with nothing useful to say ends up inventing {"ok": true} for the model to reason about.

So a UI component is declared like a tool and behaves like a display:

const ai = createClient({
  getToken,
  ui: [{
    name: "show_chart",
    description: "Draw a line chart of a numeric series over time.",
    parameters: {
      type: "object",
      properties: {
        title: { type: "string" },
        points: { type: "array", items: { type: "number" } },
      },
      required: ["title", "points"],
    },
    render: ({ title, points }) => setChart({ title, points }),
  }],
});

The agent calls it, your render runs, and the turn does not pause — it carries on in the same step. Streaming clients render the instant it is called; waiting for done would mean a chart appearing after the paragraph that refers to it. The whole list also comes back on the turn as ui, in call order, so a client that re-mounts can redraw from it rather than replaying events.

The agent is told what it drew and asked to refer to it rather than repeat it in prose, so you get "here's how that trended" beside the chart instead of the chart described twice.

A few things it deliberately is not:

• Not trusted to be listening. A component that reaches nobody still returns cleanly. A closed tab is not a failed turn.
• Not a transport. Payloads are capped — pass an id and let your client fetch the rest — and a turn can draw at most 40 things.
• Not available to subagents. A delegate's output is a report the main agent has to weigh; letting one paint the conversation would put something in front of the user that the agent answerable for the turn never saw.