Interactive widgets

widgets · per-turn flag enable_widgets

The agent can write a widget into the conversation when it explains something better than prose: a chart of numbers over time, a table comparing options, a calculator whose sliders recompute the answer on the user's device, or a short form whose answer goes back to the agent. It uses them sparingly, and most turns have none.

Nothing in a widget is code. It is a tree of components from a closed catalog, plus local state and formulas over that state. That has three consequences:

• It renders natively wherever you are. The same widget draws with the SDK in React, React Native, Vue, Svelte, Angular or a plain page. In any other language it reads as text, and the catalog is published as JSON Schema if you want to write a renderer.
• It looks like your product. The agent picks meanings, such as tone: "danger", gap: "sm" and style: "primary". It never picks colours or pixels. Your theme decides what those meanings look like, including dark mode.
• It is checked before anyone sees it. See the guarantee.

Turning it on

1. Grant Interactive widgets under the project's Capabilities. The minted token then carries widgets.
2. Render widget events, and send presses back. Both are shown below.

enable_widgets (default true) lets a client opt out per turn. Turn it off for a client that renders nothing, such as an SMS bridge. A widget sent to such a client still carries fallback, so the substance is never lost.

Rendering — React

import * as React from "react";
import { createClient } from "@oberik/sdk";
import { createWidgetComponent, type WidgetRender } from "@oberik/sdk/widgets";

const OberikWidget = createWidgetComponent(React); // React is passed in, never imported

const ai = createClient({ token });
let sessionId: string | undefined;
// One card per `call_id`: the preview grows in it, then the finished widget replaces it.
const cards = new Map<string, WidgetRender>();

const handle = ai.chat.stream(
  { session_id: sessionId, message: "How does the rate change my mortgage?" },
  {
    onWidgetPreview: (p) => cards.set(p.call_id, previewRender(p)),
    onWidget: (w) => cards.set(w.call_id ?? w.id, w),
    onWidgetDiscarded: (callId) => cards.delete(callId),
  },
);

// …in your transcript, keyed by call_id:
<OberikWidget
  render={cards.get("call_…")!}
  onReply={(press) => ai.chat.stream({ session_id: sessionId, widget_action: press })}
/>;

Import previewRender from @oberik/sdk/widgets as well. A widget arrives at the point in the turn where the agent made it, and the text after it refers to it, so draw it in place rather than at the end. The blocking /chat response carries the finished records on widgets.

Watching it build

The model writes a widget token by token, and a large one takes several seconds. While it does, widget_preview frames carry the finished part so far: the header first, then each child the moment its closing brace arrives, each one checked like the finished widget. The renderers animate each new piece in.

• Nothing half-written is drawn. A chart appears with all its data or not at all.
• Previews react. A slider dragged while the rest is still arriving keeps its value when the finished widget replaces the preview, because both carry the same call_id and stay one card.
• A preview's reply buttons wait for the finished widget, and it shows "Building…".
• If the finished widget is refused, widget_discarded takes the preview down and the agent's corrected widget builds under a new call_id.

A provider that doesn't stream tool arguments sends no previews. The finished widget then simply arrives on its own.

Rendering — React Native

The same component, built from React Native's primitives:

import * as React from "react";
import * as RN from "react-native";
import Svg, { Path, Rect, Line } from "react-native-svg"; // optional
import { createWidgetComponent, nativePlatform } from "@oberik/sdk/widgets";

const OberikWidget = createWidgetComponent(React, nativePlatform(React, RN, { Svg, Path, Rect, Line }));

Without react-native-svg, bar charts are drawn with views and look identical, and line and area charts are drawn as connected segments. Sliders, switches, chips and inputs are drawn from core primitives with no extra dependency. The SDK has no imports at all, so it bundles under Metro unchanged.

Rendering — any other framework

mountWidget draws into a DOM element and keeps that element while the widget redraws, so a slider being dragged or a box being typed into is never interrupted:

import { mountWidget, type WidgetRender } from "@oberik/sdk/widgets";

declare const w: WidgetRender;
declare function sendPress(press: object): void;

const mounted = mountWidget(document.getElementById("widget")!, w, {
  onReply: (press) => sendPress(press),
  theme: { scheme: "dark" },
});
// later: mounted.update(nextRender) / mounted.destroy()

You can also register a custom element and use it in any template language:

import { defineWidgetElement, type WidgetRender } from "@oberik/sdk/widgets";

declare const w: WidgetRender;
defineWidgetElement(); // <oberik-widget>
const el = document.createElement("oberik-widget") as HTMLElement & { render: WidgetRender };
el.render = w;
el.addEventListener("widget-reply", (e) => console.log((e as CustomEvent).detail.press));
document.body.append(el);

Outside JavaScript, show fallback: the widget as text, computed by the server at its starting state. To render natively instead, GET /widgets/catalog returns every component, prop and formula function, and the JSON Schema of a resolved widget.

When the user presses a reply button

A reply button sends the user's answer, with values attached and no paraphrase. Your onReply receives a press. Send it as widget_action on the next request in the same conversation:

POST /chat/stream
{
  "session_id": "…",
  "widget_action": {
    "widget_id": "w_3f2a9c1e7b04",
    "action_id": "actions[0].action",
    "values": { "amount": 300000, "rate": 5.5, "years": 15 }
  }
}

The server does the rest, against the widget as it was stored:

• The action must be one the widget has, and the values must be ones its inputs could produce. A slider's value must be inside its range, and a select's value must be one of its options. Anything else is refused 409.
• The user's message is worded on the server from the button's template. Your client doesn't decide what the transcript says was chosen.
• The agent receives that message with the values attached, both as the user saw them ("Rate (%) = 5.5") and raw, for arithmetic.
• The widget is marked answered. A second press, including one from another tab, is refused, and a reloaded conversation draws the widget read-only at the values that were sent (answered).

A widget from an earlier turn is still a working tool: its sliders, tabs and set-buttons keep working, and its reply button can still be pressed until it has been answered. While a turn is running, pass holdReplies so a press doesn't start a second one. That holds only the reply buttons, never the rest of the widget.

Theming

<OberikWidget
  render={w}
  theme={{
    scheme: "dark",                // default: follows the platform
    radius: 14,
    density: "compact",            // compact | regular | comfortable
    fontFamily: "Inter, sans-serif",
    colors: { accent: "#8b7bff", background: "#111116", border: "#24242d" },
  }}
/>

Every colour can be replaced: background, surface, border, text, muted, faint, accent, onAccent, success, warning, danger, info, and series for charts. On the web a CSS variable works too (accent: "var(--brand)"), though soft tints of that colour then fall back to surface, because a variable can't be mixed. Set chrome: false to draw the widget flush inside your own card.

Never a broken widget

Every widget the agent shows has passed a check on the server first. There is no other route from the agent to the user's screen. "Passed" means more than "parses":

1. Shape. Every formula parses and reads only names the widget declares. Every input is bound to state of the right type. No prop is misspelt, because the catalog is strict.
2. Every input value, not just the first. A widget with a slider is a function of the slider, so the check resolves it at the slider's minimum, middle and maximum, at every option of every select, at both states of every toggle, with every text box empty and filled, and after each button. amount / (rate * 12) with a rate slider that reaches 0 fails at rate = 0, which is exactly the value nobody would test by hand.
3. Meaning. Each resolved widget is validated: a chart series has to be in its data, a select's value has to be one of its options, an image has to be public https, a line chart needs two points, and every input must be read by something.

A widget that fails is not shown. The agent is told exactly what failed, where, and at which input values:

Not shown — the user saw nothing. Problems:
- computed.m: division by zero — guard it, e.g. if(d == 0, 0, n / d) (when rate = 0)
Fix these and call show_widget again (2 attempts left).

It fixes the widget and tries again. After five failed attempts it answers in prose.

Strict about meaning, not about spelling: where a value has exactly one reading, it is rewritten before the check rather than refused. Examples:

• CSS words: align: "right" becomes end.
• Tone and component synonyms: tone: "error" becomes danger, and type: "kpi" becomes stat.
• A name where a number goes: "value": "share" becomes the formula {"expr": "share"}.

The rewritten widget is what is checked, stored and sent, so a renderer never sees an alias.

The renderer keeps the promise too:

• Formulas between sample points: a formula that fails at a value the check didn't sample keeps the last numbers it drew rather than showing an error.
• Unknown components: a component newer than your SDK renders as its text.
• Images: an image that won't load shows its alt.
• Anything else: whatever else throws while drawing falls back to the widget's fallback.

Sparingly, by design

The agent is told to use widgets only when a chart, a comparison, a control or a form makes the subject clearer, and never for a short answer or decoration. Two rules enforce it:

• At most two widgets a turn.
• No widget that only holds text. A paragraph in a box is refused as "write it in your reply".

The catalog

Group	Components
Layout	row, column, divider, tabs, disclosure
Content	text, badge, stat, progress, image, list, table, fields, chart (bar, line, area, stackable), timeline, callout
Inputs	slider, toggle, select (chips), input (text, number, email, multiline, date, tel)
Actions	button with reply (answer the agent), set (change local state) or link

Tones are neutral, accent, success, warning, danger and info. Formats are text, number, currency, percent and compact. Any node can be hidden by a visible formula.

Formulas

A widget's live behaviour is written as formulas over its state. Each input binds one state key, and anything in the widget can read it:

{
  "title": "Loan",
  "state": { "amount": 250000, "rate": 6, "years": 30 },
  "computed": { "m": "pmt(rate / 1200, years * 12, amount)" },
  "children": [
    { "type": "slider", "bind": "rate", "label": "Rate (%)", "min": 0, "max": 12, "step": 0.25 },
    { "type": "stat", "label": "Monthly", "value": "{currency(m)}" },
    { "type": "chart", "kind": "line", "x": "year", "series": [{ "key": "paid" }],
      "data": { "expr": "map(range(0, years + 1, 5), y => {year: str(y), paid: m * 12 * y})" } }
  ]
}

• Where formulas go: any string can interpolate {formula} (write {{ for a literal brace), and any value can be {"expr": "…"}.
• Operators: + - * / %, comparisons, && || ! (also and, or, not), c ? a : b, lists, objects, a.b, a[i], and x => … inside map and filter.
• Functions: min max abs round floor ceil sqrt pow log exp clamp sum avg len range map filter first last contains if str number currency percent compact pmt upper lower plural.

The language is deliberately small. It has no loops other than range and no I/O, and every evaluation has a fixed budget, which is what makes checking every input value possible. The SDK and the server run two implementations of it, held identical by a shared set of cases that both test suites run.

The response shape

{
  "widgets": [{
    "id": "w_3f2a9c1e7b04",
    "widget": { "title": "Loan", "state": { "…": "…" }, "children": ["…"] },
    "fallback": "**Loan**\n\nMonthly: **$1,498.88**",
    "interactive": true,
    "answered": null
  }]
}

• interactive means the widget has a reply button.
• answered is {action_id, values, at} once it has been pressed.

Stored widgets come back on the conversation's messages in extra.widgets.