Handing the page back to a human

Some things only a person can do. A bot check, a login the agent has no credentials for, a consent dialog, a 2FA prompt. The agent's options are to guess, to give up, or to hand over live control of that part of the page.

We built the third one, and almost everything difficult about it is geometry and timing rather than anything to do with models.

To be clear about where the line is: the agent is told when a page is a bot check — otherwise it reads a near-empty page, decides the site is broken, and reports something untrue — and it is told not to work through it. That check exists to establish a person is present, and the agent is not one. Relaying a real person's clicks is a different thing, and it is what this is. With nobody to ask, it says the page is gated and stops.

The integration is about ten lines

let view: HandoffController | null = null;

ai.chat.stream({ message: "check my order status" }, {
  onBrowserHandoff: (handoff) => {
    if (view) return view.update(handoff);
    view = ai.chat.sessions.handoff(sessionId, handoff, {
      onView: () => render(),
      onEnded: ({ resume }) => {
        view = null;
        if (resume) ai.chat.handoffDone(sessionId);
      },
    });
  },
});

const detach = view.attach(img);   // paints into it, relays what the user does

attach owns the parts that are easy to get subtly wrong and impossible to notice.

The crop. Frames are the whole viewport, so the image is blown up and offset inside a window the size of the region. Which is what lets the challenge overlay that appears after the first click re-aim the view with no re-encoding and no round trip.

The coordinates. The picture is CSS-scaled to fit, so displayed pixels are converted back to page pixels before they are sent. A click a centimetre from where the user aimed is, on a CAPTCHA grid, simply a wrong answer — and it fails in a way that looks like the person being bad at the puzzle.

The gestures. Press, move, release relayed as a real drag, throttled to about 60/s because a pointermove handler fires far faster than any network will carry. This is not optional any more: the common bot check is now a slider you drag, and a viewer that only relays clicks hands someone a puzzle they physically cannot finish.

The keyboard. Typing, Tab, Enter, Backspace and the arrows go to the page, and so does a paste. A printable character is sent as text rather than as a key press — which is the only way anything a layout produces with a modifier arrives as what the person actually typed. An accent, a ı, anything behind AltGr.

Paste is read from the real clipboard event, because Ctrl-V inside the remote page would paste the server's clipboard, which is empty. A password out of a manager reaches a login form no other way.

What it deliberately leaves alone is the viewer's own browser. Cmd-R, Cmd-T and the address bar keep working, because a picture of a page should not swallow the shortcuts of the window it is in. Ctrl/Cmd-A and Ctrl/Cmd-Z do go to the page — select-all before retyping a field, and undo.

Render focused

onView tells you whether the page currently holds the keyboard. Render it.

A picture of a login form gives a person no way to tell whether their keystrokes are reaching it, and the failure is silent in the worst possible way, which is someone typing a password into nothing.

Ending it without asking them to say so

Someone who has just passed a bot check should not then have to report that they passed it, to a system looking straight at the result.

So after each release the region is shown to a separate, small model — separate because the turn's own context is large and this runs repeatedly — which answers one question: is this finished?

The page is asked before the model is. Passing a bot check is normally a navigation: the widget goes away and you land on the content, on a redirect, or for a moment on nothing at all. Two of the five answers therefore cost nothing and are facts rather than opinions:

Of the three the model answers, only those words can ever come back. That is a security property rather than brevity: the checker is looking at a CAPTCHA, and a closed vocabulary is what stops it being a way to read one out. It reports a state, never describes the picture, and never touches the page.

The bug in gone, which is my favourite one here

"Nothing recognisable is on screen" is only evidence of an ending if something recognisable was there to begin with.

Without that condition, gone fired on the first check of every hand-off whose subject nothing can recognise. A login form is not a bot check, and a hand-off need not carry a selector — so both signals were absent before the person had done anything. And since the check runs on their first release, clicking into the email field ended the hand-off and resumed the turn while they were still reading the page.

A hand-off for a consent dialog, a 2FA prompt or a payment form all ended the same way. So gone now requires having seen a bot check during this hand-off, and everything else goes to the model.

A selector that stops matching does not count either. A two-step login takes the named form off the page halfway through, and a selector is the agent's guess at a region worth showing rather than a definition of done.

Branch on will_check_again, not on auto_done

A hand-off that begins with auto_done: true can still run out of checks while the person is working. One hand-off gets a finite number, and somebody still going after that is not being helped by another one.

So a client that reads auto_done once and polls on the strength of it will, on a long hand-off, keep asking a question that has stopped being answered. will_check_again: false is the signal to stop polling and show the Done button — which is the fallback either way, since every failure means "not finished".

A selector that matches nothing is not the whole page

It used to be. The agent has to guess a selector from markup it may never have read, and one wrong guess threw away all the aiming silently — handing someone the entire page and no indication that the aim had missed.

Oberik already recognises these widgets well enough to tell the agent a page is a bot check, so it points the camera itself when the selector misses. Each frame says which happened: selector, detected, or viewport. And the agent is told when its selector missed, so it can pick a better one rather than believing it handed over what it named.