Let an agent see and drive your desktop, under your eyes

Agents are good with commands and files. But changing a password in a web interface with no API, ticking a box in an admin page, filling a form: those needed you at the keyboard. With a tool server that captures the screen and drives mouse and keyboard, the agent does them, and you only watch.

What you get. The last manual chores handed over: password changes and settings in web apps that offer no command line, the agent clicking, you supervising. Plus the four constants without which it fails silently.

1. One tool server per desktop

This is not a matter of taste: grim captures nothing under GNOME. Register both under the same name on their own machine, so the agent's instructions stay identical.

uvx --from wayland-mcp-kurojs wayland-mcp     # on the Hyprland laptop

2. Two conditions to switch it on

• /dev/uinput belongs to root. Add a udev rule giving it to the input group, put your account in that group, and log out and back in: the group only applies to a new session. Before that, ydotoold stays "activating" forever.
• GNOME asks for consent through its portal on first use, on the machine's own screen. Approve it once; the token is kept.

Run ydotoold as a user service, not a system one. It can type into any open window, including a root terminal: keep it inside your session at least.

3. The four constants

Without them, nothing works and nothing says so. Clicks land in a corner, a field empties, a button stays silent.

1. The mouse grid is not the screen. On our laptop, ydotool mousemove --absolute works on a grid of 768×432 for a 1920×1080 screen scaled at 125 %: send the pixel read on the capture × 0.4. Check with the compositor's cursor position after a move (hyprctl cursorpos): it is the only proof.
2. Typed text loses characters. outlook.com became outloocom, and the = of a link vanished. Put everything through the clipboard (wl-copy, then Ctrl+V), links and passwords alike.
3. Escape leaves full screen. To close a dialog, click its button.
4. Focus is lost when you type elsewhere. Check which window has the focus before each burst of keys, and compute positions from the window list, never hard-coded.

4. A referee for each gesture

A model looking at a screenshot will happily say "done". We added a tiny tool that compares the capture before and after a gesture and returns a fact:

• NO_OP: nothing moved, the gesture missed;
• CONFIRMED: a small, local change, the signature of an interface that reacted;
• DIVERGED: almost everything changed, another window took over.

Read the window title before taking a picture. It is text, it costs nothing, and a title like "My orders" proves a login as well as a 200 KB image.

Traps

Stay in the room. An agent with your keyboard can type in any window: supervise each session, and never leave it running alone on a desktop where a root terminal is open.

Found this useful? Tell us what to measure next.