# Controlling the desktop

> Start programs on a machine's desktop, name their elements, type, click, answer elevation prompts and take screenshots.

The `endia desktop` commands act in the signed-in user's session, where the screen is. Most commands name a process by its ID, and the keyboard and mouse commands first bring that process's main window to the front. `desktop pointer --screen` instead acts at a point on the screen, without naming a process.

The examples read a value from JSON with `jq`, which is a separate program that must be installed on the Linux side where the examples run.

```sh
pid=$(endia desktop launch orders-dev-dev-1 C:\\app\\Orders.exe | jq -r .processId)
endia desktop wait orders-dev-dev-1 "$pid" "Window[Orders]/Edit#CustomerName"
endia desktop set-value orders-dev-dev-1 "$pid" "Window[Orders]/Edit#CustomerName" "Ada"
endia desktop invoke orders-dev-dev-1 "$pid" "Window[Orders]/Button[Save]"
endia desktop click --button right orders-dev-dev-1 "$pid" "Window[Orders]/List/ListItem:3"
endia desktop keys orders-dev-dev-1 "$pid" "ctrl+s"
endia desktop keys --text orders-dev-dev-1 "$pid" "Order 42"
endia desktop pointer orders-dev-dev-1 "$pid" click 120,45
endia desktop pointer --screen orders-dev-dev-1 double-click 960,540
```

## Commands

| Command | What it does |
|---|---|
| `desktop launch <machine> <program> [args]` | Starts a program on the desktop and prints, in JSON, its process ID (`processId`), start time, session, account, integrity level and windows. |
| `desktop wait <machine> <pid> <element>` | Waits until the element exists, then prints it and its window. With `--gone`, waits until it no longer exists. `--timeout <seconds>` sets the longest wait; the default is the service's `desktopActionTimeoutSeconds`, 60 seconds unless changed. |
| `desktop invoke <machine> <pid> <element>` | Presses a button or other element through UI Automation, without moving the mouse. |
| `desktop click <machine> <pid> <element>` | Brings the element's window to the front and clicks the centre of the element with the mouse, for an element that responds only to mouse clicks. `--button right` and `--double` change the click. |
| `desktop set-value <machine> <pid> <element> <value>` | Sets a text box or similar element, and prints the value read back. A password field accepts only `--secret-stdin`, which reads the value from standard input and never prints or logs it. |
| `desktop keys <machine> <pid> "<chords>"` | Presses key chords such as `"ctrl+s"` or `"alt+f4 enter"`. With `--text`, types the text as written. |
| `desktop pointer <machine> <pid> <action> <x,y>` | Moves the mouse to a point in the process's main window and clicks, double-clicks or scrolls there. `--button right` and `--button middle` change the button; `--delta -3` scrolls three notches down. It prints the screen point, and the window there with the process that owns it. |
| `desktop pointer --screen <machine> <action> <x,y>` | Does the same at a point on the screen, without choosing a window or bringing one to the front. |
| `desktop windows <machine>` | Lists every visible window on the desktop, with the process that owns it, its position and size, and its title. |
| `desktop tray ls <machine>` | Lists the icons in the notification area beside the clock, including those hidden behind "Show hidden icons". |
| `desktop tray click <machine> <icon>` | Clicks the icon whose name starts with `<icon>`, opening the hidden icons first when it is there. `--button right` opens most programs' menus. |
| `desktop uac --program <exe> --consent <machine>` | Answers the Windows elevation prompt for that program. `--consent` accepts it and `--cancel` refuses it. `--user <admin> --password-stdin` enters an administrator's name and password. |

## Elements

An element is named by a path of segments separated by `/`. **The first segment always names one of the process's windows**; each later segment is searched for anywhere inside what the segment before it found. So `Window[Orders]/Button[Save]` is the Save button in the window called `Orders`, while `Button[Save]` on its own looks for a *window* that is a button, and fails with `ELEMENT_NOT_FOUND`. When you do not know the window's title, `Window/Button[Save]` works while the process shows one window.

A menu that a program opens is a window of its own. A Windows menu's window has the control type `Menu`: `Menu/MenuItem[Exit]`. A menu drawn by another framework may be an ordinary `Window`, as Avalonia's is: `Window/MenuItem[Exit]`. `desktop inspect` shows which. When the first segment matches no window, the tree in the error shows each window without its children, so an empty `children` there does not mean the window is empty.

Each segment may give, in this order:

1. a control type (`Window`, `Button`, `Edit`);
2. a name in square brackets, compared exactly and with case (`[Save]`);
3. an automation ID after `#` (`#CustomerName`);
4. a position after `:`, counting from 1 (`:2`).

It must give at least one of the first three. `Window[Orders]/Edit#CustomerName` names an edit box in the window called `Orders`, and `Window[Orders]/Button:2` the second button in it. `desktop inspect` shows the control types, names and automation IDs to use. Naming an element is sturdier than clicking a position, because it does not depend on the window's size or layout.

## Waiting, passwords and tray icons

**Waiting.** A program shows its windows some time after it starts, and a window changes after a click. Wait for the element the next step needs with `desktop wait` instead of sleeping or polling `desktop inspect`: it returns as soon as the element exists, and fails with `TIMEOUT`, showing the window's elements, if it never does. A one-segment selector such as `"Window[Orders]"` waits for the window.

**Passwords.** `set-value`, `keys` and `pointer` refuse a password field, so that a password cannot be typed by accident or end up in a log. To enter one, pass it on standard input: `printf '%s' "$password" | endia desktop set-value --secret-stdin orders-dev-dev-1 "$pid" "Window[Sign in]/Edit#Password"`. East India writes it into the field and never prints, logs or reads it back.

**Notification-area icons.** A tray program's icon belongs to the Windows taskbar, not to the program, so the commands that take a process ID cannot reach it; `desktop tray` can. Icon names are the programs' tooltips, and many programs add their state to them, so `desktop tray click` takes the start of the name: `endia desktop tray click --button right orders-dev-dev-1 "Orders"` clicks `Orders: connected`. The menu that opens is a window of the tray program, so continue with its process ID: `desktop wait` and `desktop invoke` with `"Menu/MenuItem[Settings]"` for a Windows menu. Avalonia's menu is a `Window` whose items do not answer `invoke`: use `"Window/MenuItem[Settings]"` with `desktop click`.

## Positions

Every position is in physical pixels, and there are two origins:

- **Relative to a window.** With a process ID, the position is measured from the top-left corner of the process's main window's client area: the inside of the window, below its title bar. `120,45` is 120 pixels right and 45 pixels down from there, wherever the window is.
- **Absolute on the screen.** With `--screen`, the position is measured from the top-left corner of the screen. `desktop windows` gives window positions on the same scale, so a point inside a window's rectangle there lands on that window, if no other window covers it.

`desktop inspect` gives each element two rectangles. `bounds` is measured from the window's outer top-left corner, which is not the client area's corner. `screenBounds` is measured from the top-left corner of the screen. To click an element, use `desktop click` with its selector, which brings the window to the front and clicks the centre of `screenBounds` for you.

**Process IDs.** Windows reuses process IDs. Every command that takes a process ID also takes `--started <time>`, the start time `process inspect` prints, such as `2026-10-05T14:03:22.1234567Z` (UTC), and refuses to act if the process with that ID started at another time.

## Elevation

`desktop launch --elevated` starts a program with administrator rights, and is refused when the desktop user is a standard user. In that case, start the program normally and answer its elevation prompt with `desktop uac`.

**Windows of elevated programs.** The part of East India that acts on the desktop runs as the desktop user without administrator rights. Windows does not let such a program press buttons, type or click in the windows of a program that runs with administrator rights, whether it was started with `desktop launch --elevated` or through an elevation prompt. So `invoke`, `click`, `set-value`, `keys`, `pointer`, `inspect`, and `wait` for an element inside such a window fail at once with `ACTION_UNSUPPORTED`, and the message says that the process runs elevated. What still works on those windows: `desktop screenshot`, `desktop windows`, `process inspect`, and `desktop wait` for the window itself, such as `"Window[Completely uninstall]"`. To get past a dialog of an elevated program:

- Start the program without `--elevated` when the test allows it; its dialogs can then be driven as usual.
- Use the program's own options for running without a dialog, such as a silent, headless or force switch.
- End the program with `endia process stop` once its work is done.

A machine whose image was built over a root image prepared before East India 0.16.122 reports the same cause with older, less clear messages, such as `Process <pid> shows no window` or `Access is denied`.

## Screenshots

```sh
shot=$(mktemp --suffix .png)
endia desktop screenshot orders-dev-dev-1 > "$shot"
endia desktop screenshot --pid "$pid" orders-dev-dev-1 > "$(mktemp --suffix .png)"
```

`endia` writes the screenshot as a PNG to its standard output, on the computer where it runs, so you decide where the image goes: a temporary file, as above, or a program that reads the image directly. Nothing is saved on the PC or in the machine. `-o <file>` writes the image to that file instead and prints the file's full path. Without `--pid` the command captures the whole desktop; with `--pid` it captures that process's largest visible window.
