# Driving the screen

> Read a window's elements, act on them, wait for the screen, type, click, work tray icons and answer elevation prompts.

Prefer acting on named elements over clicking positions: a name survives a change of window size, position or layout, and a position does not.

## Reading a window

```sh
endia desktop inspect "$m" "$pid" > ui.json
jq -c '.. | objects | select(.controlType?) | {controlType, name, automationId}' ui.json
```

`desktop inspect` prints the window's elements as Windows UI Automation sees them. For each element it gives the control type, name, automation ID, value (never for a password field), whether it is enabled and visible, the patterns it supports (such as `Invoke` and `Value`), and where it is. For the sample's window it shows `Window` "Sample Desktop", `Edit` "Customer name" with the automation ID `CustomerName`, and `Button` "Save".

## Acting on elements

An element is named by a path of segments separated by `/`. **The first segment always names one of the process's windows**, and each later segment is searched for anywhere inside what the segment before it found. A segment gives a control type, a name in square brackets compared exactly, an automation ID after `#`, and a position after `:` counting from 1, in that order; it needs at least one of the first three.

| Selector | Finds |
|---|---|
| `Window[Sample Desktop]/Button[Save]` | the Save button in the window titled `Sample Desktop` |
| `Window/Button[Save]` | the Save button in the process's one window, whatever its title |
| `Window[Sample Desktop]/Edit#CustomerName` | the text box whose automation ID is `CustomerName` |
| `Menu/MenuItem[Exit]` | Exit in a Windows menu the program has opened, which is a window of its own whose control type is `Menu` |
| `Window/MenuItem[Exit]` | Exit in a menu drawn by another framework, such as Avalonia, whose window is an ordinary `Window` |
| `Button[Save]` | **nothing**: its first segment asks for a window that is a button, so the command fails with `ELEMENT_NOT_FOUND` |

```sh
endia desktop wait "$m" "$pid" "Window[Sample Desktop]/Edit#CustomerName"              # returns once the box exists
endia desktop set-value "$m" "$pid" "Window[Sample Desktop]/Edit#CustomerName" "Ada"   # prints {"value":"Ada"}
endia desktop invoke "$m" "$pid" "Window[Sample Desktop]/Button[Save]"                  # prints {"invoked":true}
```

`set-value` and `invoke` act through UI Automation rather than the mouse, so they reach the element even when another window covers it. A selector that matches nothing, or several elements, is refused with the window's element tree in the error, so the test's log shows what was there.

## Waiting for the screen

A program shows its window a moment after it starts, and a window changes a moment after a click. `desktop wait` reads the window again every half second until the element exists, and prints it. `--gone` waits until the element has gone, for example a sign-in form after signing in. `--timeout <seconds>` sets the longest wait (default 60); after it, the command fails with `TIMEOUT` and the last element tree it read. Use it instead of `sleep` or a loop of your own:

```sh
endia desktop wait --timeout 120 "$m" "$pid" "Window[Sample Desktop]"   # one segment: waits for the window
endia desktop wait --gone "$m" "$pid" "Window[Sign in]"
```

## Passwords

`set-value`, `keys` and `pointer` refuse a password field, so that a password is never typed by accident or written to a log. Pass a password on standard input with `--secret-stdin`; East India writes it into the field and never prints, logs or reads it back:

```sh
printf '%s' "$password" | endia desktop set-value --secret-stdin "$m" "$pid" "Window[Sign in]/Edit#Password"
```

## Keyboard

```sh
endia desktop keys "$m" "$pid" "ctrl+a"            # key chords, separated by spaces: "alt+f4 enter"
endia desktop keys --text "$m" "$pid" "Grace"      # types the text as written
```

`keys` brings the process's main window to the front first, then presses the keys wherever its focus is.

## Mouse

```sh
endia desktop pointer "$m" "$pid" click 200,31                  # in the window's client area
endia desktop pointer "$m" "$pid" double-click 200,31
endia desktop pointer --button right "$m" "$pid" click 200,31
endia desktop pointer "$m" "$pid" scroll 200,31 --delta -3      # three notches down
endia desktop pointer --screen "$m" click 838,744               # anywhere on the screen
```

With a process ID, the point is in physical pixels from the top-left corner of the window's client area, the inside of the window below its title bar, and the window is brought to the front first. In the sample, the text box is at 20,20 in the client area and Save at 20,60, so `200,31` lands in the text box and `70,71` on Save. With `--screen`, the point is on the screen, and no window is brought forward. Each `pointer` action prints the screen point and the window and process it reached.

To click an element, name it instead of working out a point. `desktop click` brings its window to the front and clicks the centre of the element:

```sh
endia desktop click "$m" "$pid" "Window[Sample Desktop]/Button[Save]"
endia desktop click --button right "$m" "$pid" "Window[Sample Desktop]/Edit#CustomerName"
endia desktop click --double "$m" "$pid" "Window/List/ListItem:2"
```

Use `invoke` when the element supports it, because it does not depend on the window being uncovered; use `click` for an element that answers only the mouse.

## Notification-area icons

A tray program's icon belongs to the Windows taskbar, not to the tray program, so the commands that take a process ID cannot reach it; `desktop tray` can. Windows hides new icons behind the "Show hidden icons" arrow, and `desktop tray` finds them there too. The menu the icon opens is a window of the tray program's own process, so the element commands take over from there:

```sh
endia desktop tray ls "$m"                                         # NAME and HIDDEN of every icon
endia desktop tray click --button right "$m" "Sample Tray"        # opens the hidden icons when it must
endia desktop wait "$m" "$tray" "Menu/MenuItem[Exit]"              # the menu is the tray program's window
endia desktop invoke "$m" "$tray" "Menu/MenuItem[Exit]"
```

**The menu's window is not always a `Menu`.** What a menu looks like to UI Automation depends on the framework that draws it, so open the menu once and look before writing the selector: `endia desktop inspect "$m" "$tray"`. The sample's menu is a Windows menu: its window's control type is `Menu`, and its items answer `invoke`. Avalonia draws its tray menu in an ordinary window of control type `Window`, named `Window`, with a `List` inside and the `MenuItem`s inside that. For such a menu `Menu/MenuItem[Exit]` finds no window; use `Window/MenuItem[Exit]`. Avalonia's items do not answer `invoke` (`ACTION_UNSUPPORTED`), so use `desktop click`, which clicks the item with the mouse:

```sh
endia desktop wait "$m" "$tray" "Window/MenuItem[Exit]"            # a menu drawn by Avalonia
endia desktop click "$m" "$tray" "Window/MenuItem[Exit]"
```

An icon's name is the program's tooltip, and many programs add their state to it, so `tray click` takes the start of the name: `"Sample Tray"` also matches `Sample Tray: connected`. A name that matches no icon, or several, is refused with the names of the icons there are. Copy the name as `desktop tray ls` prints it: some programs' names begin with a space, which the start of the name must then include.

## Elevation prompts

A program that needs administrator rights makes Windows ask for an administrator's name and password. `endia desktop uac` waits for that prompt and answers it:

```sh
endia desktop uac --program setup.exe --user testadmin --password-stdin "$m" < admin-password.txt
endia desktop uac --program setup.exe --cancel "$m"
```

`--program` is the file name the prompt shows: the program's, or for an installer package the `.msi` file's. The command refuses a prompt for any other program, and types nothing into it. The password of `testadmin` is the one chosen when the root image was prepared.

Windows normally shows the prompt on a separate secure screen, where only Windows itself can read and press it. The image must turn that off with this line in its recipe, which the [tools layer](/docs/testing/images/#the-tools-layer) has:

```dockerfile
RUN reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v PromptOnSecureDesktop /t REG_DWORD /d 0 /f
```

Without it, `desktop uac` fails with `CAPABILITY_MISSING` and says so. `endia inspect <machine>` reports `uacPromptAutomation: true` when the setting is in place.

## Elevated programs

The part of East India that acts on the desktop runs as the desktop user without administrator rights. Windows does not let such a program press buttons, type or click in the windows of a program that runs with administrator rights, whether it was started with `desktop launch --elevated` or through an elevation prompt. So `invoke`, `click`, `set-value`, `keys`, `pointer`, `inspect`, and `wait` for an element inside such a window fail at once with `ACTION_UNSUPPORTED`, and the message says that the process runs elevated.

What still works on those windows: `desktop screenshot`, `desktop windows`, `process inspect`, and `desktop wait` for the window itself, such as `"Window[Completely uninstall]"`. To get past a dialog of an elevated program:

- Start the program without `--elevated` when the test allows it; its dialogs can then be driven as usual.
- Use the program's own options for running without a dialog, such as a silent, headless or force switch.
- End the program with `endia process stop` once its work is done.

A machine whose image was built over a root image prepared before East India 0.16.122 gives the earlier, less clear failures instead, such as `Process <pid> shows no window` or `Access is denied`; the cause is the same.
