# Building and layering images

> Prepare a root image from a Windows disc, build images on it from Endiafiles, and decide where the layers fall.

Images form layers. A root image holds Windows and East India's own programs. Each built image adds the steps of a recipe on top of its parent and is stored as a disk of differences over the parent's disk. Every parent becomes read-only, so one Windows disk serves every image built on it. Building a new application layer therefore does not repeat the slow steps below it.

## The root image

Every image starts from a Windows disk that holds East India's own programs. `endia base prepare` makes one from a Windows installation disc:

```sh
echo "$PASSWORD" | endia base prepare M:\eastindia\windows11-iot-ltsc-2024-eval-x64.iso --tag lab/win11:base --password-stdin
```

**The ISO must be in the image folder**

The ISO must be in the image folder, `M:\eastindia`, because that is the folder the `eastindia` account on the PC is allowed to read. The command takes about half an hour.

Set `PASSWORD` in your shell to the password you choose, and make it meet Windows' own password rules, because Windows Setup applies them. The password is for both accounts the new Windows has: `testuser`, a standard account that signs in automatically at every start, and `testadmin`, an administrator whose name and password answer an elevation prompt. `--desktop-user` and `--admin-user` change the names, and `--index` chooses another edition on the disc. The service keeps the password in memory only. It reaches nothing but the answer file, the list of settings Windows Setup follows, which is deleted once Windows is installed.

The command installs Windows in a machine of its own, without anyone at the screen, together with:

- East India's guest service, a Windows service in the machine through which the East India service on the PC reaches it;
- East India's desktop helper, a program that starts at every sign-in and controls the screen, keyboard and mouse for the desktop commands;
- an OpenSSH server.

It then reads which edition and build it installed, and when an evaluation copy expires, and turns Windows Update off. Every machine is short-lived, so updating Windows would only spend the machine's processor time, memory and network on work that is thrown away with it. Microsoft Defender still looks for its own updates, and its downloads fail. If any Windows Update setting cannot be made, the command fails with `CAPABILITY_MISSING` and the error, and keeps no image. Last, it imports the disk as a base, which boots it once more to check, in order:

1. `guest-service`: the guest service answers;
2. `fast-startup`: Fast Startup is off, so that stopping and starting a machine boots Windows afresh;
3. `desktop`: the desktop account is signed in with the desktop helper running;
4. `ssh`: SSH answers on port 22;
5. `evaluation`: Windows says when its evaluation period ends, or that it is not an evaluation copy;
6. `shutdown`: Windows shuts down cleanly.

A failed check fails the command with `CAPABILITY_MISSING` and names the check that failed, using the name that starts its line in the list above, such as `ssh`. When every check passes, the command's last line is the new image's digest, as `docker import` ends by printing the new image's ID. Removing the base later deletes its disk.

### Importing your own disk

A disk you prepared yourself can be imported instead, if it is a stand-alone VHDX with:

- the guest service installed as the Windows service `EastIndiaGuest`;
- the desktop helper started at sign-in;
- automatic sign-in of a desktop account;
- an OpenSSH server;
- Fast Startup off (`HiberbootEnabled` set to 0 under `HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Power`).

Import it as `docker import` imports a file:

```sh
endia import M:\eastindia\win11.vhdx lab/win11:base
```

The import copies the disk into the image folder (`--move` moves it instead), makes it read-only and runs the same import check, then prints the new image's digest. The copy needs the disk's size plus the 20 GiB that East India keeps free on the image folder's drive (`minFreeDiskGiB`); with less free, the import fails with `STORAGE_FULL` and copies nothing. A disk attached to a virtual machine, or one that is already an image's disk (the same file, or a copy that carries the same Hyper-V disk identifier), is refused with `RESOURCE_IN_USE`, naming the virtual machine or the image. Every machine made from the image waits for the account that the `desktop` check found signed in. The `evaluation` check records when an evaluation copy expires, so that `endia images` warns before it does. `-c` (`--change`) sets the image's configuration with a recipe instruction, as `docker import --change` does: `LABEL`, `ENV`, `WORKDIR`, `CMD`, `ENTRYPOINT` or `EXPOSE`, for example `-c "ENV LANG=en-GB"`.

## Built images

A recipe adds software and Windows settings to a parent image, in Dockerfile syntax. Its file is an Endiafile: `Endiafile` when a folder holds one recipe, or `name.Endiafile`, such as `tools.Endiafile`, when it holds several. East India also reads Docker's names: `endia build` without `-f` reads `Endiafile`, or `Dockerfile` when there is no `Endiafile`, and `-f` takes a file of any name, so an existing `app.Dockerfile` still builds. As with a Docker image, the recipe installs everything the test needs, .NET included: the root image holds only Windows and East India's own programs. A typical stack has two built layers: one with the runtime every test needs, and one with the application under test.

In these recipes `SHELL` names `powershell.exe`, which is Windows PowerShell 5.1, the version that ships with Windows.

```dockerfile title="tools.Endiafile"
# tools.Endiafile: .NET 10 and its desktop runtime, over Windows
# escape=`
FROM lab/win11:base
SHELL ["powershell.exe", "-NoProfile", "-Command"]
WORKDIR C:\setup
COPY dotnet-install.ps1 .
RUN .\dotnet-install.ps1 -Channel 10.0 -Runtime dotnet -InstallDir 'C:\Program Files\dotnet'
RUN .\dotnet-install.ps1 -Channel 10.0 -Runtime windowsdesktop -InstallDir 'C:\Program Files\dotnet'
RUN [Environment]::SetEnvironmentVariable('Path', $env:Path + ';C:\Program Files\dotnet', 'Machine')
```

`dotnet-install.ps1` is Microsoft's install script, from `https://dot.net/v1/dotnet-install.ps1`, kept in the build context. The desktop runtime needs the base runtime, which brings `dotnet.exe`, so the base runtime is installed first.

```dockerfile title="app.Endiafile"
# app.Endiafile: the application and the Windows settings it needs, over the tools
# escape=`
FROM lab/tools:1
SHELL ["powershell.exe", "-NoProfile", "-Command"]
COPY app/ C:/app/
RUN netsh advfirewall firewall add rule name=OrderService dir=in action=allow protocol=TCP localport=8080
RUN Add-Content C:\Windows\System32\drivers\etc\hosts "10.0.0.50 licence-server.lab"
RUN reg add HKLM\SOFTWARE\Contoso\Orders /v Mode /t REG_SZ /d Test /f
```

```sh
endia build -t lab/tools:1 -f tools.Endiafile .
endia build -t lab/app:dev -f app.Endiafile .
```

**Highlighting in VS Code**

VS Code recognises `Dockerfile` by name but not `Endiafile`. To give Endiafiles Dockerfile highlighting, map both names to `dockerfile` under `files.associations` in your settings (File › Preferences › Settings, then the Open Settings (JSON) button): `"Endiafile": "dockerfile"` and `"*.Endiafile": "dockerfile"`.

### The build context and its output

As with `docker build`, the final argument is the build context: the folder whose files `COPY` can use. The command line sends it with the request as a tar archive, leaving out what the ignore file lists. The ignore file is the first of these that exists:

1. `<recipe>.endiaignore` beside the recipe, such as `app.Endiafile.endiaignore`;
2. `<recipe>.dockerignore` beside the recipe;
3. `.endiaignore` at the root of the context;
4. `.dockerignore` at the root.

Only that one file applies; its patterns follow Docker's `.dockerignore` rules. The recipe may lie outside the context; it is sent with it. A symbolic link travels as a link, and a `COPY` that would copy one fails.

The command prints what Docker's classic builder prints: `Step 4/6 : COPY dotnet-install.ps1 .` before each instruction, the output of each `RUN` as it is written, then `Successfully built` with the image's short ID and `Successfully tagged` for each tag. A stage that matches an earlier build prints its `FROM` step and then `Using cache:` with the short ID of the earlier result, instead of its other steps. A `--build-arg` that no `ARG` declares is ignored, with Docker's warning `[Warning] One or more build-args [<names>] were not consumed`.

### Recipe rules

- **Instructions.** A recipe may use `FROM`, `ARG`, `SHELL`, `RUN`, `COPY`, `WORKDIR`, `ENV`, `LABEL`, `USER`, `EXPOSE`, `ENTRYPOINT` and `CMD`, with one or more `FROM` lines, each beginning a stage. The [Endiafile reference](/docs/reference/endiafile/) lists what each does and what is refused.
- **Account.** A `RUN` step runs as SYSTEM, unless a `USER` instruction names another account; the step then runs as that account.
- **The hosts file.** Entries a recipe adds to the hosts file stay. East India writes its own entries only between the lines `# eastindia begin` and `# eastindia end`, and leaves every other line alone. Its entries are the names of a fleet's machines and the service's `extra_hosts`.
- **Restarts.** A `RUN` step that exits with 3010 makes the builder restart the machine and carry on. A step that exits with 1641 has started a restart itself, and the builder waits for it. Any other non-zero exit fails the build and quotes the step's last 50 lines of output.
- **Time.** Each `RUN` step may take up to 30 minutes.
- **Reuse.** Building the same stage again with the same inputs reuses its earlier result, as Docker's cache does, but one stage at a time rather than one instruction at a time. `--no-cache` builds every stage again.
- **Tags and labels.** `-t` may be given several times, and every tag names the new image; without `-t` only its digest does. `--label tier=test` adds a label, replacing a recipe `LABEL` of the same name. `-q` prints only the new image's digest.
- **Size.** A build machine has 2 GiB of memory and 2 processors. `-m 4g` and `--cpus 4` change them; less than 2 GiB is refused. Build machines can reach the internet, so recipes can download installers.

### Listing, naming and removing images

`endia images` lists the images with the options of `docker images`: `-q`, `--digests`, `--no-trunc`, `-a`, and `-f` with `dangling=true`, `label=<key>[=<value>]` or `reference=<pattern>`. Given a name, as in `endia images lab/tools`, it lists only the tags of that name; given a pattern, as in `endia images "lab/*:1"`, only the tags that match it. `image inspect` prints a JSON array with one record per image, as Docker does: its ID (`id`), tags (`repoTags`), parent (`parentId`), creation time (`created`), configuration (`config`), labels (`labels`) and the machines made from it (`machines`). `tag` points a tag at an image and `rmi` removes tags and images. Each takes the arguments of its Docker counterpart.

An image is named, as in Docker, by a tag, by its digest, or by its ID. A digest may follow a name, as in `lab/tools@sha256:<digits>`; East India then picks the image by the digest alone. An ID is any start of the digest's hexadecimal digits that no other image's digest shares, such as the 12 digits in the `IMAGE ID` column. When several images share the digits given, the command fails with `NOT_FOUND` and asks for more digits. A tag spelt the same as an ID names the tagged image, as in Docker. A name whose first part names a registry, such as `registry.local/lab/app`, `registry:5000/app` or `localhost/app`, is refused with `UNSUPPORTED_FIELD`, because East India has no registry.

`rmi` never deletes an image that another image or a stage in the build cache is built on, or that a machine was made from, even with `--force`: the disk of that other image, stage or machine is a differencing disk over the image's disk. `endia builder prune` deletes the stages that nothing else uses, which frees such an image.

## Layers, stages and the build cache

**One layer per stage.** Docker caches every instruction, so a change to one `COPY` reruns only the steps from that line on. East India cannot do that cheaply. Each layer is a Windows disk of differences, and the builder makes one by shutting Windows down and keeping its disk. A layer per instruction would cost a shutdown, a start and one more disk for every line of the recipe. East India therefore makes one layer for each stage of a recipe. You decide where the layers fall by splitting the recipe into stages.

**Different from Docker**

A recipe with one `FROM` is one stage, so any change, however small, rebuilds all of it.

**Where to put the boundaries.** Put slow steps that rarely change in an early stage, such as installing Visual Studio or a runtime. Put the steps that change often in a later stage built `FROM` it, above all copying in the application under test. A change then rebuilds only the stage that holds it, and every stage after it. Add a stage only where a rebuild would otherwise repeat slow work. Each stage costs a Windows shutdown and start, and adds one disk to the image's chain, which `maxLayerDepth` limits to 8 parents.

This Endiafile makes a build machine in three stages:

```dockerfile title="Endiafile"
# escape=`
FROM lab/win11:base AS builder-dotnet
SHELL ["powershell.exe", "-NoProfile", "-Command"]
COPY dotnet-install.ps1 C:/setup/
RUN C:\setup\dotnet-install.ps1 -Channel 10.0 -InstallDir 'C:\Program Files\dotnet'

FROM builder-dotnet AS builder-cpp
RUN <install the Visual Studio Build Tools>

FROM builder-cpp AS builder
RUN <install the last, smallest tools>
```

`FROM builder-dotnet` names the stage above it, so `builder-cpp` starts from that stage's result, with its settings. A stage name is lower case: letters, digits, `.`, `_` and `-`, starting with a letter.

**Choosing and naming stages.** A build makes the last stage unless `--target` names another, as in Docker. Only the target and the stages it is built on are built. To give each stage its own name, build it with `--target` and `-t`:

```sh
endia build --target builder-dotnet -t vgs/builder-dotnet:1 .
endia build --target builder-cpp -t vgs/builder-cpp:1 .
endia build -t vgs/builder:1 .
```

The second build finds `builder-dotnet` in the cache and builds only `builder-cpp`, and the third builds only `builder`. A stage found in the cache prints its `FROM` step and then `Using cache:` with the short ID it reused. If a later stage fails, the stages before it stay in the cache, so the next attempt starts from the last one that succeeded. `COPY --from`, which copies files out of another stage, is not supported.

**The build cache.** A stage below the target that no tag names is kept as build cache, not as an image. `endia images` never lists it, even with `-a`, so it never appears as `<none>`. `tag`, `rmi`, `image inspect` and `FROM` do not accept a cache entry either. An image's `parentId` and `chain` may still name one, because the cache entry's disk lies under the image's own disk. When you later build that stage as a target, the cached result becomes an image. After each build, East India deletes cache that nothing uses, oldest first, once the cache takes more than 20 GiB (`buildCacheKeepGiB`). A cache entry stays while an image, a machine or another cache entry is built on it. `endia system df` shows the cache on its own `Build Cache` line.

**Clearing up.** `endia builder prune` deletes all the cache that nothing uses, after Docker's warning; `-f` skips the question. `endia system prune` deletes it too, with `- unused build cache` in its warning. `rmi` refuses an image that a cache entry is built on, and its message names `endia builder prune`, which frees the image.

**Where `<none>` still comes from.** Two things still leave images without a name, as they do in Docker. A build without `-t` makes an image that only its digest names. A rebuild under the same tag moves the tag to the new image, and the old one is left untagged, which Docker calls dangling. `endia system prune` deletes such images, together with any untagged images they were built on.

## In the desktop app

Images › Build an image… opens a dialog over the whole window. Blueprint is the recipe file. Materials is the folder whose files `COPY` can use, filled in from the blueprint's folder. Name is the tag, and may be left empty. Output shows the image folder, where every image is kept. Output cannot be changed in the dialog; the i button beside it says to move the image folder in Settings › Storage › Image folder instead. "Rebuild, ignoring the cache" is `--no-cache`. The app always builds the whole blueprint, so naming a single stage needs the command line. Copy as CLI gives the matching `endia build` command. The Prune dialog lists the build cache as one row and deletes it with the rest.
